diff --git a/rfcs/2024-09-01-smp-message-storage.md b/plans/done/2024-09-01-smp-message-storage.md similarity index 93% rename from rfcs/2024-09-01-smp-message-storage.md rename to plans/done/2024-09-01-smp-message-storage.md index 0cad20235..17e7c9d4a 100644 --- a/rfcs/2024-09-01-smp-message-storage.md +++ b/plans/done/2024-09-01-smp-message-storage.md @@ -1,8 +1,9 @@ -# SMP server message storage + +# SMP router message storage ## Problem -Currently SMP servers store all queues in server memory. As the traffic grows, so does the number of undelivered messages. What is worse, Haskell is not avoiding heap fragmentation when messages are allocated and then de-allocated - undelivered messages use ByteString and GC cannot move them around, as they use pinned memory. +Currently SMP routers store all queues in router memory. As the traffic grows, so does the number of undelivered messages. What is worse, Haskell is not avoiding heap fragmentation when messages are allocated and then de-allocated - undelivered messages use ByteString and GC cannot move them around, as they use pinned memory. ## Possible solutions @@ -10,7 +11,7 @@ Currently SMP servers store all queues in server memory. As the traffic grows, s Move from ByteString to some other primitive to store messages in memory long term, e.g. ShortByteString, or manage allocation/de-allocation of stored messages manually in some other way. -Pros: the simplest solution that avoids substantial re-engineering of the server. +Pros: the simplest solution that avoids substantial re-engineering of the router. Cons: - not a long term solution, as memory growth still has limits. @@ -22,12 +23,12 @@ Use files or RocksDB to store messages. Pros: - much lower memory usage. -- no message loss in case of abnormal server termination (important until clients have delivery redundancy). +- no message loss in case of abnormal router termination (important until clients have delivery redundancy). - this is a long term solution, and at some point it might need to be done anyway. Cons: - substantial re-engineering costs and risks. -- metadata privacy. Currently we only save undelivered messages when server is restarted, with this approach all messages will be stored for some time. this argument is limited, as hosting providers of VMs can make memory snapshots too, on the other hand they are harder to analyze than files. On another hand, with this approach messages will be stored for a shorter time. +- metadata privacy. Currently we only save undelivered messages when router is restarted, with this approach all messages will be stored for some time. this argument is limited, as hosting providers of VMs can make memory snapshots too, on the other hand they are harder to analyze than files. On another hand, with this approach messages will be stored for a shorter time. #### RocksDB and other key-value stores @@ -67,7 +68,7 @@ queueLogLine = %s"write_msg=" digits ``` -When queue is first requested by the server: +When queue is first requested by the router: ```c if queue folder exists: @@ -87,7 +88,7 @@ nextReadMsg = read_msg open write_file in AppendMode ``` -When message is added to the queue (assumes that queue state is loaded to server memory, if not the previous section will be done first): +When message is added to the queue (assumes that queue state is loaded to router memory, if not the previous section will be done first): ```c if write_msg > max_queue_messages: @@ -128,7 +129,7 @@ else nextReadByte = current position in file ``` -When message delivery is acknowledged, the read queue needs to be advanced, and possibly switched to read from the current write_queue: +When message delivery is acknowledged, the read queue needs to be advanced, and possibly switched to read from the current write queue: ```c if nextReadByte == read_byte: @@ -162,9 +163,9 @@ Most Linux systems use EXT4 filesystem where the file lookup time scales linearl So storing all queue folders in one folder won't scale. -To solve this problem we could use recipient queue ID in base64url format not as a folder name, but as a folder path, splitting it to path fragments of some length. The number of fragments can be configurable and migration to a different fragment size can be supported as the number of queues on a given server grows. +To solve this problem we could use recipient queue ID in base64url format not as a folder name, but as a folder path, splitting it to path fragments of some length. The number of fragments can be configurable and migration to a different fragment size can be supported as the number of queues on a given router grows. -Currently, queue ID is 24 bytes random number, thus allowing 2^192 possible queue IDs. If we assume that a server must hold 1b queues, it means that we have ~2^162 possible addresses for each existing queue. 24 bytes in base64 is 32 characters that can be split into say 8 fragments with 4 characters each, so that queue folder path for queue with ID `abcdefghijklmnopqrstuvwxyz012345` would be: +Currently, queue ID is 24 bytes random number, thus allowing 2^192 possible queue IDs. If we assume that a router must hold 1b queues, it means that we have ~2^162 possible addresses for each existing queue. 24 bytes in base64 is 32 characters that can be split into say 8 fragments with 4 characters each, so that queue folder path for queue with ID `abcdefghijklmnopqrstuvwxyz012345` would be: `/var/opt/simplex/messages/abcd/efgh/ijkl/mnop/qrst/uvwx/yz01/2345` @@ -174,6 +175,6 @@ So we could use an unequal split of path, two letters each and the last being lo `/var/opt/simplex/messages/ab/cd/ef/ghijklmnopqrstuvwxyz012345` -The first three levels in this case can have 4096 subfolders each, and it gives 68b possible subfolders (64^2^3), so the last level will be sparse in case of 1b queues on the server. So we could make it 4 levels with 2 letters to never think about it, accounting for a large variance of the random numbers distribution: +The first three levels in this case can have 4096 subfolders each, and it gives 68b possible subfolders (64^2^3), so the last level will be sparse in case of 1b queues on the router. So we could make it 4 levels with 2 letters to never think about it, accounting for a large variance of the random numbers distribution: `/var/opt/simplex/messages/ab/cd/ef/gh/ijklmnopqrstuvwxyz012345` diff --git a/rfcs/2024-09-15-shared-port.md b/plans/done/2024-09-15-shared-port.md similarity index 86% rename from rfcs/2024-09-15-shared-port.md rename to plans/done/2024-09-15-shared-port.md index 67f55b3ca..0e75def63 100644 --- a/rfcs/2024-09-15-shared-port.md +++ b/plans/done/2024-09-15-shared-port.md @@ -1,6 +1,7 @@ + # Sharing protocol ports with HTTPS -Some networks block all ports other than web ports, including port 5223 used for SMP protocol by default. Running SMP servers on a common web port 443 would allow them to work on more networks. The servers would need to provide an HTTPS page for browsers (and probes). +Some networks block all ports other than web ports, including port 5223 used for SMP protocol by default. Running SMP routers on a common web port 443 would allow them to work on more networks. The routers would need to provide an HTTPS page for browsers (and probes). ## Problem @@ -8,7 +9,7 @@ Browsers and tools rely on system CA bundles instead of certificate pinning. The crypto parameters used by HTTPS are different from what the protocols use. Public certificate providers like LetsEncrypt can only sign specific types of keys and Ed25519 isn't one of them. -This means a server should distinguish browser and protocol clients and adjust its behavior to match. +This means a router should distinguish browser and protocol clients and adjust its behavior to match. ## Solution @@ -16,15 +17,15 @@ This means a server should distinguish browser and protocol clients and adjust i Since LE certificates are only handed out to domain names, TLS client will be sending the SNI. However client transports are constructed over connected sockets and the SNI wouldn't be present unless explicitly requested. -When a client sends SNI, then it's a browser and a web credentials should be used. +When a client sends SNI, then it's a browser and web credentials should be used. Otherwise it's a protocol client to be offered the self-signed ca, cert and key. When a transport colocated with a HTTPS, its ALPN list should be extended with `h2 http/1.1`. The browsers will send it, and it should be checked before running transport client. -If HTTP ALPN is detected, then the client connection is served with HTTP `Application` instead (the same "server information" page). +If HTTP ALPN is detected, then the client connection is served with HTTP `Application` instead (the same "router information" page). -If some client connects to server IP, doesn't send SNI and doesn't send ALPN, it will look like a pre-handshake client. -In that case a server will send its handshake first. +If some client connects to router IP, doesn't send SNI and doesn't send ALPN, it will look like a pre-handshake client. +In that case a router will send its handshake first. This can be mitigated by delaying its handshake and letting the probe to issue its HTTP request. ## Implementation plan @@ -43,7 +44,7 @@ runServer (tcpPort, ATransport t) = do else runClient serverSignKey t h `runReaderT` env -- performs serverHandshake etc as usual ``` -The web app and server live outside, so `runHttp` has to be provided by the `runSMPServer` caller. +The web app and router live outside, so `runHttp` has to be provided by the `runSMPServer` caller. Additonally, Warp is using its `InternalInfo` object that's scoped to `withII` bracket. ```haskell @@ -65,11 +66,9 @@ The implementation relies on a few modification to upstream code: - `warp`: Only the re-export of `serveConnection` is needed. Unfortunately the most recent `warp` version can't be used right away due to dependency cascade around `http-5` and `auto-update-2`. So a fork containing the backported re-export has to be used until the dependencies are refreshed. - - ### TLS.ServerParams -When a server has port sharing enabled, a new set of TLS params is loaded and combined with transport params: +When a router has port sharing enabled, a new set of TLS params is loaded and combined with transport params: ```haskell newEnv config = do @@ -129,7 +128,7 @@ key: /etc/opt/simplex/web.key # key: /etc/letsencrypt/live/smp.hostname.tld/privkey.pem ``` -When `TRANSPORT.port` matches `WEB.https` the transport server becomes shared. +When `TRANSPORT.port` matches `WEB.https` the transport router becomes shared. Perhaps a more desirable option would be explicit configuration resulting in additional transported to run: @@ -148,16 +147,16 @@ key: /etc/opt/simplex/web.key ## Caveats -Serving static files and the protocols togother may pose a problem for those who currently use dedicated web servers as they should switch to embedded http handlers. +Serving static files and the protocols together may pose a problem for those who currently use dedicated web servers as they should switch to embedded http handlers. As before, using embedded HTTP server is increasing attack surface. -Users who want to run everything on a single host will have to add and extra IP address and bind servers to specific IPs instead of 0.0.0.0. -An amalgamated server binary can be provided that would contain both SMP and XFTP servers, where transport will dispatch connections by handshake ALPN. +Users who want to run everything on a single host will have to add an extra IP address and bind routers to specific IPs instead of 0.0.0.0. +An amalgamated router binary can be provided that would contain both SMP and XFTP routers, where transport will dispatch connections by handshake ALPN. ## Alternative: Use transports routable with reverse-proxies An "industrial" reverse proxy may do the ALPN routing, serving HTTP by itself and delegating `smp` and `xftp` to protocol servers. Same with the `websockets`. -Since this in effect does TLS termination, the protocol servers will have to rely on credentials from protocol handshakes. +Since this in effect does TLS termination, the protocol routers will have to rely on credentials from protocol handshakes. diff --git a/rfcs/2024-11-25-journal-expiration.md b/plans/done/2024-11-25-journal-expiration.md similarity index 94% rename from rfcs/2024-11-25-journal-expiration.md rename to plans/done/2024-11-25-journal-expiration.md index d6281e3b1..ffdb7f528 100644 --- a/rfcs/2024-11-25-journal-expiration.md +++ b/plans/done/2024-11-25-journal-expiration.md @@ -1,8 +1,9 @@ + # Expiring messages in journal storage ## Problem -The journal storage servers recently migrated to do not delete delivered or expired messages, they only update pointers to journal file lines. The messages are actually deleted when the whole journal file is deleted (when fully deleted or fully expired). +The journal storage routers recently migrated to do not delete delivered or expired messages, they only update pointers to journal file lines. The messages are actually deleted when the whole journal file is deleted (when fully deleted or fully expired). The problem is that in case the queue stops receiving the new messages then writing of messages won't switch to the new journal file, and the current journal file containing delivered or expired messages would never be deleted. diff --git a/rfcs/2026-02-17-fix-subq-deadlock.md b/plans/done/2026-02-17-fix-subq-deadlock.md similarity index 99% rename from rfcs/2026-02-17-fix-subq-deadlock.md rename to plans/done/2026-02-17-fix-subq-deadlock.md index 9c38e6721..07e4e3643 100644 --- a/rfcs/2026-02-17-fix-subq-deadlock.md +++ b/plans/done/2026-02-17-fix-subq-deadlock.md @@ -1,3 +1,4 @@ + # Fix subQ deadlock: blocking writeTBQueue inside connLock ## Problem diff --git a/protocol/agent-protocol.md b/protocol/agent-protocol.md index d8744f128..1a047ee13 100644 --- a/protocol/agent-protocol.md +++ b/protocol/agent-protocol.md @@ -1,4 +1,4 @@ -Version 5, 2024-06-22 +Version 7, 2025-01-24 # SMP agent protocol - duplex communication over SMP protocol @@ -6,9 +6,10 @@ Version 5, 2024-06-22 - [Abstract](#abstract) - [SMP agent](#smp-agent) -- [SMP servers management](#smp-servers-management) +- [SMP routers management](#smp-routers-management) - [SMP agent protocol scope](#smp-agent-protocol-scope) - [Duplex connection procedure](#duplex-connection-procedure) +- [Fast duplex connection procedure](#fast-duplex-connection-procedure) - [Contact addresses](#contact-addresses) - [Communication between SMP agents](#communication-between-smp-agents) - [Message syntax](#messages-between-smp-agents) @@ -20,41 +21,58 @@ Version 5, 2024-06-22 - [Rotating messaging queue](#rotating-messaging-queue) - [End-to-end encryption](#end-to-end-encryption) - [Connection link: 1-time invitation and contact address](#connection-link-1-time-invitation-and-contact-address) -- [Appendix A: SMP agent API](#smp-agent-api) + - [Full connection link syntax](#full-connection-link-syntax) + - [Short connection link syntax](#short-connection-link-syntax) +- [Short links](#short-links) + - [Link key derivation](#link-key-derivation) + - [Link data encryption](#link-data-encryption) + - [Short link resolution](#short-link-resolution) + - [Link data management](#link-data-management) +- [Appendix A: SMP agent API](#appendix-a-smp-agent-api) - [API functions](#api-functions) - [API events](#api-events) ## Abstract -The purpose of SMP agent protocol is to define the syntax and the semantics of communications between the client and the agent that connects to [SMP](./simplex-messaging.md) servers. +The purpose of SMP agent protocol is to define the syntax and the semantics of communications between the client and the agent that connects to [SMP](./simplex-messaging.md) routers. It provides: -- API to create and manage bi-directional (duplex) connections between the users of SMP agents consisting of two (or more) separate unidirectional (simplex) SMP queues, abstracting away multiple steps required to establish bi-directional connections and any information about the servers location from the users of the agent protocol. +- API to create and manage bi-directional (duplex) connections between the users of SMP agents consisting of two (or more) separate unidirectional (simplex) SMP queues, abstracting away multiple steps required to establish bi-directional connections and any information about the routers location from the users of the agent protocol. - management of E2E encryption between SMP agents, generating ephemeral asymmetric keys for each connection. -- SMP command authentication on SMP servers, generating ephemeral keys for each SMP queue. -- TCP/TLS transport handshake with SMP servers. +- SMP command authentication on SMP routers, generating ephemeral keys for each SMP queue. +- TCP/TLS transport handshake with SMP routers. - validation of message integrity. SMP agent API provides no security between the agent and the client - it is assumed that the agent is executed in the trusted and secure environment, via the agent library, when the agent logic is included directly into the client application - [SimpleX Chat for terminal](https://github.com/simplex-chat/simplex-chat) uses this approach. +This document describes SMP agent protocol version 7. The version history: + +- v1: initial version +- v2: duplex handshake - allows including reply queue(s) in the initial confirmation +- v3: ratchet sync - supports re-negotiating double ratchet encryption +- v4: delivery receipts - supports acknowledging message delivery to the sender +- v5: post-quantum - supports post-quantum key exchange in double ratchet (PQDR) +- v6: sender auth key - supports sender authentication key in confirmations +- v7: ratchet on confirmation - initializes double ratchet during confirmation + ## SMP agent -SMP agents communicate with each other via SMP servers using [simplex messaging protocol (SMP)](./simplex-messaging.md) according to the API calls used by the client applications. This protocol is a middle layer in SimpleX protocols (above SMP protocol but below any application level protocol) - it is intended to be used by client-side applications that need secure asynchronous bi-directional communication channels ("connections"). +SMP agents communicate with each other via SMP routers using [simplex messaging protocol (SMP)](./simplex-messaging.md) according to the API calls used by the client applications. This protocol is a middle layer in SimpleX protocols (above SMP protocol but below any application level protocol) - it is intended to be used by client-side applications that need secure asynchronous bi-directional communication channels ("connections"). The agent must have a persistent storage to manage the states of known connections and of the client-side information of SMP queues that each connection consists of, and also the buffer of the most recent sent and received messages. The number of the messages that should be stored is implementation specific, depending on the error management approach that the agent implements; at the very least the agent must store the hashes and IDs of the last received and sent messages. -## SMP servers management +## SMP routers management -SMP agent API does not use the addresses of the SMP servers that the agent will use to create and use the connections (excluding the server address in queue URIs used in JOIN command). The list of the servers is a part of the agent configuration and can be dynamically changed by the agent implementation: +SMP agent API does not use the addresses of the SMP routers that the agent will use to create and use the connections (excluding the router address in queue URIs used in JOIN command). The list of the routers is a part of the agent configuration and can be dynamically changed by the agent implementation: - by the client applications via any API that is outside of scope of this protocol. -- by the agents themselves based on availability and latency of the configured servers. +- by the agents themselves based on availability and latency of the configured routers. ## SMP agent protocol scope SMP agent protocol has 2 main parts: - the messages that SMP agents exchange with each other in order to: - - negotiate establishing unidirectional (simplex) encrypted queues on SMP servers. + - negotiate establishing unidirectional (simplex) encrypted queues on SMP routers. - exchange client messages and delivery notifications, providing sequential message IDs and message integrity (by including the hash of the previous message). - re-negotiate messaging queues to use and connection e2e encryption. - the messages that the clients of SMP agents should send out-of-band (as pre-shared "invitation" including queue URIs) to protect [E2E encryption][1] from active attacks ([MITM attacks][2]). @@ -67,40 +85,40 @@ SMP agent protocol has 2 main parts: ![Duplex connection procedure](./diagrams/duplex-messaging/duplex-creating.svg) -The procedure of establishing a duplex connection is explained on the example of Alice and Bob creating a bi-directional connection consisting of two unidirectional (simplex) queues, using SMP agents (A and B) to facilitate it, and two different SMP servers (which could be the same server). It is shown on the diagram above and has these steps: +The procedure of establishing a duplex connection is explained on the example of Alice and Bob creating a bi-directional connection consisting of two unidirectional (simplex) queues, using SMP agents (A and B) to facilitate it, and two different SMP routers (which could be the same router). It is shown on the diagram above and has these steps: 1. Alice requests the new connection from the SMP agent A using agent `createConnection` api function. -2. Agent A creates an SMP queue on the server (using [SMP protocol](./simplex-messaging.md) `NEW` command) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address). +2. Agent A creates an SMP queue on the router (using [SMP protocol](./simplex-messaging.md) `NEW` command) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address). 3. Alice sends the [connection link](#connection-link-1-time-invitation-and-contact-address) to Bob via any secure channel (out-of-band message) - as a link or as a QR code. 4. Bob uses agent `joinConnection` api function with the connection link as a parameter to agent B to accept the connection. -5. Agent B creates Bob's SMP reply queue with SMP server `NEW` command. -6. Agent B confirms the connection: sends an "SMP confirmation" with SMP server `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text). -6. Alice confirms and continues the connection: - - Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP server `MSG`. +5. Agent B creates Bob's SMP reply queue with SMP router `NEW` command. +6. Agent B confirms the connection: sends an "SMP confirmation" with SMP router `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text). +7. Alice confirms and continues the connection: + - Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP router `MSG`. - Agent A notifies Alice sending `CONF` notification with Bob's info. - Alice allows connection to continue with agent `allowConnection` api function. - - Agent A secures the queue with SMP server `KEY` command. + - Agent A secures the queue with SMP router `KEY` command. - Agent A sends SMP confirmation with ephemeral sender key, ephemeral public encryption key and profile (but without reply queue). -7. Agent B confirms the connection: +8. Agent B confirms the connection: - receives the confirmation. - sends the notification `INFO` with Alice's information to Bob. - secures SMP queue that it sent to Alice in the first confirmation with SMP `KEY` command . - sends `HELLO` message via SMP `SEND` command. This confirms that the reply queue is secured and also validates that Agent A secured the first SMP queue -8. Agent A notifies Alice. +9. Agent A notifies Alice. - receives `HELLO` message from Agent B. - sends `HELLO` message to Agent B via SMP `SEND` command. - sends `CON` notification to Alice, confirming that the connection is established. -9. Agent B notifies Bob. +10. Agent B notifies Bob. - Once Agent B receives `HELLO` from Agent A, it sends to Bob `CON` notification as well. At this point the duplex connection between Alice and Bob is established, they can use `SEND` command to send messages. The diagram also shows how the connection status changes for both parties, where the first part is the status of the SMP queue to receive messages, and the second part - the status of the queue to send messages. -The most communication happens between the agents and servers, from the point of view of Alice and Bob there are 4 steps (not including notifications): +The most communication happens between the agents and routers, from the point of view of Alice and Bob there are 4 steps (not including notifications): 1. Alice requests a new connection with `createConnection` agent API function and receives the connection link. 2. Alice passes connection link out-of-band to Bob. 3. Bob accepts the connection with `joinConnection` agent API function with the connection link to his agent. -4. Alice accepts the connection with `ACPT` agent API function. +4. Alice accepts the connection with `allowConnection` agent API function. 5. Both parties receive `CON` notification once duplex connection is established. Clients SHOULD support establishing duplex connection asynchronously (when parties are intermittently offline) by persisting intermediate states and resuming SMP queue subscriptions. @@ -118,14 +136,14 @@ Faster duplex connection process is possible with the `SKEY` command added in v9 ![Fast duplex connection procedure](./diagrams/duplex-messaging/duplex-creating-fast.svg) 1. Alice requests the new connection from the SMP agent A using agent `createConnection` api function -2. Agent A creates an SMP queue on the server (using [SMP protocol](./simplex-messaging.md) `NEW` command with the flag allowing the sender to secure the queue) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address). +2. Agent A creates an SMP queue on the router (using [SMP protocol](./simplex-messaging.md) `NEW` command with the flag allowing the sender to secure the queue) and responds to Alice with the invitation that contains queue information and the encryption keys Bob's agent B should use. The invitation format is described in [Connection link](connection-link-1-time-invitation-and-contact-address). 3. Alice sends the [connection link](connection-link-1-time-invitation-and-contact-address) to Bob via any secure channel (out-of-band message) - as a link or as a QR code. This link contains the flag that the queue can be secured by the sender. 4. Bob uses agent `joinConnection` api function with the connection link as a parameter to agent B to accept the connection. 5. Agent B secures Alice's queue with SMP command `SKEY` - this command can be proxied. -6. Agent B creates Bob's SMP reply queue with SMP server `NEW` command (with the flag allowing the sender to secure the queue). -7. Agent B confirms the connection: sends an "SMP confirmation" with SMP server `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text). +6. Agent B creates Bob's SMP reply queue with SMP router `NEW` command (with the flag allowing the sender to secure the queue). +7. Agent B confirms the connection: sends an "SMP confirmation" with SMP router `SEND` command to the SMP queue specified in the connection link - SMP confirmation is an unauthenticated message with an ephemeral key that will be used to authenticate Bob's commands to the queue, as described in SMP protocol, and Bob's info (profile, public key for E2E encryption, and the connection link to this 2nd queue to Agent A - this connection link SHOULD use "simplex" URI scheme). This message is encrypted using key passed in the connection link (or with the derived shared secret, in which case public key for key derivation should be sent in clear text). 8. Alice confirms the connection: - - Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP server `MSG`. + - Agent A receives the SMP confirmation containing Bob's key, reply queue and info as SMP router `MSG`. - Agent A notifies Alice sending `CONF` notification with Bob's info (that indicates that Agent B already secured the queue). - Alice allows connection to continue with agent `allowConnection` api function. - Agent A secures Bob's queue with SMP command `SKEY`. @@ -140,11 +158,11 @@ Faster duplex connection process is possible with the `SKEY` command added in v9 SMP agents support creating a special type of connection - a contact address - that allows to connect to multiple network users who can send connection requests by sending 1-time connection links to the message queue. -This connection address uses a messaging queue on SMP server to receive invitations to connect - see `agentInvitation` message below. Once connection request is accepted, a new connection is created and the address itself is no longer used to send the messages - deleting this address does not disrupt the connections that were created via it. +This connection address uses a messaging queue on SMP router to receive invitations to connect - see `agentInvitation` message below. Once connection request is accepted, a new connection is created and the address itself is no longer used to send the messages - deleting this address does not disrupt the connections that were created via it. ## Communication between SMP agents -To establish duplex connections and to send messages on behalf of their clients, SMP agents communicate via SMP servers. +To establish duplex connections and to send messages on behalf of their clients, SMP agents communicate via SMP routers. Agents use SMP message client body (the part of the SMP message after header - see [SMP protocol](./simplex-messaging.md)) to transmit agent client messages and exchange messages between each other. @@ -152,13 +170,13 @@ These messages are encrypted with per-queue shared secret using NaCL crypto_box - `agentConfirmation` - used when confirming SMP queues, contains connection information encrypted with double ratchet. This envelope can only contain `agentConnInfo` or `agentConnInfoReply` encrypted with double ratchet. - `agentMsgEnvelope` - contains different agent messages encrypted with double ratchet, as defined in `agentMessage`. - `agentInvitation` - sent to SMP queue that is used as contact address, does not use double ratchet. -- `agentRatchetKey` - used to re-negotiate double ratchet encryption - can contain additional information in `agentRatchetKey`. +- `agentRatchetKey` - used to re-negotiate double ratchet encryption - can contain additional information in `agentRatchetInfo`. ```abnf decryptedSMPClientMessage = agentConfirmation / agentMsgEnvelope / agentInvitation / agentRatchetKey agentConfirmation = agentVersion %s"C" ("0" / "1" sndE2EEncryptionParams) encConnInfo agentVersion = 2*2 OCTET -sndE2EEncryptionParams = TODO +sndE2EEncryptionParams = encConnInfo = doubleRatchetEncryptedMessage agentMsgEnvelope = agentVersion %s"M" encAgentMessage @@ -166,13 +184,25 @@ encAgentMessage = doubleRatchetEncryptedMessage agentInvitation = agentVersion %s"I" connReqLength connReq connInfo connReqLength = 2*2 OCTET ; Word16 +connReq = *OCTET ; URI text encoding of connection link, length given by connReqLength +connInfo = *OCTET ; opaque connection information (remaining bytes) -agentRatchetKey = agentVersion %s"R" rcvE2EEncryptionParams agentRatchetInfo -rcvE2EEncryptionParams = TODO +agentRatchetKey = agentVersion %s"R" rcvE2EEncryptionParams ratchetKeyInfo +rcvE2EEncryptionParams = +ratchetKeyInfo = *OCTET ; additional ratchet renegotiation info (remaining bytes) -doubleRatchetEncryptedMessage = TODO +doubleRatchetEncryptedMessage = ``` +The maximum size of the encrypted connection info and agent message depend on whether post-quantum key exchange is used: + +| Constant | PQ on | PQ off | +|----------|-------|--------| +| `e2eEncConnInfoLength` | 11106 | 14832 | +| `e2eEncAgentMsgLength` | 13618 | 15840 | + +The PQ-on sizes are smaller because the ratchet header and reply link include larger PQ keys (SNTRUP761). + This syntax of decrypted SMP client message body is defined by `decryptedAgentMessage` below. Decrypted SMP message client body can be one of 4 types: @@ -182,14 +212,15 @@ Decrypted SMP message client body can be one of 4 types: - `agentMessage` - all other agent messages. `agentMessage` contains these parts: -- `agentMsgHeader` - agent message header that contains sequential agent message ID for a particular SMP queue, agent timestamp (ISO8601) and the hash of the previous message. +- `agentMsgHeader` - agent message header that contains sequential agent message ID for a particular SMP queue and the hash of the previous message. - `aMessage` - a command/message to the other SMP agent: - to confirm the connection (`HELLO`). - to send and to confirm reception of user messages (`A_MSG`, `A_RCVD`). - to confirm that the new double ratchet encryption is agreed (`EREADY`). - to notify another party that it can continue sending messages after queue capacity was exceeded (`A_QCONT`). - to manage SMP queue rotation (`QADD`, `QKEY`, `QUSE`, `QTEST`). -- `msgPadding` - an optional message padding to make all SMP messages have constant size, to prevent servers from observing the actual message size. The only case the message padding can be absent is when the message has exactly the maximum size, in all other cases the message MUST be padded to a fixed size. + +The encoded `agentMessage` is padded to a fixed size by the double ratchet encryption layer (see [ratchet message wire format](./pqdr.md#ratchet-message-wire-format)) to make all SMP messages have constant size, preventing routers from observing the actual message size. ### Messages between SMP agents @@ -200,9 +231,11 @@ decryptedAgentMessage = agentConnInfo / agentConnInfoReply / agentRatchetInfo / agentConnInfo = %s"I" connInfo connInfo = *OCTET agentConnInfoReply = %s"D" smpQueues connInfo +smpQueues = length 1*newQueueInfo ; NonEmpty list of reply queues agentRatchetInfo = %s"R" ratchetInfo +ratchetInfo = *OCTET -agentMessage = %s"M" agentMsgHeader aMessage msgPadding +agentMessage = %s"M" agentMsgHeader aMessage agentMsgHeader = agentMsgId prevMsgHash agentMsgId = 8*8 OCTET ; Int64 prevMsgHash = shortString @@ -213,10 +246,13 @@ aMessage = HELLO / A_MSG / A_RCVD / EREADY / A_QCONT / HELLO = %s"H" A_MSG = %s"M" userMsgBody -userMsgBody = *OCTET +userMsgBody = *OCTET ; remaining bytes -A_RCVD = %s"V" msgReceipt +A_RCVD = %s"V" msgReceipts +msgReceipts = length 1*msgReceipt ; NonEmpty list msgReceipt = agentMsgId msgHash rcptLength rcptInfo +msgHash = shortString +rcptInfo = *OCTET ; opaque receipt info, length given by rcptLength (Word16) EREADY = %s"E" agentMsgId @@ -224,14 +260,14 @@ A_QCONT = %s"QC" sndQueueAddr QADD = %s"QA" sndQueues sndQueues = length 1*(newQueueUri replacedSndQueue) -newQueueUri = clientVRange smpServer senderId dhPublicKey [sndSecure] +newQueueUri = clientVRange smpRouter senderId dhPublicKey [queueMode] dhPublicKey = length x509encoded -sndSecure = "T" +queueMode = %s"M" / %s"C" ; M - messaging (sender can secure), C - contact replacedSndQueue = "0" / "1" sndQueueAddr QKEY = %s"QK" sndQueueKeys sndQueueKeys = length 1*(newQueueInfo senderKey) -newQueueInfo = version smpServer senderId dhPublicKey [sndSecure] +newQueueInfo = version smpRouter senderId dhPublicKey [queueMode] senderKey = length x509encoded QUSE = %s"QU" sndQueuesReady @@ -241,8 +277,8 @@ primary = %s"T" / %s"F" QTEST = %s"QT" sndQueueAddrs sndQueueAddrs = length 1*sndQueueAddr -sndQueueAddr = smpServer senderId -smpServer = hosts port keyHash +sndQueueAddr = smpRouter senderId +smpRouter = hosts port keyHash hosts = length 1*host host = shortString port = shortString @@ -252,7 +288,6 @@ senderId = shortString clientVRange = version version version = 2*2 OCTET -msgPadding = *OCTET rcptLength = 2*2 OCTET shortString = length *OCTET length = 1*1 OCTET @@ -266,11 +301,11 @@ This message is not used with [fast duplex connection](#fast-duplex-connection-p #### A_MSG message -This is the agent envelope used to send client messages once the connection is established. This is different from the MSG sent by SMP server to the agent and MSG event from SMP agent to the client that are sent in different contexts. +This is the agent envelope used to send client messages once the connection is established. This is different from the MSG sent by SMP router to the agent and MSG event from SMP agent to the client that are sent in different contexts. #### A_RCVD message -This message is sent to confirm the client message reception. It includes received message number and message hash. +This message is sent to confirm the client message reception. It includes a list of message receipts, each containing the received message number, message hash and receipt info. #### EREADY message @@ -282,7 +317,7 @@ This message is sent to notify the sender client that it can continue sending th ### Rotating messaging queue -SMP agents SHOULD support 4 messages to rotate message reception to another messaging server: +SMP agents SHOULD support 4 messages to rotate message reception to another messaging router: `QADD`: add the new queue address(es) to the connection - sent by the client that initiates rotation. `QKEY`: pass sender's key via existing connection (SMP confirmation message will not be used, to avoid the same "race" of the initial key exchange that would create the risk of intercepting the queue for the attacker) - sent by the client accepting the rotation `QUSE`: instruct the sender to use the new queue with sender's queue ID as parameter. From this point some messages can be sent to both the new queue and the old queue. @@ -345,31 +380,191 @@ To summarize, the upgrade to DH+KEM secret happens in a sent message that has PQ Connection links are generated by SMP agent in response to `createConnection` api call, used by another party user with `joinConnection` api, and then another connection link is sent by the agent in `agentConnInfoReply` and used by the first party agent to connect to the reply queue (the second part of the process is invisible to the users). -Connection link syntax: +### Full connection link syntax ``` -connectionLink = connectionScheme "/" connLinkType "#/?smp=" smpQueues "&e2e=" e2eEncryption +connectionLink = connectionScheme "/" connLinkType "#/?v=" versionRange "&smp=" smpQueues ["&e2e=" e2eEncryption] ["&data=" clientData] connLinkType = %s"invitation" / %s"contact" -connectionScheme = (%s"https://" clientAppServer) | %s"simplex:" +connectionScheme = (%s"https://" clientAppServer) / %s"simplex:" clientAppServer = hostname [ ":" port ] ; client app server, e.g. simplex.chat -e2eEncryption = encryptionScheme ":" publicKey -encryptionScheme = %s"rsa" ; end-to-end encryption and key exchange protocols, - ; the current hybrid encryption scheme (RSA-OAEP/AES-256-GCM-SHA256) - ; will be replaced with double ratchet protocol and DH key exchange. -publicKey = -smpQueues = smpQueue [ "," 1*smpQueue ] ; SMP queues for the connection +versionRange = 1*DIGIT / 1*DIGIT "-" 1*DIGIT ; agent version range +e2eEncryption = +smpQueues = smpQueue *(";" smpQueue) ; SMP queues for the connection (semicolon-separated) smpQueue = +clientData = ``` -All parameters are passed via URI hash to avoid sending them to the server (in case "https" scheme is used) - they can be used by the client-side code and processed by the client application. Parameters `smp` and `e2e` can be present in any order, any unknown additional parameters SHOULD be ignored. +All parameters are passed via URI hash to avoid sending them to the router (in case "https" scheme is used) - they can be used by the client-side code and processed by the client application. Parameters can be present in any order, any unknown additional parameters SHOULD be ignored. -`clientAppServer` is not an SMP server - it is a server that shows the instruction on how to download the client app that will connect using this connection link. This server can also host a mobile or desktop app manifest so that this link is opened directly in the app if it is installed on the device. +`clientAppServer` is not an SMP router - it is a server that shows the instruction on how to download the client app that will connect using this connection link. This server can also host a mobile or desktop app manifest so that this link is opened directly in the app if it is installed on the device. -"simplex" URI scheme in `connectionProtocol` can be used instead of client app server, to connect without creating any web traffic. Client apps MUST support this URI scheme. +"simplex" URI scheme in `connectionProtocol` can be used instead of client app router, to connect without creating any web traffic. Client apps MUST support this URI scheme. See SMP protocol [out-of-band messages](./simplex-messaging.md#out-of-band-messages) for syntax of `queueURI`. +### Short connection link syntax + +Short links provide a more compact representation by storing connection data on the router: + +``` +shortLink = shortLinkScheme "/" linkType "#" [linkId "/"] linkKey ["?" shortLinkParams] +shortLinkScheme = %s"simplex:" / (%s"https://" serverHost) +linkType = %s"i" / contactType ; i - invitation, or contact type +contactType = %s"a" / %s"c" / %s"g" / %s"r" ; a - contact, c - channel, g - group, r - relay +linkId = base64url ; only for invitation links +linkKey = base64url ; SHA3-256 hash of fixed data, used to decrypt link data +shortLinkParams = hostParam ["&" portParam] ["&" keyHashParam] +hostParam = %s"h=" hostList +hostList = host *("," host) +portParam = %s"p=" port +keyHashParam = %s"c=" base64url ; router certificate fingerprint +``` + +Contact types: +- `a` (CCTContact) - direct contact connection +- `c` (CCTChannel) - channel connection +- `g` (CCTGroup) - group connection +- `r` (CCTRelay) - relay connection + +Short links can use either the `simplex:` scheme or `https://` with a router hostname. When using the simplex scheme, router information is included in query parameters. + +## Short links + +Short links provide a compact representation of connection links by storing encrypted connection data on the SMP router. The link key in the URI fragment (after `#`) is never sent to the router, ensuring the router cannot decrypt the stored connection data. + +### Link key derivation + +The link key is derived from the fixed link data using SHA3-256 hash function: + +``` +linkKey = SHA3-256(fixedLinkData) +``` + +The fixed link data includes: +- Agent version range +- Root public key (Ed25519) for signing +- SMP queue connection request (router, queue IDs, encryption keys) +- Optional link entity ID + +For contact links, the link ID and encryption key are derived from the link key using HKDF: + +``` +(linkId, encryptionKey) = HKDF(info="SimpleXContactLink", key=linkKey, outputLen=56) +; linkId = first 24 bytes, encryptionKey = remaining 32 bytes +``` + +For invitation links, the link ID is stored separately (usually included in the URI), and only the encryption key is derived: + +``` +encryptionKey = HKDF(info="SimpleXInvLink", key=linkKey, outputLen=32) +``` + +### Link data encryption + +Link data stored on the router consists of two encrypted parts: fixed data and user data. Both are encrypted using NaCl secret_box (XSalsa20-Poly1305) with the derived encryption key: + +```abnf +queueLinkData = encFixedData encUserData +encFixedData = largeString ; encrypted padded(signedFixedData, 2008) +encUserData = largeString ; encrypted padded(signedUserData, 13784) + +signedFixedData = signature fixedData +signedUserData = signature userData +signature = length 64*64 OCTET ; Ed25519 signature + +fixedData = agentVersionRange rootKey linkConnReq [linkEntityId] +agentVersionRange = version version ; min and max agent protocol version +version = 2*2 OCTET +rootKey = length x509encoded ; Ed25519 public key +linkConnReq = invitationConnReq / contactConnReq ; binary encoding of connection request +invitationConnReq = %s"I" connReqData e2eRatchetParams +contactConnReq = %s"C" connReqData +linkEntityId = shortString +userData = invitationLinkData / contactLinkData +invitationLinkData = %s"I" agentVersionRange userLinkData +contactLinkData = %s"C" agentVersionRange userContactData +userLinkData = shortString / (%xFF largeString) ; opaque application data (e.g., user profile) + ; shortString length byte 0x00-0xFE (max 254 bytes); 0xFF is reserved as largeString sentinel +userContactData = direct ownersList relaysList userLinkData +direct = %s"T" / %s"F" ; whether direct connection via connReq is allowed +ownersList = length *ownerAuth +ownerAuth = shortString ; length-prefixed encoding of (ownerId ownerKey authOwnerSig) +ownerId = shortString ; application-specific owner ID (e.g., MemberId) +ownerKey = length x509encoded ; Ed25519 public key +authOwnerSig = length 64*64 OCTET ; Ed25519 signature of (ownerId || ownerKey) by previous owner +relaysList = length *connShortLink ; alternative relay short links + +; Binary encoding of connection request (used in linkConnReq) +connReqData = agentVersionRange smpQueueUris clientData +smpQueueUris = length 1*smpQueueUri +clientData = %s"0" / (%s"1" largeString) ; Maybe (Large ByteString) +smpQueueUri = smpClientVersionRange smpServer senderId smpDhPublicKey [queueMode] +smpClientVersionRange = version version ; min and max SMP client versions +smpServer = hosts port serverKeyHash +hosts = length 1*host +host = shortString ; text-encoded hostname or IP address +port = shortString ; text-encoded port number +serverKeyHash = shortString ; CA certificate fingerprint +senderId = shortString ; queue sender ID +smpDhPublicKey = length x509encoded ; X25519 DH public key +queueMode = %s"M" / %s"C" ; messaging or contact (version-dependent trailing field) +e2eRatchetParams = e2eVersionRange e2eDhKey e2eDhKey kemParams +e2eVersionRange = version version ; min and max e2e encryption versions +e2eDhKey = length x509encoded ; X448 DH public key +kemParams = %s"0" / (%s"1" ratchetKEMParams) +ratchetKEMParams = %s"P" kemPublicKey / %s"A" kemCiphertext kemPublicKey +kemPublicKey = largeString ; sntrup761 public key +kemCiphertext = largeString ; sntrup761 ciphertext + +; Binary encoding of short link (used in relaysList) +connShortLink = invShortLink / contactShortLink +invShortLink = %s"I" smpServer linkId linkKey +contactShortLink = %s"C" contactConnType smpServer linkKey +contactConnType = %s"A" / %s"C" / %s"G" / %s"R" ; contact / channel / group / relay +linkId = shortString +linkKey = shortString + +x509encoded = *OCTET ; DER-encoded X.509 SubjectPublicKeyInfo +largeString = 2*2 OCTET *OCTET ; Word16 length prefix +length = 1*1 OCTET +shortString = length *OCTET +``` + +The fixed data is signed with the root key and its hash becomes the link key. The user data is signed either with the root key (for invitations) or with an owner key (for contact addresses). + +### Short link resolution + +When a user receives a short link, the agent resolves it as follows: + +1. Extract the link key from the URI fragment +2. Send `LGET` command to the SMP router with the link ID +3. Receive encrypted link data from the router +4. Decrypt the link data using the link key +5. Extract the full connection information (SMP queue URI, encryption keys, profile) +6. Proceed with the standard connection procedure using `joinConnection` + +For invitation links, the `LKEY` command is used to set the sender key when getting link data. Repeated `LKEY` would require using the same key. + +### Link data management + +The recipient who created the queue can manage the short link data: + +- **LSET** - Set or update the link data associated with a queue. This is used when creating a short link or updating the user data (e.g., profile changes). +- **LDEL** - Delete the link data from the router. This effectively invalidates the short link. + +Short links support different connection modes: +- **invitation** - One-time invitation links that can only be used once +- **contact** - Reusable contact address links that can be used multiple times + +For contact addresses, the link data includes additional information about the contact type: +- **contact** - Direct contact connection +- **channel** - Channel connection +- **group** - Group connection +- **relay** - Relay connection + +The agent maintains the link data and updates it when connection parameters change, ensuring short links remain valid and reflect current connection information. + ## Appendix A: SMP agent API The exact specification of agent library API and of the events that the agent sends to the client application is out of scope of the protocol specification. @@ -380,7 +575,7 @@ The list of some of the API functions and events below is supported by the refer The list of APIs below is not exhaustive and provided for information only. Please consult the source code for more information. -#### Create conection +#### Create connection `createConnection` api is used to create a connection - it returns the connection link that should be sent out-of-band to another protocol user (the joining party). It should be used by the client of the agent that initiates creating a duplex connection (the initiating party). @@ -408,13 +603,13 @@ Client can `acceptContact` and `rejectContact`, with `OK` and `ERR` events in ca #### Send message -`sendMessage` api is always asynchronous. The api call returns message ID, `SENT` event once the message is sent to the server, `MWARN` event in case of temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client) and `MERR` in case of permanent delivery failure. +`sendMessage` api is always asynchronous. The api call returns message ID, `SENT` event once the message is sent to the router, `MWARN` event in case of temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client) and `MERR` in case of permanent delivery failure. #### Acknowledge received message Messages are delivered to the client application via `MSG` event. -Client application must always `ackMessage` to receive the next one - failure to call it in reference implementation will prevent the delivery of subsequent messages until the client reconnects to the server. +Client application must always `ackMessage` to receive the next one - failure to call it in reference implementation will prevent the delivery of subsequent messages until the client reconnects to the router. This api is also used to acknowledge message delivery to the sending party - that party client application will receive `RCVD` event. @@ -426,9 +621,17 @@ This api is also used to acknowledge message delivery to the sending party - tha `getNotificationMessage` is used by push notification subsystem of the client application to receive the message from a specific messaging queue mentioned in the notification. The client application would receive `MSG` and any other events from the agent, and then `MSGNTF` event once the message related to this notification is received. -#### Rotate message queue to another server +#### Set short link data + +`setConnectionLink` api (`LSET` command) is used to set or update short link data associated with a contact address queue. Returns `LINK` event with the short link URI. + +#### Get short link data + +`getConnectionLink` api (`LGET` command) is used to retrieve and decrypt the short link data from the router. Returns `LDATA` event with the decrypted link data. + +#### Rotate message queue to another router -`switchConnection` api is used to rotate connection queues to another messaging server. +`switchConnection` api is used to rotate connection queues to another messaging router. #### Renegotiate e2e encryption @@ -436,7 +639,7 @@ This api is also used to acknowledge message delivery to the sending party - tha #### Delete connection -`deleteConnection` api is used to delete connection. In case of asynchronous call, the connection deletion will be confirmed with `DEL_RCVQ` and `DEL_CONN` events. +`deleteConnection` api is used to delete connection. In case of asynchronous call, the connection deletion will be confirmed with `DEL_RCVQS` and `DEL_CONNS` events. #### Suspend connection @@ -451,25 +654,80 @@ Agent API uses these events dispatch to notify client application about events r - `INFO` - information from the party that initiated the connection with `createConnection` sent to the party accepting the connection with `joinConnection`. - `CON` - notification that connection is established sent to both parties of the connection. - `END` - notification that connection subscription is terminated when another client subscribed to the same messaging queue. -- `DOWN` - notification that connection server is temporarily unavailable. -- `UP` - notification that the subscriptions made in the current client session are resumed after the server became available. +- `DOWN` - notification that connection router is temporarily unavailable. +- `UP` - notification that the subscriptions made in the current client session are resumed after the router became available. - `SWITCH` - notification about queue rotation process. - `RSYNC` - notification about e2e encryption re-negotiation process. -- `SENT` - notification to confirm that the message was delivered to at least one of SMP servers. This notification contains the same message ID as returned to `sendMessage` api. `SENT` notification, depending on network availability, can be sent at any time later, potentially in the next client session. +- `SENT` - notification to confirm that the message was delivered to at least one of SMP routers. This notification contains the same message ID as returned to `sendMessage` api. `SENT` notification, depending on network availability, can be sent at any time later, potentially in the next client session. - `MWARN` - temporary delivery failure that can be resolved by the user (e.g., by connecting via Tor or by upgrading the client). - `MERR` - notification about permanent message delivery failure. - `MERRS` - notification about permanent message delivery failure for multiple messages (e.g., when multiple messages expire). -- `MSG` - sent when agent receives the message from the SMP server. +- `MSG` - sent when agent receives the message from the SMP router. - `MSGNTF` - sent after agent received and processed the message referenced in the push notification. - `RCVD` - notification confirming message receipt by another party. - `QCONT` - notification that the agent continued sending messages after queue capacity was exceeded and recipient received all messages. -- `DEL_RCVQ` - confirmation that message queue was deleted. -- `DEL_CONN` - confirmation that connection was deleted. +- `LINK` - short link URI created or updated for a contact address. +- `LDATA` - decrypted short link data received from the router. +- `DELD` - notification that the connection was deleted. +- `JOINED` - notification that a member joined via a contact address. +- `STAT` - connection statistics event. +- `DEL_RCVQS` - confirmation that receiver message queues were deleted. +- `DEL_CONNS` - confirmation that connections were deleted. - `OK` - confirmation that asynchronous api call was successful. - `ERR` - error of asynchronous api call or some other error event. This list of events is not exhaustive and provided for information only. Please consult the source code for more information. +## Threat model + +This threat model complements SimpleX Messaging Protocol [threat model](./security.md#threat-model) with agent-level concerns: duplex connections, end-to-end encryption with [post-quantum double ratchet](./pqdr.md), message integrity, connection establishment and queue rotation. Only additional properties not covered in the SMP threat model are listed below. + +#### Additional global assumptions + + - The connection link is shared via a trusted out-of-band channel. + - Both agents support post-quantum double ratchet (PQDR). + +#### A passive adversary + +*cannot:* + - learn the contents of packets, which are additionally encrypted with the double ratchet independently from per-queue encryption. + +#### Destination router (chosen by the receiving client application) + +*can:* + - correlate queues belonging to the same duplex connection when queue rotation creates a new queue on the same router. + - when both peers of a connection chose the same router, correlate the two directions of the duplex connection. + +*cannot:* + - compromise end-to-end encryption even with full access to the per-queue NaCl DH secret. + - correlate queues belonging to the same connection after queue rotation to a different router. + +#### An attacker who obtained a client application's (decrypted) database + +*can:* + - learn the full communication graph: all communication peers, associated router addresses, and queue identifiers. + +*cannot:* + - decrypt future messages once the client application resumes communication and the double ratchet completes a new ratchet step, provided PQDR is active. + +#### A communication peer + +*can:* + - send malformed agent messages that may affect the client application processing them. + - skip message IDs, causing the recipient to generate and store excessive intermediate ratchet keys. + - prevent double ratchet advancement by not sending messages, delaying break-in recovery. + +*cannot:* + - disrupt packet delivery in other queues. + +#### An attacker who obtained a connection link + +*can:* + - learn the initiating party's chosen router address and public keys. + +*cannot:* + - use the link after the intended recipient has completed the connection. + [1]: https://en.wikipedia.org/wiki/End-to-end_encryption [2]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack [3]: https://tools.ietf.org/html/rfc5234 diff --git a/protocol/overview-tjr.md b/protocol/overview-tjr.md index d30f55235..2fa8f717e 100644 --- a/protocol/overview-tjr.md +++ b/protocol/overview-tjr.md @@ -1,4 +1,4 @@ -Revision 2, 2024-06-22 +Revision 4, 2026-03-09 Evgeny Poberezkin @@ -8,16 +8,17 @@ Evgeny Poberezkin - [Introduction](#introduction) - [What is SimpleX](#what-is-simplex) + - [Network model](#network-model) + - [Applications](#applications) - [SimpleX objectives](#simplex-objectives) - [In Comparison](#in-comparison) - [Technical Details](#technical-details) - - [Trust in Servers](#trust-in-servers) - - [Client -> Server Communication](#client---server-communication) + - [Trust in Routers](#trust-in-routers) + - [Client -> Router Communication](#client---router-communication) - [2-hop Onion Message Routing](#2-hop-onion-message-routing) - [SimpleX Messaging Protocol](#simplex-messaging-protocol) - [SimpleX Agents](#simplex-agents) - - [Encryption Primitives Used](#encryption-primitives-used) -- [Threat model](#threat-model) +- [Security](#security) - [Acknowledgements](#acknowledgements) @@ -27,27 +28,27 @@ Evgeny Poberezkin SimpleX as a whole is a platform upon which applications can be built. [SimpleX Chat](https://github.com/simplex-chat/simplex-chat) is one such application that also serves as an example and reference application. - - [SimpleX Messaging Protocol](./simplex-messaging.md) (SMP) is a protocol to send messages in one direction to a recipient, relying on a server in-between. The messages are delivered via uni-directional queues created by recipients. - - - SMP protocol allows to send message via a SMP server playing proxy role using 2-hop onion routing (referred to as "private routing" in messaging clients) to protect transport information of the sender (IP address and session) from the server chosen (and possibly controlled) by the recipient. + - [SimpleX Messaging Protocol](./simplex-messaging.md) (SMP) is a protocol to send messages in one direction to a recipient, relying on a router in-between. The messages are delivered via uni-directional queues created by recipients. + + - SMP protocol allows to send message via a SMP router playing proxy role using 2-hop onion routing (referred to as "private routing" in messaging clients) to protect transport information of the sender (IP address and session) from the router chosen (and possibly controlled) by the recipient. - SMP runs over a transport protocol (shown below as TLS) that provides integrity, server authentication, confidentiality, and transport channel binding. - - A SimpleX Server is one of those servers. + - A SimpleX router is one of those routers. - - The SimpleX Network is the term used for the collective of SimpleX Servers that facilitate SMP. + - The SimpleX Network is the term used for the collective of SimpleX routers that facilitate SMP. - - SimpleX Client libraries speak SMP to SimpleX Servers and provide a low-level API not generally intended to be used by applications. + - SimpleX Client libraries speak SMP to SimpleX routers and provide a low-level API not generally intended to be used by applications. - SimpleX Agents interface with SimpleX Clients to provide a more high-level API intended to be used by applications. Typically they are embedded as libraries, but can also be abstracted into local services. - SimpleX Agents communicate with other agents inside e2e encrypted envelopes provided by SMP protocol - the syntax and semantics of the messages exchanged by the agent are defined by [SMP agent protocol](./agent-protocol.md) -*Diagram showing the SimpleX Chat app, with logical layers of the chat application interfacing with a SimpleX Agent library, which in turn interfaces with a SimpleX Client library. The Client library in turn speaks the Messaging Protocol to a SimpleX Server.* +*Diagram showing the SimpleX Chat app, with logical layers of the chat application interfacing with a SimpleX Agent library, which in turn interfaces with a SimpleX Client library. The Client library in turn speaks the Messaging Protocol to a SimpleX router.* ``` - User's Computer Internet Third-Party Server + User's Computer Internet Third-Party Router ------------------ | ---------------------- | ------------------------- | | SimpleX Chat | | @@ -57,11 +58,43 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX +----------------+ | | | SimpleX Agent | | | +----------------+ -------------- TLS ---------------- +----------------+ -| SimpleX Client | ------ SimpleX Messaging Protocol ------> | SimpleX Server | +| SimpleX Client | ------ SimpleX Messaging Protocol ------> | SimpleX router | +----------------+ ----------------------------------- +----------------+ | | ``` +#### Network model + +SimpleX is a general-purpose packet routing network built on top of the Internet. Network endpoints — end-user devices, automated services, AI-enabled applications, IoT devices — exchange data packets through SimpleX network nodes (SMP routers), which accept, buffer, and deliver packets. Each router operates independently and can be operated by any party on standard computing hardware. + +SimpleX routers use resource-based addressing: each address identifies a resource on a router, similar to how the World Wide Web addresses resources via URLs. Internet routers, by comparison, use endpoint-based addressing, where IP addresses identify destination devices. Because of this design, SimpleX network participants do not need globally unique addresses to communicate. + +SimpleX network has two resource-based addressing schemes: + +- *Messaging queues* ([SMP](./simplex-messaging.md)). A queue is a unidirectional, ordered sequence of fixed-size data packets (16,384 bytes each). Each queue has a resource address on a specific router, gated by cryptographic credentials that separately authorize sending and receiving. + +- *Data packets* ([XFTP](./xftp.md)). A data packet is an individually addressed block in one of the standard sizes. Each packet has a unique resource address on a specific router, gated by cryptographic credentials. Data packet addressing is more efficient for delivery of larger payloads than queues. + +Packet delivery follows a two-router path. The sending endpoint submits a packet to a first router, which forwards it to a second router, where the receiving endpoint retrieves it. The sending endpoint's IP address is known only to the first router; the receiving endpoint's IP address is known only to the second router. See [2-hop Onion Message Routing](#2-hop-onion-message-routing) for details. + +Routers buffer packets between submission and retrieval — from seconds to days, enabling asynchronous delivery when endpoints are online at different times. Packets are removed after delivery or after a configured expiration period. + + +#### Applications + +Applications currently using SimpleX network: + +- **SimpleX Chat** — a peer-to-peer messenger using SimpleX network as a transport layer, in the same way that communication applications use WebRTC, Tor, i2p, or Nym. All communication logic — contacts, conversations, groups, message formats, end-to-end encryption — runs on endpoint devices. + +- **IoT devices** — using the SimpleX queue protocol directly for sensor data collection and device control. + +- **AI-based services** — automated services built on the SimpleX Chat application core. + +- **Secure monitoring and control systems** — applications for equipment monitoring and control, including robotics, using the network for command delivery and telemetry collection. + +[SimpleGo](https://simplego.dev), developed by an independent organization, is a microcontroller-based device running a SimpleX Chat-compatible messenger directly on a microcontroller without a general-purpose operating system. Running over 20 days on a single battery charge, it demonstrates the energy efficiency of resource-based addressing: the device receives packets without continuous polling. A microcontroller-based router implementation that functions simultaneously as a WiFi router is also in development. + + #### SimpleX objectives 1. Provide messaging infrastructure for distributed applications. This infrastructure needs to have the following qualities: @@ -70,7 +103,7 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX - Privacy: protect against traffic correlation attacks to determine the contacts that the users communicate with. - - Reliability: the messages should be delivered even if some participating network servers or receiving clients fail, with “at least once” delivery guarantee. + - Reliability: the messages should be delivered even if some participating network routers or receiving clients fail, with "at least once" delivery guarantee. - Integrity: the messages sent in one direction are ordered in a way that sender and recipient agree on; the recipient can detect when a message was removed or changed. @@ -78,63 +111,63 @@ SimpleX as a whole is a platform upon which applications can be built. [SimpleX - Low latency: the delay introduced by the network should not be higher than 100ms-1s in addition to the underlying TCP network latency. -2. Provide better communication security and privacy than the alternative instant messaging solutions. In particular SimpleX provides better privacy of metadata (who talks to whom and when) and better security against active network attackers and malicious servers. +2. Provide better communication security and privacy than the alternative instant messaging solutions. In particular SimpleX provides better privacy of metadata (who talks to whom and when) and better security against active network attackers and malicious routers. 3. Balance user experience with privacy requirements, prioritizing experience of mobile device users. #### In Comparison -SimpleX network has a design similar to P2P networks, but unlike most P2P networks it consists of clients and servers without depending on any centralized component. +SimpleX network has a design similar to P2P networks, but unlike most P2P networks it consists of clients and routers without depending on any centralized component. In comparison to more traditional messaging applications (e.g. WhatsApp, Signal, Telegram) the key differences of SimpleX network are: - participants do not need to have globally unique addresses to communicate, instead they use redundant unidirectional (simplex) messaging queues, with a separate set of queues for each contact. - connection requests are passed out-of-band, non-optionally protecting key exchange against man-in-the-middle attack. -- simple message queues provided by network servers are used by the clients to create more complex communication scenarios, such as duplex one-to-one communication, transmitting files, group communication without central servers, and content/communication channels. +- simple message queues provided by network routers are used by the clients to create more complex communication scenarios, such as duplex one-to-one communication, transmitting files, group communication without central routers, and content/communication channels. -- servers do not store any user information (no user profiles or contacts, or messages once they are delivered), and primarily use in-memory persistence. +- routers do not store any user information (no user profiles or contacts, or messages once they are delivered), and primarily use in-memory persistence. -- users can change servers with minimal disruption - even after an in-use server disappears, simply by changing the configuration on which servers the new queues are created. +- users can change routers with minimal disruption - even after an in-use router disappears, simply by changing the configuration on which routers the new queues are created. ## Technical Details -#### Trust in Servers +#### Trust in Routers -Clients communicate directly with servers (but not with other clients) using SimpleX Messaging Protocol (SMP) running over some transport protocol that provides integrity, server authentication, confidentiality, and transport channel binding. By default, we assume this transport protocol is TLS. +Clients communicate directly with routers (but not with other clients) using SimpleX Messaging Protocol (SMP) running over some transport protocol that provides integrity, server authentication, confidentiality, and transport channel binding. By default, we assume this transport protocol is TLS. -Users use multiple servers, and choose where to receive their messages. Accordingly, they send messages to their communication partners' chosen servers either directly, if this is a known/trusted server, or via another SMP server providing proxy functionality to protect IP address and session of the sender. +Users use multiple routers, and choose where to receive their messages. Accordingly, they send messages to their communication partners' chosen routers either directly, if this is a known/trusted router, or via another SMP router providing proxy functionality to protect IP address and session of the sender. -Although end-to-end encryption is always present, users place a degree of trust in servers they connect to. This trust decision is very similar to a user's choice of email provider; however the trust placed in a SimpleX server is significantly less. Notably, there is no re-used identifier or credential between queues on the same (or different) servers. While a user *may* re-use a transport connection to fetch messages from multiple queues, or connect to a server from the same IP address, both are choices a user may opt into to break the promise of un-correlatable queues. +Although end-to-end encryption is always present, users place a degree of trust in routers they connect to. This trust decision is very similar to a user's choice of email provider; however the trust placed in a SimpleX router is significantly less. Notably, there is no re-used identifier or credential between queues on the same (or different) routers. While a user *may* re-use a transport connection to fetch messages from multiple queues, or connect to a router from the same IP address, both are choices a user may opt into to break the promise of un-correlatable queues. -Users may trust a server because: +Users may trust a router because: -- They deploy and control the servers themselves from the available open-source code. This has the trade-offs of strong trust in the server but limited metadata obfuscation to a passive network observer. Techniques such as noise traffic, traffic mixing (incurring latency), and using an onion routing transport protocol can mitigate that. +- They deploy and control the routers themselves from the available open-source code. This has the trade-offs of strong trust in the router but limited metadata obfuscation to a passive network observer. Techniques such as noise traffic, traffic mixing (incurring latency), and using an onion routing transport protocol can mitigate that. -- They use servers from a trusted commercial provider. The more clients the provider has, the less metadata about the communication times is leaked to the network observers. +- They use routers from a trusted commercial provider. The more clients the provider has, the less metadata about the communication times is leaked to the network observers. -By default, servers do not retain access logs, and permanently delete messages and queues when requested. Messages persist only in memory until they cross a threshold of time, typically on the order of days.[0] There is still a risk that a server maliciously records all queues and messages (even though encrypted) sent via the same transport connection to gain a partial knowledge of the user’s communications graph and other meta-data. +By default, routers do not retain access logs, and permanently delete messages and queues when requested. Messages persist in memory or in a database until they cross a threshold of time, typically on the order of days.[0] There is still a risk that a router maliciously records all queues and messages (even though encrypted) sent via the same transport connection to gain a partial knowledge of the user's communications graph and other meta-data. -SimpleX supports measures (managed transparently to the user at the agent level) to mitigate the trust placed in servers. These include rotating the queues in use between users, noise traffic, supporting overlay networks such as Tor, and isolating traffic to different queues to different transport connections (and Tor circuits, if Tor is used). +SimpleX supports measures (managed transparently to the user at the agent level) to mitigate the trust placed in routers. These include rotating the queues in use between users, noise traffic, supporting overlay networks such as Tor, and isolating traffic to different queues to different transport connections (and Tor circuits, if Tor is used). -[0] While configurable by servers, a minimum value is enforced by the default software. SimpleX Agents can provide redundant routing over queues to mitigate against message loss. +[0] While configurable by routers, a minimum value is enforced by the default software. SimpleX Agents can provide redundant routing over queues to mitigate against message loss. -#### Client -> Server Communication +#### Client -> Router Communication Utilizing TLS grants the SimpleX Messaging Protocol (SMP) server authentication and metadata protection to a passive network observer. But SMP does not rely on the transport protocol for message confidentiality or client authentication. The SMP protocol itself provides end-to-end confidentiality, authentication, and integrity of messages between communicating parties. -Servers have long-lived, self-signed, offline certificates whose hash is pre-shared with clients over secure channels - either provided with the client library or provided in the secure introduction between clients, as part of the server address. The offline certificate signs an online certificate used in the transport protocol handshake. [0] +Routers have long-lived, self-signed, offline certificates whose hash is pre-shared with clients over secure channels - either provided with the client library or provided in the secure introduction between clients, as part of the router address. The offline certificate signs an online certificate used in the transport protocol handshake. [0] -If the transport protocol's confidentiality is broken, incoming and outgoing messages to the server cannot be correlated by message contents. Additionally, because of encryption at the SMP layer, impersonating the server is not sufficient to pass (and therefore correlate) a message from a sender to recipient - the only attack possible is to drop the messages. Only by additionally *compromising* the server can one pass and correlate messages. +If the transport protocol's confidentiality is broken, incoming and outgoing messages to the router cannot be correlated by message contents. Additionally, because of encryption at the SMP layer, impersonating the router is not sufficient to pass (and therefore correlate) a message from a sender to recipient - the only attack possible is to drop the messages. Only by additionally *compromising* the router can one pass and correlate messages. -It's important to note that the SMP protocol does not do server authentication. Instead we rely upon the fact that an attacker who tricks the transport protocol into authenticating the server incorrectly cannot do anything with the SMP messages except drop them. +It's important to note that the SMP protocol does not do server authentication. Instead we rely upon the fact that an attacker who tricks the transport protocol into authenticating the router incorrectly cannot do anything with the SMP messages except drop them. -After the connection is established, the client sends blocks of a fixed size 16KB, and the server replies with the blocks of the same size to reduce metadata observable to a network adversary. The protocol has been designed to make traffic correlation attacks difficult, adapting ideas from Tor, remailers, and more general onion and mix networks. It does not try to replace Tor though - SimpleX servers can be deployed as onion services and SimpleX clients can communicate with servers over Tor to further improve participants privacy. +After the connection is established, the client sends blocks of a fixed size 16KB, and the router replies with the blocks of the same size to reduce metadata observable to a network adversary. The protocol has been designed to make traffic correlation attacks difficult, adapting ideas from Tor, remailers, and more general onion and mix networks. It does not try to replace Tor though - SimpleX routers can be deployed as onion services and SimpleX clients can communicate with routers over Tor to further improve participants privacy. -By using fixed-size blocks, oversized for the expected content, the vast majority of traffic is uniform in nature. When enough traffic is transiting a server simultaneously, the server acts as a low-latency mix node. We can't rely on this behavior to make a security claim, but we have engineered to take advantage of it when we can. As mentioned, this holds true even if the transport connection is compromised. +By using fixed-size blocks, oversized for the expected content, the vast majority of traffic is uniform in nature. When enough traffic is transiting a router simultaneously, the router acts as a low-latency mix node. We can't rely on this behavior to make a security claim, but we have engineered to take advantage of it when we can. As mentioned, this holds true even if the transport connection is compromised. The protocol does not protect against attacks targeted at particular users with known identities - e.g., if the attacker wants to prove that two known users are communicating, they can achieve it by observing their local traffic. At the same time, it substantially complicates large-scale traffic correlation, making determining the real user identities much less effective. @@ -143,39 +176,39 @@ The protocol does not protect against attacks targeted at particular users with #### 2-hop Onion Message Routing -As SimpleX Messaging Protocol servers providing messaging queues are chosen by the recipients, in case senders connect to these servers directly the server owners (who potentially can be the recipients themselves) can learn senders' IP addresses (if Tor is not used) and which other queues on the same server are accessed by the user in the same transport connection (even if Tor is used). +As SimpleX Messaging Protocol routers providing messaging queues are chosen by the recipients, in case senders connect to these routers directly the router owners (who potentially can be the recipients themselves) can learn senders' IP addresses (if Tor is not used) and which other queues on the same router are accessed by the user in the same transport connection (even if Tor is used). While the clients support isolating the messages sent to different queues into different transport connections (and Tor circuits), this is not practical, as it consumes additional traffic and system resources. -To mitigate this problem SimpleX Messaging Protocol servers support 2-hop onion message routing when the SMP server chosen by the sender forwards the messages to the servers chosen by the recipients, thus protecting both the senders IP addresses and sessions, even if connection isolation and Tor are not used. +To mitigate this problem SimpleX Messaging Protocol routers support 2-hop onion message routing when the SMP router chosen by the sender forwards the messages to the routers chosen by the recipients, thus protecting both the senders IP addresses and sessions, even if connection isolation and Tor are not used. The design of 2-hop onion message routing prevents these potential attacks: -- MITM by proxy (SMP server that forwards the messages). +- MITM by proxy (SMP router that forwards the messages). -- Identification by the proxy which and how many queues the sender sends messages to (as messages are additionally e2e encrypted between the sender and the destination SMP server). +- Identification by the proxy which and how many queues the sender sends messages to (as messages are additionally e2e encrypted between the sender and the destination SMP router). - Correlation of messages sent to different queues via the same user session (as random correlation IDs and keys are used for each message). See more details about 2-hop onion message routing design in [SimpleX Messaging Protocol](./simplex-messaging.md#proxying-sender-commands) -Also see [Threat model](#threat-model) +Also see [Security](./security.md) #### SimpleX Messaging Protocol -SMP is initialized with an in-person or out-of-band introduction message, where Alice provides Bob with details of a server (including IP address or host name, port, and hash of the long-lived offline certificate), a queue ID, and Alice's public keys to agree e2e encryption. These introductions are similar to the PANDA key-exchange, in that if observed, the adversary can race to establish the communication channel instead of the intended participant. [0] +SMP is initialized with an in-person or out-of-band introduction message, where Alice provides Bob with details of a router (including IP address or host name, port, and hash of the long-lived offline certificate), a queue ID, and Alice's public keys to agree e2e encryption. These introductions are similar to the PANDA key-exchange, in that if observed, the adversary can race to establish the communication channel instead of the intended participant. [0] Because queues are uni-directional, Bob provides an identically-formatted introduction message to Alice over Alice's now-established receiving queue. -When setting up a queue, the server will create separate sender and recipient queue IDs (provided to Alice during set-up and Bob during initial connection). Additionally, during set-up Alice will perform a DH exchange with the server to agree upon a shared secret. This secret will be used to re-encrypt Bob's incoming message before Alice receives it, creating the anti-correlation property earlier-described should the transport encryption be compromised. +When setting up a queue, the router will create separate sender and recipient queue IDs (provided to Alice during set-up and Bob during initial connection). Additionally, during set-up Alice will perform a DH exchange with the router to agree upon a shared secret. This secret will be used to re-encrypt Bob's incoming message before Alice receives it, creating the anti-correlation property earlier-described should the transport encryption be compromised. -[0] Users can additionally create public 'contact queues' that are only used to receive connection requests. +[0] Users can additionally create public 'contact queues' that are only used to receive connection requests. #### SimpleX Agents -SimpleX agents provide higher-level operations compared to SimpleX Clients, who are primarily concerned with creating queues and communicating with servers using SMP. Agent operations include: +SimpleX agents provide higher-level operations compared to SimpleX Clients, who are primarily concerned with creating queues and communicating with routers using SMP. Agent operations include: - Managing sets of bi-directional, redundant queues for communication partners @@ -186,195 +219,21 @@ SimpleX agents provide higher-level operations compared to SimpleX Clients, who - Noise traffic -#### Encryption Primitives Used - -- Ed25519 or Curve25519 to authorize/verify commands to SMP servers (authorization algorithm is set via client/server configuration). -- Curve25519 for DH exchange to agree: - - the shared secret between server and recipient (to encrypt message bodies - it avoids shared cipher-text in sender and recipient traffic) - - the shared secret between sender and recipient (to encrypt messages end-to-end in each queue - it avoids shared cipher-text in redundant queues). -- [NaCl crypto_box](https://nacl.cr.yp.to/box.html) encryption scheme (curve25519xsalsa20poly1305) for message body encryption between server and recipient and for E2E per-queue encryption. -- SHA256 to validate server offline certificates. -- [double ratchet](https://signal.org/docs/specifications/doubleratchet/) protocol for end-to-end message encryption between the agents: - - Curve448 keys to agree shared secrets required for double ratchet initialization (using [X3DH](https://signal.org/docs/specifications/x3dh/) key agreement with 2 ephemeral keys for each side), - - AES-GCM AEAD cipher, - - SHA512-based HKDF for key derivation. - - -## Threat Model - -#### Global Assumptions - - - A user protects their local database and key material. - - The user's application is authentic, and no local malware is running. - - The cryptographic primitives in use are not broken. - - A user's choice of servers is not directly tied to their identity or otherwise represents distinguishing information about the user. - - The user's client uses 2-hop onion message routing. - -#### A passive adversary able to monitor the traffic of one user - -*can:* - - - identify that and when a user is using SimpleX. - - - determine which servers the user receives the messages from. - - - observe how much traffic is being sent, and make guesses as to its purpose. - -*cannot:* - - - see who sends messages to the user and who the user sends the messages to. - - - determine the servers used by users' contacts. - -#### A passive adversary able to monitor a set of senders and recipients - - *can:* - - - identify who and when is using SimpleX. - - - learn which SimpleX Messaging Protocol servers are used as receive queues for which users. - - - learn when messages are sent and received. - - - perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the servers. - - - observe how much traffic is being sent, and make guesses as to its purpose - -*cannot, even in case of a compromised transport protocol:* - - - perform traffic correlation attacks with any increase in efficiency over a non-compromised transport protocol - -#### SimpleX Messaging Protocol server - -*can:* - -- learn when a queue recipient is online - -- know how many messages are sent via the queue (although some may be noise or not content messages). - -- learn which messages would trigger notifications even if a user does not use [push notifications](./push-notifications.md). - -- perform the correlation of the queue used to receive messages (matching multiple queues to a single user) via either a re-used transport connection, user's IP Address, or connection timing regularities. - -- learn a recipient's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, as long as Tor is not used. - -- drop all future messages inserted into a queue, detectable only over other, redundant queues. - -- lie about the state of a queue to the recipient and/or to the sender (e.g. suspended or deleted when it is not). - -- spam a user with invalid messages. - -*cannot:* - -- undetectably add, duplicate, or corrupt individual messages. - -- undetectably drop individual messages, so long as a subsequent message is delivered. - -- learn the contents or type of messages. - -- distinguish noise messages from regular messages except via timing regularities. - -- compromise the users' end-to-end encryption with an active attack. - -- learn a sender's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, even if Tor is not used (provided messages are sent via proxy SMP server). - -- perform senders' queue correlation (matching multiple queues to a single sender) via either a re-used transport connection, user's IP Address, or connection timing regularities, unless it has additional information from the proxy SMP server (provided messages are sent via proxy SMP server). - -#### SimpleX Messaging Protocol server that proxies the messages to another SMP server - -*can:* - -- learn a sender's IP address, as long as Tor is not used. - -- learn when a sender with a given IP address is online. - -- know how many messages are sent from a given IP address and to a given destination SMP server. - -- drop all messages from a given IP address or to a given destination server. - -- unless destination SMP server detects repeated public DH keys of senders, replay messages to a destination server within a single session, causing either duplicate message delivery (which will be detected and ignored by the receiving clients), or, when receiving client is not connected to SMP server, exhausting capacity of destination queues used within the session. - -*cannot:* - -- perform queue correlation (matching multiple queues to a single user), unless it has additional information from the destination SMP server. - -- undetectably add, duplicate, or corrupt individual messages. - -- undetectably drop individual messages, so long as a subsequent message is delivered. - -- learn the contents or type of messages. - -- learn which messages would trigger notifications. - -- learn the destination queues of messages. - -- distinguish noise messages from regular messages except via timing regularities. - -- compromise the user's end-to-end encryption with another user via an active attack. - -- compromise the user's end-to-end encryption with the destination SMP servers via an active attack. - -#### An attacker who obtained Alice's (decrypted) chat database - -*can:* - -- see the history of all messages exchanged by Alice with her communication partners. - -- see shared profiles of contacts and groups. - -- surreptitiously receive new messages sent to Alice via existing queues; until communication queues are rotated or the Double-Ratchet advances forward. - -- prevent Alice from receiving all new messages sent to her - either surreptitiously by emptying the queues regularly or overtly by deleting them. - -- send messages from the user to their contacts; recipients will detect it as soon as the user sends the next message, because the previous message hash won’t match (and potentially won’t be able to decrypt them in case they don’t keep the previous ratchet keys). - -*cannot:* - -- impersonate a sender and send messages to the user whose database was stolen. Doing so requires also compromising the server (to place the message in the queue, that is possible until the Double-Ratchet advances forward) or the user's device at a subsequent time (to place the message in the database). - -- undetectably communicate at the same time as Alice with her contacts. Doing so would result in the contact getting different messages with repeated IDs. - -- undetectably monitor message queues in realtime without alerting the user they are doing so, as a second subscription request unsubscribes the first and notifies the second. - -#### A user’s contact - -*can:* - -- spam the user with messages. - -- forever retain messages from the user. - -*cannot:* - -- cryptographically prove to a third-party that a message came from a user (assuming the user’s device is not seized). - -- prove that two contacts they have is the same user. - -- cannot collaborate with another of the user's contacts to confirm they are communicating with the same user. - -#### An attacker who observes Alice showing an introduction message to Bob - -*can:* - - - Impersonate Bob to Alice. - -*cannot:* - - - Impersonate Alice to Bob. +## Security -#### An attacker with Internet access +For encryption primitives, threat model, and detailed security analysis, see [Security](./security.md). -*can:* +SimpleX provides these security properties: -- Denial of Service SimpleX messaging servers. +- **End-to-end encryption** with forward secrecy via double ratchet protocol, with optional post-quantum protection. -- spam a user's public “contact queue” with connection requests. +- **No shared identifiers** across connections — contacts cannot prove they communicate with the same user. -*cannot:* +- **Sender deniability** — neither routers nor recipients can cryptographically prove message origin. -- send messages to a user who they are not connected with. +- **Transport metadata protection** — fixed-size blocks, 2-hop onion routing, and connection isolation frustrate traffic correlation. -- enumerate queues on a SimpleX server. +- **Out-of-band key exchange** — connection requests passed outside the network protect against MITM attacks. ## Acknowledgements diff --git a/protocol/pqdr.md b/protocol/pqdr.md index 27f7082c8..d3d3e2b48 100644 --- a/protocol/pqdr.md +++ b/protocol/pqdr.md @@ -13,6 +13,11 @@ Version 1, 2024-06-22 - [Initialization](#initialization) - [Encrypting messages](#encrypting-messages) - [Decrypting messages](#decrypting-messages) +- [Ratchet message wire format](#ratchet-message-wire-format) + - [Encrypted ratchet message](#encrypted-ratchet-message) + - [Encrypted message header](#encrypted-message-header) + - [Plaintext message header](#plaintext-message-header) + - [KEM state machine](#kem-state-machine) - [Implementation considerations](#implementation-considerations) - [Chosen KEM algorithm](#chosen-kem-algorithm) - [Summary](#summary) @@ -71,11 +76,10 @@ def RatchetInitAlicePQ2HE(state, SK, bob_dh_public_key, shared_hka, shared_nhkb, // below added for post-quantum KEM state.PQRs = GENERATE_PQKEM() state.PQRr = bob_pq_kem_encapsulation_key - state.PQRss = random // shared secret for KEM - state.PQRct = PQKEM-ENC(state.PQRr, state.PQRss) // encapsulated additional shared secret + state.PQRct, state.PQRss = PQKEM-ENC(state.PQRr) // encapsulate: generates shared secret and ciphertext // above added for KEM // the next line augments DH key agreement with PQ shared secret - state.RK, state.CKs, state.NHKs = KDF_RK_HE(SK, DH(state.DHRs, state.DHRr) || state.PQRss) + state.RK, state.CKs, state.NHKs = KDF_RK_HE(SK, DH(state.DHRs, state.DHRr) || state.PQRss) state.CKr = None state.Ns = 0 state.Nr = 0 @@ -176,8 +180,7 @@ def DHRatchetPQ2HE(state, header): state.DHRs = GENERATE_DH() // below is added for KEM state.PQRs = GENERATE_PQKEM() // generate new PQ key pair - state.PQRss = random // shared secret for KEM - state.PQRct = PQKEM-ENC(state.PQRr, state.PQRss) // encapsulated additional shared secret KEM #1 + state.PQRct, state.PQRss = PQKEM-ENC(state.PQRr) // encapsulate: generates shared secret and ciphertext KEM #1 // above is added for KEM // use new shared secret with sending ratchet state.RK, state.CKs, state.NHKs = KDF_RK_HE(state.RK, DH(state.DHRs, state.DHRr) || state.PQRss) @@ -191,6 +194,80 @@ Other than augmenting DH key agreements with the shared secrets from KEM, the ab It is worth noting that while DH agreements work as ping-pong, when the new received DH key is used for both DH agreements (and only the sent DH key is updated for the second DH key agreement), PQ KEM agreements in the proposed scheme work as a "parallel ping-pong", with two balls in play all the time (two KEM agreements run in parallel). +## Ratchet message wire format + +The pseudocode above describes the algorithm. This section specifies the actual binary encoding used in SimpleX implementation with Curve448 DH keys, sntrup761 KEM and AES-256-GCM AEAD. + +The ratchet-encrypted message has three encoding layers, from outermost to innermost: + +1. **Encrypted ratchet message** — the complete ratchet message envelope, referenced as an opaque encrypted body in [agent protocol](./agent-protocol.md). +2. **Encrypted message header** — the encrypted header within the ratchet message, used as associated data for message body encryption. +3. **Plaintext message header** — the DH and KEM ratchet keys and counters. + +### Encrypted ratchet message + +The outer envelope contains the encrypted header (used as associated data for body authentication), the body authentication tag, and the encrypted message body. + +The message body is encrypted with AES-256-GCM using the message key derived from the sending chain key (`KDF_CK`). The associated data for body encryption is the concatenation of the ratchet associated data and the encoded encrypted header. + +```abnf +encRatchetMessage = versionedLength encMessageHeader msgAuthTag encMsgBody +; encMessageHeader is used as associated data for body decryption: AD = rcAD || encMessageHeader +msgAuthTag = 16*16 OCTET ; AES-256-GCM authentication tag for the message body +encMsgBody = *OCTET ; AES-256-GCM encrypted padded message body (remaining bytes) +``` + +### Encrypted message header + +The encrypted header wraps the current ratchet e2e encryption version, an initialization vector, an authentication tag, and the encrypted padded header body. + +The header body is encrypted with AES-256-GCM using the header key (`HKs`). The associated data for header encryption is the ratchet associated data. The header is padded before encryption to a fixed size to prevent leaking information about the KEM state. + +```abnf +encMessageHeader = currentVersion headerIV headerAuthTag versionedLength encHeaderBody +currentVersion = 2*2 OCTET ; Word16, current ratchet e2e encryption version +headerIV = 16*16 OCTET ; AES-256 initialization vector for header encryption +headerAuthTag = 16*16 OCTET ; AES-256-GCM authentication tag for the header +encHeaderBody = *OCTET ; AES-256-GCM encrypted padded header (see plaintext format below) +``` + +`versionedLength` uses a 2-byte length prefix (Word16) when the current e2e version supports PQ encryption, or a 1-byte length prefix otherwise. The parser distinguishes the two encodings by peeking at the first byte: values below 32 indicate a 2-byte prefix (as the header is always at least 69 bytes). + +```abnf +versionedLength = largeLength / length ; 2-byte for PQ versions, 1-byte for pre-PQ versions +``` + +The padded header sizes before encryption are: 2310 bytes when PQ is supported, 88 bytes when PQ is not supported. Padding uses a 2-byte big-endian length prefix followed by the plaintext header and `#` fill bytes. + +### Plaintext message header + +```abnf +msgHeader = maxVersion dhPublicKey [kemParams] prevMsgCount msgCount +maxVersion = 2*2 OCTET ; Word16, max supported e2e encryption version +dhPublicKey = length x509encoded ; Curve448 public DH ratchet key +kemParams = noKEM / proposedKEM / acceptedKEM + ; present only when current ratchet version >= pqRatchetE2EEncryptVersion +noKEM = %x30 ; "0" - no KEM parameters +proposedKEM = %x31 %s"P" kemEncapsulationKey ; KEM proposed, not yet accepted +acceptedKEM = %x31 %s"A" kemCiphertext kemEncapsulationKey ; KEM accepted +kemEncapsulationKey = largeLength 1158*1158 OCTET ; sntrup761 encapsulation key +kemCiphertext = largeLength 1039*1039 OCTET ; sntrup761 ciphertext +prevMsgCount = 4*4 OCTET ; Word32, number of messages in previous sending chain +msgCount = 4*4 OCTET ; Word32, message number in current sending chain +length = 1*1 OCTET +largeLength = 2*2 OCTET ; Word16 +``` + +### KEM state machine + +PQ encryption can be enabled or disabled during a connection's lifetime. The KEM parameters in the header reflect three states: + +- **No KEM** (`noKEM`): PQ encryption is not active. The header contains only the DH key, as in the original double ratchet. +- **Proposed** (`proposedKEM`): One party generated a KEM key pair and includes the encapsulation key in the header, proposing PQ encryption. No ciphertext is included because the other party has not yet sent its encapsulation key. +- **Accepted** (`acceptedKEM`): The party received the other's encapsulation key, performed encapsulation (KEM #1), and includes both the ciphertext and its own new encapsulation key (for KEM #2). This is the steady state for active PQ encryption. + +The transition from Proposed to Accepted happens when a party receives a message containing KEM parameters (either Proposed or Accepted) and responds with its own Accepted parameters. Once both parties are in Accepted state, the double PQ KEM augmentation described in the algorithm above operates in each DH ratchet step. + ## Implementation considerations for SimpleX Messaging Protocol As SimpleX Messaging Protocol pads messages to a fixed size, using 16kb transport blocks, the size increase introduced by this scheme can be compensated for by using ZSTD encryption of JSON bodies and image previews encoded as base64. While there may be some rare cases of random texts that would fail to compress, in all real scenarios it would not cause the message size reduction. diff --git a/protocol/push-notifications.md b/protocol/push-notifications.md index 6d5e1dea0..88645c4c2 100644 --- a/protocol/push-notifications.md +++ b/protocol/push-notifications.md @@ -1,14 +1,19 @@ -Version 2, 2024-06-22 +Version 3, 2025-01-24 -# Overview of push notifications for SimpleX Messaging Servers +# Overview of push notifications for SimpleX Messaging Routers + +This document describes Notification Router protocol version 3. Version history: +- v1: initial version +- v2: authenticated commands, command batching +- v3: detailed invalid token reason ## Table of contents - [Introduction](#introduction) -- [Participating servers](#participating-servers) +- [Participating routers](#participating-routers) - [Register device token to receive push notifications](#register-device-token-to-receive-push-notifications) - [Subscribe to connection notifications](#subscribe-to-connection-notifications) -- [SimpleX Notification Server protocol](#simplex-notification-server-protocol) +- [SimpleX Notification Router protocol](#simplex-notification-router-protocol) - [Register new notification token](#register-new-notification-token) - [Verify notification token](#verify-notification-token) - [Check notification token status](#check-notification-token-status) @@ -23,35 +28,35 @@ Version 2, 2024-06-22 ## Introduction -SimpleX Messaging servers already operate as push servers and deliver the messages to subscribed clients as soon as they are sent to the servers. +SimpleX Messaging routers already operate as push routers and deliver the messages to subscribed clients as soon as they are sent to the routers. The reason for push notifications is to support instant message notifications on iOS that does not allow background services. -## Participating servers +## Participating routers -The diagram below shows which servers participate in message notification delivery. +The diagram below shows which routers participate in message notification delivery. -While push provider (e.g., APN) can learn how many notifications are delivered to the user, it cannot access message content, even encrypted, or any message metadata - the notifications are e2e encrypted between SimpleX Notification Server and the user's device. +While push provider (e.g., APN) can learn how many notifications are delivered to the user, it cannot access message content, even encrypted, or any message metadata - the notifications are e2e encrypted between SimpleX Notification Router and the user's device. ``` - User's iOS device Internet Servers + User's iOS device Internet Routers --------------------- . ------------------------ . ----------------------------- . . . . can be self-hosted now +--------------+ . . +----------------+ | SimpleX Chat | -------------- TLS --------------- | SimpleX | | client |------> SimpleX Messaging Protocol (SMP) ------> | Messaging | -+--------------+ ---------------------------------- | Server | ++--------------+ ---------------------------------- | Router | ^ | . . +----------------+ | | . . . . . | . . . | | . . | V | | | . . |SMP| TLS | | . . | | | SimpleX - | | . . . . . V . . . NTF Server + | | . . . . . V . . . NTF Router | | . . +----------------------------------+ | | . . | +---------------+ | | | -------------- TLS --------------- | | SimpleX | can be | - | |-----------> Notification Server Protocol -----> | | Notifications | self-hosted | + | |-----------> Notification Router Protocol -----> | | Notifications | self-hosted | | ---------------------------------- | | Subscriber | in the future | | . . | +---------------+ | | . . | | | @@ -59,7 +64,7 @@ While push provider (e.g., APN) can learn how many notifications are delivered t | . . | +---------------+ | | . . | | SimpleX | | | . . | | Push | | - | . . | | Server | | + | . . | | Router | | | . . | +---------------+ | | . . +----------------------------------+ | . . . . . | . . . @@ -85,25 +90,28 @@ This diagram shows the process of subscription to notifications, notification de ![Subscribe to notifications](./diagrams/notifications/subscription.svg) -## SimpleX Notification Server protocol +## SimpleX Notification Router protocol + +To manage notification subscriptions to SMP routers, SimpleX Notification Router provides an RPC protocol with a similar design to SimpleX Messaging Protocol router. -To manage notification subscriptions to SMP servers, SimpleX Notification Server provides an RPC protocol with a similar design to SimpleX Messaging Protocol server. +This protocol sends requests and responses in a fixed size blocks of 512 bytes over TLS, uses the same [syntax of protocol transmissions](./simplex-messaging.md#smp-transmission-and-transport-block-structure) as SMP protocol, and has the same transport [handshake syntax](./simplex-messaging.md#transport-handshake) (except the router certificate is not included in the handshake). -This protocol sends requests and responses in a fixed size blocks of 512 bytes over TLS, uses the same [syntax of protocol transmissions](./simplex-messaging.md#smp-transmission-and-transport-block-structure) as SMP protocol, and has the same transport [handshake syntax](./simplex-messaging.md#transport-handshake) (except the server certificate is not included in the handshake). +The client and router use ALPN extension with `ntf/1` protocol name to agree handshake version. Protocol commands have this syntax: -``` -ntfServerTransmission = -ntfServerCmd = newTokenCmd / verifyTokenCmd / checkTokenCmd / +```abnf +ntfRouterTransmission = authorization corrId entityId ntfRouterCmd + ; same transmission structure as SMP, see simplex-messaging.md +ntfRouterCmd = newTokenCmd / verifyTokenCmd / checkTokenCmd / replaceTokenCmd / deleteTokenCmd / cronCmd / - newSubCmd / checkSubCmd / deleteSubCmd + newSubCmd / checkSubCmd / deleteSubCmd / pingCmd ``` ### Register new notification token -This command should be used after the client app obtains a token from push notifications provider to register the token with the server. +This command should be used after the client app obtains a token from push notifications provider to register the token with the router. -Having received this command the server will deliver a test notification via the push provider to validate that the client has this token. +Having received this command the router will deliver a test notification via the push provider to validate that the client has this token. The command syntax: @@ -111,23 +119,24 @@ The command syntax: newTokenCmd = %s"TNEW" SP newToken newToken = %s"T" deviceToken authPubKey clientDhPubKey deviceToken = pushProvider tokenString -pushProvider = apnsDev / apnsProd / apnsNull +pushProvider = apnsDev / apnsProd / apnsTest / apnsNull apnsDev = "AD" ; APNS token for development environment apnsProd = "AP" ; APNS token for production environment -apnsNull = "AN" ; token that does not trigger any notification delivery - used for server testing +apnsTest = "AT" ; APNS token for test environment (mock server) +apnsNull = "AN" ; token that does not trigger any notification delivery - used for router testing tokenString = shortString authPubKey = length x509encoded ; Ed25519 key used to verify clients commands -clientDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the server and client +clientDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the router and client shortString = length *OCTET length = 1*1 OCTET ``` -The server response syntax: +The router response syntax: ```abnf -tokenIdResp = %s"IDTKN" SP entityId serverDhPubKey +tokenIdResp = %s"IDTKN" SP entityId routerDhPubKey entityId = shortString -serverDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the server and client +routerDhPubKey = length x509encoded ; X25519 key to agree e2e encryption between the router and client ``` ### Verify notification token @@ -159,7 +168,9 @@ The response to this command: ```abnf tokenStatusResp = %s"TKN" SP tokenStatus -tokenStatus = %s"NEW" / %s"REGISTERED" / %s"INVALID" / %s"CONFIRMED" / %s"ACTIVE" / %s"EXPIRED" +tokenStatus = %s"NEW" / %s"REGISTERED" / tokenInvalid / %s"CONFIRMED" / %s"ACTIVE" / %s"EXPIRED" +tokenInvalid = %s"INVALID" ["," invalidReason] ; optional reason added in v3 +invalidReason = %s"BAD" / %s"TOPIC" / %s"EXPIRED" / %s"UNREGISTERED" ``` ### Replace notification token @@ -200,8 +211,8 @@ After this command all message notification subscriptions will be removed and no This command enables or disables periodic notifications sent to the client device irrespective of message notifications. This is useful for two reasons: -- it provides better privacy from notification server, as while the server learns the device token, it doesn't learn anything else about user communications. -- it allows to receive messages when notifications were dropped by push provider, e.g. while the device was offline, or lost by notification server, e.g. while it was restarting. +- it provides better privacy from notification router, as while the router learns the device token, it doesn't learn anything else about user communications. +- it allows to receive messages when notifications were dropped by push provider, e.g. while the device was offline, or lost by notification router, e.g. while it was restarting. The command syntax: @@ -214,18 +225,18 @@ The interval for periodic notifications is set in minutes, with the minimum of 2 ### Create SMP message notification subscription -This command makes notification server subscribe to message notifications from SMP server and to deliver them to push provider: +This command makes notification router subscribe to message notifications from SMP router and to deliver them to push provider: ```abnf -newSubCmd = %s"SNEW" newSub -newSub = %s "S" tokenId smpServer notifierId notifierKey +newSubCmd = %s"SNEW" SP newSub +newSub = %s"S" tokenId smpRouter notifierId notifierKey tokenId = shortString ; returned in response to `TNEW` command -smpServer = smpServer = hosts port fingerprint +smpRouter = hosts port fingerprint hosts = length 1*host host = shortString port = shortString fingerprint = shortString -notifierId = shortString ; returned by SMP server in response to `NKEY` SMP command +notifierId = shortString ; returned by SMP router in response to `NKEY` SMP command notifierKey = length x509encoded ; private key used to authorize requests to subscribe to message notifications ``` @@ -247,10 +258,10 @@ The response: ```abnf subStatusResp = %s"SUB" SP subStatus -subStatus = %s"NEW" / %s"PENDING" / ; e.g., after SMP server disconnect/timeout while ntf server is retrying to connect - %s"ACTIVE" / %s"INACTIVE" / %s"END" / ; if another server subscribed to notifications - %s"AUTH" / subErrStatus -subErrStatus = %s"ERR" SP shortString +subStatus = %s"NEW" / %s"PENDING" / ; e.g., after SMP router disconnect/timeout while ntf router is retrying to connect + %s"ACTIVE" / %s"INACTIVE" / %s"END" / ; if another router subscribed to notifications + %s"AUTH" / %s"DELETED" / %s"SERVICE" / subErrStatus +subErrStatus = %s"ERR" SP *OCTET ``` ### Delete notification subscription @@ -265,6 +276,17 @@ The response to this command is `okResp` or `errorResp`. After this command no more message notifications will be sent from this queue. +### Keep-alive command + +To keep the transport connection alive the clients should use `PING` command: + +```abnf +pingCmd = %s"PING" +pongResp = %s"PONG" +``` + +This command is sent unsigned and without entity ID. + ### Error responses All commands can return error response: @@ -277,7 +299,7 @@ Where `errorType` has the same syntax as in [SimpleX Messaging Protocol](./simpl ## Threat Model -This threat model compliments SimpleX Messaging Protocol [threat model](./overview-tjr.md#threat-model) +This threat model compliments SimpleX Messaging Protocol [threat model](./security.md#threat-model) #### A passive adversary able to monitor the traffic of one user @@ -287,21 +309,21 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi *cannot:* - - determine which servers a user subscribed to the notifications from. + - determine which routers a user subscribed to the notifications from. #### A passive adversary able to monitor a set of senders and recipients *can:* - - perform more efficient traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the servers. + - perform more efficient traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the routers. -#### SimpleX Messaging Protocol server +#### SimpleX Messaging Protocol router *can:* - learn which messages trigger push notifications. -- learn IP address of SimpleX notification servers used by the user. +- learn IP address of SimpleX notification routers used by the user. - drop message notifications. @@ -313,13 +335,13 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi - learn which queues belong to the same users with any additional efficiency compared with not using push notifications. -#### SimpleX Notification Server subscribed to message notifications +#### SimpleX Notification Router subscribed to message notifications *can:* - learn a user device token. -- learn how many messaging queues and servers a user receives messages from. +- learn how many messaging queues and routers a user receives messages from. - learn how many message notifications are delivered to the user from each queue. @@ -339,7 +361,7 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi - add, duplicate, or corrupt individual messages that will be shown to the user. -#### SimpleX Notification Server subscribed ONLY to periodic notifications +#### SimpleX Notification Router subscribed ONLY to periodic notifications *can:* @@ -351,7 +373,7 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi *cannot:* -- learn how many messaging queues and servers a user receives messages from. +- learn how many messaging queues and routers a user receives messages from. - learn how many message notifications are delivered to the user from each queue. @@ -383,7 +405,7 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi *cannot:* -- learn which SimpleX Messaging Protocol servers are used by a user (notifications are e2e encrypted). +- learn which SimpleX Messaging Protocol routers are used by a user (notifications are e2e encrypted). - learn which or how many messaging queues a user receives notifications from. @@ -395,4 +417,4 @@ This threat model compliments SimpleX Messaging Protocol [threat model](./overvi - register notification token not present on attacker's device. -- enumerate tokens or subscriptions on a SimpleX Notification Server. +- enumerate tokens or subscriptions on a SimpleX Notification Router. diff --git a/protocol/security.md b/protocol/security.md new file mode 100644 index 000000000..ea236daf4 --- /dev/null +++ b/protocol/security.md @@ -0,0 +1,215 @@ +Revision 1, 2026-03-09 + +# SimpleX Network: Security + +This document describes the cryptographic primitives and threat model for the SimpleX network. For a general introduction, see [SimpleX: messaging and application platform](./overview-tjr.md). + +## Table of contents + +- [Encryption primitives](#encryption-primitives) +- [Threat model](#threat-model) + - [Global Assumptions](#global-assumptions) + - [A passive adversary able to monitor the traffic of one user](#a-passive-adversary-able-to-monitor-the-traffic-of-one-user) + - [A passive adversary able to monitor a set of senders and recipients](#a-passive-adversary-able-to-monitor-a-set-of-senders-and-recipients) + - [SimpleX Messaging Protocol router](#simplex-messaging-protocol-router) + - [SimpleX Messaging Protocol router that proxies the messages to another SMP router](#simplex-messaging-protocol-router-that-proxies-the-messages-to-another-smp-router) + - [An attacker who obtained Alice's (decrypted) chat database](#an-attacker-who-obtained-alices-decrypted-chat-database) + - [A user's contact](#a-users-contact) + - [An attacker who observes Alice showing an introduction message to Bob](#an-attacker-who-observes-alice-showing-an-introduction-message-to-bob) + - [An attacker with Internet access](#an-attacker-with-internet-access) + + +## Encryption primitives + +- **Router command authorization**: X25519 DH-based authenticated encryption (SMP v7+), providing sender deniability. Ed25519 signatures used for recipient commands and notifier commands. + +- **Per-queue key agreement**: Curve25519 DH exchange to agree: + - the shared secret between router and recipient (to encrypt message bodies — avoids shared ciphertext in sender and recipient traffic), + - the shared secret between sender and recipient (to encrypt messages end-to-end in each queue — avoids shared ciphertext in redundant queues). + +- **SMP-layer encryption**: [NaCl crypto_box](https://nacl.cr.yp.to/box.html) (curve25519xsalsa20poly1305) for message body encryption between router and recipient, and for e2e per-queue encryption. + +- **Certificate validation**: SHA256 to validate router offline certificates. + +- **End-to-end encryption**: [Double ratchet](https://signal.org/docs/specifications/doubleratchet/) protocol: + - Curve448 keys for shared secret agreement via [X3DH](https://signal.org/docs/specifications/x3dh/) with 2 ephemeral keys per side, + - optional [SNTRUP761](https://ntruprime.cr.yp.to/) post-quantum KEM running in parallel with the DH ratchet (see [PQDR](./pqdr.md)), providing post-quantum forward secrecy, + - AES-GCM AEAD cipher, + - SHA512-based HKDF for key derivation. + + +## Threat Model + +### Global Assumptions + +- A user protects their local database and key material. +- The user's application is authentic, and no local malware is running. +- The cryptographic primitives in use are not broken. +- A user's choice of routers is not directly tied to their identity or otherwise represents distinguishing information about the user. +- The user's client uses 2-hop onion message routing. + +### A passive adversary able to monitor the traffic of one user + +*can:* + +- identify that and when a user is using SimpleX. + +- determine which routers the user receives messages from. + +- observe how much traffic is being sent, and make guesses as to its purpose. + +*cannot:* + +- see who sends messages to the user and who the user sends messages to. + +- determine the routers used by users' contacts. + +### A passive adversary able to monitor a set of senders and recipients + +*can:* + +- identify who and when is using SimpleX. + +- learn which SimpleX Messaging Protocol routers are used as receive queues for which users. + +- learn when messages are sent and received. + +- perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the routers. + +- observe how much traffic is being sent, and make guesses as to its purpose. + +*cannot, even in case of a compromised transport protocol:* + +- perform traffic correlation attacks with any increase in efficiency over a non-compromised transport protocol. + +### SimpleX Messaging Protocol router + +*can:* + +- learn when a queue recipient is online. + +- know how many messages are sent via the queue (although some may be noise or not content messages). + +- learn which messages would trigger notifications even if a user does not use [push notifications](./push-notifications.md). + +- perform the correlation of the queue used to receive messages (matching multiple queues to a single user) via either a re-used transport connection, user's IP Address, or connection timing regularities. + +- learn a recipient's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, as long as Tor is not used. + +- drop all future messages inserted into a queue, detectable only over other, redundant queues. + +- lie about the state of a queue to the recipient and/or to the sender (e.g. suspended or deleted when it is not). + +- spam a user with invalid messages. + +*cannot:* + +- undetectably add, duplicate, or corrupt individual messages. + +- undetectably drop individual messages, so long as a subsequent message is delivered. + +- learn the contents or type of messages. + +- distinguish noise messages from regular messages except via timing regularities. + +- compromise the users' end-to-end encryption with an active attack. + +- learn a sender's IP address, track them through other IP addresses they use to access the same queue, and infer information (e.g. employer) based on the IP addresses, even if Tor is not used (provided messages are sent via proxy SMP router). + +- perform senders' queue correlation (matching multiple queues to a single sender) via either a re-used transport connection, user's IP Address, or connection timing regularities, unless it has additional information from the proxy SMP router (provided messages are sent via proxy SMP router). + +### SimpleX Messaging Protocol router that proxies the messages to another SMP router + +*can:* + +- learn a sender's IP address, as long as Tor is not used. + +- learn when a sender with a given IP address is online. + +- know how many messages are sent from a given IP address and to a given destination SMP router. + +- drop all messages from a given IP address or to a given destination router. + +- unless destination SMP router detects repeated public DH keys of senders, replay messages to a destination router within a single session, causing either duplicate message delivery (which will be detected and ignored by the receiving clients), or, when receiving client is not connected to SMP router, exhausting capacity of destination queues used within the session. + +*cannot:* + +- perform queue correlation (matching multiple queues to a single user), unless it has additional information from the destination SMP router. + +- undetectably add, duplicate, or corrupt individual messages. + +- undetectably drop individual messages, so long as a subsequent message is delivered. + +- learn the contents or type of messages. + +- learn which messages would trigger notifications. + +- learn the destination queues of messages. + +- distinguish noise messages from regular messages except via timing regularities. + +- compromise the user's end-to-end encryption with another user via an active attack. + +- compromise the user's end-to-end encryption with the destination SMP routers via an active attack. + +### An attacker who obtained Alice's (decrypted) chat database + +*can:* + +- see the history of all messages exchanged by Alice with her communication partners. + +- see shared profiles of contacts and groups. + +- surreptitiously receive new messages sent to Alice via existing queues; until communication queues are rotated or the Double-Ratchet advances forward. + +- prevent Alice from receiving all new messages sent to her - either surreptitiously by emptying the queues regularly or overtly by deleting them. + +- send messages from the user to their contacts; recipients will detect it as soon as the user sends the next message, because the previous message hash won't match (and potentially won't be able to decrypt them in case they don't keep the previous ratchet keys). + +*cannot:* + +- impersonate a sender and send messages to the user whose database was stolen. Doing so requires also compromising the router (to place the message in the queue, that is possible until the Double-Ratchet advances forward) or the user's device at a subsequent time (to place the message in the database). + +- undetectably communicate at the same time as Alice with her contacts. Doing so would result in the contact getting different messages with repeated IDs. + +- undetectably monitor message queues in realtime without alerting the user they are doing so, as a second subscription request unsubscribes the first and notifies the first. + +### A user's contact + +*can:* + +- spam the user with messages. + +- forever retain messages from the user. + +*cannot:* + +- cryptographically prove to a third-party that a message came from a user (assuming the user's device is not seized). + +- prove that two contacts they have is the same user. + +- cannot collaborate with another of the user's contacts to confirm they are communicating with the same user. + +### An attacker who observes Alice showing an introduction message to Bob + +*can:* + +- Impersonate Bob to Alice. + +*cannot:* + +- Impersonate Alice to Bob. + +### An attacker with Internet access + +*can:* + +- Denial of Service SimpleX messaging routers. + +- spam a user's public "contact queue" with connection requests. + +*cannot:* + +- send messages to a user who they are not connected with. + +- enumerate queues on a SimpleX router. diff --git a/protocol/simplex-messaging.md b/protocol/simplex-messaging.md index 16e4e6606..f1d1f77ce 100644 --- a/protocol/simplex-messaging.md +++ b/protocol/simplex-messaging.md @@ -1,4 +1,4 @@ -Version 9, 2024-06-22 +Version 19, 2025-01-24 # Simplex Messaging Protocol (SMP) @@ -16,8 +16,12 @@ Version 9, 2024-06-22 - [Cryptographic algorithms](#cryptographic-algorithms) - [Deniable client authentication scheme](#deniable-client-authentication-scheme) - [Simplex queue IDs](#simplex-queue-ids) -- [Server security requirements](#server-security-requirements) +- [Router security requirements](#router-security-requirements) - [Message delivery notifications](#message-delivery-notifications) +- [Client services](#client-services) + - [Service roles](#service-roles) + - [Service certificates](#service-certificates) + - [Service subscriptions](#service-subscriptions) - [SMP Transmission and transport block structure](#smp-transmission-and-transport-block-structure) - [SMP commands](#smp-commands) - [Correlating responses with commands](#correlating-responses-with-commands) @@ -26,7 +30,11 @@ Version 9, 2024-06-22 - [Recipient commands](#recipient-commands) - [Create queue command](#create-queue-command) - [Subscribe to queue](#subscribe-to-queue) + - [Subscribe to multiple queues](#subscribe-to-multiple-queues) - [Secure queue by recipient](#secure-queue-by-recipient) + - [Set queue recipient keys](#set-queue-recipient-keys) + - [Set short link](#set-short-link) + - [Delete short link](#delete-short-link) - [Enable notifications command](#enable-notifications-command) - [Disable notifications command](#disable-notifications-command) - [Get message command](#get-message-command) @@ -40,60 +48,89 @@ Version 9, 2024-06-22 - [Proxying sender commands](#proxying-sender-commands) - [Request proxied session](#request-proxied-session) - [Send command via proxy](#send-command-via-proxy) - - [Forward command to destination server](#forward-command-to-destination-server) + - [Forward command to destination router](#forward-command-to-destination-router) + - [Short link commands](#short-link-commands) + - [Set link key](#set-link-key) + - [Get link data](#get-link-data) - [Notifier commands](#notifier-commands) - [Subscribe to queue notifications](#subscribe-to-queue-notifications) - - [Server messages](#server-messages) + - [Subscribe to multiple queue notifications](#subscribe-to-multiple-queue-notifications) + - [Router messages](#router-messages) + - [Link response](#link-response) + - [Queue subscription response](#queue-subscription-response) + - [Service subscription response](#service-subscription-response) + - [All service messages received](#all-service-messages-received) - [Deliver queue message](#deliver-queue-message) - [Deliver message notification](#deliver-message-notification) - [Subscription END notification](#subscription-end-notification) + - [Service subscription END notification](#service-subscription-end-notification) + - [Queue deleted notification](#queue-deleted-notification) - [Error responses](#error-responses) - [OK response](#ok-response) -- [Transport connection with the SMP server](#transport-connection-with-the-SMP-server) +- [Transport connection with the SMP router](#transport-connection-with-the-SMP-router) - [General transport protocol considerations](#general-transport-protocol-considerations) - [TLS transport encryption](#tls-transport-encryption) - - [Server certificate](#server-certificate) + - [Router certificate](#router-certificate) - [ALPN to agree handshake version](#alpn-to-agree-handshake-version) - [Transport handshake](#transport-handshake) - [Additional transport privacy](#additional-transport-privacy) ## Abstract -Simplex Messaging Protocol is a transport agnostic client-server protocol for asynchronous distributed secure unidirectional message transmission via persistent simplex message queues. +Simplex Messaging Protocol is a transport agnostic client-router protocol for asynchronous distributed secure unidirectional message transmission via persistent simplex message queues. It's designed with the focus on communication security and integrity, under the assumption that any part of the message transmission network can be compromised. It is designed as a low level protocol for other application protocols to solve the problem of secure and private message transmission, making [MITM attack][1] very difficult at any part of the message transmission system. -This document describes SMP protocol versions 6 and 7, the previous versions are discontinued. +This document describes SMP protocol version 19. Versions 1-5 are discontinued. The version history: + +- v1: binary protocol encoding +- v2: message flags (used to control notifications) +- v3: encrypt message timestamp and flags together with the body when delivered to recipient +- v4: support command batching +- v5: basic auth for SMP routers +- v6: allow creating queues without subscribing (current minimum version) +- v7: support authenticated encryption to verify senders' commands +- v8: SMP proxy for sender commands (PRXY, PFWD, RFWD, PKEY, PRES, RRES) +- v9: faster handshake with SKEY command for sender to secure queue +- v10: DELD event to subscriber when queue is deleted via another connection +- v11: additional encryption of transport blocks with forward secrecy +- v12: BLOCKED error for blocked queues +- v14: proxyRouter handshake property to disable transport encryption between router and proxy +- v15: short links with associated data passed in NEW or LSET command +- v16: service certificates +- v17: create notification credentials with NEW command +- v18: support client notices in BLOCKED error +- v19: service subscriptions to messages (SUBS, NSUBS, SOKS, ENDS, ALLS commands) ## Introduction -The objective of Simplex Messaging Protocol (SMP) is to facilitate the secure and private unidirectional transfer of messages from senders to recipients via persistent simplex queues managed by the message brokers (servers). +The objective of Simplex Messaging Protocol (SMP) is to facilitate the secure and private unidirectional transfer of messages from senders to recipients via persistent simplex queues managed by the message routers. SMP is independent of any particular transmission system and requires only a reliable ordered data stream channel. While this document describes transport over TCP, other transports are also possible. -The protocol describes the set of commands that recipients and senders can exchange with SMP servers to create and to operate unidirectional "queues" (a data abstraction identifying one of many communication channels managed by the server) and to send messages from the sender to the recipient via the SMP server. +The protocol describes the set of commands that recipients and senders can exchange with SMP routers to create and to operate unidirectional "queues" (a data abstraction identifying one of many communication channels managed by the router) and to send messages from the sender to the recipient via the SMP router. More complex communication scenarios can be designed using multiple queues - for example, a duplex communication channel can be made of 2 simplex queues. -The protocol is designed with the focus on privacy and security, to some extent deprioritizing reliability by requiring that SMP servers only store messages until they are acknowledged by the recipients and, in any case, for a limited period of time. For communication scenarios requiring more reliable transmission the users should use several SMP servers to pass each message and implement some additional protocol to ensure that messages are not removed, inserted or changed - this is out of scope of this document. +The protocol is designed with the focus on privacy and security, to some extent deprioritizing reliability by requiring that SMP routers only store messages until they are acknowledged by the recipients and, in any case, for a limited period of time. For communication scenarios requiring more reliable transmission the users should use several SMP routers to pass each message and implement some additional protocol to ensure that messages are not removed, inserted or changed - this is out of scope of this document. SMP does not use any form of participants' identities and provides [E2EE][2] without the possibility of [MITM attack][1] relying on two pre-requisites: -- the users can establish a secure encrypted transport connection with the SMP server. [Transport connection](#transport-connection-with-the-smp-server) section describes SMP transport protocol of such connection over TCP, but any other transport connection protocol can be used. +- the users can establish a secure encrypted transport connection with the SMP router. [Transport connection](#transport-connection-with-the-smp-router) section describes SMP transport protocol of such connection over TCP, but any other transport connection protocol can be used. -- the recipient can pass a single message to the sender via a pre-existing secure and private communication channel (out-of-band message) - the information in this message is used to encrypt messages and to establish connection with SMP server. +- the recipient can pass a single message to the sender via a pre-existing secure and private communication channel (out-of-band message) - the information in this message is used to encrypt messages and to establish connection with SMP router. ## SMP Model -The SMP model has three communication participants: the recipient, the message broker (SMP server) that is chosen and, possibly, controlled by the recipient, and the sender. +The SMP model has three communication participants: the recipient, the message router (SMP router) that is chosen and, possibly, controlled by the recipient, and the sender. -SMP server manages multiple "simplex queues" - data records on the server that identify communication channels from the senders to the recipients. The same communicating party that is the sender in one queue, can be the recipient in another - without exposing this fact to the server. +SMP router manages multiple "simplex queues" - data records on the router that identify communication channels from the senders to the recipients. The same communicating party that is the sender in one queue, can be the recipient in another - without exposing this fact to the router. -The queue record consists of 2 unique random IDs generated by the server, one for the recipient and another for the sender, and 2 keys to verify the recipient's and the sender's commands, provided by the clients. The users of SMP protocol must use a unique ephemeral keys for each queue, to prevent aggregating their queues by keys in case SMP server is compromised. +The queue record consists of 2 unique random IDs generated by the router, one for the recipient and another for the sender, and 2 keys to verify the recipient's and the sender's commands, provided by the clients. The users of SMP protocol must use a unique ephemeral keys for each queue, to prevent aggregating their queues by keys in case SMP router is compromised. -Creating and using the queue requires sending commands to the SMP server from the recipient and the sender - they are described in detail in [SMP commands](#smp-commands) section. +Creating and using the queue requires sending commands to the SMP router from the recipient and the sender - they are described in detail in [SMP commands](#smp-commands) section. ## Out-of-band messages @@ -105,17 +142,17 @@ The approach to out-of-band message passing and their syntax should be defined i The simplex queue is the main unit of SMP protocol. It is used by: -- Sender of the queue (who received out-of-band message) to send messages to the server using sender's queue ID, authorized by sender's key. +- Sender of the queue (who received out-of-band message) to send messages to the router using sender's queue ID, authorized by sender's key. -- Recipient of the queue (who created the queue and sent out-of-band message) will use it to retrieve messages from the server, authorizing the commands by the recipient key. Recipient decrypts the messages with the key negotiated during the creation of the queue. +- Recipient of the queue (who created the queue and sent out-of-band message) will use it to retrieve messages from the router, authorizing the commands by the recipient key. Recipient decrypts the messages with the key negotiated during the creation of the queue. -- Participant identities are not shared with the server - new unique keys and queue IDs are used for each queue. +- Participant identities are not shared with the router - new unique keys and queue IDs are used for each queue. -This simplex queue can serve as a building block for more complex communication network. For example, two (or more, for redundancy) simplex queues can be used to create a duplex communication channel. Higher level primitives that are only known to system participants in their client applications can be created as well - e.g., contacts, conversations, groups and broadcasts. Simplex messaging servers only have the information about the low-level simplex queues. In this way a high level of privacy and security of the communication is provided. Application level primitives are not in scope of this protocol. +This simplex queue can serve as a building block for more complex communication network. For example, two (or more, for redundancy) simplex queues can be used to create a duplex communication channel. Higher level primitives that are only known to system participants in their client applications can be created as well - e.g., contacts, conversations, groups and broadcasts. Simplex messaging routers only have the information about the low-level simplex queues. In this way a high level of privacy and security of the communication is provided. Application level primitives are not in scope of this protocol. This approach is based on the concept of [unidirectional networks][4] that are used for applications with high level of information security. -Access to each queue is controlled with unique (not shared with other queues) asymmetric key pairs, separate for the sender and the recipient. The sender and the receiver have private keys, and the server has associated public keys to authenticate participants' commands by verifying cryptographic authorizations. +Access to each queue is controlled with unique (not shared with other queues) asymmetric key pairs, separate for the sender and the recipient. The sender and the receiver have private keys, and the router has associated public keys to authenticate participants' commands by verifying cryptographic authorizations. The messages sent over the queue are end-to-end encrypted using the DH secret agreed via out-of-band message and SMP confirmation. @@ -123,22 +160,22 @@ The messages sent over the queue are end-to-end encrypted using the DH secret ag ![Simplex queue](./diagrams/simplex-messaging/simplex.svg) -Queue is defined by recipient ID `RID` and sender ID `SID`, unique for the server. Sender key (`SK`) is used by the server to verify sender's commands (identified by `SID`) to send messages. Recipient key (`RK`) is used by the server to verify recipient's commands (identified by `RID`) to retrieve messages. +Queue is defined by recipient ID `RID` and sender ID `SID`, unique for the router. Sender key (`SK`) is used by the router to verify sender's commands (identified by `SID`) to send messages. Recipient key (`RK`) is used by the router to verify recipient's commands (identified by `RID`) to retrieve messages. -The protocol uses different IDs for sender and recipient in order to provide an additional privacy by preventing the correlation of senders and recipients commands sent over the network - in case the encrypted transport is compromised, it would still be difficult to correlate senders and recipients without access to the queue records on the server. +The protocol uses different IDs for sender and recipient in order to provide an additional privacy by preventing the correlation of senders and recipients commands sent over the network - in case the encrypted transport is compromised, it would still be difficult to correlate senders and recipients without access to the queue records on the router. ## SMP queue URI -The SMP queue URIs MUST include server identity, queue hostname, an optional port, sender queue ID, and the recipient's public key to agree shared secret for e2e encryption, and an optional query string parameter `k=s` to indicate that the queue can be secured by the sender using `SKEY` command (see [Fast SMP procedure](#fast-smp-procedure) and [Secure queue by sender](#secure-queue-by-sender)). Server identity is used to establish secure connection protected from MITM attack with SMP server (see [Transport connection](#transport-connection-with-the-smp-server) for SMP transport protocol). +The SMP queue URIs MUST include router identity, queue hostname, an optional port, sender queue ID, and the recipient's public key to agree shared secret for e2e encryption, and an optional query string parameter `k=s` to indicate that the queue can be secured by the sender using `SKEY` command (see [Fast SMP procedure](#fast-smp-procedure) and [Secure queue by sender](#secure-queue-by-sender)). Router identity is used to establish secure connection protected from MITM attack with SMP router (see [Transport connection](#transport-connection-with-the-smp-router) for SMP transport protocol). The [ABNF][8] syntax of the queue URI is: ```abnf -queueURI = %s"smp://" smpServer "/" queueId "#/?" versionParam keyParam [sndSecureParam] -smpServer = serverIdentity "@" srvHosts [":" port] +queueURI = %s"smp://" smpRouter "/" queueId "#/?" versionParam keyParam [sndSecureParam] +smpRouter = routerIdentity "@" srvHosts [":" port] srvHosts = ["," srvHosts] ; RFC1123, RFC5891 port = 1*DIGIT -serverIdentity = base64url +routerIdentity = base64url queueId = base64url versionParam = %s"v=" versionRange versionRange = 1*DIGIT / 1*DIGIT "-" 1*DIGIT @@ -157,49 +194,49 @@ x509UrlEncoded = `port` is optional, the default TCP port for SMP protocol is 5223. -`serverIdentity` is a required hash of the server certificate SPKI block (without line breaks, header and footer) used by the client to validate server certificate during transport handshake (see [Transport connection](#transport-connection-with-the-smp-server)) +`routerIdentity` is a required hash of the router certificate SPKI block (without line breaks, header and footer) used by the client to validate router certificate during transport handshake (see [Transport connection](#transport-connection-with-the-smp-router)) ## SMP procedure -The SMP procedure of creating a simplex queue on SMP server is explained using participants Alice (the recipient) who wants to receive messages from Bob (the sender). +The SMP procedure of creating a simplex queue on SMP router is explained using participants Alice (the recipient) who wants to receive messages from Bob (the sender). To create and start using a simplex queue Alice and Bob follow these steps: -1. Alice creates a simplex queue on the server: +1. Alice creates a simplex queue on the router: - 1. Decides which SMP server to use (can be the same or different server that Alice uses for other queues) and opens secure encrypted transport connection to the chosen SMP server (see [Transport connection](#transport-connection-with-the-smp-server)). + 1. Decides which SMP router to use (can be the same or different router that Alice uses for other queues) and opens secure encrypted transport connection to the chosen SMP router (see [Transport connection](#transport-connection-with-the-smp-router)). 2. Generates a new random public/private key pair (encryption key - `EK`) that she did not use before to agree a shared secret with Bob to encrypt the messages. - 3. Generates another new random public/private key pair (recipient key - `RK`) that she did not use before for her to authorize commands to the server. + 3. Generates another new random public/private key pair (recipient key - `RK`) that she did not use before for her to authorize commands to the router. - 4. Generates one more random key pair (recipient DH key - `RDHK`) to negotiate symmetric key that will be used by the server to encrypt message bodies delivered to Alice (to avoid shared cipher-text inside transport connection). + 4. Generates one more random key pair (recipient DH key - `RDHK`) to negotiate symmetric key that will be used by the router to encrypt message bodies delivered to Alice (to avoid shared cipher-text inside transport connection). - 5. Sends `"NEW"` command to the server to create a simplex queue (see `create` in [Create queue command](#create-queue-command)). This command contains previously generated unique "public" keys `RK` and `RDHK`. `RK` will be used by the server to verify the subsequent commands related to the same queue authorized by its private counterpart, for example to subscribe to the messages received to this queue or to update the queue, e.g. by setting the key required to send the messages (initially Alice creates the queue that accepts unauthorized messages, so anybody could send the message via this queue if they knew the queue sender's ID and server address). + 5. Sends `"NEW"` command to the router to create a simplex queue (see `create` in [Create queue command](#create-queue-command)). This command contains previously generated unique "public" keys `RK` and `RDHK`. `RK` will be used by the router to verify the subsequent commands related to the same queue authorized by its private counterpart, for example to subscribe to the messages received to this queue or to update the queue, e.g. by setting the key required to send the messages (initially Alice creates the queue that accepts unauthorized messages, so anybody could send the message via this queue if they knew the queue sender's ID and router address). - 6. The server sends `IDS` response with queue IDs (`queueIds`): + 6. The router sends `IDS` response with queue IDs (`queueIds`): - Recipient ID `RID` for Alice to manage the queue and to receive the messages. - Sender ID `SID` for Bob to send messages to the queue. - - Server public DH key (`SDHK`) to negotiate a shared secret for message body encryption, that Alice uses to derive a shared secret with the server `SS`. + - Router public DH key (`SDHK`) to negotiate a shared secret for message body encryption, that Alice uses to derive a shared secret with the router `SS`. 2. Alice sends an out-of-band message to Bob via the alternative channel that both Alice and Bob trust (see [protocol abstract](#simplex-messaging-protocol-abstract)). The message must include [SMP queue URI](#smp-queue-uri) with: - Unique "public" key (`EK`) that Bob must use to agree a shared secret for E2E encryption. - - SMP server hostname and information to open secure encrypted transport connection (see [Transport connection](#transport-connection-with-the-smp-server)). + - SMP router hostname and information to open secure encrypted transport connection (see [Transport connection](#transport-connection-with-the-smp-router)). - Sender queue ID `SID` for Bob to use. 3. Bob, having received the out-of-band message from Alice, connects to the queue: - 1. Generates a new random public/private key pair (sender key - `SK`) that he did not use before for him to authorize messages sent to Alice's server and another key pair for e2e encryption agreement. + 1. Generates a new random public/private key pair (sender key - `SK`) that he did not use before for him to authorize messages sent to Alice's router and another key pair for e2e encryption agreement. 2. Prepares the confirmation message for Alice to secure the queue. This message includes: - - Previously generated "public" key `SK` that will be used by Alice's server to verify Bob's messages, once the queue is secured. + - Previously generated "public" key `SK` that will be used by Alice's router to verify Bob's messages, once the queue is secured. - Public key to agree a shared secret with Alice for e2e encryption. @@ -207,9 +244,9 @@ To create and start using a simplex queue Alice and Bob follow these steps: 3. Encrypts the confirmation body with the shared secret agreed using public key `EK` (that Alice provided via the out-of-band message). - 4. Sends the encrypted message to the server with queue ID `SID` (see `send` in [Send message](#send-message)). This initial message to the queue must not be authorized - authorized messages will be rejected until Alice secures the queue (below). + 4. Sends the encrypted message to the router with queue ID `SID` (see `send` in [Send message](#send-message)). This initial message to the queue must not be authorized - authorized messages will be rejected until Alice secures the queue (below). -4. Alice receives Bob's message from the server using recipient queue ID `RID` (possibly, via the same transport connection she already has opened - see `message` in [Deliver queue message](#deliver-queue-message)): +4. Alice receives Bob's message from the router using recipient queue ID `RID` (possibly, via the same transport connection she already has opened - see `message` in [Deliver queue message](#deliver-queue-message)): 1. She decrypts received message body using the secret `SS`. @@ -217,11 +254,11 @@ To create and start using a simplex queue Alice and Bob follow these steps: 3. Anybody can send the message to the queue with ID `SID` before it is secured (e.g. if communication is compromised), so it's a "race" to secure the queue. Optionally, in the client application, Alice may identify Bob using the information provided, but it is out of scope of SMP protocol. -5. Alice secures the queue `RID` with `"KEY"` command so only Bob can send messages to it (see [Secure queue command](#secure-queue-command)): +5. Alice secures the queue `RID` with `"KEY"` command so only Bob can send messages to it (see [Secure queue by recipient](#secure-queue-by-recipient)): 1. She sends the `KEY` command with `RID` signed with "private" key `RK` to update the queue to only accept requests authorized by "private" key `SK` provided by Bob. This command contains unique "public" key `SK` previously generated by Bob. - 2. From this moment the server will accept only authorized commands to `SID`, so only Bob will be able to send messages to the queue `SID` (corresponding to `RID` that Alice has). + 2. From this moment the router will accept only authorized commands to `SID`, so only Bob will be able to send messages to the queue `SID` (corresponding to `RID` that Alice has). 3. Once queue is secured, Alice deletes `SID` and `SK` - even if Alice's client is compromised in the future, the attacker would not be able to send messages pretending to be Bob. @@ -239,19 +276,19 @@ Bob now can securely send messages to Alice: 1. He encrypts the message to Alice with the agreed shared secret (using "public" key `EK` provided by Alice, only known to Bob, used only for one simplex queue). - 2. He authorizes `"SEND"` command to the server queue `SID` using the "private" key `SK` (that only he knows, used only for this queue). + 2. He authorizes `"SEND"` command to the router queue `SID` using the "private" key `SK` (that only he knows, used only for this queue). - 3. He sends the command to the server (see `send` in [Send message](#send-message)), that the server will verify using the "public" key `SK` (that Alice earlier received from Bob and provided to the server via `"KEY"` command). + 3. He sends the command to the router (see `send` in [Send message](#send-message)), that the router will verify using the "public" key `SK` (that Alice earlier received from Bob and provided to the router via `"KEY"` command). 2. Alice receives the message(s): - 1. She authorizes `"SUB"` command to the server to subscribe to the queue `RID` with the "private" key `RK` (see `subscribe` in [Subscribe to queue](#subscribe-to-queue)). + 1. She authorizes `"SUB"` command to the router to subscribe to the queue `RID` with the "private" key `RK` (see `subscribe` in [Subscribe to queue](#subscribe-to-queue)). - 2. The server, having verified Alice's command with the "public" key `RK` that she provided, delivers Bob's message(s) (see `message` in [Deliver queue message](#deliver-queue-message)). + 2. The router, having verified Alice's command with the "public" key `RK` that she provided, delivers Bob's message(s) (see `message` in [Deliver queue message](#deliver-queue-message)). 3. She decrypts Bob's message(s) with the shared secret agreed using "private" key `EK`. - 4. She acknowledges the message reception to the server with `"ACK"` so that the server can delete the message and deliver the next messages. + 4. She acknowledges the message reception to the router with `"ACK"` so that the router can delete the message and deliver the next messages. This flow is show on sequence diagram below. @@ -263,11 +300,11 @@ This flow is show on sequence diagram below. ![Simplex queue operations](./diagrams/simplex-messaging/simplex-op.svg) -Sequence diagram does not show E2E encryption - server knows nothing about encryption between the sender and the receiver. +Sequence diagram does not show E2E encryption - router knows nothing about encryption between the sender and the receiver. A higher level application protocol should define the semantics that allow to use two simplex queues (or two sets of queues for redundancy) for the bi-directional or any other communication scenarios. -The SMP is intentionally unidirectional - it provides no answer to how Bob will know that the transmission succeeded, and whether Alice received any messages. There may be a scenario when Alice wants to securely receive the messages from Bob, but she does not want Bob to have any proof that she received any messages - this low-level protocol can be used in this scenario, as all Bob knows as a fact is that he was able to send one unsigned message to the server that Alice provided, and now he can only send messages signed with the key `SK` that he sent to the server - it does not prove that any message was received by Alice. +The SMP is intentionally unidirectional - it provides no answer to how Bob will know that the transmission succeeded, and whether Alice received any messages. There may be a scenario when Alice wants to securely receive the messages from Bob, but she does not want Bob to have any proof that she received any messages - this low-level protocol can be used in this scenario, as all Bob knows as a fact is that he was able to send one unsigned message to the router that Alice provided, and now he can only send messages signed with the key `SK` that he sent to the router - it does not prove that any message was received by Alice. For bi-directional conversation, now that Bob can securely send encrypted messages to Alice, Bob can create the second simplex queue that will allow Alice to send messages to Bob in the same way, sending the second queue details via the first queue. If both Alice and Bob have their respective unique "public" keys (Alice's and Bob's `EK`s of two separate queues), or pass additional keys to sign the messages, the conversation can be both encrypted and signed. @@ -293,37 +330,37 @@ Simplex Messaging Protocol: - Defines only message-passing protocol: - - Transport agnostic - the protocol does not define how clients connect to the servers. It can be implemented over any ordered data stream channel: TCP connection, HTTP with long polling, websockets, etc. + - Transport agnostic - the protocol does not define how clients connect to the routers. It can be implemented over any ordered data stream channel: TCP connection, HTTP with long polling, websockets, etc. - - Not semantic - the protocol does not assign any meaning to queues and messages. While on the application level the queues and messages can have different meaning (e.g., for messages: text or image chat message, message acknowledgement, participant profile information, status updates, changing "public" key to encrypt messages, changing servers, etc.), on SMP protocol level all the messages are binary and their meaning can only be interpreted by client applications and not by the servers - this interpretation is out of scope of this protocol. + - Not semantic - the protocol does not assign any meaning to queues and messages. While on the application level the queues and messages can have different meaning (e.g., for messages: text or image chat message, message acknowledgement, participant profile information, status updates, changing "public" key to encrypt messages, changing routers, etc.), on SMP protocol level all the messages are binary and their meaning can only be interpreted by client applications and not by the routers - this interpretation is out of scope of this protocol. -- Client-server architecture: +- Client-router architecture: - - Multiple servers, that can be deployed by the system users, can be used to send and retrieve messages. + - Multiple routers, that can be deployed by the system users, can be used to send and retrieve messages. - - Servers do not communicate with each other, except when used as proxy to forward commands to another server, and do not "know" about other servers. + - Routers do not communicate with each other, except when used as proxy to forward commands to another router, and do not "know" about other routers. - - Clients only communicate with servers (excluding the initial out-of-band message), so the message passing is asynchronous. + - Clients only communicate with routers (excluding the initial out-of-band message), so the message passing is asynchronous. - - For each queue, the message recipient defines the server through which the sender should send messages. To protect transport anonymity the sender can use their chosen server to forward commands to the server chosen by the recipient. + - For each queue, the message recipient defines the router through which the sender should send messages. To protect transport anonymity the sender can use their chosen router to forward commands to the router chosen by the recipient. - - While multiple servers and multiple queues can be used to pass each message, it is in scope of application level protocol(s), and out of scope of this protocol. + - While multiple routers and multiple queues can be used to pass each message, it is in scope of application level protocol(s), and out of scope of this protocol. - - Servers store messages only until they are retrieved by the recipients, and in any case, for a limited time. + - Routers store messages only until they are retrieved by the recipients, and in any case, for a limited time. - - Servers are required to NOT store any message history or delivery log, but even if the server is compromised, it does not allow to decrypt the messages or to determine the list of queues established by any participant - this information is only stored on client devices. + - Routers are required to NOT store any message history or delivery log, but even if the router is compromised, it does not allow to decrypt the messages or to determine the list of queues established by any participant - this information is only stored on client devices. -- The only element provided by SMP servers is simplex queues: +- The only element provided by SMP routers is simplex queues: - Each queue is created and managed by the queue recipient. - Asymmetric encryption is used to authorize and verify the requests to send and receive the messages. - - One ephemeral public key is used by the servers to verify requests to send the messages into the queue, and another ephemeral public key - to verify requests to retrieve the messages from the queue. These ephemeral keys are used only for one queue, and are not used for any other context - this key does not represent any participant identity. + - One ephemeral public key is used by the routers to verify requests to send the messages into the queue, and another ephemeral public key - to verify requests to retrieve the messages from the queue. These ephemeral keys are used only for one queue, and are not used for any other context - this key does not represent any participant identity. - - Both recipient and sender public keys are provided to the server by the queue recipient. "Public" key `RK` is provided when the queue is created, public key `SK` is provided when the queue is secured. V9 of SMP protocol allows senders to provide their key to the server directly or via proxy, to avoid waiting until the recipient is online to secure the queue. + - Both recipient and sender public keys are provided to the router by the queue recipient. "Public" key `RK` is provided when the queue is created, public key `SK` is provided when the queue is secured. V9 of SMP protocol allows senders to provide their key to the router directly or via proxy, to avoid waiting until the recipient is online to secure the queue. - - The "public" keys known to the server and used to verify commands from the participants are unrelated to the keys used to encrypt and decrypt the messages - the latter keys are also unique per each queue but they are only known to participants, not to the servers. + - The "public" keys known to the router and used to verify commands from the participants are unrelated to the keys used to encrypt and decrypt the messages - the latter keys are also unique per each queue but they are only known to participants, not to the routers. - Messaging graph can be asymmetric: Bob's ability to send messages to Alice does not automatically lead to the Alice's ability to send messages to Bob. @@ -331,7 +368,7 @@ Simplex Messaging Protocol: Simplex messaging clients must cryptographically authorize commands for the following operations: -- With the recipient's key `RK` (server to verify): +- With the recipient's key `RK` (router to verify): - create the queue (`NEW`) - subscribe to queue (`SUB`) - secure the queue (`KEY`) @@ -340,19 +377,19 @@ Simplex messaging clients must cryptographically authorize commands for the foll - acknowledge received messages (`ACK`) - suspend the queue (`OFF`) - delete the queue (`DEL`) -- With the sender's key `SK` (server to verify): +- With the sender's key `SK` (router to verify): - secure queue (`SKEY`) - send messages (`SEND`) - With the optional notifier's key: - subscribe to message notifications (`NSUB`) -To authorize/verify transmissions clients and servers MUST use either signature algorithm Ed25519 algorithm defined in [RFC8709][15] or [deniable authentication scheme](#deniable-client-authentication-scheme) based on NaCL crypto_box. +To authorize/verify transmissions clients and routers MUST use either signature algorithm Ed25519 algorithm defined in [RFC8709][15] or [deniable authentication scheme](#deniable-client-authentication-scheme) based on NaCL crypto_box. It is recommended that clients use signature algorithm for the recipient commands and deniable authentication scheme for sender commands (to have non-repudiation quality in the whole protocol stack). -To encrypt/decrypt message bodies delivered to the recipients, servers/clients MUST use NaCL crypto_box. +To encrypt/decrypt message bodies delivered to the recipients, routers/clients MUST use NaCL crypto_box. -Clients MUST encrypt message bodies sent via SMP servers using use NaCL crypto_box. +Clients MUST encrypt message bodies sent via SMP routers using use NaCL crypto_box. ## Deniable client authentication scheme @@ -360,47 +397,88 @@ While e2e encryption algorithms used in the client applications have repudiation SMP protocol supports repudiable authenticators to authorize client commands. These authenticators use NaCl crypto_box that proves authentication and third party unforgeability and, unlike signature, provides repudiation guarantee. See [crypto_box docs](https://nacl.cr.yp.to/box.html). -When queue is created or secured, the recipient would provide a DH key (X25519) to the server (either their own or received from the sender, in case of KEY command), and the server would provide its own random X25519 key per session in the handshake header. The authenticator is computed in this way: +When queue is created or secured, the recipient would provide a DH key (X25519) to the router (either their own or received from the sender, in case of KEY command), and the router would provide its own random X25519 key per session in the handshake header. The authenticator is computed in this way: ```abnf transmission = authenticator authorized -authenticator = crypto_box(sha512(authorized), secret = dh(client long term queue key, server session key), nonce = correlation ID) +authenticator = crypto_box(sha512(authorized), secret = dh(client long term queue key, router session key), nonce = correlation ID) authorized = sessionIdentifier corrId queueId protocol_command ; same as the currently signed part of the transmission ``` ## Simplex queue IDs -Simplex messaging servers MUST generate 2 different IDs for each new queue - for the recipient (that created the queue) and for the sender. It is REQUIRED that: +Simplex messaging routers MUST generate 2 different IDs for each new queue - for the recipient (that created the queue) and for the sender. It is REQUIRED that: -- These IDs are different and unique within the server. +- These IDs are different and unique within the router. - Based on random bytes generated with cryptographically strong pseudo-random number generator. -## Server security requirements +## Router security requirements -Simplex messaging server implementations MUST NOT create, store or send to any other servers: +Simplex messaging router implementations MUST NOT create, store or send to any other routers: - Logs of the client commands and transport connections in the production environment. - History of deleted queues, retrieved or acknowledged messages (deleted queues MAY be stored temporarily as part of the queue persistence implementation). -- Snapshots of the database they use to store queues and messages (instead simplex messaging clients must manage redundancy by using more than one simplex messaging server). In-memory persistence is recommended. +- Snapshots of the database they use to store queues and messages (instead simplex messaging clients must manage redundancy by using more than one simplex messaging router). In-memory persistence is recommended. -- Any other information that may compromise privacy or [forward secrecy][4] of communication between clients using simplex messaging servers (the servers cannot compromise forward secrecy of any application layer protocol, such as double ratchet). +- Any other information that may compromise privacy or [forward secrecy][4] of communication between clients using simplex messaging routers (the routers cannot compromise forward secrecy of any application layer protocol, such as double ratchet). ## Message delivery notifications Supporting message delivery while the client mobile app is not running requires sending push notifications with the device token. All alternative mechanisms for background message delivery are unreliable, particularly on iOS platform. -To protect the privacy of the recipients, there are several commands in SMP protocol that allow enabling and subscribing to message notifications from SMP queues, using separate set of "notifier keys" and via separate queue IDs - as long as SMP server is not compromised, these notifier queue IDs cannot be correlated with recipient or sender queue IDs. +To protect the privacy of the recipients, there are several commands in SMP protocol that allow enabling and subscribing to message notifications from SMP queues, using separate set of "notifier keys" and via separate queue IDs - as long as SMP router is not compromised, these notifier queue IDs cannot be correlated with recipient or sender queue IDs. -The clients can optionally instruct a dedicated push notification server to subscribe to notifications and deliver push notifications to the device, which can then retrieve the messages in the background and send local notifications to the user - this is out of scope of SMP protocol. The commands that SMP protocol provides to allow it: +The clients can optionally instruct a dedicated push notification router to subscribe to notifications and deliver push notifications to the device, which can then retrieve the messages in the background and send local notifications to the user - this is out of scope of SMP protocol. The commands that SMP protocol provides to allow it: -- `enableNotifications` (`"NKEY"`) with `notifierId` (`"NID"`) response - see [Enable notifications command](#enable-notifications-command). +- `enableNotifications` (`"NKEY"`) with `notifierIdResp` (`"NID"`) response - see [Enable notifications command](#enable-notifications-command). - `disableNotifications` (`"NDEL"`) - see [Disable notifications command](#disable-notifications-command). - `subscribeNotifications` (`"NSUB"`) - see [Subscribe to queue notifications](#subscribe-to-queue-notifications). - `messageNotification` (`"NMSG"`) - see [Deliver message notification](#deliver-message-notification). -[`SEND` command](#send-message) includes the notification flag to instruct SMP server whether to send the notification - this flag is forwarded to the recipient inside encrypted envelope, together with the timestamp and the message body, so even if TLS is compromised this flag cannot be used for traffic correlation. +[`SEND` command](#send-message) includes the notification flag to instruct SMP router whether to send the notification - this flag is forwarded to the recipient inside encrypted envelope, together with the timestamp and the message body, so even if TLS is compromised this flag cannot be used for traffic correlation. + +## Client services + +SMP protocol supports client services - high capacity clients that act as services. Client services allow scalable message and notification delivery services. + +### Service roles + +A client service can have one of three roles: + +- **Messaging** (`"M"`) - Message receiver service that subscribes to and receives messages from multiple SMP queues with a single command. + +- **Notifications** (`"N"`) - Notification service that subscribes to queue notifications and delivers push notifications to user devices. + +- **Proxy** (`"P"`) - Proxy service that forwards sender commands to destination routers. + +Service role is identified in the transport handshake and determines what commands the service is authorized to send. + +### Service certificates + +To send service commands, services should authenticate themselves to SMP routers using service certificates. This provides: + +- **Service identity** - The router assigns a unique service ID based on the service certificate, allowing associating multiple SMP queues with a service. +- **Subscription management** - Services can efficiently manage subscriptions across reconnections without re-subscribing to individual queues. +- **Rate limiting** - Routers can apply rate limits per service identity rather than per connection. + +Service certificates are included in the client handshake and verified by the router. The service receives a service ID in the handshake response, which is then used as entity ID in service transmissions. + +```abnf +clientService = serviceRole serviceCertKey +serviceRole = %s"M" / %s"N" / %s"P" ; Messaging / Notifier / Proxy +serviceCertKey = certChain signedServiceKey +``` + +### Service subscriptions + +Services use batch subscription commands to subscribe to multiple queues: + +- **SUBS** - Subscribe to messages from all associated SMP queues at once. The service provides a count and hash of queue IDs, and receives `SOKS` response with the service ID. +- **NSUBS** - Subscribe to notifications from all associated SMP queues. Similar to SUBS. +- **SOKS** - Router response confirming batch subscription success. +- **ENDS** - Router notification when batch subscriptions are terminated (e.g., when another instance of service connects). ## SMP Transmission and transport block structure @@ -421,7 +499,7 @@ paddedNotation = ; paddedLength - required length after padding, including 2 bytes for originalLength ``` -Transport block for SMP transmission between the client and the server must have this syntax: +Transport block for SMP transmission between the client and the router must have this syntax: ```abnf paddedTransportBlock = @@ -430,17 +508,19 @@ transmissionCount = 1*1 OCTET ; equal or greater than 1 transmissions = transmissionLength transmission [transmissions] transmissionLength = 2*2 OCTET ; word16 encoded in network byte order -transmission = authorization authorized +transmission = authorization [serviceSig] authorized authorized = sessionIdentifier corrId entityId smpCommand corrId = %x18 24*24 OCTET / %x0 "" - ; corrId is required in client commands and server responses, - ; it is empty (0-length) in server notifications. + ; corrId is required in client commands and router responses, + ; it is empty (0-length) in router notifications. ; %x18 is 24 - the random correlation ID must be 24 bytes as it is used as a nonce for NaCL crypto_box in some contexts. entityId = shortString ; queueId or proxySessionId - ; empty entityId ID is used with "create" command and in some server responses + ; empty entityId ID is used with "create" command and in some router responses authorization = shortString ; signature or authenticator ; empty authorization can be used with "send" before the queue is secured with secure command - ; authorization is always empty with "ping" and server responses + ; authorization is always empty with "ping" and router responses +serviceSig = shortString ; optional Ed25519 service signature (v16+) + ; present only in service sessions when authorization is non-empty sessionIdentifier = "" ; sessionIdentifierForAuth = shortString ; sessionIdentifierForAuth MUST be included in authorized transmission body. @@ -455,76 +535,105 @@ Commands syntax below is provided using [ABNF][8] with [case-sensitive strings e ```abnf smpCommand = ping / recipientCmd / senderCommand / - proxyCommand / subscribeNotifications / serverMsg -recipientCmd = create / subscribe / rcvSecure / - enableNotifications / disableNotifications / getMessage - acknowledge / suspend / delete / getQueueInfo + proxyCommand / notifierCommand / linkCommand / routerMsg +recipientCmd = create / subscribe / subscribeMultiple / rcvSecure / recipientKeys / + enableNotifications / disableNotifications / getMessage / + acknowledge / suspend / delete / getQueueInfo / setShortLink / deleteShortLink senderCommand = send / sndSecure -proxyCommand = proxySession / proxyCommand / relayCommand -serverMsg = queueIds / message / notifierId / messageNotification / - proxySessionKey / proxyResponse / relayResponse - unsubscribed / queueInfo/ ok / error +linkCommand = setLinkKey / getLinkData +proxyCommand = proxySession / proxyForward / relayForward +notifierCommand = subscribeNotifications / subscribeNotificationsMultiple +routerMsg = queueIds / linkResponse / serviceOk / serviceOkMultiple / + message / allReceived / notifierIdResponse / messageNotification / + proxySessionKey / proxyResponse / relayResponse / + unsubscribed / serviceUnsubscribed / deleted / + queueInfo / ok / error / pong ``` The syntax of specific commands and responses is defined below. ### Correlating responses with commands -The server should send `queueIds`, `error` and `ok` responses in the same order within each queue ID as the commands received in the transport connection, so that they can be correlated by the clients. To simplify correlation of commands and responses, the server must use the same `corrId` in the response as in the command sent by the client. +The router should send `queueIds`, `error` and `ok` responses in the same order within each queue ID as the commands received in the transport connection, so that they can be correlated by the clients. To simplify correlation of commands and responses, the router must use the same `corrId` in the response as in the command sent by the client. If the transport connection is closed before some responses are sent, these responses should be discarded. ### Command verification -SMP servers must verify all transmissions (excluding `ping` and initial `send` commands) by verifying the client authorizations. Command authorization should be generated by applying the algorithm specified for the queue to the `signed` block of the transmission, using the key associated with the queue ID (recipient's, sender's or notifier's, depending on which queue ID is used). +SMP routers must verify all transmissions (excluding `ping` and initial `send` commands) by verifying the client authorizations. Command authorization should be generated by applying the algorithm specified for the queue to the `signed` block of the transmission, using the key associated with the queue ID (recipient's, sender's or notifier's, depending on which queue ID is used). ### Keep-alive command -To keep the transport connection alive and to generate noise traffic the clients should use `ping` command to which the server responds with `ok` response. This command should be sent unsigned and without queue ID. +To keep the transport connection alive and to generate noise traffic the clients should use `ping` command to which the router responds with `pong` response. This command should be sent unsigned and without queue ID. ```abnf ping = %s"PING" +pong = %s"PONG" ``` -This command is always send unsigned. +This command is always sent unsigned. ### Recipient commands -Sending any of the commands in this section (other than `create`, that is sent without queue ID) is only allowed with recipient's ID (`RID`). If sender's ID is used the server must respond with `"ERR AUTH"` response (see [Error responses](#error-responses)). +Sending any of the commands in this section (other than `create`, that is sent without queue ID) is only allowed with recipient's ID (`RID`). If sender's ID is used the router must respond with `"ERR AUTH"` response (see [Error responses](#error-responses)). #### Create queue command -This command is sent by the recipient to the SMP server to create a new queue. +This command is sent by the recipient to the SMP router to create a new queue. -Servers SHOULD support basic auth with this command, to allow only server owners and trusted users to create queues on the destiation servers. +Routers SHOULD support basic auth with this command, to allow only router owners and trusted users to create queues on the destination routers. The syntax is: ```abnf -create = %s"NEW " recipientAuthPublicKey recipientDhPublicKey basicAuth subscribe sndSecure +create = %s"NEW " recipientAuthPublicKey recipientDhPublicKey optBasicAuth subscribeMode optQueueReqData optNtfCreds recipientAuthPublicKey = length x509encoded ; the recipient's Ed25519 or X25519 public key to verify commands for this queue recipientDhPublicKey = length x509encoded ; the recipient's Curve25519 key for DH exchange to derive the secret -; that the server will use to encrypt delivered message bodies +; that the router will use to encrypt delivered message bodies ; using [NaCl crypto_box][16] encryption scheme (curve25519xsalsa20poly1305). -basicAuth = "0" / "1" shortString ; server password +optBasicAuth = %s"0" / (%s"1" shortString) ; optional router password subscribeMode = %s"S" / %s"C" ; S - create and subscribe, C - only create -sndSecure = %s"T" / %s"F" ; T - sender can secure the queue, from v9 +optQueueReqData = %s"0" / (%s"1" queueReqData) ; optional queue request data +queueReqData = queueReqMessaging / queueReqContact +queueReqMessaging = %s"M" optMessagingLinkData +queueReqContact = %s"C" optContactLinkData +optMessagingLinkData = %s"0" / (%s"1" senderId encFixedData encUserData) +optContactLinkData = %s"0" / (%s"1" linkId senderId encFixedData encUserData) +senderId = shortString ; first 24 bytes of SHA3-384(corrId) +linkId = shortString +encFixedData = largeString ; encrypted fixed link data +encUserData = largeString ; encrypted user data +optNtfCreds = %s"0" / (%s"1" ntfKey ntfDhKey) ; optional notification credentials +ntfKey = length x509encoded +ntfDhKey = length x509encoded x509encoded = +shortString = length *OCTET +largeString = length2 *OCTET length = 1*1 OCTET +length2 = 2*2 OCTET ; Word16, network byte order ``` -If the queue is created successfully, the server must send `queueIds` response with the recipient's and sender's queue IDs and public key to encrypt delivered message bodies: +If the queue is created successfully, the router must send `queueIds` response with the recipient's and sender's queue IDs and public key to encrypt delivered message bodies: ```abnf -queueIds = %s"IDS " recipientId senderId srvDhPublicKey sndSecure -serverDhPublicKey = length x509encoded -; the server's Curve25519 key for DH exchange to derive the secret -; that the server will use to encrypt delivered message bodies to the recipient +queueIds = %s"IDS " recipientId senderId srvDhPublicKey optQueueMode optLinkId optServiceId optRouterNtfCreds +srvDhPublicKey = length x509encoded +; the router's Curve25519 key for DH exchange to derive the secret +; that the router will use to encrypt delivered message bodies to the recipient recipientId = shortString ; 16-24 bytes senderId = shortString ; 16-24 bytes +optQueueMode = %s"0" / (%s"1" queueMode) +queueMode = %s"M" / %s"C" ; M - messaging (sender can secure), C - contact +optLinkId = %s"0" / (%s"1" linkId) +linkId = shortString +optServiceId = %s"0" / (%s"1" serviceId) +serviceId = shortString +optRouterNtfCreds = %s"0" / (%s"1" srvNtfId srvNtfDhKey) +srvNtfId = shortString +srvNtfDhKey = length x509encoded ``` Once the queue is created, depending on `subscribeMode` parameter of `NEW` command the recipient gets automatically subscribed to receive the messages from that queue, until the transport connection is closed. To start receiving the messages from the existing queue when the new transport connection is opened the client must use `subscribe` command. @@ -541,17 +650,29 @@ When the simplex queue was not created in the current transport connection, the subscribe = %s"SUB" ``` -If subscription is successful the server must respond with the first available message or with `ok` response if no messages are available. The recipient will continue receiving the messages from this queue until the transport connection is closed or until another transport connection subscribes to the same simplex queue - in this case the first subscription should be cancelled and [subscription END notification](#subscription-end-notification) delivered. +If subscription is successful the router must respond with the first available message or with [queue subscription response](#queue-subscription-response) (`SOK`) if no messages are available. The recipient will continue receiving the messages from this queue until the transport connection is closed or until another transport connection subscribes to the same simplex queue - in this case the first subscription should be cancelled and [subscription END notification](#subscription-end-notification) delivered. The first message will be delivered either immediately or as soon as it is available; to receive the following message the recipient must acknowledge the reception of the message (see [Acknowledge message delivery](#acknowledge-message-delivery)). This transmission and its response MUST be signed. +#### Subscribe to multiple queues + +This command is used by recipient services to subscribe to multiple queues at once: + +```abnf +subscribeMultiple = %s"SUBS " count idsHash +count = 8*8 OCTET ; Int64, network byte order (big-endian) +idsHash = length 16*16 OCTET ; XOR of MD5 hashes of all queue IDs +``` + +The count and idsHash allow the router to detect subscription drift. The router responds with `serviceOkMultiple` (`SOKS`) response. + #### Secure queue by recipient -This command is only used until v8 of SMP protocol. V9 uses [SKEY](#secure-queue-by-sender). +This command was used before v9 of SMP protocol. V9+ uses [SKEY](#secure-queue-by-sender). KEY is still supported for backwards compatibility. -This command is sent by the recipient to the server to add sender's key to the queue: +This command is sent by the recipient to the router to add sender's key to the queue: ```abnf rcvSecure = %s"KEY " senderAuthPublicKey @@ -565,9 +686,47 @@ Once the queue is secured only authorized messages can be sent to it. This command MUST be used in transmission with recipient queue ID. +#### Set queue recipient keys + +This command is used to set additional recipient keys to support shared management of the queue: + +```abnf +recipientKeys = %s"RKEY " recipientKeysList +recipientKeysList = count 1*recipientKey ; non-empty list +count = 1*1 OCTET ; number of keys (1-255) +recipientKey = length x509encoded +``` + +This command added to allow multiple group owners manage data of the same queue link. + +#### Set short link + +This command is used to associate a short link with the queue: + +```abnf +setShortLink = %s"LSET " linkId encFixedData encUserData +linkId = shortString +encFixedData = largeString ; encrypted fixed link data +encUserData = largeString ; encrypted user data (e.g., profile) +largeString = length2 *OCTET +length2 = 2*2 OCTET ; Word16, network byte order (big-endian) +``` + +The router responds with `OK` response if successful. + +#### Delete short link + +This command is used to remove a short link association from the queue: + +```abnf +deleteShortLink = %s"LDEL" +``` + +The router responds with `OK` or `ERR` + #### Enable notifications command -This command is sent by the recipient to the server to add notifier's key to the queue, to allow push notifications server to receive notifications when the message arrives, via a separate queue ID, without receiving message content. +This command is sent by the recipient to the router to add notifier's key to the queue, to allow push notifications router to receive notifications when the message arrives, via a separate queue ID, without receiving message content. ```abnf enableNotifications = %s"NKEY " notifierKey recipientNotificationDhPublicKey @@ -576,18 +735,18 @@ notifierKey = length x509encoded recipientNotificationDhPublicKey = length x509encoded ; the recipient's Curve25519 key for DH exchange to derive the secret -; that the server will use to encrypt notification metadata (encryptedNMsgMeta in NMSG) +; that the router will use to encrypt notification metadata (encryptedNMsgMeta in NMSG) ; using [NaCl crypto_box][16] encryption scheme (curve25519xsalsa20poly1305). ``` -The server will respond with `notifierId` response if notifications were enabled and the notifier's key was successfully added to the queue: +The router will respond with `NID` response if notifications were enabled and the notifier's key was successfully added to the queue: ```abnf -notifierId = %s"NID " notifierId srvNotificationDhPublicKey +notifierIdResponse = %s"NID " notifierId srvNotificationDhPublicKey notifierId = shortString ; 16-24 bytes srvNotificationDhPublicKey = length x509encoded -; the server's Curve25519 key for DH exchange to derive the secret -; that the server will use to encrypt notification metadata to the recipient (encryptedNMsgMeta in NMSG) +; the router's Curve25519 key for DH exchange to derive the secret +; that the router will use to encrypt notification metadata to the recipient (encryptedNMsgMeta in NMSG) ``` This response is sent with the recipient's queue ID (the third part of the transmission). @@ -596,15 +755,15 @@ To receive the message notifications, `subscribeNotifications` command ("NSUB") #### Disable notifications command -This command is sent by the recipient to the server to remove notifier's credentials from the queue: +This command is sent by the recipient to the router to remove notifier's credentials from the queue: ```abnf disableNotifications = %s"NDEL" ``` -The server must respond `ok` to this command if it was successful. +The router must respond `ok` to this command if it was successful. -Once notifier's credentials are removed server will no longer send "NMSG" for this queue to notifier. +Once notifier's credentials are removed router will no longer send "NMSG" for this queue to notifier. #### Get message command @@ -618,18 +777,18 @@ getMessage = %s"GET" #### Acknowledge message delivery -The recipient should send the acknowledgement of message delivery once the message was stored in the client, to notify the server that the message should be deleted: +The recipient should send the acknowledgement of message delivery once the message was stored in the client, to notify the router that the message should be deleted: ```abnf acknowledge = %s"ACK" SP msgId msgId = shortString ``` -Client must send message ID to acknowledge a particular message - to prevent double acknowledgement (e.g., when command response times out) resulting in message being lost. If the message was not delivered or if the ID of the message does not match the last delivered message, the server SHOULD respond with `ERR NO_MSG` error. +Client must send message ID to acknowledge a particular message - to prevent double acknowledgement (e.g., when command response times out) resulting in message being lost. If the message was not delivered or if the ID of the message does not match the last delivered message, the router SHOULD respond with `ERR NO_MSG` error. -The server should limit the time the message is stored, even if the message was not delivered or if acknowledgement is not sent by the recipient. +The router should limit the time the message is stored, even if the message was not delivered or if acknowledgement is not sent by the recipient. -Having received the acknowledgement, SMP server should delete the message and then send the next available message or respond with `ok` if there are no more messages available in this simplex queue. +Having received the acknowledgement, SMP router should delete the message and then send the next available message or respond with `ok` if there are no more messages available in this simplex queue. #### Suspend queue @@ -639,13 +798,13 @@ The recipient can suspend a queue prior to deleting it to make sure that no mess suspend = %s"OFF" ``` -The server must respond with `"ERR AUTH"` to any messages sent after the queue was suspended (see [Error responses](#error-responses)). +The router must respond with `"ERR AUTH"` to any messages sent after the queue was suspended (see [Error responses](#error-responses)). -The server must respond `ok` to this command if it was successful. +The router must respond `ok` to this command if it was successful. -This command can be sent multiple times (in case transport connection was interrupted and the response was not delivered), the server should still respond `ok` even if the queue is already suspended. +This command can be sent multiple times (in case transport connection was interrupted and the response was not delivered), the router should still respond `ok` even if the queue is already suspended. -There is no command to resume the queue. Servers must delete suspended queues that were not deleted after some period of time. +There is no command to resume the queue. Routers must delete suspended queues that were not deleted after some period of time. #### Delete queue @@ -669,7 +828,7 @@ queueInfo = %s"INFO " info info = ``` -The format of queue information is implementation specific, and is not part of the specification. For information, [JTD schema][17] for queue information returned by the reference implementation of SMP server is: +The format of queue information is implementation specific, and is not part of the specification. For information, [JTD schema][17] for queue information returned by the reference implementation of SMP router is: ```json { @@ -700,13 +859,13 @@ The format of queue information is implementation specific, and is not part of t ### Sender commands -Currently SMP defines only one command that can be used by senders - `send` message. This command must be used with sender's ID, if recipient's ID is used the server must respond with `"ERR AUTH"` response (see [Error responses](#error-responses)). +SMP defines two commands that can be used by senders - `sndSecure` and `send` message. These commands must be used with sender's ID, if recipient's ID is used the router must respond with `"ERR AUTH"` response (see [Error responses](#error-responses)). #### Secure queue by sender -This command is used from v8 of SMP protocol. V8 and earlier uses [KEY](#secure-queue-by-recipient). +This command is used from v9 of SMP protocol. V8 and earlier uses [KEY](#secure-queue-by-recipient). -This command is sent by the sender to the server to add sender's key to the queue: +This command is sent by the sender to the router to add sender's key to the queue: ```abnf sndSecure = %s"SKEY " senderAuthPublicKey @@ -720,23 +879,23 @@ This command MUST be used in transmission with sender queue ID. #### Send message -This command is sent to the server by the sender both to confirm the queue after the sender received out-of-band message from the recipient and to send messages after the queue is secured: +This command is sent to the router by the sender both to confirm the queue after the sender received out-of-band message from the recipient and to send messages after the queue is secured: ```abnf send = %s"SEND " msgFlags SP smpEncMessage -msgFlags = notificationFlag reserved +msgFlags = notificationFlag notificationFlag = %s"T" / %s"F" -smpEncMessage = smpEncClientMessage / smpEncConfirmation ; message up to 16064 bytes +smpEncMessage = smpEncClientMessage / smpEncConfirmation ; message up to 16048 bytes (v11+) -smpEncClientMessage = smpPubHeaderNoKey msgNonce sentClientMsgBody ; message up to 16064 bytes +smpEncClientMessage = smpPubHeaderNoKey msgNonce sentClientMsgBody ; message up to maxMessageLength bytes smpPubHeaderNoKey = smpClientVersion "0" -sentClientMsgBody = 16016*16016 OCTET +sentClientMsgBody = 16016*16016 OCTET ; = e2eEncMessageLength(16000) + authTagSize(16) smpEncConfirmation = smpPubHeaderWithKey msgNonce sentConfirmationBody smpPubHeaderWithKey = smpClientVersion "1" senderPublicDhKey ; sender's Curve25519 public key to agree DH secret for E2E encryption in this queue ; it is only sent in confirmation message -sentConfirmationBody = 15920*15920 OCTET ; E2E-encrypted smpClientMessage padded to 16016 bytes before encryption +sentConfirmationBody = 15920*15920 OCTET ; E2E-encrypted smpConfirmation padded to e2eEncConfirmationLength(15904), + authTagSize(16) senderPublicDhKey = length x509encoded smpClientVersion = word16 @@ -745,38 +904,40 @@ msgNonce = 24*24 OCTET word16 = 2*2 OCTET ``` -The first message is sent to confirm the queue - it should contain sender's server key (see decrypted message syntax below) - this first message may be sent without authorization. +The first message is sent to confirm the queue - it should contain sender's router key (see decrypted message syntax below) - this first message may be sent without authorization. Once the queue is secured (see [Secure queue by sender](#secure-queue-by-sender)), the subsequent `SEND` commands must be sent with the authorization. -The server must respond with `"ERR AUTH"` response in the following cases: +The router must respond with `"ERR AUTH"` response in the following cases: - the queue does not exist or is suspended - the queue is secured but the transmission does NOT have a authorization - the queue is NOT secured but the transmission has a authorization -The server must respond with `"ERR QUOTA"` response when queue capacity is exceeded. The number of messages that the server can hold is defined by the server configuration. When sender reaches queue capacity the server will not accept any further messages until the recipient receives ALL messages from the queue. After the last message is delivered, the server will deliver an additional special message indicating that the queue capacity was reached. See [Deliver queue message](#deliver-queue-message) +The router must respond with `"ERR QUOTA"` response when queue capacity is exceeded. The number of messages that the router can hold is defined by the router configuration. When sender reaches queue capacity the router will not accept any further messages until the recipient receives ALL messages from the queue. After the last message is delivered, the router will deliver an additional special message indicating that the queue capacity was reached. See [Deliver queue message](#deliver-queue-message) -Until the queue is secured, the server should accept any number of unsigned messages (up to queue capacity) - it allows the sender to resend the confirmation in case of failure. +Until the queue is secured, the router should accept any number of unsigned messages (up to queue capacity) - it allows the sender to resend the confirmation in case of failure. The body should be encrypted with the shared secret based on recipient's "public" key (`EK`); once decrypted it must have this format: ```abnf -sentClientMsgBody = +sentClientMsgBody = + ; e2eEncMessageLength = 16000 smpClientMessage = emptyHeader clientMsgBody emptyHeader = "_" -clientMsgBody = *OCTET ; up to 16016 - 2 +clientMsgBody = *OCTET ; up to e2eEncMessageLength - 2 -sentConfirmationBody = +sentConfirmationBody = + ; e2eEncConfirmationLength = 15904 smpConfirmation = smpConfirmationHeader confirmationBody smpConfirmationHeader = emptyHeader / %s"K" senderKey ; emptyHeader is used when queue is already secured by sender -confirmationBody = *OCTET ; up to 15920 - 2 +confirmationBody = *OCTET ; up to e2eEncConfirmationLength - 2 senderKey = length x509encoded ; the sender's Ed25519 or X25519 public key to authorize SEND commands for this queue ``` -`clientHeader` in the initial unsigned message is used to transmit sender's server key and can be used in the future revisions of SMP protocol for other purposes. +`clientHeader` in the initial unsigned message is used to transmit sender's router key and can be used in the future revisions of SMP protocol for other purposes. SMP transmission structure for directly sent messages: @@ -785,15 +946,15 @@ SMP transmission structure for directly sent messages: 1 | transmission count (= 1) 2 | originalLength 299- | authorization sessionId corrId queueId %s"SEND" SP (1+114 + 1+32? + 1+24 + 1+24 + 4+1 = 203) - ....... smpEncMessage (= 16064 bytes = 16384 - 320 bytes) + ....... smpEncMessage (= 16048 bytes for v11+, within 16384 - 320 bytes) 8- | smpPubHeader (for messages it is only version and '0' to mean "no DH key" = 3 bytes) 24 | nonce for smpClientMessage 16 | auth tag for smpClientMessage - ------- smpClientMessage (E2E encrypted, = 16016 bytes = 16064 - 48) + ------- smpClientMessage (E2E encrypted, = 16000 bytes = 16048 - 48, for v11+) 2 | originalLength 2- | smpPrivHeader ....... - | clientMsgBody (<= 16012 bytes = 16016 - 4) + | clientMsgBody (<= 15996 bytes = 16000 - 4) ....... 0+ | smpClientMessage pad ------- smpClientMessage end @@ -812,21 +973,21 @@ SMP transmission structure for received messages: 2 | originalLength 283- | authorization sessionId corrId queueId %s"MSG" SP msgId (1+114 + 1+32? + 1+24 + 1+24 + 3+1 + 1+24 = 227) 16 | auth tag (msgId is used as nonce) - ------- serverEncryptedMsg (= 16082 bytes = 16384 - 302 bytes) + ------- routerEncryptedMsg (= 16082 bytes = 16384 - 302 bytes) 2 | originalLength 8 | timestamp 8- | message flags - ....... smpEncMessage (= 16064 bytes = 16082 - 18 bytes) + ....... smpEncMessage (= 16048 bytes for v11+, padded within 16082 - 18 = 16064 bytes) 8- | smpPubHeader (empty header for the message) 24 | nonce for smpClientMessage 16 | auth tag for smpClientMessage - ------- smpClientMessage (E2E encrypted, = 16016 bytes = 16064 - 48 bytes) + ------- smpClientMessage (E2E encrypted, = 16000 bytes = 16048 - 48 bytes, for v11+) 2 | originalLength 2- | smpPrivHeader (empty header for the message) - ....... clientMsgBody (<= 16012 bytes = 16016 - 4) + ....... clientMsgBody (<= 15996 bytes = 16000 - 4) -- TODO move internal structure (below) to agent protocol 20- | agentPublicHeader (the size is for user messages post handshake, without E2E X3DH keys - it is version and 'M' for the messages - 3 bytes in total) - ....... E2E double-ratchet encrypted (<= 15996 bytes = 16016 - 20) + ....... E2E double-ratchet encrypted (<= 15980 bytes = 16000 - 20) 1 | encoded double ratchet header length (it is 123 now) 123 | encoded double ratchet header, including: 2 | version @@ -834,12 +995,12 @@ SMP transmission structure for received messages: 16 | double-ratchet header auth tag 1+88 | double-ratchet header (actual size is 69 bytes, the rest is reserved) 16 | message auth tag (IV generated from chain ratchet) - ------- encrypted agent message (= 15856 bytes = 15996 - 140) + ------- encrypted agent message (= 15840 bytes = 15980 - 140) 2 | originalLength 64- | agentHeader (the actual size is 41 = 8 + 1+32) 2 | %s"MM" ....... - | application message (<= 15788 bytes = 15856 - 68) + | application message (<= 15772 bytes = 15840 - 68) ....... 0+ | encrypted agent message pad ------- encrypted agent message end @@ -852,22 +1013,22 @@ SMP transmission structure for received messages: ------- smpClientMessage end | ....... smpEncMessage end - 0+ | serverEncryptedMsg pad - ------- serverEncryptedMsg end + 0+ | routerEncryptedMsg pad + ------- routerEncryptedMsg end 0+ | transmission pad ------- transmission end ``` ### Proxying sender commands -To protect transport (IP address and session) anonymity of the sender from the server chosen (and, potentially, controlled) by the recipient SMP v8 added support for proxying sender's command to the recipient's server via the server chosen by the sender. +To protect transport (IP address and session) anonymity of the sender from the router chosen (and, potentially, controlled) by the recipient SMP v8 added support for proxying sender's command to the recipient's router via the router chosen by the sender. Sequence diagram for sending the message and `SKEY` commands via SMP proxy: ``` ------------- ------------- ------------- ------------- | sending | | SMP | | SMP | | receiving | -| client | | proxy | | server | | client | +| client | | proxy | | router | | client | ------------- ------------- ------------- ------------- | `PRXY` | | | | -------------------------> | | | @@ -891,55 +1052,55 @@ Sequence diagram for sending the message and `SKEY` commands via SMP proxy: | | | | ``` -1. The client requests (`PRXY` command) the chosen server to connect to the destination SMP server and receives (`PKEY` response) the session information, including server certificate and the session key signed by this certificate. To protect client session anonymity the proxy MUST re-use the same session with all clients that request connection with any given destination server. +1. The client requests (`PRXY` command) the chosen router to connect to the destination SMP router and receives (`PKEY` response) the session information, including router certificate and the session key signed by this certificate. To protect client session anonymity the proxy MUST re-use the same session with all clients that request connection with any given destination router. -2. The client encrypts the transmission (`SKEY` or `SEND`) to the destination server using the shared secret computed from per-command random key and server's session key and sends it to proxying server in `PFWD` command. +2. The client encrypts the transmission (`SKEY` or `SEND`) to the destination router using the shared secret computed from per-command random key and router's session key and sends it to proxying router in `PFWD` command. -3. Proxy additionally encrypts the body to prevent correlation by ciphertext (in case TLS is compromised) and forwards it to proxy in `RFWD` command. +3. Proxy additionally encrypts the body to prevent correlation by ciphertext (in case TLS is compromised) and forwards it to the destination router in `RFWD` command. -4. Proxy receives the double-encrypted response from the destination server, removes one encryption layer and forwards it to the client. +4. Proxy receives the double-encrypted response from the destination router, removes one encryption layer and forwards it to the client. The diagram below shows the encryption layers for `PFWD`/`RFWD` commands and `RRES`/`PRES` responses: -- s2r - encryption between client and SMP relay, with relay key returned in relay handshake, with MITM by proxy mitigated by verifying the certificate fingerprint included in the relay address. This encryption prevents proxy server from observing commands and responses - proxy does not know how many different queues a connected client sends messages and commands to. +- s2r - encryption between client and SMP router, with router key returned in router handshake, with MITM by proxy mitigated by verifying the certificate fingerprint included in the router address. This encryption prevents proxy router from observing commands and responses - proxy does not know how many different queues a connected client sends messages and commands to. - e2e - end-to-end encryption per SMP queue, with additional client encryption inside it. -- p2r - additional encryption between proxy and SMP relay with the shared secret agreed in the handshake, to mitigate traffic correlation inside TLS. -- r2c - additional encryption between SMP relay and client to prevent traffic correlation inside TLS. +- p2r - additional encryption between proxy and SMP router with the shared secret agreed in the handshake, to mitigate traffic correlation inside TLS. +- r2c - additional encryption between SMP router and client to prevent traffic correlation inside TLS. ``` ----------------- ----------------- -- TLS -- ----------------- ----------------- | | -- TLS -- | | -- p2r -- | | -- TLS -- | | | | -- s2r -- | | -- s2r -- | | -- r2c -- | | | sending | -- e2e -- | | -- e2e -- | | -- e2e -- | receiving | -| client | MSG | SMP proxy | MSG | SMP server | MSG | client | +| client | MSG | SMP proxy | MSG | SMP router | MSG | client | | | -- e2e -- | | -- e2e -- | | -- e2e -- | | | | -- s2r -- | | -- s2r -- | | -- r2c -- | | | | -- TLS -- | | -- p2r -- | | -- TLS -- | | ----------------- ----------------- -- TLS -- ----------------- ----------------- ``` -SMP proxy is not another type of the server, it is a role that any SMP server can play when forwarding the commands. +SMP proxy is not another type of the router, it is a role that any SMP router can play when forwarding the commands. #### Request proxied session -The sender uses this command to request the session with the destination proxy. +The sender uses this command to request the session with the destination router. -Servers SHOULD support basic auth with this command, to allow only server owners and trusted users to proxy commands to the destination servers. +Routers SHOULD support basic auth with this command, to allow only router owners and trusted users to proxy commands to the destination routers. ```abnf -proxySession = %s"PRXY" SP smpServer basicAuth -smpServer = hosts port fingerprint +proxySession = %s"PRXY" SP smpRouter basicAuth +smpRouter = hosts port fingerprint hosts = length 1*host host = shortString port = shortString fingerprint = shortString -basicAuth = "0" / "1" shortString ; server password +basicAuth = "0" / "1" shortString ; router password ``` ```abnf proxySessionKey = %s"PKEY" SP sessionId smpVersionRange certChain signedKey sessionId = shortString - ; Session ID (tlsunique) of the proxy with the destination server. + ; Session ID (tlsunique) of the proxy with the destination router. ; This session ID should be used as entity ID in transmission with `PFWD` command certChain = length 1*cert cert = originalLength x509encoded @@ -948,92 +1109,177 @@ originalLength = 2*2 OCTET ``` When the client receives PKEY response it MUST validate that: -- the fingerprint of the received certificate matches fingerprint in the server address - it mitigates MITM attack by proxy. -- the server session key is correctly signed with the received certificate. +- the fingerprint of the received certificate matches fingerprint in the router address - it mitigates MITM attack by proxy. +- the router session key is correctly signed with the received certificate. -The proxy server may respond with error response in case the destination server is not available or in case it has an earlier version that does not support proxied commands. +The proxy router may respond with error response in case the destination router is not available or in case it has an earlier version that does not support proxied commands. #### Send command via proxy Sender can send `SKEY` and `SEND` commands via proxy after obtaining the session ID with `PRXY` command (see [Request proxied session](#request-proxied-session)). -Transmission sent to proxy server should use session ID as entity ID and use a random correlation ID of 24 bytes as a nonce for crypto_box encryption of transmission to the destination server. The random ephemeral X25519 key to encrypt transmission should be unique per command, and it should be combined with the key sent by the server in the handshake header to proxy and to the client in `PKEY` command. +Transmission sent to proxy router should use session ID as entity ID and use a random correlation ID of 24 bytes as a nonce for crypto_box encryption of transmission to the destination router. The random ephemeral X25519 key to encrypt transmission should be unique per command, and it should be combined with the key sent by the router in the handshake header to proxy and to the client in `PKEY` command. -Encrypted transmission should use the received session ID from the connection between proxy server and destination server in the authorized body. +Encrypted transmission should use the received session ID from the connection between proxy router and destination router in the authorized body. ```abnf -proxyCommand = %s"PFWD" SP smpVersion commandKey +proxyForward = %s"PFWD" SP smpVersion commandKey smpVersion = 2*2 OCTET commandKey = length x509encoded ``` -The proxy server will forward the encrypted transmission in `RFWD` command (see below). +The proxy router will forward the encrypted transmission in `RFWD` command (see below). -Having received the `RRES` response from the destination server, proxy server will forward `PRES` response to the client. `PRES` response should use the same correlation ID as `PFWD` command. The destination server will use this correlation ID increased by 1 as a nonce for encryption of the response. +Having received the `RRES` response from the destination router, proxy router will forward `PRES` response to the client. `PRES` response should use the same correlation ID as `PFWD` command. The destination router will use this correlation ID increased by 1 as a nonce for encryption of the response. ```abnf -proxyResponse = %s"PRES" SP +proxyResponse = %s"PRES" SP +forwardedResponse = *OCTET ; client-encrypted SMP response, decrypted by client using per-command DH secret ``` -#### Forward command to destination server +#### Forward command to destination router -Having received `PFWD` command from the client, the server should additionally encrypt it (without padding, as the received transmission is already encrypted by the client and padded to a fixed size) together with the correlation ID, sender command key, and protocol version, and forward it to the destination server as `RFWD` command: +Having received `PFWD` command from the client, the router should additionally encrypt it (without padding, as the received transmission is already encrypted by the client and padded to a fixed size) together with the correlation ID, sender command key, and protocol version, and forward it to the destination router as `RFWD` command: -Transmission forwarded to relay uses empty entity ID and its unique random correlation ID is used as a nonce to encrypt forwarded transmission. Correlation ID increased by 1 is used by the destination server as a nonce to encrypt responses. +Transmission forwarded to destination router uses empty entity ID and its unique random correlation ID is used as a nonce to encrypt forwarded transmission. Correlation ID increased by 1 is used by the destination router as a nonce to encrypt responses. ```abnf -relayCommand = %s"RFWD" SP +relayForward = %s"RFWD" SP forwardedTransmission = fwdCorrId fwdSmpVersion fwdCommandKey transmission fwdCorrId = length 24*24 OCTET ; `fwdCorrId` - correlation ID used in `PFWD` command transmission - it is used as a nonce for client encryption, - ; and `fwdCorrId + 1` is used as a nonce for the destination server response encryption. + ; and `fwdCorrId + 1` is used as a nonce for the destination router response encryption. fwdSmpVersion = 2*2 OCTET fwdCommandKey = length x509encoded transmission = *OCTET ; note that it is not prefixed with the length ``` -The destination server having received this command decrypts both encryption layers (proxy and client), verifies client authorization as usual, processes it, and send the double encrypted `RRES` response to proxy. +The destination router having received this command decrypts both encryption layers (proxy and client), verifies client authorization as usual, processes it, and send the double encrypted `RRES` response to proxy. -The shared secret for encrypting transmission bodies between proxy server and destination server is agreed from proxy and destination server keys exchanged in handshake headers - proxy and server use the same shared secret during the session for the encryption between them. +The shared secret for encrypting transmission bodies between proxy router and destination router is agreed from proxy and destination router keys exchanged in handshake headers - proxy and router use the same shared secret during the session for the encryption between them. ```abnf relayResponse = %s"RRES" SP +responseTransmission = fwdCorrId forwardedResponse + ; fwdCorrId and forwardedResponse defined above in RFWD section +``` + +### Short link commands + +These commands are used by senders to access queues via short links (added in v15). + +#### Set link key + +This command is used to set the sender key and to get link data associated with a "messaging" queue: + +```abnf +setLinkKey = %s"LKEY " senderAuthPublicKey +senderAuthPublicKey = length x509encoded +``` + +The router secures the queue with the provided key and responds with `LNK` response containing the sender ID and encrypted link data. + +Once this command is used, the queue is secured, and the command can only be repeated with the same key. + +#### Get link data + +This command is used to retrieve the link data associated with a "contact" queue: + +```abnf +getLinkData = %s"LGET" ``` +The router responds with `LNK` response containing the sender ID and encrypted link data. + +This command may be repeated multiple times. + ### Notifier commands #### Subscribe to queue notifications -The push notifications server (notifier) must use this command to start receiving message notifications from the queue: +The push notifications router (notifier) must use this command to start receiving message notifications from the queue: ```abnf subscribeNotifications = %s"NSUB" ``` -If subscription is successful the server must respond with `ok` response if no messages are available. The notifier will be receiving the message notifications from this queue until the transport connection is closed or until another transport connection subscribes to notifications from the same simplex queue - in this case the first subscription should be cancelled and [subscription END notification](#subscription-end-notification) delivered. +If subscription is successful the router must respond with [queue subscription response](#queue-subscription-response) (`SOK`). The notifier will be receiving the message notifications from this queue until the transport connection is closed or until another transport connection subscribes to notifications from the same simplex queue - in this case the first subscription should be cancelled and [subscription END notification](#subscription-end-notification) delivered. The first message notification will be delivered either immediately or as soon as the message is available. -### Server messages +#### Subscribe to multiple queue notifications + +This command is used by notifier services to subscribe to multiple queues at once: + +```abnf +subscribeNotificationsMultiple = %s"NSUBS " count idsHash +count = 8*8 OCTET ; Int64, network byte order (big-endian) +idsHash = length 16*16 OCTET ; XOR of MD5 hashes of all queue IDs +``` + +The router responds with `serviceOkMultiple` (`SOKS`) response. + +### Router messages -This section includes server events and generic command responses used for several commands. +This section includes router events and generic command responses used for several commands. The syntax for command-specific responses is shown together with the commands. +#### Link response + +Sent in response to `LKEY` and `LGET` commands: + +```abnf +linkResponse = %s"LNK " senderId encFixedData encUserData +senderId = shortString ; the sender ID for the queue +encFixedData = largeString ; encrypted fixed link data +encUserData = largeString ; encrypted user data +``` + +#### Queue subscription response + +Sent in response to `SUB` and `NSUB` commands: + +```abnf +serviceOk = %s"SOK " optServiceId +optServiceId = %s"0" / (%s"1" serviceId) +serviceId = shortString +``` + +If response contains `serviceId`, it means that queue is associated with the service. + +#### Service subscription response + +Sent in response to `SUBS` or `NSUBS` commands: + +```abnf +serviceOkMultiple = %s"SOKS " count idsHash +count = 8*8 OCTET ; Int64, network byte order (big-endian) +idsHash = length 16*16 OCTET ; XOR of MD5 hashes of all subscribed queue IDs +``` + +#### All service messages received + +Sent to indicate all messages have been delivered from all queues associated with the service: + +```abnf +allReceived = %s"ALLS" +``` + #### Deliver queue message -When server delivers the messages to the recipient, message body should be encrypted with the secret derived from DH exchange using the keys passed during the queue creation and returned with `queueIds` response. +When router delivers the messages to the recipient, message body should be encrypted with the secret derived from DH exchange using the keys passed during the queue creation and returned with `queueIds` response. -This is done to prevent the possibility of correlation of incoming and outgoing traffic of SMP server inside transport protocol. +This is done to prevent the possibility of correlation of incoming and outgoing traffic of SMP router inside transport protocol. -The server must deliver messages to all subscribed simplex queues on the currently open transport connection. The syntax for the message delivery is: +The router must deliver messages to all subscribed simplex queues on the currently open transport connection. The syntax for the message delivery is: ```abnf message = %s"MSG" SP msgId encryptedRcvMsgBody encryptedRcvMsgBody = - ; server-encrypted padded sent msgBody - ; maxMessageLength = 16064 + ; router-encrypted padded sent msgBody + ; maxMessageLength = 16048 (v11+) rcvMsgBody = timestamp msgFlags SP sentMsgBody / msgQuotaExceeded msgQuotaExceeded = %s"QUOTA" SP timestamp msgId = length 24*24OCTET @@ -1042,34 +1288,32 @@ timestamp = 8*8OCTET If the sender exceeded queue capacity the recipient will receive a special message indicating the quota was exceeded. This can be used in the higher level protocol to notify sender client that it can continue sending messages. -`msgId` - unique message ID generated by the server based on cryptographically strong random bytes. It should be used by the clients to detect messages that were delivered more than once (in case the transport connection was interrupted and the server did not receive the message delivery acknowledgement). Message ID is used as a nonce for server/recipient encryption of message bodies. +`msgId` - unique message ID generated by the router based on cryptographically strong random bytes. It should be used by the clients to detect messages that were delivered more than once (in case the transport connection was interrupted and the router did not receive the message delivery acknowledgement). Message ID is used as a nonce for router/recipient encryption of message bodies. -`timestamp` - system time when the server received the message from the sender as **a number of seconds** since Unix epoch (1970-01-01) encoded as 64-bit integer in network byte order. If a client system/language does not support 64-bit integers, until 2106 it is safe to simply skip the first 4 zero bytes and decode 32-bit unsigned integer (or as signed integer until 2038). +`timestamp` - system time when the router received the message from the sender as **a number of seconds** since Unix epoch (1970-01-01) encoded as 64-bit integer in network byte order. If a client system/language does not support 64-bit integers, until 2106 it is safe to simply skip the first 4 zero bytes and decode 32-bit unsigned integer (or as signed integer until 2038). `sentMsgBody` - message sent by `SEND` command. See [Send message](#send-message). #### Deliver message notification -The server must deliver message notifications to all simplex queues that were subscribed with `subscribeNotifications` command (`NSUB`) on the currently open transport connection. The syntax for the message notification delivery is: +The router must deliver message notifications to all simplex queues that were subscribed with `subscribeNotifications` command (`NSUB`) on the currently open transport connection. The syntax for the message notification delivery is: ```abnf messageNotification = %s"NMSG " nmsgNonce encryptedNMsgMeta - -encryptedNMsgMeta = -; metadata E2E encrypted between server and recipient containing server's message ID and timestamp (allows extension), -; to be passed to the recipient by the notifier for them to decrypt -; with key negotiated in NKEY and NID commands using nmsgNonce - -nmsgNonce = -; nonce used by the server for encryption of message metadata, to be passed to the recipient by the notifier -; for them to use in decryption of E2E encrypted metadata +nmsgNonce = 24*24 OCTET ; 192-bit NaCl crypto_box nonce +encryptedNMsgMeta = shortString + ; NaCl crypto_box encrypted padded(nmsgMeta, 128): 128 + 16 (auth tag) = 144 bytes + ; metadata E2E encrypted between router and recipient, + ; to be passed to the recipient by the notifier for them to decrypt + ; with key negotiated in NKEY and NID commands using nmsgNonce +nmsgMeta = msgId timestamp ; message ID and timestamp, allows future extension ``` Message notification does not contain any message data or non E2E encrypted metadata. #### Subscription END notification -When another transport connection is subscribed to the same simplex queue, the server should unsubscribe and to send the notification to the previously subscribed transport connection: +When another transport connection is subscribed to the same simplex queue, the router should unsubscribe and to send the notification to the previously subscribed transport connection: ```abnf unsubscribed = %s"END" @@ -1077,6 +1321,24 @@ unsubscribed = %s"END" No further messages should be delivered to unsubscribed transport connection. +#### Service subscription END notification + +Sent when service subscription is terminated (can be sent when service re-connects): + +```abnf +serviceUnsubscribed = %s"ENDS " count idsHash +count = 8*8 OCTET ; Int64, network byte order (big-endian) +idsHash = length 16*16 OCTET ; XOR of MD5 hashes of terminated queue IDs +``` + +#### Queue deleted notification + +Sent when a queue has been deleted via another connection: + +```abnf +deleted = %s"DELD" +``` + #### Error responses - incorrect block format, encoding or authorization size (`BLOCK`). @@ -1090,135 +1352,153 @@ No further messages should be delivered to unsubscribed transport connection. - transmission has no required authorization or queue ID (`NO_AUTH`) - transmission has unexpected credentials (`HAS_AUTH`) - transmission has no required queue ID (`NO_ENTITY`) -- proxy server errors (`PROXY`): +- proxy router errors (`PROXY`): - `PROTOCOL` - any error. - `BASIC_AUTH` - incorrect basic auth. - - `NO_SESSION` - no destination server session with passed ID. - - `BROKER` - destination server error: - - `RESPONSE` - invalid server response (failed to parse). + - `NO_SESSION` - no destination router session with passed ID. + - `BROKER` - destination router error: + - `RESPONSE` - invalid router response (failed to parse). - `UNEXPECTED` - unexpected response. - `NETWORK` - network error. - `TIMEOUT` - command response timeout. - - `HOST` - no compatible server host (e.g. onion when public is required, or vice versa) + - `HOST` - no compatible router host (e.g. onion when public is required, or vice versa) + - `NO_SERVICE` - service unavailable client-side. - `TRANSPORT` - handshake or other transport error: - `BLOCK` - error parsing transport block. - - `VERSION` - incompatible client or server version. + - `VERSION` - incompatible client or router version. - `LARGE_MSG` - message too large. - `SESSION` - incorrect session ID. - - `NO_AUTH` - absent server key - when the server did not provide a DH key to authorize commands for the queue that should be authorized with a DH key. + - `NO_AUTH` - absent router key - when the router did not provide a DH key to authorize commands for the queue that should be authorized with a DH key. - `HANDSHAKE` - transport handshake error: - `PARSE` - handshake syntax (parsing) error. - - `IDENTITY` - incorrect server identity (certificate fingerprint does not match server address). - - `BAD_AUTH` - incorrect or missing server credentials in handshake. + - `IDENTITY` - incorrect router identity (certificate fingerprint does not match router address). + - `BAD_AUTH` - incorrect or missing router credentials in handshake. - authentication error (`AUTH`) - incorrect authorization, unknown (or suspended) queue, sender's ID is used in place of recipient's and vice versa, and some other cases (see [Send message](#send-message) command). +- blocked entity error (`BLOCKED`) - the entity (queue or message) was blocked due to policy violation (added in v12). Contains blocking information: + - `reason` - blocking reason (`spam` or `content`). + - `notice` - optional client notice with additional information. +- service error (`SERVICE`) - service-related error. +- crypto error (`CRYPTO`) - cryptographic operation failed. - message queue quota exceeded error (`QUOTA`) - too many messages were sent to the message queue. Further messages can only be sent after the recipient retrieves the messages. -- sent message is too large (> 16064) to be delivered (`LARGE_MSG`). -- internal server error (`INTERNAL`). +- store error (`STORE`) - router storage error with error message. +- router public key expired (`EXPIRED`) - router public key has expired. +- no message (`NO_MSG`) - no message available or message ID mismatch. +- sent message is too large (> maxMessageLength) to be delivered (`LARGE_MSG`). +- internal router error (`INTERNAL`). +- duplicate error (`DUPLICATE_`) - internal duplicate detection error (not returned by router). The syntax for error responses: ```abnf error = %s"ERR " errorType -errorType = %s"BLOCK" / %s"SESSION" / %s"CMD" SP cmdError / %s"PROXY" proxyError / - %s"AUTH" / %s"QUOTA" / %s"LARGE_MSG" / %s"INTERNAL" -cmdError = %s"SYNTAX" / %s"PROHIBITED" / %s"NO_AUTH" / %s"HAS_AUTH" / %s"NO_ENTITY" +errorType = %s"BLOCK" / %s"SESSION" / %s"CMD" SP cmdError / %s"PROXY" SP proxyError / + %s"AUTH" / %s"BLOCKED" SP blockingInfo / %s"SERVICE" / %s"CRYPTO" / + %s"QUOTA" / %s"STORE" SP storeError / %s"EXPIRED" / %s"NO_MSG" / + %s"LARGE_MSG" / %s"INTERNAL" / %s"DUPLICATE_" +cmdError = %s"UNKNOWN" / %s"SYNTAX" / %s"PROHIBITED" / %s"NO_AUTH" / %s"HAS_AUTH" / %s"NO_ENTITY" proxyError = %s"PROTOCOL" SP errorType / %s"BROKER" SP brokerError / %s"BASIC_AUTH" / %s"NO_SESSION" brokerError = %s"RESPONSE" SP shortString / %s"UNEXPECTED" SP shortString / - %s"NETWORK" / %s"TIMEOUT" / %s"HOST" / + %s"NETWORK" [SP networkError] / %s"TIMEOUT" / %s"HOST" / %s"NO_SERVICE" / %s"TRANSPORT" SP transportError +networkError = %s"CONNECT" SP shortString / %s"TLS" SP shortString / + %s"UNKNOWNCA" / %s"FAILED" / %s"TIMEOUT" / %s"SUBSCRIBE" SP shortString transportError = %s"BLOCK" / %s"VERSION" / %s"LARGE_MSG" / %s"SESSION" / %s"NO_AUTH" / %s"HANDSHAKE" SP handshakeError -handshakeError = %s"PARSE" / %s"IDENTITY" / %s"BAD_AUTH" +handshakeError = %s"PARSE" / %s"IDENTITY" / %s"BAD_AUTH" / %s"BAD_SERVICE" +blockingInfo = %s"reason=" blockingReason ["," %s"notice=" jsonNotice] +blockingReason = %s"spam" / %s"content" +jsonNotice = +storeError = *OCTET ``` -Server implementations must aim to respond within the same time for each command in all cases when `"ERR AUTH"` response is required to prevent timing attacks (e.g., the server should verify authorization even when the queue does not exist on the server or the authorization of different type is sent, using any dummy key compatible with the used authorization). +Router implementations must aim to respond within the same time for each command in all cases when `"ERR AUTH"` response is required to prevent timing attacks (e.g., the router should verify authorization even when the queue does not exist on the router or the authorization of different type is sent, using any dummy key compatible with the used authorization). ### OK response -When the command is successfully executed by the server, it should respond with OK response: +When the command is successfully executed by the router, it should respond with OK response: ```abnf ok = %s"OK" ``` -## Transport connection with the SMP server +## Transport connection with the SMP router ### General transport protocol considerations -Both the recipient and the sender can use TCP or some other, possibly higher level, transport protocol to communicate with the server. The default TCP port for SMP server is 5223. +Both the recipient and the sender can use TCP or some other, possibly higher level, transport protocol to communicate with the router. The default TCP port for SMP router is 5223. The transport protocol should provide the following: -- server authentication (by matching server certificate hash with `serverIdentity`), +- router authentication (by matching router certificate hash with `routerIdentity`), - forward secrecy (by encrypting the traffic using ephemeral keys agreed during transport handshake), - integrity (preventing data modification by the attacker without detection), - unique channel binding (`sessionIdentifier`) to include in the signed part of SMP transmissions. ### TLS transport encryption -The client and server communicate using [TLS 1.3 protocol][13] restricted to: +The client and router communicate using [TLS 1.3 protocol][13] restricted to: - TLS_CHACHA20_POLY1305_SHA256 cipher suite (for better performance on mobile devices), - ed25519 EdDSA algorithms for signatures, - x25519 ECDHE groups for key exchange. -- servers must send the chain of 2, 3 or 4 self-signed certificates in the handshake (see [Server certificate](#server-certificate)), with the first (offline) certificate one signing the second (online) certificate. Offline certificate fingerprint is used as a server identity - it is a part of SMP server address. +- routers must send the chain of 2, 3 or 4 self-signed certificates in the handshake (see [Router certificate](#router-certificate)), with the first (offline) certificate one signing the second (online) certificate. Offline certificate fingerprint is used as a router identity - it is a part of SMP router address. - The clients must abort the connection in case a different number of certificates is sent. -- server and client TLS configuration should not allow resuming the sessions. +- router and client TLS configuration should not allow resuming the sessions. -During TLS handshake the client must validate that the fingerprint of the online server certificate is equal to the `serverIdentity` the client received as part of SMP server address; if the server identity does not match the client must abort the connection. +During TLS handshake the client must validate that the fingerprint of the online router certificate is equal to the `routerIdentity` the client received as part of SMP router address; if the router identity does not match the client must abort the connection. -### Server certificate +### Router certificate -Servers use self-signed certificates that the clients validate by comparing the fingerprint of one of the certificates in the chain with the certificate fingerprint present in the server address. +Routers use self-signed certificates that the clients validate by comparing the fingerprint of one of the certificates in the chain with the certificate fingerprint present in the router address. -Clients SHOULD support the chains of 2, 3 and 4 server certificates: +Clients SHOULD support the chains of 2, 3 and 4 router certificates: **2 certificates**: -1. offline server certificate: - - its fingerprint is present in the server address. - - its private key is not stored on the server. -2. online server certificate: +1. offline router certificate: + - its fingerprint is present in the router address. + - its private key is not stored on the router. +2. online router certificate: - it must be signed by offline certificate. - - its private key is stored on the server and is used in TLS session. + - its private key is stored on the router and is used in TLS session. **3 certificates**: -1. offline server certificate - same as with 2 certificates. -2. online server certificate: +1. offline router certificate - same as with 2 certificates. +2. online router certificate: - it must be signed by offline certificate. - - its private key is stored on the server. + - its private key is stored on the router. 3. session certificate: - - generated automatically on every server start and/or on schedule. - - signed by online server certificate. + - generated automatically on every router start and/or on schedule. + - signed by online router certificate. - its private key is used in TLS session. **4 certificates**: 0. offline operator identity certificate: - - used for all servers operated by the same entity. - - its private key is not stored on the server. -1. offline server certificate: + - used for all routers operated by the same entity. + - its private key is not stored on the router. +1. offline router certificate: - signed by offline operator certificate. - same as with 2 certificates. -2. online server certificate - same as with 3 certificates. +2. online router certificate - same as with 3 certificates. 3. session certificate - same as with 3 certificates. ### ALPN to agree handshake version -Client and server use [ALPN extension][18] of TLS to agree handshake version. +Client and router use [ALPN extension][18] of TLS to agree handshake version. -Server SHOULD send `smp/1` protocol name and the client should confirm this name in order to use the current protocol version. This is added to allow support of older clients without breaking backward compatibility and to extend or modify handshake syntax. +Router SHOULD send `smp/1` protocol name and the client should confirm this name in order to use the current protocol version. This is added to allow support of older clients without breaking backward compatibility and to extend or modify handshake syntax. -If the client does not confirm this protocol name, the server would fall back to v6 of SMP protocol. +If the client does not confirm this protocol name, the router would fall back to v6 of SMP protocol. ### Transport handshake -Once TLS handshake is complete, client and server will exchange blocks of fixed size (16384 bytes). +Once TLS handshake is complete, client and router will exchange blocks of fixed size (16384 bytes). -The first block sent by the server should be `paddedServerHello` and the client should respond with `paddedClientHello` - these blocks are used to agree SMP protocol version: +The first block sent by the router should be `paddedRouterHello` and the client should respond with `paddedClientHello` - these blocks are used to agree SMP protocol version: ```abnf -paddedServerHello = -serverHello = smpVersionRange sessionIdentifier [serverCert signedServerKey] ignoredPart +paddedRouterHello = +routerHello = smpVersionRange sessionIdentifier [routerCertKey] ignoredPart smpVersionRange = minSmpVersion maxSmpVersion minSmpVersion = smpVersion maxSmpVersion = smpVersion @@ -1226,28 +1506,49 @@ sessionIdentifier = shortString ; unique session identifier derived from transport connection handshake ; it should be included in authorized part of all SMP transmissions sent in this transport connection, ; but it must not be sent as part of the transmission in the current protocol version. -serverCert = originalLength x509encoded -signedServerKey = originalLength x509encoded ; signed by server certificate +routerCertKey = certChain signedRouterKey +certChain = count 1*cert ; 2-4 certificates +cert = originalLength x509encoded +signedRouterKey = originalLength x509encoded ; X25519 key signed by router certificate paddedClientHello = -clientHello = smpVersion [clientKey] ignoredPart +clientHello = smpVersion keyHash [clientKey] proxyRouter optClientService ignoredPart ; chosen SMP protocol version - it must be the maximum supported version -; within the range offered by the server -clientKey = length x509encoded +; within the range offered by the router +keyHash = shortString ; router identity - CA certificate fingerprint +clientKey = length x509encoded ; X25519 public key for session encryption - only present if needed +proxyRouter = %s"T" / %s"F" ; true if connecting client is a proxy router +optClientService = %s"0" / (%s"1" clientService) ; optional service client credentials +clientService = serviceRole serviceCertKey +serviceRole = %s"M" / %s"N" / %s"P" ; Messaging / Notifier / Proxy +serviceCertKey = certChain signedServiceKey +signedServiceKey = originalLength x509encoded ; Ed25519 key signed by service certificate smpVersion = 2*2OCTET ; Word16 version number originalLength = 2*2OCTET +count = 1*1OCTET ignoredPart = *OCTET pad = *OCTET ``` -`signedServerKey` is used to compute a shared secret to authorize client transmission - it is combined with the per-queue key that was used when the queue was created. +`signedRouterKey` is used to compute a shared secret to authorize client transmissions - it is combined with the per-queue key that was used when the queue was created. + +`clientKey` is used only by SMP proxy router when it connects to the destination router to agree shared secret for the additional encryption layer, end user clients do not use this key. + +`proxyRouter` flag (v14+) disables additional transport encryption inside TLS for proxy connections, since proxy router connection already has additional encryption. -`clientKey` is used only by SMP proxy server when it connects to the destination server to agree shared secret for the additional encryption layer, end user clients do not use this key. +`clientService` (v16+) provides long-term service client certificate for high-volume services using SMP router (chat relays, notification routers, high traffic bots). The router responds with a third handshake message containing the assigned service ID: + +```abnf +paddedRouterHandshakeResponse = +routerHandshakeResponse = %s"R" serviceId / %s"E" handshakeError +serviceId = shortString +handshakeError = transportError +``` -`ignoredPart` in handshake allows to add additional parameters in handshake without changing protocol version - the client and servers must ignore any extra bytes within the original block length. +`ignoredPart` in handshake allows to add additional parameters in handshake without changing protocol version - the client and routers must ignore any extra bytes within the original block length. -For TLS transport client should assert that `sessionIdentifier` is equal to `tls-unique` channel binding defined in [RFC 5929][14] (TLS Finished message struct); we pass it in `serverHello` block to allow communication over some other transport protocol (possibly, with another channel binding). +For TLS transport client should assert that `sessionIdentifier` is equal to `tls-unique` channel binding defined in [RFC 5929][14] (TLS Finished message struct); we pass it in `routerHello` block to allow communication over some other transport protocol (possibly, with another channel binding). ### Additional transport privacy @@ -1257,7 +1558,7 @@ For scenarios when meta-data privacy is critical, it is recommended that clients - establish a separate connection for each SMP queue, - send noise traffic (using PING command). -In addition to that, the servers can be deployed as Tor onion services. +In addition to that, the routers can be deployed as Tor onion services. [1]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack [2]: https://en.wikipedia.org/wiki/End-to-end_encryption diff --git a/protocol/xftp.md b/protocol/xftp.md index bd1d9e664..4180e652b 100644 --- a/protocol/xftp.md +++ b/protocol/xftp.md @@ -1,4 +1,4 @@ -Version 2, 2024-06-22 +Version 3, 2025-01-24 # SimpleX File Transfer Protocol @@ -11,12 +11,12 @@ Version 2, 2024-06-22 - [XFTP procedure](#xftp-procedure) - [File description](#file-description) - [URIs syntax](#uris-syntax) - - [XFTP server URI](#xftp-server-uri) + - [XFTP router URI](#xftp-router-uri) - [File description URI](#file-description-URI) - [XFTP qualities and features](#xftp-qualities-and-features) - [Cryptographic algorithms](#cryptographic-algorithms) -- [File chunk IDs](#file-chunk-ids) -- [Server security requirements](#server-security-requirements) +- [Data packet IDs](#data-packet-ids) +- [Router security requirements](#router-security-requirements) - [Transport protocol](#transport-protocol) - [TLS ALPN](#tls-alpn) - [Connection handshake](#connection-handshake) @@ -26,13 +26,14 @@ Version 2, 2024-06-22 - [Command authentication](#command-authentication) - [Keep-alive command](#keep-alive-command) - [File sender commands](#file-sender-commands) - - [Register new file chunk](#register-new-file-chunk) - - [Add file chunk recipients](#add-file-chunk-recipients) - - [Upload file chunk](#upload-file-chunk) - - [Delete file chunk](#delete-file-chunk) + - [Register new data packet](#register-new-data-packet) + - [Add data packet recipients](#add-data-packet-recipients) + - [Upload data packet](#upload-data-packet) + - [Delete data packet](#delete-data-packet) - [File recipient commands](#file-recipient-commands) - - [Download file chunk](#download-file-chunk) - - [Acknowledge file chunk download](#acknowledge-file-chunk-download) + - [Download data packet](#download-data-packet) + - [Acknowledge data packet download](#acknowledge-data-packet-download) + - [Error responses](#error-responses) - [Threat model](#threat-model) ## Abstract @@ -45,23 +46,31 @@ It is designed as a application level protocol to solve the problem of secure an ## Introduction -The objective of SimpleX File Transfer Protocol (XFTP) is to facilitate the secure and private unidirectional transfer of files from senders to recipients via persistent file chunks stored by the xftp server. +The objective of SimpleX File Transfer Protocol (XFTP) is to facilitate the secure and private unidirectional transfer of files from senders to recipients via persistent data packets stored by the xftp router. XFTP is implemented as an application level protocol on top of HTTP2 and TLS. -The protocol describes the set of commands that senders and recipients can send to XFTP servers to create, upload, download and delete file chunks of several pre-defined sizes. XFTP servers SHOULD support chunks of 4 sizes: 64KB, 256KB, 1MB and 4MB (1KB = 1024 bytes, 1MB = 1024KB). +This document describes XFTP protocol version 3. The version history: -The protocol is designed with the focus on meta-data privacy and security. While using TLS, the protocol does not rely on TLS security by using additional encryption to achieve that there are no identifiers or ciphertext in common in received and sent server traffic, frustrating traffic correlation even if TLS is compromised. +- v1: initial version +- v2: authenticated commands - added basic auth support for commands +- v3: blocked files - added BLOCKED error type for policy violations -XFTP does not use any form of participants' identities. It relies on out-of-band passing of "file description" - a human-readable YAML document with the list of file chunk locations, hashes and necessary cryptographic keys. +The protocol describes the set of commands that senders and recipients can send to XFTP routers to create, upload, download and delete data packets of several pre-defined sizes. XFTP routers SHOULD support packets of 4 sizes: 64KB, 256KB, 1MB and 4MB (1KB = 1024 bytes, 1MB = 1024KB). + +The protocol is designed with the focus on meta-data privacy and security. While using TLS, the protocol does not rely on TLS security by using additional encryption to achieve that there are no identifiers or ciphertext in common in received and sent router traffic, frustrating traffic correlation even if TLS is compromised. + +XFTP does not use any form of participants' identities. It relies on out-of-band passing of "file description" - a human-readable YAML document with the list of data packet locations, hashes and necessary cryptographic keys. + +> **Note:** While this protocol was originally designed for file transfer, it handles generic addressed data packets. File-specific semantics (splitting files into packets, assembly, naming) are application-level concerns defined in the [agent protocol](./agent-protocol.md). ## XFTP Model -The XFTP model has three communication participants: the recipient, the file server (XFTP server) that is chosen and, possibly, controlled by the sender, and the sender. +The XFTP model has three communication participants: the recipient, the XFTP router that is chosen and, possibly, controlled by the sender, and the sender. -XFTP server allows uploading fixed size file chunks, with or without basic authentication. The same party that can be the sender of one file chunk can be the recipient of another, without exposing it to the server. +XFTP router allows uploading fixed size data packets, with or without basic authentication. The same party that can be the sender of one data packet can be the recipient of another, without exposing it to the router. -Each file chunk allows multiple recipients, each recipient can download the same chunk multiple times. It allows depending on the threat model use the same recipient credentials for multiple parties, thus reducing server ability to understand the number of intended recipients (but server can still track IP addresses to determine it), or use one unique set of credentials for each recipient, frustrating traffic correlation on the assumption of compromised TLS. In the latter case, senders can create a larger number of recipient credentials to hide the actual number of intended recipients from the servers (which is what SimpleX clients do). +Each data packet allows multiple recipients, each recipient can download the same packet multiple times. It allows depending on the threat model use the same recipient credentials for multiple parties, thus reducing router ability to understand the number of intended recipients (but router can still track IP addresses to determine it), or use one unique set of credentials for each recipient, frustrating traffic correlation on the assumption of compromised TLS. In the latter case, senders can create a larger number of recipient credentials to hide the actual number of intended recipients from the routers (which is what SimpleX clients do). ``` Sender Internet XFTP relays Internet Recipient @@ -69,7 +78,7 @@ Each file chunk allows multiple recipients, each recipient can download the same | | | | | | (can be self-hosted) | | | | +---------+ | | - chunk 1 ----- HTTP2 over TLS ------ | XFTP | ---- HTTP2 / TLS ----- chunk 1 + packet 1 ----- HTTP2 over TLS ------ | XFTP | ---- HTTP2 / TLS ----- packet 1 |---> SimpleX File Transfer Protocol (XFTP) --> | Relay | ---> XFTP ------------->| | --------------------------- +---------+ ---------------------- | | | | | | | @@ -83,21 +92,21 @@ file ---> | XFTP | ------> XFTP ----> | Relay | ---> | | | +---------+ | | | | ------- HTTP2 / TLS ------- | XFTP | ---- HTTP2 / TLS ---- | |-------------> XFTP ----> | Relay | ---> XFTP ------------->| - chunk N --------------------------- +---------+ --------------------- chunk N - | | (store file chunks) | | + packet N --------------------------- +---------+ --------------------- packet N + | | (store data packets) | | | | | | | | | | ``` -When sender client uploads a file chunk, it has to register it first with one sender ID and multiple recipient IDs, and one random unique key per ID to authenticate sender and recipients, and also provide its size and hash that will be validated when chunk is uploaded. +When sender client uploads a data packet, it has to register it first with one sender ID and multiple recipient IDs, and one random unique key per ID to authenticate sender and recipients, and also provide its size and hash that will be validated when packet is uploaded. -To send the actual file, the sender client MUST pad it and encrypt it with a random symmetric key and distribute chunks of fixed sized across multiple XFTP servers. Information about chunk locations, keys, hashes and required keys is passed to the recipients as "[file description](#file-description)" out-of-band. +To send the actual file, the sender client MUST pad it and encrypt it with a random symmetric key and distribute packets of fixed sized across multiple XFTP routers. Information about packet locations, keys, hashes and required keys is passed to the recipients as "[file description](#file-description)" out-of-band. -Creating, uploading, downloading and deleting file chunks requires sending commands to the XFTP server - they are described in detail in [XFTP commands](#xftp-commands) section. +Creating, uploading, downloading and deleting data packets requires sending commands to the XFTP router - they are described in detail in [XFTP commands](#xftp-commands) section. ## Persistence model -Server stores file chunk records in memory, with optional adding to append-only log, to allow restoring them on server restart. File chunk bodies can be stored as files or as objects in any object store (e.g. S3). +Router stores data packet records in memory, with optional adding to append-only log, to allow restoring them on router restart. Data packet bodies can be stored as files or as objects in any object store (e.g. S3). ## XFTP procedure @@ -107,28 +116,28 @@ To send the file, the sender will: 1) Prepare file - compute its SHA512 digest. - - prepend header with the name and pad the file to match the whole number of chunks in size. It is RECOMMENDED to use 2 of 4 allowed chunk sizes, to balance upload size and metadata privacy. + - prepend header with the name and pad the file to match the whole number of packets in size. It is RECOMMENDED to use 2 of 4 allowed packet sizes, to balance upload size and metadata privacy. - encrypt it with a randomly chosen symmetric key and IV (e.g., using NaCL secret_box). - - split into allowed size chunks. + - split into allowed size packets. - generate per-recipient keys. It is recommended that the sending client generates more per-recipient keys than the actual number of recipients, rounding up to a power of 2, to conceal the actual number of intended recipients. -2) Upload file chunks - - register each chunk record with randomly chosen one or more (for redundancy) XFTP server(s). +2) Upload data packets + - register each packet record with randomly chosen one or more (for redundancy) XFTP router(s). - optionally request additional recipient IDs, if required number of recipient keys didn't fit into register request. - - upload each chunk to chosen server(s). + - upload each packet to chosen router(s). 3) Prepare file descriptions, one per recipient. -The sending client combines addresses of all chunks and other information into "file description", different for each file recipient, that will include: +The sending client combines addresses of all packets and other information into "file description", different for each file recipient, that will include: - an encryption key used to encrypt/decrypt the full file (the same for all recipients). - file SHA512 digest to validate download. -- list of chunk descriptions; information for each chunk: - - private Ed25519 key to sign commands for file transfer server. - - chunk address (server host and chunk ID). - - chunk sha512 digest. +- list of packet descriptions; information for each packet: + - private Ed25519 key to sign commands for file transfer router. + - packet address (router host and packet ID). + - packet sha256 digest. -To reduce the size of file description, chunks are grouped by the server host. +To reduce the size of file description, packets are grouped by the router host. 4) Send file description(s) to the recipient(s) out-of-band, via pre-existing secure and authenticated channel. E.g., SimpleX clients send it as messages via SMP protocol, but it can be done via any other channel. @@ -138,16 +147,16 @@ To reduce the size of file description, chunks are grouped by the server host. Having received the description, the recipient will: -1) Download all chunks. +1) Download all packets. -The receiving client can fall back to secondary servers, if necessary: -- if the server is not available. -- if the chunk is not present on the server (ERR AUTH response). -- if the hash of the downloaded file chunk does not match the description. +The receiving client can fall back to secondary routers, if necessary: +- if the router is not available. +- if the packet is not present on the router (ERR AUTH response). +- if the hash of the downloaded data packet does not match the description. -Optionally recipient can acknowledge file chunk reception to delete file ID from server for this recipient. +Optionally recipient can acknowledge data packet reception to delete file ID from router for this recipient. -2) Combine the chunks into a file. +2) Combine the packets into a file. 3) Decrypt the file using the key in file description. @@ -163,35 +172,35 @@ Optionally recipient can acknowledge file chunk reception to delete file ID from It includes these fields: - `party` - "sender" or "recipient". Sender's file description is required to delete the file. -- `size` - padded file size equal to total size of all chunks, see `fileSize` syntax below. +- `size` - padded file size equal to total size of all packets, see `fileSize` syntax below. - `digest` - SHA512 hash of encrypted file, base64url encoded string. - `key` - symmetric encryption key to decrypt the file, base64url encoded string. - `nonce` - nonce to decrypt the file, base64url encoded string. -- `chunkSize` - default chunk size, see `fileSize` syntax below. -- `replicas` - the array of file chunk replicas descriptions. +- `chunkSize` - default packet size, see `fileSize` syntax below. +- `replicas` - the array of data packet replicas descriptions. - `redirect` - optional property for redirect information indicating that the file is itself a description to another file, allowing to use file description as a short URI. Each replica description is an object with 2 fields: -- `chunks` - and array of chunk replica descriptions stored on one server. -- `server` - [server address](#xftp-server-uri) where the chunks can be downloaded from. +- `chunks` - an array of packet replica descriptions stored on one server. +- `server` - [router address](#xftp-router-uri) where the packets can be downloaded from. -Each server replica description is a string with this syntax: +Each router replica description is a string with this syntax: ```abnf -chunkReplica = chunkNo ":" replicaId ":" replicaKey [":" chunkDigest [":" chunkSize]] -chunkNo = 1*DIGIT - ; a sequential 1-based chunk number in the original file. +packetReplica = packetNo ":" replicaId ":" replicaKey [":" packetDigest [":" packetSize]] +packetNo = 1*DIGIT + ; a sequential 1-based packet number in the original file. replicaId = base64url - ; server-assigned random chunk replica ID. + ; router-assigned random packet replica ID. replicaKey = base64url - ; sender-generated random key to receive (or to delete, in case of sender's file description) the chunk replica. -chunkDigest = base64url - ; chunk digest that MUST be specified for the first replica of each chunk, + ; sender-generated random key to receive (or to delete, in case of sender's file description) the packet replica. +packetDigest = base64url + ; packet digest that MUST be specified for the first replica of each packet, ; and SHOULD be omitted (or be the same) on the subsequent replicas -chunkSize = fileSize +packetSize = fileSize fileSize = sizeInBytes / sizeInUnits - ; chunk size SHOULD only be specified on the first replica and only if it is different from default chunk size + ; packet size SHOULD only be specified on the first replica and only if it is different from default packet size sizeInBytes = 1*DIGIT sizeInUnits = 1*DIGIT sizeUnit sizeUnit = %s"kb" / %s"mb" / %s"gb" @@ -204,28 +213,28 @@ Optional redirect information has two fields: ## URIs syntax -### XFTP server URI +### XFTP router URI -The XFTP server address is a URI with the following syntax: +The XFTP router address is a URI with the following syntax: ```abnf -xftpServerURI = %s"xftp://" xftpServer -xftpServer = serverIdentity [":" basicAuth] "@" srvHost [":" port] +xftpRouterURI = %s"xftp://" xftpRouter +xftpRouter = routerIdentity [":" basicAuth] "@" srvHost [":" port] srvHost = ; RFC1123, RFC5891 port = 1*DIGIT -serverIdentity = base64url +routerIdentity = base64url basicAuth = base64url ``` ### File description URI -This file description URI can be generated by the client application to share a small file description as a QR code or as a link. Practically, to be able to scan a QR code it should be under 1000 characters, so only file descriptions with 1-2 chunks can be used in this case. This is supported with `redirect` property when file description leads to a file which in itself is a larger file description to another file - akin to URL shortener. +This file description URI can be generated by the client application to share a small file description as a QR code or as a link. Practically, to be able to scan a QR code it should be under 1000 characters, so only file descriptions with 1-2 packets can be used in this case. This is supported with `redirect` property when file description leads to a file which in itself is a larger file description to another file - akin to URL shortener. File description URI syntax: ```abnf fileDescriptionURI = serviceScheme "/file" "#/?desc=" description [ "&data=" userData ] -serviceScheme = (%s"https://" clientAppServer) | %s"simplex:" +serviceScheme = (%s"https://" clientAppServer) / %s"simplex:" clientAppServer = hostname [ ":" port ] ; client app server, e.g. simplex.chat description = @@ -240,50 +249,50 @@ clientAppServer is not a server the client connects to - it is a server that sho XFTP stands for SimpleX File Transfer Protocol. Its design is based on the same ideas and has some of the qualities of SimpleX Messaging Protocol: -- recipient cannot see sender's IP address, as the file fragments (chunks) are temporarily stored on multiple XFTP relays. +- recipient cannot see sender's IP address, as the file fragments (packets) are temporarily stored on multiple XFTP relays. - file can be sent asynchronously, without requiring the sender to be online for file to be received. - there is no network of peers that can observe this transfer - sender chooses which XFTP relays to use, and can self-host their own. -- XFTP relays do not have any file metadata - they only see individual chunks, with access to each chunk authorized with anonymous credentials (using Edwards curve cryptographic signature) that are random per chunk. -- chunks have one of the sizes allowed by the servers - 64KB, 256KB, 1MB and 4MB chunks, so sending a large file looks indistinguishable from sending many small files to XFTP server. If the same transport connection is reused, server would only know that chunks are sent by the same user. -- each chunk can be downloaded by multiple recipients, but each recipient uses their own key and chunk ID to authorize access, and the chunk is encrypted by a different key agreed via ephemeral DH keys (NaCl crypto_box (SalsaX20Poly1305 authenticated encryption scheme ) with shared secret derived from Curve25519 key exchange) on the way from the server to each recipient. XFTP protocol as a result has the same quality as SMP protocol - there are no identifiers and ciphertext in common between sent and received traffic inside TLS connection, so even if TLS is compromised, it complicates traffic correlation attacks. -- XFTP protocol supports redundancy - each file chunk can be sent via multiple relays, and the recipient can choose the one that is available. Current implementation of XFTP protocol in SimpleX Chat does not support redundancy though. +- XFTP relays do not have any file metadata - they only see individual packets, with access to each packet authorized with anonymous credentials (using Edwards curve cryptographic signature) that are random per packet. +- packets have one of the sizes allowed by the routers - 64KB, 256KB, 1MB and 4MB packets, so sending a large file looks indistinguishable from sending many small files to XFTP router. If the same transport connection is reused, router would only know that packets are sent by the same user. +- each packet can be downloaded by multiple recipients, but each recipient uses their own key and packet ID to authorize access, and the packet is encrypted by a different key agreed via ephemeral DH keys (NaCl crypto_box (SalsaX20Poly1305 authenticated encryption scheme ) with shared secret derived from Curve25519 key exchange) on the way from the router to each recipient. XFTP protocol as a result has the same quality as SMP protocol - there are no identifiers and ciphertext in common between sent and received traffic inside TLS connection, so even if TLS is compromised, it complicates traffic correlation attacks. +- XFTP protocol supports redundancy - each data packet can be sent via multiple relays, and the recipient can choose the one that is available. Current implementation of XFTP protocol in SimpleX Chat does not support redundancy though. - the file as a whole is encrypted with a random symmetric key using NaCl secret_box. ## Cryptographic algorithms Clients must cryptographically authorize XFTP commands, see [Command authentication](#command-authentication). -To authorize/verify transmissions clients and servers MUST use either signature algorithm Ed25519 algorithm defined in RFC8709 or using deniable authentication scheme based on NaCL crypto_box (see Simplex Messaging Protocol). +To authorize/verify transmissions clients and routers MUST use either signature algorithm Ed25519 algorithm defined in RFC8709 or using deniable authentication scheme based on NaCL crypto_box (see Simplex Messaging Protocol). -To encrypt/decrypt file chunk bodies delivered to the recipients, servers/clients MUST use NaCL crypto_box. +To encrypt/decrypt data packet bodies delivered to the recipients, routers/clients MUST use NaCL crypto_box. -Clients MUST encrypt file chunk bodies sent via XFTP servers using use NaCL crypto_box. +Clients MUST encrypt data packet bodies sent via XFTP routers using use NaCL crypto_box. -## File chunk IDs +## Data packet IDs -XFTP servers MUST generate a separate new set of IDs for each new chunk - for the sender (that uploads the chunk) and for each intended recipient. It is REQUIRED that: +XFTP routers MUST generate a separate new set of IDs for each new packet - for the sender (that uploads the packet) and for each intended recipient. It is REQUIRED that: -- These IDs are different and unique within the server. +- These IDs are different and unique within the router. - Based on random bytes generated with cryptographically strong pseudo-random number generator. -## Server security requirements +## Router security requirements -XFTP server implementations MUST NOT create, store or send to any other servers: +XFTP router implementations MUST NOT create, store or send to any other routers: - Logs of the client commands and transport connections in the production environment. - History of retrieved files. -- Snapshots of the database they use to store file chunks (instead clients can manage redundancy by creating chunk replicas using more than one XFTP server). In-memory persistence is recommended for file chunks records. +- Snapshots of the database they use to store data packets (instead clients can manage redundancy by creating packet replicas using more than one XFTP router). In-memory persistence is recommended for data packets records. -- Any other information that may compromise privacy or [forward secrecy][4] of communication between clients using XFTP servers. +- Any other information that may compromise privacy or [forward secrecy][4] of communication between clients using XFTP routers. ## Transport protocol -- binary-encoded commands sent as fixed-size padded block in the body of HTTP2 POST request, similar to SMP and notifications server protocol transmission encodings. +- binary-encoded commands sent as fixed-size padded block in the body of HTTP2 POST request, similar to SMP and notifications router protocol transmission encodings. - HTTP2 POST with a fixed size padded block body for file upload and download. -Block size - 4096 bytes (it would fit ~120 Ed25519 recipient keys). +Block size - 16384 bytes (it would fit ~350 Ed25519 recipient keys). The reasons to use HTTP2: @@ -299,40 +308,41 @@ The reason not to use URI segments / HTTP verbs / REST semantics is to have cons ### ALPN to agree handshake version -Client and server use [ALPN extension][18] of TLS to agree handshake version. +Client and router use [ALPN extension][18] of TLS to agree handshake version. -Server SHOULD send `xftp/1` protocol name and the client should confirm this name in order to use the current protocol version. This is added to allow support of older clients without breaking backward compatibility and to extend or modify handshake syntax. +Router SHOULD send `xftp/1` protocol name and the client should confirm this name in order to use the current protocol version. This is added to allow support of older clients without breaking backward compatibility and to extend or modify handshake syntax. -If the client does not confirm this protocol name, the server would fall back to v1 of XFTP protocol. +If the client does not confirm this protocol name, the router would fall back to v1 of XFTP protocol. ### Transport handshake -When a client and a server agree on handshake version using ALPN extension, they should proceed with XFTP handshake. +When a client and a router agree on handshake version using ALPN extension, they should proceed with XFTP handshake. -As with SMP, a client doesn't reveal its version range to avoid version fingerprinting. Unlike SMP, XFTP runs a HTTP2 protocol over TLS and the server can't just send its handshake right away. So a session handshake is driven by client-sent requests: +As with SMP, a client doesn't reveal its version range to avoid version fingerprinting. Unlike SMP, XFTP runs a HTTP2 protocol over TLS and the router can't just send its handshake right away. So a session handshake is driven by client-sent requests: -1. To pass initiative to the server, the client sends a request with empty body. -2. Server responds with its `paddedServerHello` block. +1. To pass initiative to the router, the client sends a request with empty body. +2. Router responds with its `paddedRouterHello` block. 3. Clients sends a request containing `paddedClientHello` block, -4. Server sends an empty response, finalizing the handshake. +4. Router sends an empty response, finalizing the handshake. -Once TLS handshake is complete, client and server will exchange blocks of fixed size (16384 bytes). +Once TLS handshake is complete, client and router will exchange blocks of fixed size (16384 bytes). ```abnf -paddedServerHello = -serverHello = xftpVersionRange sessionIdentifier serverCert signedServerKey ignoredPart +paddedRouterHello = +routerHello = xftpVersionRange sessionIdentifier routerCerts signedRouterKey ignoredPart xftpVersionRange = minXftpVersion maxXftpVersion minXftpVersion = xftpVersion maxXftpVersion = xftpVersion sessionIdentifier = shortString ; unique session identifier derived from transport connection handshake -serverCert = originalLength -signedServerKey = originalLength ; signed by server certificate +routerCerts = length 1*routerCert ; NonEmpty list of certificates in chain +routerCert = originalLength +signedRouterKey = originalLength ; signed by router certificate paddedClientHello = clientHello = xftpVersion keyHash ignoredPart ; chosen XFTP protocol version - must be the maximum supported version -; within the range offered by the server +; within the range offered by the router xftpVersion = 2*2OCTET ; Word16 version number keyHash = shortString @@ -342,47 +352,47 @@ originalLength = 2*2OCTET ignoredPart = *OCTET ``` -In XFTP v2 the handshake is only used for version negotiation, but `serverCert` and `signedServerKey` must be validated by the client. +In XFTP v2 the handshake is only used for version negotiation, but `routerCert` and `signedRouterKey` must be validated by the client. -`keyHash` is the CA fingerprint used by client to validate TLS certificate chain and is checked by a server against its own key. +`keyHash` is the CA fingerprint used by client to validate TLS certificate chain and is checked by a router against its own key. -`ignoredPart` in handshake allows to add additional parameters in handshake without changing protocol version - the client and servers must ignore any extra bytes within the original block length. +`ignoredPart` in handshake allows to add additional parameters in handshake without changing protocol version - the client and routers must ignore any extra bytes within the original block length. -For TLS transport client should assert that `sessionIdentifier` is equal to `tls-unique` channel binding defined in [RFC 5929][14] (TLS Finished message struct); we pass it in `serverHello` block to allow communication over some other transport protocol (possibly, with another channel binding). +For TLS transport client should assert that `sessionIdentifier` is equal to `tls-unique` channel binding defined in [RFC 5929][14] (TLS Finished message struct); we pass it in `routerHello` block to allow communication over some other transport protocol (possibly, with another channel binding). ### Requests and responses - File sender: - - create file chunk record. + - create data packet record. - Parameters: - Ed25519 key for subsequent sender commands and Ed25519 keys for commands of each recipient. - - chunk size. + - packet size. - Response: - - chunk ID for the sender and different IDs for all recipients. - - add recipients to file chunk + - packet ID for the sender and different IDs for all recipients. + - add recipients to data packet - Parameters: - - sender's chunk ID + - sender's packet ID - Ed25519 keys for commands of each recipient. - Response: - - chunk IDs for new recipients. - - upload file chunk. - - delete file chunk (invalidates all recipient IDs). + - packet IDs for new recipients. + - upload data packet. + - delete data packet (invalidates all recipient IDs). - File recipient: - - download file chunk: - - chunk ID - - DH key for additional encryption of the chunk. - - command should be signed with the key passed by the sender when creating chunk record. - - delete file chunk ID (only for one recipient): signed with the same key. + - download data packet: + - packet ID + - DH key for additional encryption of the packet. + - command should be signed with the key passed by the sender when creating packet record. + - delete data packet ID (only for one recipient): signed with the same key. ## XFTP commands Commands syntax below is provided using ABNF with case-sensitive strings extension. ```abnf -xftpCommand = ping / senderCommand / recipientCmd / serverMsg +xftpCommand = ping / senderCommand / recipientCmd / routerMsg senderCommand = register / add / put / delete recipientCmd = get / ack -serverMsg = pong / sndIds / rcvIds / ok / file +routerMsg = pong / sndIds / rcvIds / ok / file / error ``` The syntax of specific commands and responses is defined below. @@ -393,11 +403,11 @@ Commands are made via HTTP2 requests, responses to commands are correlated as HT ### Command authentication -XFTP servers must authenticate all transmissions (excluding `ping`) by verifying the client signatures. Command signature should be generated by applying the algorithm specified for the file to the `signed` block of the transmission, using the key associated with the file chunk ID (recipient's or sender's depending on which file chunk ID is used). +XFTP routers must authenticate all transmissions (excluding `ping`) by verifying the client signatures. Command signature should be generated by applying the algorithm specified for the file to the `signed` block of the transmission, using the key associated with the data packet ID (recipient's or sender's depending on which data packet ID is used). ### Keep-alive command -To keep the transport connection alive and to generate noise traffic the clients should use `ping` command to which the server responds with `pong` response. This command should be sent unsigned and without file chunk ID. +To keep the transport connection alive and to generate noise traffic the clients should use `ping` command to which the router responds with `pong` response. This command should be sent unsigned and without data packet ID. ```abnf ping = %s"PING" @@ -405,21 +415,19 @@ ping = %s"PING" This command is always sent unsigned. - data FileResponse = ... | FRPong | ... - ```abnf pong = %s"PONG" ``` ### File sender commands -Sending any of the commands in this section (other than `register`, that is sent without file chunk ID) is only allowed with sender's ID. +Sending any of the commands in this section (other than `register`, that is sent without data packet ID) is only allowed with sender's ID. The `register` command must be signed (using `sndKey` included in `fileInfo` for verification) but must NOT include a data packet ID. -#### Register new file chunk +#### Register new data packet -This command is sent by the sender to the XFTP server to register a new file chunk. +This command is sent by the sender to the XFTP router to register a new data packet. -Servers SHOULD support basic auth with this command, to allow only server owners and trusted users to create file chunks on the servers. +Routers SHOULD support basic auth with this command, to allow only router owners and trusted users to create data packets on the routers. The syntax is: @@ -427,7 +435,7 @@ The syntax is: register = %s"FNEW " fileInfo rcvPublicAuthKeys basicAuth fileInfo = sndKey size digest sndKey = length x509encoded -size = 1*DIGIT +size = 4*4 OCTET ; Word32 big-endian digest = length *OCTET rcvPublicAuthKeys = length 1*rcvPublicAuthKey rcvPublicAuthKey = length x509encoded @@ -438,7 +446,7 @@ x509encoded = length = 1*1 OCTET ``` -If the file chunk is registered successfully, the server must send `sndIds` response with the sender's and recipients' file chunk IDs: +If the data packet is registered successfully, the router must send `sndIds` response with the sender's and recipients' data packet IDs: ```abnf sndIds = %s"SIDS " senderId recipientIds @@ -447,9 +455,9 @@ recipientIds = length 1*recipientId recipientId = length *OCTET ``` -#### Add file chunk recipients +#### Add data packet recipients -This command is sent by the sender to the XFTP server to add additional recipient keys to the file chunk record, in case number of keys requested by client didn't fit into `register` command. The syntax is: +This command is sent by the sender to the XFTP router to add additional recipient keys to the data packet record, in case number of keys requested by client didn't fit into `register` command. The syntax is: ```abnf add = %s"FADD " rcvPublicAuthKeys @@ -457,7 +465,7 @@ rcvPublicAuthKeys = length 1*rcvPublicAuthKey rcvPublicAuthKey = length x509encoded ``` -If additional keys were added successfully, the server must send `rcvIds` response with the added recipients' file chunk IDs: +If additional keys were added successfully, the router must send `rcvIds` response with the added recipients' data packet IDs: ```abnf rcvIds = %s"RIDS " recipientIds @@ -465,66 +473,100 @@ recipientIds = length 1*recipientId recipientId = length *OCTET ``` -#### Upload file chunk +#### Upload data packet -This command is sent by the sender to the XFTP server to upload file chunk body to server. The syntax is: +This command is sent by the sender to the XFTP router to upload data packet body to router. The syntax is: ```abnf put = %s"FPUT" ``` -Chunk body is streamed via HTTP2 request. +Packet body is streamed via HTTP2 request. -If file chunk body was successfully received, the server must send `ok` response. +If data packet body was successfully received, the router must send `ok` response. ```abnf ok = %s"OK" ``` -#### Delete file chunk +#### Delete data packet -This command is sent by the sender to the XFTP server to delete file chunk from the server. The syntax is: +This command is sent by the sender to the XFTP router to delete data packet from the router. The syntax is: ```abnf delete = %s"FDEL" ``` -Server should delete file chunk record, invalidating all recipient IDs, and delete file body from file storage. If file chunk was successfully deleted, the server must send `ok` response. +Router should delete data packet record, invalidating all recipient IDs, and delete file body from file storage. If data packet was successfully deleted, the router must send `ok` response. ### File recipient commands Sending any of the commands in this section is only allowed with recipient's ID. -#### Download file chunk +#### Download data packet -This command is sent by the recipient to the XFTP server to download file chunk body from the server. The syntax is: +This command is sent by the recipient to the XFTP router to download data packet body from the router. The syntax is: ```abnf get = %s"FGET " rDhKey rDhKey = length x509encoded ``` -If requested file is successfully located, the server must send `file` response. File chunk body is sent as HTTP2 response body. +If requested file is successfully located, the router must send `file` response. Data packet body is sent as HTTP2 response body. ```abnf file = %s"FILE " sDhKey cbNonce sDhKey = length x509encoded -cbNonce = +cbNonce = 24*24 OCTET ; NaCl crypto_box nonce ``` -Chunk is additionally encrypted on the way from the server to the recipient using a key agreed via ephemeral DH keys `rDhKey` and `sDhKey`, so there is no ciphertext in common between sent and received traffic inside TLS connection, in order to complicate traffic correlation attacks, if TLS is compromised. +Packet is additionally encrypted on the way from the router to the recipient using a key agreed via ephemeral DH keys `rDhKey` and `sDhKey`, so there is no ciphertext in common between sent and received traffic inside TLS connection, in order to complicate traffic correlation attacks, if TLS is compromised. -#### Acknowledge file chunk download +#### Acknowledge data packet download -This command is sent by the recipient to the XFTP server to acknowledge file reception, deleting file ID from server for this recipient. The syntax is: +This command is sent by the recipient to the XFTP router to acknowledge file reception, deleting file ID from router for this recipient. The syntax is: ```abnf ack = %s"FACK" ``` -If file recipient ID is successfully deleted, the server must send `ok` response. +If file recipient ID is successfully deleted, the router must send `ok` response. + +In current implementation of XFTP protocol in SimpleX Chat clients don't use FACK command. Files are automatically expired on routers after configured time interval. + +### Error responses + +The router responds with `ERR` followed by the error type: + +```abnf +error = %s"ERR " errorType +errorType = %s"BLOCK" / %s"SESSION" / %s"HANDSHAKE" / + %s"CMD" SP cmdError / %s"AUTH" / %s"BLOCKED" SP blockingInfo / + %s"SIZE" / %s"QUOTA" / %s"DIGEST" / %s"CRYPTO" / + %s"NO_FILE" / %s"HAS_FILE" / %s"FILE_IO" / + %s"TIMEOUT" / %s"INTERNAL" +cmdError = %s"UNKNOWN" / %s"SYNTAX" / %s"PROHIBITED" / %s"NO_AUTH" / %s"HAS_AUTH" / %s"NO_ENTITY" +blockingInfo = %s"reason=" blockingReason ["," %s"notice=" jsonNotice] +blockingReason = %s"spam" / %s"content" +jsonNotice = *OCTET ; JSON-encoded notice object +``` -In current implementation of XFTP protocol in SimpleX Chat clients don't use FACK command. Files are automatically expired on servers after configured time interval. +Error types: +- `BLOCK` - incorrect block format, encoding or signature size. +- `SESSION` - incorrect session ID (TLS Finished message / tls-unique binding). +- `HANDSHAKE` - incorrect handshake command. +- `CMD` - command syntax errors (UNKNOWN, SYNTAX, PROHIBITED, NO_AUTH, HAS_AUTH, NO_ENTITY). +- `AUTH` - command authorization error - bad signature or non-existing data packet. +- `BLOCKED` - data packet was blocked due to policy violation (added in v3). Contains blocking reason and optional notice. +- `SIZE` - incorrect file size. +- `QUOTA` - storage quota exceeded. +- `DIGEST` - incorrect file digest. +- `CRYPTO` - file encryption/decryption failed. +- `NO_FILE` - no expected file body in request/response or no file on the router. +- `HAS_FILE` - unexpected file body. +- `FILE_IO` - file IO error. +- `TIMEOUT` - file sending or receiving timeout. +- `INTERNAL` - internal router error. ## Threat model @@ -533,7 +575,7 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC - A user protects their local database and key material. - The user's application is authentic, and no local malware is running. - The cryptographic primitives in use are not broken. - - A user's choice of servers is not directly tied to their identity or otherwise represents distinguishing information about the user. + - A user's choice of routers is not directly tied to their identity or otherwise represents distinguishing information about the user. #### A passive adversary able to monitor the traffic of one user @@ -541,7 +583,7 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC - identify that and when a user is sending files over XFTP protocol. - - determine which servers the user sends/receives files to/from. + - determine which routers the user sends/receives files to/from. - observe how much traffic is being sent, and make guesses as to its purpose. @@ -553,11 +595,11 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC *can:* - - learn which XFTP servers are used to send and receive files for which users. + - learn which XFTP routers are used to send and receive files for which users. - learn when files are sent and received. - - perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the servers. + - perform traffic correlation attacks against senders and recipients and correlate senders and recipients within the monitored set, frustrated by the number of users on the routers. - observe how much traffic is being sent, and make guesses as to its purpose. @@ -567,31 +609,31 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC - perform traffic correlation attacks. -#### XFTP server +#### XFTP router *can:* - learn when file senders and recipients are online. -- know how many file chunks and chunk sizes are sent via the server. +- know how many data packets and packet sizes are sent via the router. -- perform the correlation of the file chunks as belonging to one file via either a re-used transport connection, user's IP address, or connection timing regularities. +- perform the correlation of the data packets as belonging to one file via either a re-used transport connection, user's IP address, or connection timing regularities. - learn file senders' and recipients' IP addresses, and infer information (e.g. employer) based on the IP addresses, as long as Tor is not used. -- delete file chunks, preventing file delivery, as long as redundant delivery is not used. +- delete data packets, preventing file delivery, as long as redundant delivery is not used. -- lie about the state of a file chunk to the recipient and/or to the sender (e.g. deleted when it is not). +- lie about the state of a data packet to the recipient and/or to the sender (e.g. deleted when it is not). - refuse deleting the file when instructed by the sender. *cannot:* -- undetectably corrupt file chunks. +- undetectably corrupt data packets. - learn the contents, name or the exact size of sent files. -- learn approximate size of sent files, as long as more than one server is used to send file chunks. +- learn approximate size of sent files, as long as more than one router is used to send data packets. - compromise the users' end-to-end encryption of files with an active attack. @@ -603,7 +645,7 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC - receive all files sent and received by Alice that did not expire yet, as long as information about these files was not removed from the database. -- prevent Alice's contacts from receiving the files she sent by deleting all or some of the file chunks from XFTP servers. +- prevent Alice's contacts from receiving the files she sent by deleting all or some of the data packets from XFTP routers. #### A user's contact @@ -625,10 +667,10 @@ In current implementation of XFTP protocol in SimpleX Chat clients don't use FAC *can:* -- Denial of Service XFTP servers. +- Denial of Service XFTP routers. *cannot:* - send files to a user who they are not connected with. -- enumerate file chunks on an XFTP server. +- enumerate data packets on an XFTP router. diff --git a/protocol/xrcp.md b/protocol/xrcp.md index c8042f858..1b2c320a7 100644 --- a/protocol/xrcp.md +++ b/protocol/xrcp.md @@ -10,7 +10,7 @@ Version 1, 2024-06-22 - [Session invitation](#session-invitation) - [Establishing TLS connection](#establishing-tls-connection) - [Session verification and protocol negotiation](#session-verification-and-protocol-negotiation) - - [Controller/host session operation](#сontrollerhost-session-operation) + - [Controller/host session operation](#controllerhost-session-operation) - [Key agreement for announcement packet and for session](#key-agreement-for-announcement-packet-and-for-session) - [Threat model](#threat-model) @@ -104,12 +104,11 @@ Multicast session announcement is a binary encoded packet with this syntax: ```abnf sessionAddressPacket = dhPubKey nonce encrypted(unpaddedSize sessionAddress packetPad) dhPubKey = length x509encoded ; same as announced -nonce = length *OCTET -sessionAddress = largeLength sessionAddressUri ; as above +nonce = 24*24 OCTET ; NaCl 192-bit nonce, no length prefix +sessionAddress = sessionAddressUri ; length given by unpaddedSize length = 1*1 OCTET ; for binary data up to 255 bytes largeLength = 2*2 OCTET ; for binary data up to 65535 bytes -packetPad = ; possibly, we may need to move KEM agreement one step later, -; with encapsulation key in HELLO block and KEM ciphertext in reply to HELLO. +packetPad = ``` ### Establishing TLS connection @@ -143,7 +142,7 @@ hostHello = %s"HELLO " dhPubKey nonce encrypted(unpaddedSize hostHelloJSON hello unpaddedSize = largeLength dhPubKey = length x509encoded pad = -helloPad = +helloPad = largeLength = 2*2 OCTET ``` @@ -157,10 +156,7 @@ The controller decrypts (including the first session) and validates the received { "definitions": { "version": { - "type": "string", - "metadata": { - "format": "[0-9]+" - } + "type": "uint16" }, "base64url": { "type": "string", @@ -172,9 +168,7 @@ The controller decrypts (including the first session) and validates the received "properties": { "v": {"ref": "version"}, "ca": {"ref": "base64url"}, - "kem": {"ref": "base64url"} - }, - "optionalProperties": { + "kem": {"ref": "base64url"}, "app": {"properties": {}, "additionalProperties": true} }, "additionalProperties": true @@ -190,7 +184,7 @@ ctrlHello = %s"HELLO " kemCiphertext encrypted(unpaddedSize ctrlHelloJSON helloP unpaddedSize = largeLength kemCiphertext = largeLength *OCTET pad = -helloPad = +helloPad = largeLength = 2*2 OCTET ctrlError = %s"ERROR " nonce encrypted(unpaddedSize ctrlErrorMessage helloPad) pad @@ -206,7 +200,7 @@ JTD schema for the encrypted part of controller HELLO block `ctrlHelloJSON`: } ``` -Controller `hello` block and all subsequent protocol messages are encrypted with the chain keys derived from the hybrid key (see key exchange below) - that is why conntroller hello block does not include nonce. That provides forward secrecy within the XRCP session. Receiving this `hello` block allows host to compute the same hybrid keys and to derive the same chain keys. +Controller `hello` block and all subsequent protocol messages are encrypted with the chain keys derived from the hybrid key (see key exchange below) - that is why controller hello block does not include nonce. That provides forward secrecy within the XRCP session. Receiving this `hello` block allows host to compute the same hybrid keys and to derive the same chain keys. Once the controller replies HELLO to the valid host HELLO block, it should stop accepting new TCP connections. @@ -261,7 +255,7 @@ kemCiphertext(1) = enc(kemSecret(1), kemEncKey(1)) kemSecret(1) = dec(kemCiphertext(1), kemDecKey(1)) // multicast announcement for session n -announcementSecret(n) = sha256(dhSecret(n')) +announcementSecret(n) = dhSecret(n') dhSecret(n') = dh(hostHelloDhKey(n - 1), controllerDhKey(n)) // session n @@ -277,11 +271,11 @@ If controller fails to store the new host DH key after receiving HELLO block, th To decrypt a multicast announcement, the host should try to decrypt it using the keys of all known (paired) remote controllers. -Once kemSecret is agreed for the session, it is used to derive two chain keys, to receive and to send messages: +Once sessionSecret is agreed for the session, it is used to derive two chain keys, to receive and to send messages: ``` -host: sndKey, rcvKey = HKDF(kemSecret, "SimpleXSbChainInit", 64) -controller: rcvKey, sndKey = HKDF(kemSecret, "SimpleXSbChainInit", 64) +controller: sndKey, rcvKey = HKDF(sessionSecret, "SimpleXSbChainInit", 64) +host: rcvKey, sndKey = HKDF(sessionSecret, "SimpleXSbChainInit", 64) ``` where HKDF is based on SHA512, with empty salt. diff --git a/rfcs/2022-04-20-smp-conf-timeout-recovery.md b/rfcs/2022-04-20-smp-conf-timeout-recovery.md index 7c7f84caa..5941e0259 100644 --- a/rfcs/2022-04-20-smp-conf-timeout-recovery.md +++ b/rfcs/2022-04-20-smp-conf-timeout-recovery.md @@ -3,9 +3,9 @@ ## Problem When sending an SMP confirmation a network timeout can lead to the following race condition: -- server receives the confirmation while the joining party fails to receive the server's response; +- router receives the confirmation while the joining party fails to receive the router's response; - joining party deletes the connection together with credentials sent in the confirmation for securing the queue; -- initiating party will receive the confirmation from the server and secure the queue; +- initiating party will receive the confirmation from the router and secure the queue; - on subsequent attempt to join via the same invitation link initiating party will generate new credentials and fail authorization. This renders the joining party permanently unable to join via that invitation link and complete the connection. diff --git a/rfcs/2024-07-06-ios-notifications.md b/rfcs/2024-07-06-ios-notifications.md index c60d7baf3..c2d7b5472 100644 --- a/rfcs/2024-07-06-ios-notifications.md +++ b/rfcs/2024-07-06-ios-notifications.md @@ -3,12 +3,12 @@ ## Problem iOS notifications may fail to deliver for several reasons, but there are two important reasons that we could address: -- when notification server is not subscribed to SMP server(s), the notifications can be dropped - it can happen because either notification server restarts or becuase SMP server restarted and some messages are received before notification server resubscribed. We lose approximately 3% of notifications because of this reason. +- when notification router is not subscribed to SMP router(s), the notifications can be dropped - it can happen because either notification router restarts or becuase SMP router restarted and some messages are received before notification router resubscribed. We lose approximately 3% of notifications because of this reason. - when user device is offline or has low power condition, Apple does not deliver notification, but puts them to storage. If while the notification is in storage a new one arrives it would overwrite the previous notification. If it was the message to the same message queue, the client will download messages anyway, up to a limit, but if the message was to another queue, it will not be delivered until the app is opened. Apple delivers about 88% of notifications that should be delivered (not accounting for uninstalled apps), the rest is replaced with the newer notifications. ## Solution -The first problem can be solved by preserving notifications for a limited time (say 1 hour) in case there is no subscription to notification from notification server. At the very least, they can be preserved in SMP server memory but can also be stored to a file on restart, similar to messages, and be delivered when notification server resubscribes. It is sufficient to store one notification per messaging queue. +The first problem can be solved by preserving notifications for a limited time (say 1 hour) in case there is no subscription to notification from notification router. At the very least, they can be preserved in SMP router memory but can also be stored to a file on restart, similar to messages, and be delivered when notification router resubscribes. It is sufficient to store one notification per messaging queue. The second problem is both more damaging and more complex to solve. The solution could be to always deliver several last notifications to different queues in one packet (Apple allows up to ~4-5kb notification size, and we are sending packets of fixed size 512 bytes, so we could fit up to 8-10 of them in each notification). diff --git a/rfcs/2024-09-05-queue-storage.md b/rfcs/2024-09-05-queue-storage.md index 9b67cc4cf..0e8fa53a9 100644 --- a/rfcs/2024-09-05-queue-storage.md +++ b/rfcs/2024-09-05-queue-storage.md @@ -8,7 +8,7 @@ See [Short invitation links](./2024-06-21-short-links.md). 2) clients only delete queue records based on some user action, pending connections do not expire. -While part 2 should be improved in the client, indefinite storage of queue records becomes a much bigger issue if each of them would result in a permanent storage of 4-16kb blob in server memory, without server-side expiration for short invitation links. +While part 2 should be improved in the client, indefinite storage of queue records becomes a much bigger issue if each of them would result in a permanent storage of 4-16kb blob in router memory, without router-side expiration for short invitation links. ## Possible solutions @@ -16,15 +16,15 @@ While part 2 should be improved in the client, indefinite storage of queue recor The problem with this approach is that contact addresses are also unsecured queues, and they should not be expired. -We could set really large expiration time, and require that clients "update" the unsecured queues they need at least every 1-2 years, but it would not solve the problem of storing a large number of blobs in the server memory for unused/abandoned 1-time invitations. +We could set really large expiration time, and require that clients "update" the unsecured queues they need at least every 1-2 years, but it would not solve the problem of storing a large number of blobs in the router memory for unused/abandoned 1-time invitations. -2) Do not store blobs in memory / append-only log, and instead use something like RocksDB. While it may be a correct long term solution, it may be not expedient enough at the current POC stage for this feature. Also, the lack of expiration is wrong in any case and would indefinitely grow server storage. +2) Do not store blobs in memory / append-only log, and instead use something like RocksDB. While it may be a correct long term solution, it may be not expedient enough at the current POC stage for this feature. Also, the lack of expiration is wrong in any case and would indefinitely grow router storage. -3) Add flag allowing the server to differentiate permanent queues used as contact addresses, also using different blob sizes for them. In this case, messaging queues will be expired if not secured after 3 weeks, and contact address queues would be expired if not "updated" by the owner within 2 years. +3) Add flag allowing the router to differentiate permanent queues used as contact addresses, also using different blob sizes for them. In this case, messaging queues will be expired if not secured after 3 weeks, and contact address queues would be expired if not "updated" by the owner within 2 years. Probably all three solutions need to be used, to avoid creating a non-expiring blob storage in memory, as in case too many of such blobs are created it would not be possible to differentiate between real users and resource exhaustion attacks, and unlike with messages, they won't be expiring too. -Servers already can differentiate messaging queues and contact address queues, if they want to: +Routers already can differentiate messaging queues and contact address queues, if they want to: - with the old 4-message handshake, the confirmation message on a normal queue was different, and also KEY command was eventually used. - with the fast 2-message handshake, while the confirmation message has the same syntax, and the differences are inside encrypted envelope, the client still uses SKEY command. - in both cases, the usual messaging queues are secured, and contact addresses are not, so this difference is visible in the storage as well (although it is not easy to differentiate between abandoned 1-time invitations and contact addresses). @@ -33,7 +33,7 @@ Differentiating these queues can also allow different message retention times - ## Proposed solution -1. Add queue updated_at date into queue records. While it adds some metadata, it seems necessary to manage retention and quality of service. It will not include exact time, only date, and the time of creation will be replaced by the time of any update - queue secured, a message is sent, or queue owner subscribes to the queue. To avoid the need to update store log on every message this information can be appended to store log on server termination. Or given that only one update per day is needed it may be ok to make these updates as they happen (temporarily making the sequence and time of these events available in storage). +1. Add queue updated_at date into queue records. While it adds some metadata, it seems necessary to manage retention and quality of service. It will not include exact time, only date, and the time of creation will be replaced by the time of any update - queue secured, a message is sent, or queue owner subscribes to the queue. To avoid the need to update store log on every message this information can be appended to store log on router termination. Or given that only one update per day is needed it may be ok to make these updates as they happen (temporarily making the sequence and time of these events available in storage). 2. Add flag to indicate the queue usage - messaging queue or queue for contact address connection requests. This would result in different queue size and different retention policy for queue and its messages. We already have "sender can secure flag" which is, effectively, this flag - contact address queues are never secured. So this does not increase stored metadata in any way. @@ -41,11 +41,11 @@ Differentiating these queues can also allow different message retention times - This is a design considerations and a concept, not a design yet. -Instead of implementing a generic blob storage that can be used as an attack vector, and adds additional failure point (another server storing blob that is necessary to connect to the queue on the current server), but instead adds an extended queue information blobs, most of which could be dropped without the loss of connectivity, so that the attack can be mitigated by deleting these blobs without users losing the ability to connect, as long as the queue and minimal extended information is retained. +Instead of implementing a generic blob storage that can be used as an attack vector, and adds additional failure point (another router storing blob that is necessary to connect to the queue on the current router), but instead adds an extended queue information blobs, most of which could be dropped without the loss of connectivity, so that the attack can be mitigated by deleting these blobs without users losing the ability to connect, as long as the queue and minimal extended information is retained. So, to make the connection there need to be these elements: -- queue server and queue ID - mandatory part, that can be included in short link +- queue router and queue ID - mandatory part, that can be included in short link - SMP key - mandatory part for all queues. We are considering initializing ratchets earlier for contact addresses, and include ratchet keys and pre-keys into queue data as well, but it is out of scope here. - Ratchet keys - mandatory part for 1-time invitation that won't fit in short link. - PQ key - optional part that can be stored with addresses if ratchet keys are added and with 1-time invitations. @@ -56,8 +56,8 @@ So rather that storing one blob with a large address inside it, not associated w Also, we need the address shared with the sender (party accepting the connection) to be short. We could use a similar approach that was proposed for data blobs, using a single random seed per queues to derive multiple keys and IDs from it. For example: 1. The queue owner: - - generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the server, same as now sent in NEW command. - - generates queue recipient ID (this ID can still be server-generated). + - generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the router, same as now sent in NEW command. + - generates queue recipient ID (this ID can still be router-generated). - generates X25519 key pair `(k, pk)` to use with the accepting party. - derives from `k`: - sender ID. @@ -73,9 +73,9 @@ The algorithm used to derive key and ID from `k` needs to be cryptographically s So, coupling blob storage with messaging queues has these pros/cons: Cons: -- no additional layer of privacy - the server used for connection is visible in the link, even after the blobs are removed from the server. +- no additional layer of privacy - the router used for connection is visible in the link, even after the blobs are removed from the router. Pros: -- no additional point of failure in the connection process - the same server will be used to retrieve necessary blobs as for connection. +- no additional point of failure in the connection process - the same router will be used to retrieve necessary blobs as for connection. - queue blobs of messaging blobs will be automatically removed once the queue is secured or expired, without additional request from the recipient - reducing the storage and the time these blobs are available. - queue blobs for contact addresses will be structured and some of the large blobs can be removed in case of resource exhaustion attack (and recreated by the client if needed), with the only downside that PQ handshake will be postponed (which is the case now) and profile will not be available at a point of connection. diff --git a/rfcs/2024-09-10-private-rendezvous.md b/rfcs/2024-09-10-private-rendezvous.md index 744596ea6..4f549d2ef 100644 --- a/rfcs/2024-09-10-private-rendezvous.md +++ b/rfcs/2024-09-10-private-rendezvous.md @@ -2,25 +2,25 @@ ## Problem -Our current handshake protocol is open to this attack: whoever observes the link exchange, knows on which server connection is being made, and if the traffic on this server is observed, then it can confirm communication between parties. Further, even with the [last proposal](./2024-09-09-smp-blobs.md#possible-privacy-improvement), having real-time access to the server data allows to establish the exact messaging queue that is used to send messages. +Our current handshake protocol is open to this attack: whoever observes the link exchange, knows on which router connection is being made, and if the traffic on this router is observed, then it can confirm communication between parties. Further, even with the [last proposal](./2024-09-09-smp-blobs.md#possible-privacy-improvement), having real-time access to the router data allows to establish the exact messaging queue that is used to send messages. ## Solution -We could make the initial link exchange more private by making it harder for any observer to discover which server will be used for messaging by hiding this information from the server that hosts the initial link. +We could make the initial link exchange more private by making it harder for any observer to discover which router will be used for messaging by hiding this information from the router that hosts the initial link. Preliminary, the protocol could be the following: -1. Connection initiator stores 224-256 bytes of encrypted connection link on a rendezvous server (link contains server host and linkId on another messaging server, not a rendezvous one). +1. Connection initiator stores 224-256 bytes of encrypted connection link on a rendezvous router (link contains router host and linkId on another messaging router, not a rendezvous one). -2. Rendezvous server adds these links to buckets, up to 64 links per bucket. Bucket ID is the timestamp when the bucket was created + a sequential bucket number, in case more than one bucket is created per second. +2. Rendezvous router adds these links to buckets, up to 64 links per bucket. Bucket ID is the timestamp when the bucket was created + a sequential bucket number, in case more than one bucket is created per second. -3. The server responds to the link creator with a bucket ID where this link was added. That bucket ID is its timestamp + a number prevents server "fingerprinting" clients and using say one bucket for each client. If timestamp is different or a bucket number within this timestamp is too large, the client can refuse to use it, depending on the client settings. +3. The router responds to the link creator with a bucket ID where this link was added. That bucket ID is its timestamp + a number prevents router "fingerprinting" clients and using say one bucket for each client. If timestamp is different or a bucket number within this timestamp is too large, the client can refuse to use it, depending on the client settings. -4. The initiating party will pass to the accepting party the rendezvous server host, the hash of this bucket ID (bucket link) and the passphrase to derive the key from. The initiating party has an option to pass a link and passphrase via two channels - in which case the link will only contain the bucket ID. +4. The initiating party will pass to the accepting party the rendezvous router host, the hash of this bucket ID (bucket link) and the passphrase to derive the key from. The initiating party has an option to pass a link and passphrase via two channels - in which case the link will only contain the bucket ID. -5. The accepting party would then request the bucket via its ID hash (the server would store hashes to be able to look up - hash is used to prevent showing time in the link) and attempt to decrypt all contained links using the provided key. +5. The accepting party would then request the bucket via its ID hash (the router would store hashes to be able to look up - hash is used to prevent showing time in the link) and attempt to decrypt all contained links using the provided key. The accepting party then will continue the connection via the decrypted link. -This obviously does not protect accepting party from the initiating party, if it can choose rendezvous server it controls. It also does not protect from the malicious rendezvous server that would collaborate with link observers. I think reunion doesn’t protect from it too. +This obviously does not protect accepting party from the initiating party, if it can choose rendezvous router it controls. It also does not protect from the malicious rendezvous router that would collaborate with link observers. I think reunion doesn’t protect from it too. But it does protect connection from whoever observes the link, particularly if this link only contains the bucket and the key is passed separately, via some other channel. diff --git a/rfcs/2024-09-25-ios-notifications-2.md b/rfcs/2024-09-25-ios-notifications-2.md index 79416b83f..e17ee90ae 100644 --- a/rfcs/2024-09-25-ios-notifications-2.md +++ b/rfcs/2024-09-25-ios-notifications-2.md @@ -2,7 +2,7 @@ ## Problem -For iOS notifications to be delivered the client has to create credentials for notification subscription on SMP server using NKEY command and after that create a subscription on notification server using SNEW command. These two commands are sent in sequence, after the connections are created, and for it to happen the client needs to be online and in foreground. +For iOS notifications to be delivered the client has to create credentials for notification subscription on SMP router using NKEY command and after that create a subscription on notification router using SNEW command. These two commands are sent in sequence, after the connections are created, and for it to happen the client needs to be online and in foreground. iOS users tend to close the app when it is not used, and iOS has very limited permissions for background activities, so these notification subscriptions are created with a substantial delay, and notifications do not work. @@ -12,19 +12,19 @@ This problem is distinct from and probably more common than other problems affec 1. When the new connection is created, the client already knows if it needs to create notification subscription or not, based on the conversation setting (e.g., if the group is muted, the client will not create notification subscription as well.). We should extend NEW command to avoid the need to send additional NKEY command with an option to create notification subscription at the point where connection is created. NDEL would still be used to disable this notification, and NKEY will be used to re-enable it. -2. In the same way we stopped using SDEL command (NDEL sends notification DELD to subscribed notification server) to delete notificaiton subscriptions from notification server, we should delegate creating notification subscription on notification server to SMP servers. Clients could use keys agreed with ntf server for e2e encryption and for command authorization to encrypt and sign instruction to create notification subscription that will be forwarded to notification server using protocol similar to SMP proxies. This will avoid the need for clients to separately contact notification servers that won't happen until they are online. +2. In the same way we stopped using SDEL command (NDEL sends notification DELD to subscribed notification router) to delete notificaiton subscriptions from notification router, we should delegate creating notification subscription on notification router to SMP routers. Clients could use keys agreed with ntf router for e2e encryption and for command authorization to encrypt and sign instruction to create notification subscription that will be forwarded to notification router using protocol similar to SMP proxies. This will avoid the need for clients to separately contact notification routers that won't happen until they are online. -3. Instead of making Ntf server trust DELD notifications, we could send deletion instructions signed by the client, which will only fail to send in case notification server is down (and they won't be sent later after server restart). +3. Instead of making Ntf router trust DELD notifications, we could send deletion instructions signed by the client, which will only fail to send in case notification router is down (and they won't be sent later after router restart). Cons: -- If SMP servers were to retain in the storage the information about which notification server is used for which queue, it would reduce metadata privacy. While currently it is not an issue, as all notification servers are known and operated by us, once there are other client apps, this can be used for app users fingerprinting, which would act as a deterrence from using new apps – but only if app users use servers of operators who are different from the app provider. To mitigate it, we could only store it in server memory and include notification instruction in subscription commands (SUB) and include notification subscription status in SUB responses. We don't need to mitigate the problem of server being able to store this information, as messaging servers can observe which notification servers connect to them anyway. -- If SMP server is restarted before the subscription request is forwared to the notification server, then it will have to be forwarded again, once the client subscribes. The problem here is that if the client is offline, it will neither subscribe to the queue to send notification subscription request, nor receive notifications from this queue. Storing notification server and subscription request would mitigate that, as in this case we could send all pending requests on server start, without depending on client subscriptions. -- "Small" agent will need to support connections to ntf servers and manage workers that retry sending pending subscription requests. -- Until the client learns the public keys of notification server, it will not be able to decrypt notifications. It potentially can be mitigated by using the public key of the server returned when token is created, in this way different client keys (per-queue) will be combined with the same ntf server key (per-token). +- If SMP routers were to retain in the storage the information about which notification router is used for which queue, it would reduce metadata privacy. While currently it is not an issue, as all notification routers are known and operated by us, once there are other client apps, this can be used for app users fingerprinting, which would act as a deterrence from using new apps – but only if app users use routers of operators who are different from the app provider. To mitigate it, we could only store it in router memory and include notification instruction in subscription commands (SUB) and include notification subscription status in SUB responses. We don't need to mitigate the problem of router being able to store this information, as messaging routers can observe which notification routers connect to them anyway. +- If SMP router is restarted before the subscription request is forwared to the notification router, then it will have to be forwarded again, once the client subscribes. The problem here is that if the client is offline, it will neither subscribe to the queue to send notification subscription request, nor receive notifications from this queue. Storing notification router and subscription request would mitigate that, as in this case we could send all pending requests on router start, without depending on client subscriptions. +- "Small" agent will need to support connections to ntf routers and manage workers that retry sending pending subscription requests. +- Until the client learns the public keys of notification router, it will not be able to decrypt notifications. It potentially can be mitigated by using the public key of the router returned when token is created, in this way different client keys (per-queue) will be combined with the same ntf router key (per-token). ## Implementation details -1. NEW and NKEY commands will need to be extended to include notification subscription request. As the notifier ID needs to be sent to notification server, this notifier ID will have to be client-generated and supplied as part of NEW command. +1. NEW and NKEY commands will need to be extended to include notification subscription request. As the notifier ID needs to be sent to notification router, this notifier ID will have to be client-generated and supplied as part of NEW command. now: @@ -46,4 +46,4 @@ NKEY :: NtfPublicAuthKey -> RcvNtfPublicDhKey -> Maybe NtfServerRequest -> Comma -- NotifierID is passed in entity ID field of the transmission ``` -2. Notification server will need to support an additional command to receive "proxied" subscription commands, `SFWD`, that would include `NtfServerRequest`. This command can include both `SNEW` and `SDEL` commands. +2. Notification router will need to support an additional command to receive "proxied" subscription commands, `SFWD`, that would include `NtfServerRequest`. This command can include both `SNEW` and `SDEL` commands. diff --git a/rfcs/2024-11-25-queue-blobs-2.md b/rfcs/2024-11-25-queue-blobs-2.md index a4913c118..e18162c12 100644 --- a/rfcs/2024-11-25-queue-blobs-2.md +++ b/rfcs/2024-11-25-queue-blobs-2.md @@ -5,7 +5,7 @@ This document evolves the design proposed [here](./2024-09-09-smp-blobs.md). ## Problems In addition to problems in the first doc, we have these issues with in-memory queue record storage: -- many queues are idle or rarely used, but they are loaded to memory, and currently just loading all queues uses 20gb RAM on each server, and takes 10 min to process, increasing downtimes during restarts. +- many queues are idle or rarely used, but they are loaded to memory, and currently just loading all queues uses 20gb RAM on each router, and takes 10 min to process, increasing downtimes during restarts. - adding blobs to memory would make this problem much worse. ## Proposed solution diff --git a/rfcs/2025-03-30-ios-notifications-3.md b/rfcs/2025-03-30-ios-notifications-3.md index 441525767..0922b5a3e 100644 --- a/rfcs/2025-03-30-ios-notifications-3.md +++ b/rfcs/2025-03-30-ios-notifications-3.md @@ -6,65 +6,65 @@ iOS notifications have these problems: - iOS notification service crashes exceeding memory limit. This is being addressed by changes in GHC RTS. - there is a large number of connections, because each member in a group requires individual connection. This will improve with chat relays when each group would require 2-3 connections. - some notification may be not shown if notification with reply/mention is skipped, and instead some other message is delivered, which may be muted. This would not improve without some changes, as notifications may be skipped anyway. -- client devices delay communication with ntf server because it is done in background, and by that time the app may be suspended. -- notification server represents a bottleneck, as it has to be owned by the app vendor, and the current design when ntf server subscribes to notifications scales very badly. +- client devices delay communication with ntf router because it is done in background, and by that time the app may be suspended. +- notification router represents a bottleneck, as it has to be owned by the app vendor, and the current design when ntf router subscribes to notifications scales very badly. This RFC is based on the previous [RFC related to notifications](./2024-09-25-ios-notifications-2.md). ## Solution -As notification server has to know client token and currently it associates subscriptions with this token anyway, we are not gaining any privacy and security by using per-subscription keys - both authorization and encryption keys of notification subscription can be dropped. +As notification router has to know client token and currently it associates subscriptions with this token anyway, we are not gaining any privacy and security by using per-subscription keys - both authorization and encryption keys of notification subscription can be dropped. -We still need to store the list of queue IDs associated with the token on the notification server, but we do not need any per-queue keys on the notification server, and we don't need subscriptions - it's effectively a simple set of IDs, with no other information. +We still need to store the list of queue IDs associated with the token on the notification router, but we do not need any per-queue keys on the notification router, and we don't need subscriptions - it's effectively a simple set of IDs, with no other information. In this case, when queue is created the client would supply notifier ID - it has to be derived from correlation ID, to prevent existense check (see previous RFC). As we also supply sender ID, instead of deriving it as sha3-192 of correlation ID, they both can be derived as sha3-384 and split to two IDs - 24 bytes each. -The notification server will maintain a rotating list of server keys with the latest key communicated to the client every time the token is registered and checked. The keys would expire after, say, 1 week or 1 month, and removed from notification server on expiration. +The notification router will maintain a rotating list of router keys with the latest key communicated to the client every time the token is registered and checked. The keys would expire after, say, 1 week or 1 month, and removed from notification router on expiration. -The packet containing association between notifier queue ID and token will be crypto_box encrypted using key agreement between identified notification server master key and an ephemeral per packet (effectively, per-queue) client-key. +The packet containing association between notifier queue ID and token will be crypto_box encrypted using key agreement between identified notification router master key and an ephemeral per packet (effectively, per-queue) client-key. Deleting the queue may also include encrypted packet that would verify that the client deleted the queue. -Instead of notification server subscribing to the notifications creating a lot of traffic for the queues without messages, the SMP server would push notifications via NTF server connection (whether via NTF or via SMP protocol). This could be used as a mechanism to migrate existing queues when with the next subscription the notification server would communicate it's address to SMP server and this association would be stored together with the queue. +Instead of notification router subscribing to the notifications creating a lot of traffic for the queues without messages, the SMP router would push notifications via NTF router connection (whether via NTF or via SMP protocol). This could be used as a mechanism to migrate existing queues when with the next subscription the notification router would communicate it's address to SMP router and this association would be stored together with the queue. ## Protocol design Additional/changed SMP commands: ```haskell --- register notification server --- should be signed with server key +-- register notification router +-- should be signed with router key NSRV :: NtfServerCreds -> Command NtfServer -- response NSID :: NtfServerId -> BrokerMsg --- to communicate which server is responsible for the queue +-- to communicate which router is responsible for the queue -- should be signed with queue key NSUB :: Maybe NtfServerId -> Command Notifier --- subscribe to notificaions from all queues associated with the server --- should be signed with server key +-- subscribe to notificaions from all queues associated with the router +-- should be signed with router key -- entity ID - NtfServerId NSSUB :: Command NtfServer data NtfServerCreds = NtfServerCreds { server :: NtfServer, - -- NTF server certificate chain that should match fingerpring in address + -- NTF router certificate chain that should match fingerpring in address cert :: X.CertificateChain, - -- server autorizatio key to sign server subscription requests + -- router autorizatio key to sign router subscription requests authKey :: X.SignedExact X.PubKey } -- entity ID is recipient ID -NSKEY :: NtfSubscription -> Command Recipient +NSKEY :: NtfSubscription -> Command Recipient data NtfSubscription = NtfSubscription -- key to encrypt notifications e2e with the client { ntfPubDbKey :: RcvNtfPublicDhKey, ntfServer :: NtfServer, -- should be linked to correlation ID to prevent existense check - -- the ID sent to notification server could be its hash? + -- the ID sent to notification router could be its hash? ntfId :: NotifierId, encNtfTokenAssoc :: EncDataBytes } @@ -77,12 +77,12 @@ data NtfTokenAssoc = NtfTokenAssoc } ``` -SMP server will need to maintain the list of Ntf servers and their credentials, and when NSSUB arrives to make only one subscription. When message arrives it would deliver notification to the correct connection via queue / ntf server association. +SMP router will need to maintain the list of Ntf routers and their credentials, and when NSSUB arrives to make only one subscription. When message arrives it would deliver notification to the correct connection via queue / ntf router association. -Ntf server needs to maintain three indices to the same data: +Ntf router needs to maintain three indices to the same data: - `(smpServer, queueId) -> tokenId` - to deliver notification to the correct token -- `tokenId -> [smpServer -> [queueId]]` - to remove all queues when token is removed, and to store/update these associations effficiently - store log may have one compact line per token (after compacting), or per token/server combination. -- `[smpServer]` - array of SMP servers to subscribe to. +- `tokenId -> [smpServer -> [queueId]]` - to remove all queues when token is removed, and to store/update these associations effficiently - store log may have one compact line per token (after compacting), or per token/router combination. +- `[smpServer]` - array of SMP routers to subscribe to. ## Mention notifications @@ -90,4 +90,4 @@ Currently we are marking messages with T (true) for messages that require notifi The proposal is to: - add additional values to this metadata, e.g. 2 (priority) and 3 (high priority) (and T/F could be sent as 0/1 respectively) - that is, to deliver notifications even if notifications are generally disabled (they can still be further filtered by the client). -- instead of deleting notification credentials when notifications are disabled - which is costly - communicate to SMP server the change of notificaion priority level, e.g. the client could set minimal notification priority to deliver notifications, where 0 would mean disabling it completely, 1 enable for all, 2 for priority 2+, 3 for priority 3. The downside here is that it could be used for timing correlation of queues in the group, but it already can be used on bulk deletions of ntf credentials for these queues and when sending messages. +- instead of deleting notification credentials when notifications are disabled - which is costly - communicate to SMP router the change of notificaion priority level, e.g. the client could set minimal notification priority to deliver notifications, where 0 would mean disabling it completely, 1 enable for all, 2 for priority 2+, 3 for priority 3. The downside here is that it could be used for timing correlation of queues in the group, but it already can be used on bulk deletions of ntf credentials for these queues and when sending messages. diff --git a/rfcs/2025-04-04-short-links-for-groups.md b/rfcs/2025-04-04-short-links-for-groups.md index 90938acec..835f2592c 100644 --- a/rfcs/2025-04-04-short-links-for-groups.md +++ b/rfcs/2025-04-04-short-links-for-groups.md @@ -35,18 +35,18 @@ This could possibly be evolved into the requirement to have a direct connection 3. Allow "joint management" of SMP queues. -SMP servers can support multiple recipients for contact queues:\ +SMP routers can support multiple recipients for contact queues:\ - subscription would be possible to the "subscriber recipient". - all other changes (update data, change subscriber recipient, add or remove recipients) would require multiple recipient signatures on SMP command in line with n-of-m multisig rules, that the command sender would have to collect out-of-band (from SMP protocol point of view). Pros: allows joint ownership, and protects from losing access to master owner device. Cons: - complicates queue abstraction with approach that is not needed for most queues. -- still retains the server as a single point of failure. +- still retains the router as a single point of failure. -4. Introduce "group" as a new type of entity managed by SMP servers. +4. Introduce "group" as a new type of entity managed by SMP routers. -SMP servers would provide a separate set of commands for managing group records that would include in an encrypted container: +SMP routers would provide a separate set of commands for managing group records that would include in an encrypted container: - the group profile - the list of chat relay links - the list of owner member IDs with their public keys @@ -54,7 +54,7 @@ SMP servers would provide a separate set of commands for managing group records - alternative group entity locations - possibly, a globally unique group identity (as the hash of the initial/seed group data). -While the server domain would be used as the hostname in group link, it may contain alternative hosts (not just hostnames of the same server), both in the link and in the group record data. +While the router domain would be used as the hostname in group link, it may contain alternative hosts (not just hostnames of the same router), both in the link and in the group record data. Pros: separates additional complexity to where it is needed, allowing reliability and redundancy for group ownership. Cons: complexity, coupling between SMP and chat protocol. @@ -86,7 +86,7 @@ Cons: - if no messages are accepted, this is not even a queue. - no way to directly contact owners (maybe it is not a downside, as for relays there would be a communication channel anyway as part of the group). -Option 2 looks more simple and attractive, implementing server broadcast for SMP seems unnecessary, as while it could have been used for simple groups, it does not solve such problems as spam and pre-moderation anyway - it requires a higher level protocol. +Option 2 looks more simple and attractive, implementing router broadcast for SMP seems unnecessary, as while it could have been used for simple groups, it does not solve such problems as spam and pre-moderation anyway - it requires a higher level protocol. The command to update owner keys would be `RKEY` with the list of keys, and we can make `NEW` accept multiple keys too, although the use case here is less clear. @@ -96,7 +96,7 @@ Option 1: Use the same keys in SMP as when signing queue data. Option 2: Use different keys. -The value here could be that the server could validate these signatures too, and also maintain the chain of key changes. While tempting, it is probably unnecessary, and this chain of ownership is better to be maintained on chat relay level, as there are no size constraints on the size of this chain. Also, it is better for metadata privacy to not couple transport and chat protocol keys. +The value here could be that the router could validate these signatures too, and also maintain the chain of key changes. While tempting, it is probably unnecessary, and this chain of ownership is better to be maintained on chat relay level, as there are no size constraints on the size of this chain. Also, it is better for metadata privacy to not couple transport and chat protocol keys. We still need to bind the mutable data updates to the "genesis" signature key (the one included in the immutable data). @@ -147,12 +147,12 @@ The size of the OwnerInfo record encoding is: ~189 bytes, so we should practically limit the number of owners to say 8 - 1 original + 7 addiitonal. Original creator could use a different key as a "genesis" key, to conceal creator identity from other members, and it needs to include the record with memberId anyway. -The structure is simplified, and it does not allow arbitrary ownership changes. Its purpose is not to comprehensively manage ownership changes - while it is possible with a generic blockchain, it seems not appropriate at this stage, - but rather to ensure access continuity and that the server cannot modify the data (although nothing prevents the server from removing the data completely or from serving the previous version of the data). +The structure is simplified, and it does not allow arbitrary ownership changes. Its purpose is not to comprehensively manage ownership changes - while it is possible with a generic blockchain, it seems not appropriate at this stage, - but rather to ensure access continuity and that the router cannot modify the data (although nothing prevents the router from removing the data completely or from serving the previous version of the data). For example it would only allow any given owner to remove subsequenty added owners, preserving the group link and identity, but it won't allow removing owners that signed this owner authorization. So owners are not equal, with the creator having the highest rank and being able to remove all additional owners, and owners authorise by creator can remove all other owners but themselves and creator, and so on - they have to maintain the chain that authorized themselves, at least. We could explicitely include owner rank into OwnerInfo, or we could require that they are sorted by rank, or the rank can be simply derived from signatures. When additional owners want to be added to the group, they would have to provide any of the current owners: -- the key for SMP commands authorization - this will be passed to SMP server together with other keys. There could be either RKEY to pass all keys (some risk to miss some, or of race conditions), or RADD/RGET/RDEL to add and remove recipient keys, which has no risk of race conditions. +- the key for SMP commands authorization - this will be passed to SMP router together with other keys. There could be either RKEY to pass all keys (some risk to miss some, or of race conditions), or RADD/RGET/RDEL to add and remove recipient keys, which has no risk of race conditions. - the signature of the immutable data by their member key included in their profile. - the current owner would then include their member key into the queue data, and update it with LSET command. In any case there should be some simple consensus protocol between owners for owner changes, and it has to be maintained as a blockchain by owners and by chat relays, as otherwise it may lead to race conditions with LSET command. diff --git a/rfcs/2025-07-15-multi-device.md b/rfcs/2025-07-15-multi-device.md index 56a09fea2..65e5e30a6 100644 --- a/rfcs/2025-07-15-multi-device.md +++ b/rfcs/2025-07-15-multi-device.md @@ -12,13 +12,13 @@ In addition to that, the specific implementation of this approach in Signal comp While this limitation can be addressed with notifications when a new device is added and per-device keys, we still find the remaining attack vectors on user security and privacy to be unacceptable, and opening unsuspecting users to various criminal actions - and it is wrong to say that would only affect security conscious users, and most people would not be affected by these risks. Allowing potential criminals in groups to know which device you are currently using is a real risk for all users. -Another approach was offered by Threema that is ["mediator" server](https://threema.com/en/blog/md-architectural-overview) where the state of encryption ratchets is stored server-side. While it protects the user from their communication peers, it increases required level of trust to the servers, and in case of SimpleX network it would expose the knowledge of who communicates to whom. So while the idea of server-side storage of encryption state is promising, it has to be per-connection, to retain "no-accounts" property of SimpleX messaging network. +Another approach was offered by Threema that is ["mediator" router](https://threema.com/en/blog/md-architectural-overview) where the state of encryption ratchets is stored router-side. While it protects the user from their communication peers, it increases required level of trust to the routers, and in case of SimpleX network it would expose the knowledge of who communicates to whom. So while the idea of router-side storage of encryption state is promising, it has to be per-connection, to retain "no-accounts" property of SimpleX messaging network. Also see [FAQ](https://simplex.chat/faq/#why-cant-i-use-the-same-profile-on-different-devices) and [this issue](https://github.com/simplex-chat/simplex-chat/issues/444#issuecomment-3066968358). ## Proposed solution -One of the ideas presented in FAQ - to store the state of Double Ratchet algorithm in the encrypted container on the server seems promising. The RFC develops this idea. +One of the ideas presented in FAQ - to store the state of Double Ratchet algorithm in the encrypted container on the router seems promising. The RFC develops this idea. ### Considerations for the design @@ -26,21 +26,21 @@ One of the ideas presented in FAQ - to store the state of Double Ratchet algorit 2. Protocol commands and events may be changed (even if at the cost of slightly reducing message size) can fit the hash of the ratchet state (32 bytes sha256 would be sufficient), so that the client can determine whether it has the most recent ratchet state or if it needs to retrieve the latest copy. Message size reduction won't affect the users because we use compression, and there is a substantial reserve. -3. Client commands that modify ratchet state would include the hash of the previous ratchet state so that the server can reject or ignore the command in case the previous ratchet state is different or in case command is repeated in case of lost response). +3. Client commands that modify ratchet state would include the hash of the previous ratchet state so that the router can reject or ignore the command in case the previous ratchet state is different or in case command is repeated in case of lost response). 4. The client does not need to retrieve message state for each encryption and decryption operation - it can "speculatively" use the ratchet state it has, and receive correct ratchet state in the "error" response after attempting encryption based on incorrect ratchet state. ## Proposed protocol design -Ratchet state will be stored on the same server that stores message queue, as part of message queue record. 8kb is a sufficient size for this blob (the actual max size is 7800 bytes). The server would also store the hashes of the current and, possibly, the previous ratchet states (TBC). +Ratchet state will be stored on the same router that stores message queue, as part of message queue record. 8kb is a sufficient size for this blob (the actual max size is 7800 bytes). The router would also store the hashes of the current and, possibly, the previous ratchet states (TBC). While ratchet is used for duplex connection, the connection still has primary queue, and with redundancy the same ratchet state can be stored on all secondary queues. -Ratchet state will be encrypted using secret_box - a symmetric encryption scheme, so PQ-resistant. If ratchet state is stored on more than one server, it has to be encrypted with a different key for each server. +Ratchet state will be encrypted using secret_box - a symmetric encryption scheme, so PQ-resistant. If ratchet state is stored on more than one router, it has to be encrypted with a different key for each router. Questions: how to rotate the key used to store ratchet? Should key used to encrypt ratchet rotate at the same time when queue is rotated? The latter is a logical option, as it prevents additional complexity and solves the problem anyway. A possible option is to have "ratchet version" that will be used to advance the key used to encrypt ratchet via HKDF. -Security considerations: the scheme may reduce break-in recovery to the points queues are rotated, unless there is some randomness mixed-in into the key derivation (the key used to encrypt ratchet state). But including randomness would defeat the purpose, as other devices wouldn't be able to access the ratchets. Another approach would be to have each device use its own key for encryption, and encrypt to all keys of all devices (or to encrypt key, to avoid size increase). Having multiple encryptions would show how many devices use the queue, but servers already can observe it, so it is a better tradeoff. Another idea would be to rotate the key used to authorize queue commands - we already support multiple recipient keys, and it can be used for multi-device scenario. That would partially mitigate break-in attacks as the attacker who obtained the key from ratchet state would be able to decrypt it, but won't be able to decrypt it (the attacker collusion with the server is not mitigated). Yet another idea would be for each party (device) to share its private (or encapsulation) key and to have a symmetric key (used to encrypt the ratchet state) encrypted (encapsulated) separately for each device. This would reduce the size of the stored data to `ratchet size` + `encrypted key size` * N, so even in case of PQ encryption (e.g. sntrup) the size required to store the ratchet would be under transport block size, while limiting it to say 4-8 devices, which is sufficient. +Security considerations: the scheme may reduce break-in recovery to the points queues are rotated, unless there is some randomness mixed-in into the key derivation (the key used to encrypt ratchet state). But including randomness would defeat the purpose, as other devices wouldn't be able to access the ratchets. Another approach would be to have each device use its own key for encryption, and encrypt to all keys of all devices (or to encrypt key, to avoid size increase). Having multiple encryptions would show how many devices use the queue, but routers already can observe it, so it is a better tradeoff. Another idea would be to rotate the key used to authorize queue commands - we already support multiple recipient keys, and it can be used for multi-device scenario. That would partially mitigate break-in attacks as the attacker who obtained the key from ratchet state would be able to decrypt it, but won't be able to decrypt it (the attacker collusion with the router is not mitigated). Yet another idea would be for each party (device) to share its private (or encapsulation) key and to have a symmetric key (used to encrypt the ratchet state) encrypted (encapsulated) separately for each device. This would reduce the size of the stored data to `ratchet size` + `encrypted key size` * N, so even in case of PQ encryption (e.g. sntrup) the size required to store the ratchet would be under transport block size, while limiting it to say 4-8 devices, which is sufficient. To participate in multi-device scheme the devices would join the usual group that will be used to share public (encapsulation) device keys and to communicate updates to conversations that were received by the currently "active" device. "Active" means the device that received or sent and processed the message, and while only one device can receive messages from a given queue, device "active" state may be determined per queue, allowing concurrent usage. @@ -50,7 +50,7 @@ The scheme must be resilient to state updates being lost, and in case of direct `rsi` - ratchet state on device `i`. -`enc(rs)` - current authoritative ratchet state on the server. +`enc(rs)` - current authoritative ratchet state on the router. `pt` and `ct` - plaintext and ciphertext messages. @@ -58,13 +58,13 @@ Encryption is a state transition function ratchetEnc: `(ct, rs') = ratchetEnc(pt 1. Device encrypts the message using the stored ratchet state: `(ct, rsi') = ratchetEnc(pt, rsi)` -2. Device sends modified encrypted ratchet state and the hash of the previous encrypted state to the server that stores the queue: `RSET (hash(enc(rsi)), enc(rsi'))`. +2. Device sends modified encrypted ratchet state and the hash of the previous encrypted state to the router that stores the queue: `RSET (hash(enc(rsi)), enc(rsi'))`. -3. If the hash of the previous state matches state stored on the server (`hash(enc(rsi)) == hash(enc(rs))`), the server updates the state and responds with `ratchet_ok` (that may include the current state or it's hash, for validation). If the hash is different, the server responds with `bad_ratchet(enc(rs))` message that includes the correct ratchet state. These updates must be atomic. In this case device has to update the local ratchet state (provided it can decrypt it), and repeat encryption attempt. If device cannot decrypt the provided ratchet state, it means that the connection is disrupted (possibly, device is removed from device group, but missed the notifications). +3. If the hash of the previous state matches state stored on the router (`hash(enc(rsi)) == hash(enc(rs))`), the router updates the state and responds with `ratchet_ok` (that may include the current state or it's hash, for validation). If the hash is different, the router responds with `bad_ratchet(enc(rs))` message that includes the correct ratchet state. These updates must be atomic. In this case device has to update the local ratchet state (provided it can decrypt it), and repeat encryption attempt. If device cannot decrypt the provided ratchet state, it means that the connection is disrupted (possibly, device is removed from device group, but missed the notifications). 4. After successful state update in primary receiving queue, the device would update it in secondary receiving queues. -5. Device sends encrypted message as usual, via proxy that must be different both from the server that stores the ratchet and from the destination server. +5. Device sends encrypted message as usual, via proxy that must be different both from the router that stores the ratchet and from the destination router. 6. Device broadcasts sent message and new ratchet state to other devices in the device group. @@ -74,17 +74,17 @@ This protocol is simple, and it minimizes requests when sending the message to o Decryption is also a state transition function: `(pt, rs') = ratchetDec(ct, rs)` -1. Server sends the message to the device (can be in response to SUB or ACK commands, or with active subscription). Pushed message would include the hash of the currently stored ratchet state: `hash(enc(rs))`. +1. Router sends the message to the device (can be in response to SUB or ACK commands, or with active subscription). Pushed message would include the hash of the currently stored ratchet state: `hash(enc(rs))`. 2. If device has the ratchet state with the same hash (`hash(enc(rs)) == hash(enc(rsi))`), it decrypts the message: `(pt, rsi') = ratchetDec(ct, rsi)`. -3. If device has ratchet state with a different hash, it requests ratchet from the server with additional protocol command `RGET` with response `RCHT (enc(rs))` and updates the local state. +3. If device has ratchet state with a different hash, it requests ratchet from the router with additional protocol command `RGET` with response `RCHT (enc(rs))` and updates the local state. 4. Device decrypts the message `(pt, rsi') = ratchetDec(ct, rsi)` and processes it as usual. -5. Device sends acknowledgement to the server as usual, but now it includes the new ratchet state and the hash of the previous state: `ACK msgId (hash(enc(rsi)), enc(rsi'))` +5. Device sends acknowledgement to the router as usual, but now it includes the new ratchet state and the hash of the previous state: `ACK msgId (hash(enc(rsi)), enc(rsi'))` -6. The server compares ratchet state with stored state hash, and in case it matches it processes `ACK` and responds with `OK` as usual (or `NO_MSG` in case msgId is incorrect, also as usual - it would happen in repeated ACK requests). If ratchet state hash does not match, the server would respond with `bad_ratchet(enc(rs))` - which means that the message was already processed by another device and ratchet was advanced. This is a complex scenario, as the client has to either revert the change from message processing or somehow combine the change with the updates communicated via device group (as a side note, device group can simply re-broadcast messages, not state updates, but it will result in state divergence between devices when different messages are lost). +6. The router compares ratchet state with stored state hash, and in case it matches it processes `ACK` and responds with `OK` as usual (or `NO_MSG` in case msgId is incorrect, also as usual - it would happen in repeated ACK requests). If ratchet state hash does not match, the router would respond with `bad_ratchet(enc(rs))` - which means that the message was already processed by another device and ratchet was advanced. This is a complex scenario, as the client has to either revert the change from message processing or somehow combine the change with the updates communicated via device group (as a side note, device group can simply re-broadcast messages, not state updates, but it will result in state divergence between devices when different messages are lost). Unlike sending messages, this flow does not require any additional requests in most cases, only requiring requesting message state reconciliation when the same message was received and processed by more than one client, but it does not require re-acknowledgement. diff --git a/rfcs/2025-08-20-service-subs-drift.md b/rfcs/2025-08-20-service-subs-drift.md new file mode 100644 index 000000000..d4182fd5d --- /dev/null +++ b/rfcs/2025-08-20-service-subs-drift.md @@ -0,0 +1,101 @@ +# Detecting and fixing state with service subscriptions + +## Problem + +While service certificates and subscriptions hugely decrease startup time and delivery delays on router restarts, they introduce the risk of losing subscriptions in case of state drifts. They also do not provide efficient mechanism for validating that the list of subscribed queues is in sync. + +How can the state drift happen? + +There are several possibilities: +- lost broker response would make the broker consider that the queue is associated, but the client won't know it, and will have to re-associate. While in itself it is not a problem, as it'll be resolved, it would make drift detected more frequently (regardless of the detection logic used). That service certificates are used on clients with good connection would make it less likely though. +- router state restored from the backup, in case of some failure. Nothing can be done to recover lost queues, but we may restore lost service associations. +- queue blocking or removal by router operator because of policy violation. +- router downgrade (when it loses all service associations) with subsequent upgrade - the client would think queues are associated, while they are not, and won't receive any messages at all in this scenario. +- any other router-side error or logic error. + +In addition to the possibility of the drift, we simply need to have confidence that service subscriptions work as intended, without skipping queues. We ignored this consideration for notifications, as the tolerance to lost notifications is higher, but we can't ignore it for messages. + +## Solution + +Previously considered approach of sending NIL to all queues without messages is very expensive for traffic (most queues don't have messages), and it is also very expensive to detect and validate drift in the client because of asynchronous / concurrent events. + +We cannot read all queues into memory, and we cannot aggregate all responses in memory, and we cannot create database writes on every single service subscription to say 1m queues (a realistic number), as it simply won't work well even at the current scale. + +An approach of having an efficient way to detect drift, but load the full list of IDs when drift is detected, also won't work well, as drifts may be common, so we need both efficient way to detect there is diff and also to reconcile it. + +### Drift detection + +Both client and router would maintain the number of associated queues and the "symmetric" hash over the set of queue IDs. The requirements for this hash algorithm are: +- not cryptographically strong, to be fast. +- 128 bits to minimize collisions over the large set of millions of queues. +- symmetric - the result should not depend on ID order. +- allows fast additions and removals. + +In this way, every time association is added or removed (including queue marked as deleted), both peers would recompute this hash in the same transaction. + +The client would suspend sending and processing any other commands on the router and the queues of this router until SOKS response is received from this router, to prevent drift. It can be achieved with per-router semaphores/locks in memory. UI clients need to become responsive sooner than these responses are received, but we do not service certificates on UI clients, and chat relays may prevent operations on router queues until SOKS response is received. + +SOKS response would include both the count of associated queues (as now) and the hash over all associated queue IDs (to be added). If both count and hash match, the client will not do anything. If either does not match the client would perform full sync (see below). + +There is a value from doing the same in notification router as well to detect and "fix" drifts. + +The algorithm to compute hashes can be the following. + +1. Compute hash of each queue ID using xxHash3_128 ([xxhash-ffi](https://hackage.haskell.org/package/xxhash-ffi) library). They don't need to be stored or loaded at once, initially, it can be done with streaming if it is detected on start that there is no pre-computed hash. +2. Combine hashes using XOR. XOR is both commutative and associative, so it would produce the same aggregate hash irrespective of the ID order. +3. Adding queue ID to pre-computed hash requires a single XOR with ID hash: `new_aggregate = aggregate XOR hash(queue_id)`. +4. Removing queue ID from pre-computed hash also requires the same XOR (XOR is involutory, it undoes itself): `new_aggregate = aggregate XOR hash(queue_id)`. + +These hashes need to be computed per user/router in the client and per service certificate in the router - on startup both have to validate and compute them once if necessary. + +There can be also a start-up option to recompute hashe(s) to detect and fix any errors. + +This is all rather simple and would help detecting drifts. + +### Synchronization when drift is detected + +The assumption here is that in most cases drifts are rare, and isolated to few IDs (e.g., this is the case with notification router). + +But the algorithm should be resilient to losing all associations, and it should not be substantially worse than simply restoring all associations or loading all IDs. + +We have `c_n` and `c_hash` for client-side count and hash of queue IDs and `s_n` and `s_hash` for router-side, which are returned in SOKS response to SUBS command. + +1. If `c_n /= s_n || c_hash /= s_hash`, the client must perform sync. + +2. If `abs(c_n - s_n) / max(c_n, s_n) > 0.5`, the client will request the full list of queues (more than half of the queues are different), and will perform diff with the queues it has. While performing the diff the client will continue block operations with this user/router. + +3. Otherwise would perform some algorithm for determining the difference between queue IDs between client and router. This algorithm can be made efficient (`O(log N)`) by relying on efficient sorting of IDs and database loading of ranges, via computing and communicating hashes of ranges, and performing a binary search on ranges, with batching to optimize network traffic. + +This algorithm is similar to Merkle tree reconcilliation, but it is optimized for database reading of ordered ranges, and for our 16kb block size to minimize network requests. + +The algorithm: +1. The client would request all ranges from the router. +2. The router would compute hashes for N ranges of IDs and send them to the client. Each range would include start_id, optional end_id (for single ID ranges) and XOR-hash of the range. N is determined based on the block size and the range size. +3. The client would perform the same computation for the same ranges, and compare them with the returned ranges from the router, while detecting any gaps between ranges and missing range boundaries. +4. If more than half of the ranges don't match, the client would request the full list. Otherwise it would repeat the same algorithm for each mismatched range and for gaps. + +It can be further optimized by merging adjacent ranges and by batching all range requests, it is quite simple. + +Once the client determines the list of missing and extra queues it can: +- create associations (via SUB) for missing queues, +- request removal of association (a new command, e.g. BUS) for extra queues on the router. + +The pseudocode for the algorightm: + +For the router to return all ranges or subranges of requested range: + +```haskell +getSubRanges :: Maybe (RecipientId, RecipientId) -> [(RecipientId, Maybe RecipientId, Hash)] +getSubRanges range_ = do + ((min_id, max_id), s_n) <- case range_ of + Nothing -> getAssociatedQueueRange -- with the certificate in the client session. + Just range -> (range,) <$> getAssociatedQueueCount range + if + | s_n <= max_N -> reply_with_single_queue_ranges + | otherwise -> do + let range_size = s_n `div` max_N + read_all_ranges -- in a recursive loop, with max_id, range_hash and next_min_id in each step + reply_ranges +``` + +We don't need to implement this synchronization logic right now, so not including client logic here, it's sufficient to implement drift detection, and the action to fix the drift would be to disable and to re-enable certificates via some command-line parameter of CLI. diff --git a/rfcs/README.md b/rfcs/README.md index a98f8aa10..cf25e0652 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -143,8 +143,8 @@ As more protocols are designated as Core IP, development naturally transitions t | Location | Contents | Count | |----------|----------|-------| -| `protocol/` | Consolidated specs (SMP v9, Agent v5, XFTP v2, XRCP v1, Push v2, PQDR v1) | 6 specs + overview | -| `rfcs/` root | Active draft proposals | 19 | -| `rfcs/done/` | Implemented, not yet verified | 25 | -| `rfcs/standard/` | Verified against implementation | (to be populated) | +| `protocol/` | Consolidated specs (SMP v19, Agent v7, XFTP v3, XRCP v1, NTF v3, PQDR v1) | 6 specs + overview | +| `rfcs/` root | Active draft proposals | 10 | +| `rfcs/done/` | Implemented, not yet verified | 1 (+10 sub-RFCs) | +| `rfcs/standard/` | Verified against implementation | 31 | | `rfcs/rejected/` | Draft proposals not accepted | 7 | diff --git a/rfcs/2026-01-30-send-file-page.md b/rfcs/done/2026-01-30-send-file-page.md similarity index 88% rename from rfcs/2026-01-30-send-file-page.md rename to rfcs/done/2026-01-30-send-file-page.md index 0e35d4499..9080a784c 100644 --- a/rfcs/2026-01-30-send-file-page.md +++ b/rfcs/done/2026-01-30-send-file-page.md @@ -1,15 +1,16 @@ + # Send File Page — Web-based XFTP File Transfer ## 1. Problem & Business Case -There is no way to send or receive files using SimpleX without installing the app. A static web page that implements the XFTP protocol client-side would allow anyone with a browser to upload and download files via XFTP servers, promoting app adoption. +There is no way to send or receive files using SimpleX without installing the app. A static web page that implements the XFTP protocol client-side would allow anyone with a browser to upload and download files via XFTP routers, promoting app adoption. **Business constraints:** - Web page allows up to 100 MB uploads; app allows up to 1 GB. - Page must promote app installation (e.g., banner, messaging around limits). **Security constraint:** -- The server hosting the page must never access file content or file descriptions. The file description is carried in the URL hash fragment (`#`), which browsers do not send to the server. +- The router hosting the page must never access file content or file descriptions. The file description is carried in the URL hash fragment (`#`), which browsers do not send to the router. - The only way to compromise transfer security is page substitution (serving malicious JS). Mitigations: standard web security (HTTPS, CSP, SRI) and IPFS hosting with page fingerprints published in multiple independent locations. ## 2. Design Overview @@ -29,7 +30,7 @@ There is no way to send or receive files using SimpleX without installing the ap │ fetch() over HTTP/2 │ fetch() over HTTP/2 ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ -│ XFTP Server 1 │ │ XFTP Server 2 │ +│ XFTP Router 1 │ │ XFTP Router 2 │ │ (SNI→web cert) │ │ (SNI→web cert) │ │ (+CORS headers) │ │ (+CORS headers) │ └─────────────────┘ └─────────────────┘ @@ -59,7 +60,7 @@ There is no way to send or receive files using SimpleX without installing the ap ### 3.3 Error States - File too large (> 100 MB): Show limit message with app install CTA. -- Server unreachable: Retry with exponential backoff, show error after exhausting retries. +- Router unreachable: Retry with exponential backoff, show error after exhausting retries. - File expired: "This file is no longer available" message. - Decryption failure: "File corrupted or link invalid" message. @@ -71,7 +72,7 @@ There is no way to send or receive files using SimpleX without installing the ap https://example.com/file/# ``` -- Hash fragment is never sent to the server. +- Hash fragment is never sent to the router. - Compression: DEFLATE (raw, no gzip/zlib wrapper) — better ratio than LZW for structured text like YAML. - Encoding: Base64url (RFC 4648 §5) — no `+`, `/`, `=`, or `%` characters. @@ -79,18 +80,18 @@ Alternative: LZW + base64url if DEFLATE proves problematic. Both should be evalu ### 4.2 Redirect Mechanism -For files with many chunks, the YAML file description can exceed a practical URL length. The threshold is ~600 bytes of compressed+encoded description (configurable). +For files with many data packets, the YAML file description can exceed a practical URL length. The threshold is ~600 bytes of compressed+encoded description (configurable). **Flow when description is too large:** 1. Serialize recipient file description to YAML. 2. Encrypt YAML using fresh key + nonce (same XSalsa20-Poly1305 as files). -3. Upload encrypted YAML as a single-chunk "file" to one randomly chosen XFTP server. +3. Upload encrypted YAML as a single-packet "file" to one randomly chosen XFTP router. 4. Create redirect description pointing to this uploaded description. -5. Encode redirect description into URL (always small — single chunk). +5. Encode redirect description into URL (always small — single data packet). **Download with redirect:** 1. Parse URL → redirect description (has `redirect` field with `size` and `digest`). -2. Download the description "file" using the single chunk reference. +2. Download the description "file" using the single data packet reference. 3. Decrypt → get full YAML description. 4. Validate size and digest match redirect metadata. 5. Proceed with normal download using full description. @@ -99,11 +100,11 @@ For files with many chunks, the YAML file description can exceed a practical URL These estimates are preliminary and may be incorrect. -| Scenario | Chunks | Compressed+encoded size | URL length | +| Scenario | Data packets | Compressed+encoded size | URL length | |----------|--------|------------------------|------------| -| Small file (1 chunk, 1 server) | 1 | ~300 bytes | ~350 chars | -| Medium file (5 chunks, 1 server) | 5 | ~500 bytes | ~550 chars | -| Large file (25+ chunks) | 25 | Exceeds threshold → redirect | ~350 chars | +| Small file (1 data packet, 1 router) | 1 | ~300 bytes | ~350 chars | +| Medium file (5 data packets, 1 router) | 5 | ~500 bytes | ~550 chars | +| Large file (25+ data packets) | 25 | Exceeds threshold → redirect | ~350 chars | ## 5. TypeScript XFTP Client Library @@ -141,7 +142,7 @@ The XFTP wire format uses a custom binary encoding (from `Simplex.Messaging.Enco - Fields separated by space (0x20). - `signature`: Ed25519 signature over `(sessionId ++ corrId ++ entityId ++ encodedCommand)`. - `corrId`: Correlation ID (arbitrary, echoed in response). - - `entityId`: File/chunk ID on server. + - `entityId`: File/data packet ID on router. - Command: tag + space-separated fields. - **Padding:** 2-byte big-endian length prefix + message + `#` (0x23) fill to block size (16384 bytes). @@ -154,7 +155,7 @@ The XFTP wire format uses a custom binary encoding (from `Simplex.Messaging.Enco | Transit decryption (download) | XSalsa20-Poly1305 (streaming: `cbInit` + `sbDecryptChunk`) | DH shared secret | 24 B | 16 B | libsodium.js | | Command signing | Ed25519 | 64 B (private) | — | 64 B (sig) | libsodium.js | | DH key exchange | X25519 | 32 B | — | — | libsodium.js | -| Chunk digest | SHA-256 | — | — | 32 B | Web Crypto API | +| Data packet digest | SHA-256 | — | — | 32 B | Web Crypto API | | File digest | SHA-512 | — | — | 64 B | Web Crypto API | | Random bytes | ChaCha20-DRBG | — | — | — | libsodium.js `randombytes_buf` | @@ -204,7 +205,7 @@ async function sendXFTPCommand( - Firefox 102+: Supported - Safari 16.4+: Supported -For older browsers, fall back to `ArrayBuffer` body (buffer entire chunk in memory). +For older browsers, fall back to `ArrayBuffer` body (buffer entire data packet in memory). ### 5.5 Upload Orchestration @@ -220,22 +221,22 @@ For older browsers, fall back to `ArrayBuffer` body (buffer entire chunk in memo d. Encrypt `'#'` padding in 65536-byte chunks to fill `encSize - authTagSize - fileSize' - 8` e. Finalize: `sbAuth(state)` → append 16-byte auth tag 6. Compute SHA-512 digest of encrypted data -7. Split into chunks using prepareChunkSizes algorithm: - - > 75% of 4MB → 4MB chunks - - > 75% of 1MB → 1MB + 4MB chunks - - Otherwise → 64KB + 256KB chunks -8. For each chunk (parallel, up to 8 concurrent): +7. Split into data packets using prepareChunkSizes algorithm: + - > 75% of 4MB → 4MB data packets + - > 75% of 1MB → 1MB + 4MB data packets + - Otherwise → 64KB + 256KB data packets +8. For each data packet (parallel, up to 8 concurrent): a. Generate Ed25519 sender keypair b. Generate Ed25519 recipient keypair (1 recipient for web) - c. Compute SHA-256 chunk digest - d. Connect to XFTP server (handshake if new connection) + c. Compute SHA-256 data packet digest + d. Connect to XFTP router (handshake if new connection) e. Send FNEW { sndKey, size, digest } + recipient keys → receive (senderId, [recipientId]) - f. Send FPUT with chunk data → receive OK + f. Send FPUT with data packet content → receive OK g. Report progress -9. Build FileDescription YAML from all chunk metadata +9. Build FileDescription YAML from all data packet metadata 10. If YAML size (compressed+encoded) > threshold: a. Encrypt YAML as a file - b. Upload encrypted YAML (single chunk) → get redirect description + b. Upload encrypted YAML (single data packet) → get redirect description c. Use redirect description for URL 11. Compress + base64url encode description 12. Display URL: https://example.com/file/# @@ -247,32 +248,32 @@ For older browsers, fall back to `ArrayBuffer` body (buffer entire chunk in memo 1. Parse URL hash fragment 2. Base64url decode + decompress → YAML 3. Parse YAML → FileDescription -4. Validate description (sequential chunks, sizes match) +4. Validate description (sequential data packets, sizes match) 5. If redirect field present: - a. Download redirect file (single chunk) + a. Download redirect file (single data packet) b. Decrypt, validate size+digest, parse inner description c. Continue with inner description -6. For each chunk (parallel, up to 8 concurrent): +6. For each data packet (parallel, up to 8 concurrent): a. Generate ephemeral X25519 keypair - b. Connect to XFTP server (web handshake) + b. Connect to XFTP router (web handshake) c. Send FGET { recipientDhPubKey } → receive (serverDhPubKey, cbNonce) + encrypted body d. Compute DH shared secret - e. Transit-decrypt chunk body (XSalsa20-Poly1305 with DH secret) - f. Verify chunk digest (SHA-256) + e. Transit-decrypt data packet body (XSalsa20-Poly1305 with DH secret) + f. Verify data packet digest (SHA-256) g. Send FACK → receive OK h. Report progress -7. Concatenate all transit-decrypted chunks (in order) → encrypted file +7. Concatenate all transit-decrypted data packets (in order) → encrypted file 8. Verify file digest (SHA-512) 9. File-decrypt entire stream (XSalsa20-Poly1305 with file key + nonce) 10. Extract FileHeader → get original fileName 11. Trigger browser download (Blob + or File System Access API) ``` -## 6. XFTP Server Changes +## 6. XFTP Router Changes ### 6.1 SNI-Based Certificate Switching -The SMP server already implements SNI-based certificate switching (see `Transport/Server.hs:255-269`). The same mechanism must be added to the XFTP server. +The SMP router already implements SNI-based certificate switching (see `Transport/Server.hs:255-269`). The same mechanism must be added to the XFTP router. **Current SMP implementation:** ```haskell @@ -292,14 +293,14 @@ T.onServerNameIndication = case sniCredential of **Certificate setup:** - XFTP identity certificate: Existing self-signed CA chain (used for protocol identity via fingerprint). -- Web certificate: Standard CA-issued TLS certificate (e.g., Let's Encrypt) for the server's FQDN. +- Web certificate: Standard CA-issued TLS certificate (e.g., Let's Encrypt) for the router's FQDN. - Both certificates served on the same port (443). ### 6.2 CORS Support -Browsers enforce same-origin policy. The web page (served from `example.com`) must make cross-origin requests to XFTP servers (`xftp1.simplex.im`, etc.). +Browsers enforce same-origin policy. The web page (served from `example.com`) must make cross-origin requests to XFTP routers (`xftp1.simplex.im`, etc.). -**Required server changes:** +**Required router changes:** 1. **Handle OPTIONS preflight requests:** ``` @@ -319,45 +320,45 @@ Browsers enforce same-origin policy. The web page (served from `example.com`) mu Access-Control-Expose-Headers: * ``` -3. **Implementation location:** In `runHTTP2Server` handler or a wrapper around the XFTP request handler. Detect the `Origin` header → add CORS headers. This can be conditional on web mode being enabled in config. +3. **Implementation location:** In `runHTTP2Server` handler or a wrapper around the XFTP request handler. Detect the `Origin` header → add CORS headers. This can be conditional on web mode being enabled in the router config. **Security consideration:** `Access-Control-Allow-Origin: *` is safe here because: -- All XFTP commands require Ed25519 authentication (per-chunk keys from file description). +- All XFTP commands require Ed25519 authentication (per-packet keys from file description). - No cookies or browser credentials are involved. - File content is end-to-end encrypted. -### 6.3 Web Handshake with Server Identity Proof +### 6.3 Web Handshake with Router Identity Proof **Both SNI and web handshake are required.** They solve different problems: -1. **SNI certificate switching** is required because browsers reject self-signed certificates. The XFTP identity certificate is self-signed (CA chain with offline root), so the server must present a standard CA-issued web certificate (e.g., Let's Encrypt) when a browser connects. SNI is how the server detects this. +1. **SNI certificate switching** is required because browsers reject self-signed certificates. The XFTP identity certificate is self-signed (CA chain with offline root), so the router must present a standard CA-issued web certificate (e.g., Let's Encrypt) when a browser connects. SNI is how the router detects this. 2. **Web handshake with challenge-response** is required because browsers cannot access the TLS certificate fingerprint or the TLS-unique channel binding (`sessionId`). The native client validates XFTP identity by checking the certificate chain fingerprint against the known `keyHash` and binding it to the TLS session. The browser gets none of this — it only knows TLS succeeded with some CA-issued cert. So the XFTP identity must be proven at the protocol level. **Standard handshake (unchanged for native clients):** ``` -1. Client → empty POST body → Server -2. Server → padded { vRange, sessionId, CertChainPubKey } → Client -3. Client → padded { version, keyHash } → Server -4. Server → empty → Client +1. Client → empty POST body → Router +2. Router → padded { vRange, sessionId, CertChainPubKey } → Client +3. Client → padded { version, keyHash } → Router +4. Router → empty → Client ``` **Web handshake (new, when SNI is detected):** ``` -1. Client → padded { challenge: 32 random bytes } → Server -2. Server → padded { vRange, sessionId, CertChainPubKey } (header block) +1. Client → padded { challenge: 32 random bytes } → Router +2. Router → padded { vRange, sessionId, CertChainPubKey } (header block) + extended body { fullCertChain, signature(challenge ++ sessionId) } → Client 3. Client validates: - Certificate chain CA fingerprint matches known keyHash - Signature over (challenge ++ sessionId) is valid under cert's public key - - This proves: server controls XFTP identity key AND is live (not replay) -4. Client → padded { version, keyHash } → Server -5. Server → empty → Client + - This proves: router controls XFTP identity key AND is live (not replay) +4. Client → padded { version, keyHash } → Router +5. Router → empty → Client ``` -**Detection mechanism:** The server detects web clients by the `sniCredUsed` flag (already available from the TLS layer). When SNI is detected, the server expects a challenge in the first POST body (non-empty, unlike standard handshake where it is empty). No marker byte is needed — SNI presence is the discriminator. +**Detection mechanism:** The router detects web clients by the `sniCredUsed` flag (already available from the TLS layer). When SNI is detected, the router expects a challenge in the first POST body (non-empty, unlike standard handshake where it is empty). No marker byte is needed — SNI presence is the discriminator. -**Block size note:** The XFTP block size is 16384 bytes (`Protocol.hs:65`). The XFTP identity certificate chain fits within this block. The signed challenge response is sent as an extended body (streamed after the 16384-byte header block), same mechanism as file chunk data. +**Block size note:** The XFTP block size is 16384 bytes (`Protocol.hs:65`). The XFTP identity certificate chain fits within this block. The signed challenge response is sent as an extended body (streamed after the 16384-byte header block), same mechanism as data packet content. ### 6.4 Protocol Version and Handshake Extension @@ -373,11 +374,11 @@ The XFTP handshake is binary-encoded via the `Encoding` typeclass (`Transport.hs ### 6.5 Serving the Static Page -The XFTP server can optionally serve the static web page itself (similar to how SMP servers serve info pages). When a browser connects via SNI and sends a GET request (not POST), the server serves the HTML/JS/CSS bundle. +The XFTP router can optionally serve the static web page itself (similar to how SMP routers serve info pages). When a browser connects via SNI and sends a GET request (not POST), the router serves the HTML/JS/CSS bundle. -This can be implemented identically to the SMP server's static page serving (`apps/smp-server/web/Static.hs`), using Warp to handle HTTP requests on the same TLS connection. +This can be implemented identically to the SMP router's static page serving (`apps/smp-server/web/Static.hs`), using Warp to handle HTTP requests on the same TLS connection. -Alternatively, the page is hosted on a separate web server (e.g., `files.simplex.chat`). The XFTP servers only need to handle XFTP protocol requests (POST) with CORS headers. +Alternatively, the page is hosted on a separate web server (e.g., `files.simplex.chat`). The XFTP routers only need to handle XFTP protocol requests (POST) with CORS headers. ## 7. Security Analysis @@ -386,24 +387,24 @@ Alternatively, the page is hosted on a separate web server (e.g., `files.simplex | Threat | Mitigation | Residual Risk | |--------|-----------|---------------| | Page substitution (malicious JS) | HTTPS, CSP, SRI; IPFS hosting with fingerprints in multiple locations | If web server is compromised and IPFS is not used, all guarantees lost. Fundamental limitation of web-based E2E crypto, mitigated by IPFS. | -| MITM between browser and XFTP server | XFTP identity verification via challenge-response handshake | Attacker can relay traffic (see §7.2) but cannot read file content due to E2E encryption. | -| File description leakage | Hash fragment (`#`) is never sent to server | If browser extension or malware reads URL bar, description is exposed. | -| Server learns file content | File encrypted client-side before upload (XSalsa20-Poly1305) | Server sees encrypted chunks only. | +| MITM between browser and XFTP router | XFTP identity verification via challenge-response handshake | Attacker can relay traffic (see §7.2) but cannot read file content due to E2E encryption. | +| File description leakage | Hash fragment (`#`) is never sent to router | If browser extension or malware reads URL bar, description is exposed. | +| Router learns file content | File encrypted client-side before upload (XSalsa20-Poly1305) | Router sees encrypted data packets only. | | Traffic analysis | File size visible to network observers | Same as native XFTP client. | ### 7.2 Relay Attack Analysis -An attacker who controls the network could relay all traffic between the browser and the real XFTP server: +An attacker who controls the network could relay all traffic between the browser and the real XFTP router: -1. Browser sends challenge to "attacker's server" -2. Attacker relays to real server -3. Real server signs challenge + sessionId with XFTP identity key +1. Browser sends challenge to "attacker's router" +2. Attacker relays to real router +3. Real router signs challenge + sessionId with XFTP identity key 4. Attacker relays signed response to browser -5. Browser validates ✓ (signature is from the real server) +5. Browser validates ✓ (signature is from the real router) However, the attacker **cannot read file content** because: - File encryption key is in the hash fragment (never sent over network) -- Transit encryption uses DH key exchange (FGET) — attacker doesn't have server's DH private key +- Transit encryption uses DH key exchange (FGET) — attacker doesn't have router's DH private key - The attacker can observe transfer sizes and timing, but this is already visible via traffic analysis The relay attack is equivalent to a passive network observer, which is the same threat model as native XFTP. @@ -414,6 +415,7 @@ The relay attack is equivalent to a passive network observer, which is the same |----------|--------------|------------| | TLS certificate validation | XFTP identity cert via fingerprint pinning | Web CA cert via browser + XFTP identity via challenge-response | | Session binding | TLS-unique binds to XFTP identity cert | TLS-unique binds to web cert; challenge binds to XFTP identity | + | Code integrity | Binary signed/distributed via app stores | Served over HTTPS; SRI for subresources; IPFS hosting option; vulnerable to server compromise | | File encryption | XSalsa20-Poly1305 | Same | | Transit encryption | DH + XSalsa20-Poly1305 | Same | @@ -421,8 +423,8 @@ The relay attack is equivalent to a passive network observer, which is the same ### 7.4 Layman Security Summary (Displayed on Page) The web page should display a brief, non-technical security summary explaining to users: -- Files are encrypted in the browser before upload — the server never sees file contents. -- The file link (URL) contains the decryption key in the hash fragment, which the browser never sends to any server. +- Files are encrypted in the browser before upload — the router never sees file contents. +- The file link (URL) contains the decryption key in the hash fragment, which the browser never sends to any router. - Only someone with the exact link can download and decrypt the file. - The main risk is if the web page itself is tampered with (page substitution attack). IPFS hosting mitigates this. - For maximum security, use the SimpleX app instead. @@ -445,10 +447,10 @@ The web page should display a brief, non-technical security summary explaining t - Well-understood, readable, auditable by the community. - Rich crypto ecosystem (libsodium.js provides all needed NaCl primitives as WASM). - Direct access to browser APIs (fetch, File, ReadableStream, Blob). -- Testable in Node.js against Haskell XFTP server. +- Testable in Node.js against Haskell XFTP router. - Small bundle size (~200 KB with libsodium WASM). -**Risk:** Exact byte-level wire compatibility requires careful encoding implementation and thorough testing against the Haskell server. +**Risk:** Exact byte-level wire compatibility requires careful encoding implementation and thorough testing against the Haskell router. ### 8.3 Option 3: C to WASM @@ -476,14 +478,14 @@ The web page should display a brief, non-technical security summary explaining t 4. Handshake encoding/decoding (protocol/handshake.ts) — 18 tests 5. Identity proof verification (crypto/identity.ts) — 15 tests 6. File descriptions: types, YAML, validation (protocol/description.ts) — 13 tests -7. Chunk sizing: prepareChunkSizes, singleChunkSize, etc. (protocol/chunks.ts) — 4 tests +7. Data packet sizing: prepareChunkSizes, singleChunkSize, etc. (protocol/chunks.ts) — 4 tests 8. Transport crypto: cbAuthenticate/cbVerify, transit encrypt/decrypt (protocol/client.ts) — 10 tests -9. Server address parsing (protocol/address.ts) — 3 tests +9. Router address parsing (protocol/address.ts) — 3 tests 10. Download helpers: DH, transit-decrypt, file-decrypt (download.ts) — 11 tests -### Phase 2: XFTP Server Changes — DONE +### Phase 2: XFTP Router Changes — DONE -**Goal:** XFTP servers support web client connections. +**Goal:** XFTP routers support web client connections. **Completed** (7 Haskell integration tests passing): 1. SNI certificate switching — `TLSServerCredential` mechanism for XFTP @@ -493,20 +495,20 @@ The web page should display a brief, non-technical security summary explaining t ### Phase 3: HTTP/2 Client + Agent Orchestration -**Goal:** Complete XFTP client that can upload and download files against a real Haskell XFTP server. +**Goal:** Complete XFTP client that can upload and download files against a real Haskell XFTP router. 1. **`client.ts`** ← `Simplex.FileTransfer.Client` — HTTP/2 client via `fetch()` / `node:http2`: connect + handshake, sendCommand, createChunk, uploadChunk, downloadChunk, deleteChunk, ackChunk, ping. -2. **`agent.ts`** ← `Simplex.FileTransfer.Client.Main` — Upload orchestration (encrypt → chunk → register → upload → build description), download orchestration (parse → download → verify → decrypt → ack), URL encoding with DEFLATE compression (§4.1). +2. **`agent.ts`** ← `Simplex.FileTransfer.Client.Main` — Upload orchestration (encrypt → split into data packets → register → upload → build description), download orchestration (parse → download → verify → decrypt → ack), URL encoding with DEFLATE compression (§4.1). ### Phase 4: Integration Testing -**Goal:** Prove the TypeScript client is wire-compatible with the Haskell server. +**Goal:** Prove the TypeScript client is wire-compatible with the Haskell router. 1. **Test harness** — Haskell-driven tests in `XFTPWebTests.hs` (same pattern as per-function tests). -2. **Upload test** — TypeScript uploads file → Haskell client downloads it → verify contents match. -3. **Download test** — Haskell client uploads file → TypeScript downloads it → verify contents match. +2. **Upload test** — TypeScript uploads file → Haskell client downloads it → verify content matches. +3. **Download test** — Haskell client uploads file → TypeScript downloads it → verify content matches. 4. **Round-trip test** — TypeScript upload → TypeScript download → verify. -5. **Edge cases** — Single chunk, many chunks, exactly-sized chunks, redirect descriptions. +5. **Edge cases** — Single data packet, many data packets, exactly-sized data packets, redirect descriptions. ### Phase 5: Web Page @@ -517,11 +519,11 @@ The web page should display a brief, non-technical security summary explaining t 3. **Download UI** — Parse URL, show file info, download button, progress circle. 4. **App install CTA** — Banner/messaging promoting SimpleX app for larger files. -### Phase 6: Server-Hosted Page (Optional) +### Phase 6: Router-Hosted Page (Optional) -**Goal:** XFTP servers can optionally serve the web page themselves. +**Goal:** XFTP routers can optionally serve the web page themselves. -1. **Static file serving** — Similar to SMP server's `attachStaticFiles`. +1. **Static file serving** — Similar to SMP router's `attachStaticFiles`. 2. **GET handler** — When web client sends HTTP GET (not POST), serve HTML page. 3. **Page generation** — Embed page bundle at server build time. @@ -588,9 +590,9 @@ cabal test --ghc-options -O0 --test-option=--match="/XFTP Web Client/" **Random inputs:** Haskell tests can use QuickCheck to generate random inputs each run, not just hardcoded values. This catches edge cases that fixed test vectors miss. -### 10.2 Integration Tests (TS-driven, spawns Haskell server) +### 10.2 Integration Tests (TS-driven, spawns Haskell router) -**Only attempted after all per-function tests (§10.1) pass.** These are end-to-end tests that verify the full upload/download pipeline works against a real XFTP server. +**Only attempted after all per-function tests (§10.1) pass.** These are end-to-end tests that verify the full upload/download pipeline works against a real XFTP router. **Approach:** Node.js test (`xftp-web/test/integration.test.ts`) spawns `xftp-server` and `xftp` CLI as subprocesses. @@ -615,7 +617,7 @@ cabal test --ghc-options -O0 --test-option=--match="/XFTP Web Client/" 3. TypeScript upload + download round-trip. 4. Web handshake with challenge-response validation. 5. Redirect descriptions (large file → compressed description upload). -6. Multiple chunks across multiple servers. +6. Multiple data packets across multiple routers. 7. Error cases: expired file, auth failure, digest mismatch. ### 10.3 Browser Tests @@ -635,7 +637,7 @@ The per-function tests (§10.1) must pass before attempting integration tests ( 5. **Protocol encoding** — command/response encoding, transmission framing (§12.2, §12.3) 6. **Handshake** — handshake type encoding/decoding (§12.9) 7. **Description** — YAML serialization, validation (§12.12–§12.14) -8. **Chunk sizing** — `prepareChunkSizes`, `getChunkDigest` (§12.11) +8. **Data packet sizing** — `prepareChunkSizes`, `getChunkDigest` (§12.11) 9. **Transport client** — `sendCommand`, `createChunk`, `uploadChunk`, `downloadChunk` (§12.10) 10. **Integration** — full upload/download round-trips (§10.2) @@ -660,7 +662,7 @@ The TypeScript implementation must reimplement the exact streaming logic using l ### 11.3 Web Client Detection -Both SNI and web handshake are mandatory (see §6.3). SNI detection (`sniCredUsed` flag) is the discriminator — when SNI is detected, the server expects the web handshake variant. +Both SNI and web handshake are mandatory (see §6.3). SNI detection (`sniCredUsed` flag) is the discriminator — when SNI is detected, the router expects the web handshake variant. ### 11.4 URL Compression @@ -677,32 +679,32 @@ XSalsa20-Poly1305 streaming encryption/decryption is sequential — each 64KB bl **Upload flow:** 1. `File.stream()` → encrypt sequentially (state threading) → buffer encrypted output 2. Compute SHA-512 digest of encrypted data -3. Split into chunks, upload in parallel to 8 randomly selected servers (from 6 default servers in `Presets.hs`) +3. Split into data packets, upload in parallel to 8 randomly selected routers (from 6 default routers in `Presets.hs`) **Download flow:** -1. Download chunks in parallel from servers → buffer encrypted data +1. Download data packets in parallel from routers → buffer encrypted data 2. Decrypt sequentially (state threading) → verify auth tag 3. Trigger browser save Both directions buffer ~100 MB of encrypted data. The approach should be symmetric. -**Option A — Memory buffer:** Buffer encrypted data as `ArrayBuffer`. 100 MB peak memory is feasible on modern devices. Simple implementation, no Web Worker needed. Chunk slicing is zero-copy via `ArrayBuffer.slice()`. +**Option A — Memory buffer:** Buffer encrypted data as `ArrayBuffer`. 100 MB peak memory is feasible on modern devices. Simple implementation, no Web Worker needed. Data packet slicing is zero-copy via `ArrayBuffer.slice()`. **Option B — OPFS ([Origin Private File System](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system)):** Write encrypted data to OPFS instead of holding in memory. OPFS storage quota is shared with IndexedDB/Cache API — typically hundreds of MB to several GB ([quota details](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria)). The fast synchronous API (`createSyncAccessHandle()`) requires a [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/FileSystemFileHandle/createSyncAccessHandle) but is [3-4x faster than IndexedDB](https://web.dev/articles/origin-private-file-system). The async API (`createWritable()`) works on the main thread. **Decision:** Use OPFS with a Web Worker. While 100 MB fits in memory, OPFS future-proofs the implementation for raising the file size limit (250 MB, 500 MB, etc.) without code changes. The Web Worker also keeps the main thread responsive during encryption/decryption. The implementation cost is modest — a single worker that runs the sequential crypto pipeline, reading/writing OPFS files. -### 11.7 Server Page Hosting +### 11.7 Router Page Hosting -Excluded from initial implementation. Added at the very end (Phase 5) as optional feature. Initial deployment serves the page from a separate web host. +Excluded from initial implementation. Added at the very end (Phase 5) as optional feature. Initial deployment serves the page from a separate web server. ### 11.8 File Expiry Communication -Hardcode 48 hours for standalone web page. Server-hosted page can use server-configurable TTL. The page should also display which XFTP servers were used for the upload. +Hardcode 48 hours for standalone web page. Router-hosted page can use router-configurable TTL. The page should also display which XFTP routers were used for the upload. ### 11.9 Concurrent Operations -8 parallel operations in the browser. The Haskell CLI uses 16, but browsers have per-origin connection limits (6-8). Since chunks typically go to different servers (different origins), 8 provides good parallelism without hitting browser limits. +8 parallel operations in the browser. The Haskell CLI uses 16, but browsers have per-origin connection limits (6-8). Since data packets typically go to different routers (different origins), 8 provides good parallelism without hitting browser limits. ## 12. Haskell-to-TypeScript Function Mapping @@ -861,13 +863,13 @@ Note: `encryptFile` does NOT use `padLazy` or `sbEncryptTailTag`. It manually pr **`decryptChunks` algorithm** (lines 57-111) — two paths: -**Single chunk (one file, line 60):** Calls `sbDecryptTailTag(key, nonce, encSize - authTagSize, data)` directly. This internally decrypts, verifies auth tag, and strips the 8-byte length prefix + padding via `unPad`. Returns `(authOk, content)`. Then parses `FileHeader` from content. +**Single data packet (one file, line 60):** Calls `sbDecryptTailTag(key, nonce, encSize - authTagSize, data)` directly. This internally decrypts, verifies auth tag, and strips the 8-byte length prefix + padding via `unPad`. Returns `(authOk, content)`. Then parses `FileHeader` from content. -**Multi-chunk (line 67):** +**Multi-packet (line 67):** 1. `sbInit(key, nonce)` → init state -2. Decrypt first chunk file: `sbDecryptChunkLazy(state, chunk)` → `splitLen` extracts 8-byte `expectedLen` → parse `FileHeader` -3. Decrypt middle chunk files: `sbDecryptChunkLazy(state, chunk)` loop, write to output, accumulate `len` -4. Decrypt last chunk file: split off last 16 bytes as auth tag → `sbDecryptChunkLazy(state, remaining)` → truncate padding using `expectedLen` vs accumulated `len` → verify `sbAuth(finalState) == authTag` +2. Decrypt first data packet: `sbDecryptChunkLazy(state, chunk)` → `splitLen` extracts 8-byte `expectedLen` → parse `FileHeader` +3. Decrypt middle data packets: `sbDecryptChunkLazy(state, chunk)` loop, write to output, accumulate `len` +4. Decrypt last data packet: split off last 16 bytes as auth tag → `sbDecryptChunkLazy(state, remaining)` → truncate padding using `expectedLen` vs accumulated `len` → verify `sbAuth(finalState) == authTag` **`FileHeader`** (`Types.hs:35`): `{fileName :: String, fileExtra :: Maybe String}`, parsed via `smpP`. @@ -888,18 +890,18 @@ XFTP handshake types and encoding. ### 12.10 `protocol/client.ts` ← `Simplex/FileTransfer/Client.hs` (crypto primitives) — DONE -Transport-level crypto for command authentication and chunk encryption/decryption. +Transport-level crypto for command authentication and data packet encryption/decryption. | TypeScript function | Haskell function | Description | Status | |---|---|---|---| | `cbAuthenticate(peerPub, ownPriv, nonce, msg)` | `C.cbAuthenticate` | 80-byte crypto_box authenticator | ✓ | | `cbVerify(peerPub, ownPriv, nonce, auth, msg)` | `C.cbVerify` | Verify authenticator | ✓ | -| `encryptTransportChunk(dhSecret, nonce, plain)` | `sendEncFile` | Encrypt chunk (tag appended) | ✓ | -| `decryptTransportChunk(dhSecret, nonce, enc)` | `receiveEncFile` | Decrypt chunk (tag verified) | ✓ | +| `encryptTransportChunk(dhSecret, nonce, plain)` | `sendEncFile` | Encrypt data packet (tag appended) | ✓ | +| `decryptTransportChunk(dhSecret, nonce, enc)` | `receiveEncFile` | Decrypt data packet (tag verified) | ✓ | ### 12.11 `protocol/chunks.ts` ← `Simplex/FileTransfer/Chunks.hs` + `Client.hs` — DONE -Chunk size selection and file splitting. +Data packet size selection and file splitting. | TypeScript function/constant | Haskell equivalent | Status | |---|---|---| @@ -944,7 +946,7 @@ HTTP/2 XFTP client using `node:http2` (Node.js) or `fetch()` (browser). Transpil **XFTPClient state** (returned by `connectXFTP`): - HTTP/2 session (node: `ClientHttp2Session`, browser: base URL for fetch) - `thParams`: `{sessionId, blockSize, thVersion, thAuth}` from handshake -- Server address for reconnection +- Router address for reconnection **sendXFTPCommand wire format:** 1. `xftpEncodeAuthTransmission(thParams, pKey, (corrId, fId, cmd))` → padded 16KB block @@ -960,16 +962,16 @@ Upload/download orchestration and URL encoding. Combines what the RFC originally | TypeScript function | Haskell function | Line | Description | |---|---|---|---| -| `encryptFileForUpload(file, fileName)` | `encryptFileForUpload` | 264 | key/nonce → encrypt → digest → chunk specs | +| `encryptFileForUpload(file, fileName)` | `encryptFileForUpload` | 264 | key/nonce → encrypt → digest → data packet specs | | `uploadFile(client, chunkSpecs, servers, numRcps)` | `uploadFile` | 285 | Parallel upload (up to 16 concurrent) | -| `uploadFileChunk(client, chunkNo, spec, server)` | `uploadFileChunk` | 301 | FNEW + FPUT for one chunk | +| `uploadFileChunk(client, chunkNo, spec, server)` | `uploadFileChunk` | 301 | FNEW + FPUT for one data packet | | `createRcvFileDescriptions(fd, sentChunks)` | `createRcvFileDescriptions` | 329 | Build per-recipient descriptions | | `createSndFileDescription(fd, sentChunks)` | `createSndFileDescription` | 361 | Build sender (deletion) description | **Upload call sequence** (`cliSendFileOpts`, line 243): 1. `encryptFileForUpload` — `randomSbKey` + `randomCbNonce` → `encryptFile` → `sha512Hash` digest → `prepareChunkSpecs` -2. `uploadFile` — for each chunk: generate sender/recipient key pairs, `createXFTPChunk`, `uploadXFTPChunk` -3. `createRcvFileDescriptions` — assemble `FileDescription` per recipient from sent chunks +2. `uploadFile` — for each data packet: generate sender/recipient key pairs, `createXFTPChunk`, `uploadXFTPChunk` +3. `createRcvFileDescriptions` — assemble `FileDescription` per recipient from sent data packets 4. `createSndFileDescription` — assemble sender description with deletion keys **Download functions:** @@ -977,17 +979,17 @@ Upload/download orchestration and URL encoding. Combines what the RFC originally | TypeScript function | Haskell function | Line | Description | |---|---|---|---| | `downloadFile(description)` | `cliReceiveFile` | 388 | Full download: parse → download → verify → decrypt | -| `downloadFileChunk(client, chunk)` | `downloadFileChunk` | 418 | FGET + transit-decrypt one chunk | -| `ackFileChunk(client, chunk)` | `acknowledgeFileChunk` | 440 | FACK one chunk | -| `deleteFile(description)` | `cliDeleteFile` | 455 | FDEL for all chunks | +| `downloadFileChunk(client, chunk)` | `downloadFileChunk` | 418 | FGET + transit-decrypt one data packet | +| `ackFileChunk(client, chunk)` | `acknowledgeFileChunk` | 440 | FACK one data packet | +| `deleteFile(description)` | `cliDeleteFile` | 455 | FDEL for all data packets | **Download call sequence** (`cliReceiveFile`, line 388): 1. Parse and validate `FileDescription` from YAML -2. Group chunks by server -3. Parallel download: `downloadXFTPChunk` per chunk (up to 16 concurrent) -4. Verify file digest (SHA-512) over concatenated encrypted chunks +2. Group data packets by router +3. Parallel download: `downloadXFTPChunk` per data packet (up to 16 concurrent) +4. Verify file digest (SHA-512) over concatenated encrypted data packets 5. `decryptChunks` — file-level decrypt with auth tag verification -6. Parallel acknowledge: `ackXFTPChunk` per chunk +6. Parallel acknowledge: `ackXFTPChunk` per data packet **URL encoding (§4.1):** @@ -1004,7 +1006,7 @@ Upload/download orchestration and URL encoding. Combines what the RFC originally 2. Send `FGET(rcvDhPubKey)` → receive `FRFile(sndDhPubKey, cbNonce)` + encrypted body 3. Compute DH shared secret: `dh'(sndDhPubKey, rcvDhPrivKey)` (`Crypto.hs:1280`) 4. Transit-decrypt body via `receiveSbFile` (`Transport.hs:176`): `cbInit(dhSecret, cbNonce)` → `sbDecryptChunk` loop (`fileBlockSize` = 16384-byte blocks, `Transport/HTTP2/File.hs:14`) → `sbAuth` tag verification at end -5. Verify chunk digest (SHA-256): `getChunkDigest` (`Client.hs:346`) +5. Verify data packet digest (SHA-256): `getChunkDigest` (`Client.hs:346`) ### 12.18 Per-Function Testing: Haskell Drives Node diff --git a/rfcs/2026-01-30-send-file-page/2026-01-31-xftp-web-server-changes.md b/rfcs/done/2026-01-30-send-file-page/2026-01-31-xftp-web-server-changes.md similarity index 89% rename from rfcs/2026-01-30-send-file-page/2026-01-31-xftp-web-server-changes.md rename to rfcs/done/2026-01-30-send-file-page/2026-01-31-xftp-web-server-changes.md index a1a2f47d5..1a20c215a 100644 --- a/rfcs/2026-01-30-send-file-page/2026-01-31-xftp-web-server-changes.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-01-31-xftp-web-server-changes.md @@ -1,12 +1,12 @@ -# XFTP Server: SNI, CORS, and Web Support +# XFTP Router: SNI, CORS, and Web Support Implementation details for Phase 3 of `rfcs/2026-01-30-send-file-page.md` (sections 6.1-6.4). ## 1. Overview -The XFTP server is extended to support web browser clients by: +The XFTP router is extended to support web browser clients by: -1. **SNI-based TLS certificate switching** — Present a CA-issued web certificate (e.g., Let's Encrypt) to browsers, while continuing to present the self-signed XFTP identity certificate to native clients. +1. **SNI-based TLS certificate switching** — Present a CA-issued web certificate (e.g., Let's Encrypt) to browsers, while continuing to present the self-signed XFTP identity certificate to native XFTP clients. 2. **CORS headers** — Add CORS response headers on SNI connections so browsers allow cross-origin XFTP requests. 3. **Configuration** — `[WEB]` INI section for HTTPS cert/key paths; opt-in (commented out by default). @@ -16,11 +16,11 @@ Web handshake (challenge-response identity proof, §6.3 of parent RFC) is not ye ### 2.1 Reusing the SMP Pattern -The SMP server already implements SNI-based certificate switching via `TLSServerCredential` and `runTransportServerState_` (see `rfcs/2024-09-15-shared-port.md`). The XFTP server applies the same pattern with one key difference: both native and web XFTP clients use HTTP/2 transport, whereas SMP switches between raw SMP protocol and HTTP entirely. +The SMP router already implements SNI-based certificate switching via `TLSServerCredential` and `runTransportServerState_` (see `rfcs/2024-09-15-shared-port.md`). The XFTP router applies the same pattern with one key difference: both native and web XFTP clients use HTTP/2 transport, whereas SMP switches between raw SMP protocol and HTTP entirely. ### 2.2 Approach -When `httpServerCreds` is configured, the XFTP server bypasses `runHTTP2Server` and uses `runTransportServerState_` directly to obtain the per-connection `sniUsed` flag. It then sets up HTTP/2 manually on each TLS connection using `withHTTP2` (same internals as `runHTTP2ServerWith_`). The `sniUsed` flag is captured in the closure and shared by all HTTP/2 requests on that connection. +When `httpServerCreds` is configured, the XFTP router bypasses `runHTTP2Server` and uses `runTransportServerState_` directly to obtain the per-connection `sniUsed` flag. It then sets up HTTP/2 manually on each TLS connection using `withHTTP2` (same internals as `runHTTP2ServerWith_`). The `sniUsed` flag is captured in the closure and shared by all HTTP/2 requests on that connection. When `httpServerCreds` is absent, the existing `runHTTP2Server` path is unchanged. @@ -33,7 +33,7 @@ Browser client (SNI) ──TLS──> Web CA cert ──HTTP/2──> The web certificate file (e.g., `web.crt`) must contain the full chain: leaf certificate followed by the signing CA certificate. `loadServerCredential` uses `T.credentialLoadX509Chain` which reads all PEM blocks from the file. -The client validates the chain by comparing `idCert` fingerprint (the CA cert, second in the 2-cert chain) against the known `keyHash`. This is the same validation as for XFTP identity certificates — the CA that signed the web cert must match the XFTP server's identity. +The client validates the chain by comparing `idCert` fingerprint (the CA cert, second in the 2-cert chain) against the known `keyHash`. This is the same validation as for XFTP identity certificates — the CA that signed the web cert must match the XFTP router's identity. ## 3. CORS Support @@ -69,7 +69,7 @@ Access-Control-Max-Age: 86400 ### 3.4 Security `Access-Control-Allow-Origin: *` is safe because: -- All XFTP commands require Ed25519 authentication (per-chunk keys from file description). +- All XFTP commands require Ed25519 authentication (per-packet keys from file description). - No cookies or browser credentials are involved. - File content is end-to-end encrypted. @@ -87,9 +87,9 @@ Commented out by default — web support is opt-in. ### 4.2 Behavior -- `[WEB]` section not configured: silently ignored, server operates normally for native clients only. +- `[WEB]` section not configured: silently ignored, router operates normally for native clients only. - `[WEB]` section configured with valid cert/key paths: SNI + CORS enabled. -- `[WEB]` section configured with missing cert files: warning + continue (non-fatal, unlike SMP where it is fatal). +- `[WEB]` section configured with missing cert files: warning + continue (non-fatal, unlike SMP router where it is fatal). ## 5. Files Modified @@ -146,9 +146,9 @@ Added SNI and CORS tests as a subsection within `xftpServerTests` (6 tests): 3. **CORS headers** — SNI POST request includes `Access-Control-Allow-Origin: *` and `Access-Control-Expose-Headers: *`. 4. **OPTIONS preflight** — SNI OPTIONS request returns all CORS preflight headers. 5. **No CORS without SNI** — Non-SNI POST request has no CORS headers. -6. **File chunk delivery** — Full XFTP file chunk upload/download through SNI-enabled server verifying no regression. +6. **Data packet delivery** — Full XFTP data packet upload/download through SNI-enabled router verifying no regression. ## 6. Remaining Work -- **Web handshake** (§6.3 of parent RFC): Challenge-response identity proof for SNI connections. The server detects web clients via the `sniUsed` flag and expects a 32-byte challenge in the first POST body (non-empty, unlike standard handshake). Response includes full cert chain + signature over `(challenge ++ sessionId)`. +- **Web handshake** (§6.3 of parent RFC): Challenge-response identity proof for SNI connections. The router detects web clients via the `sniUsed` flag and expects a 32-byte challenge in the first POST body (non-empty, unlike standard handshake). Response includes full cert chain + signature over `(challenge ++ sessionId)`. - **Static page serving** (§6.5 of parent RFC): Optional serving of the web page HTML/JS bundle on GET requests. diff --git a/rfcs/2026-01-30-send-file-page/2026-02-02-xftp-web-handshake.md b/rfcs/done/2026-01-30-send-file-page/2026-02-02-xftp-web-handshake.md similarity index 97% rename from rfcs/2026-01-30-send-file-page/2026-02-02-xftp-web-handshake.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-02-xftp-web-handshake.md index de23bbf8b..14b8d7058 100644 --- a/rfcs/2026-01-30-send-file-page/2026-02-02-xftp-web-handshake.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-02-02-xftp-web-handshake.md @@ -1,6 +1,6 @@ # Web Handshake — Challenge-Response Identity Proof -RFC §6.3: Server proves XFTP identity to web clients independently of TLS CA infrastructure. +RFC §6.3: Router proves XFTP identity to web clients independently of TLS CA infrastructure. ## 1. Protocol @@ -29,7 +29,7 @@ Server → empty → Client **Detection**: `sniUsed` per-connection flag. Non-empty hello allowed only when `sniUsed`. Empty hello with SNI → standard handshake. -**Why both steps 3 and 4**: Native clients verify `signedPubKey` using the TLS peer certificate (`serverKey` from `getServerVerifyKey`), which is the XFTP identity cert in non-SNI connections — TLS provides this binding. Web clients cannot access TLS peer certificate data (browser API limitation; TLS presents the web CA cert but provides no API to extract it). So web clients must verify at the application layer using `authPubKey.certChain`, which always contains the XFTP identity chain regardless of which cert TLS used. Step 3 proves the server holds its identity key *right now* (freshness via random challenge). Step 4 proves the DH session key was signed by the identity key holder (prevents MITM key substitution). Together they give web clients some assurance native clients get from TLS, except channel binding for commands. +**Why both steps 3 and 4**: Native clients verify `signedPubKey` using the TLS peer certificate (`serverKey` from `getServerVerifyKey`), which is the XFTP identity cert in non-SNI connections — TLS provides this binding. Web clients cannot access TLS peer certificate data (browser API limitation; TLS presents the web CA cert but provides no API to extract it). So web clients must verify at the application layer using `authPubKey.certChain`, which always contains the XFTP identity chain regardless of which cert TLS used. Step 3 proves the router holds its identity key *right now* (freshness via random challenge). Step 4 proves the DH session key was signed by the identity key holder (prevents MITM key substitution). Together they give web clients some assurance native clients get from TLS, except channel binding for commands. ## 2. Type Changes — `src/Simplex/FileTransfer/Transport.hs` @@ -56,7 +56,7 @@ Same `Tail compat` pattern as server handshake. Both types use `(..)` export — new fields auto-exported. -## 3. Server Changes — `src/Simplex/FileTransfer/Server.hs` +## 3. Router Changes — `src/Simplex/FileTransfer/Server.hs` ### `XFTPTransportRequest` (line 88) @@ -176,7 +176,7 @@ Remove `extractCertEd25519Key` (replaced by generic path). Keep `extractCertPubl ### 10.5 Tests — `tests/XFTPWebTests.hs` -**Integration test**: Switch from `withXFTPServerEd25519SNI` (Ed25519 fixtures) to `withXFTPServerSNI` (default Ed448 fixtures). Update fingerprint source from `tests/fixtures/ed25519/ca.crt` to `tests/fixtures/ca.crt`. +**Integration test**: Switch from `withXFTPServerEd25519SNI` (Ed25519 fixtures) to `withXFTPServerSNI` (default Ed448 fixtures). Update fingerprint source from `tests/fixtures/ed25519/ca.crt` to the default `tests/fixtures/ca.crt`. Optionally add a second integration test with Ed25519 to cover both paths, or rely on existing unit tests for Ed25519 coverage. diff --git a/rfcs/2026-01-30-send-file-page/2026-02-03-xftp-web-browser-tests.md b/rfcs/done/2026-01-30-send-file-page/2026-02-03-xftp-web-browser-tests.md similarity index 100% rename from rfcs/2026-01-30-send-file-page/2026-02-03-xftp-web-browser-tests.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-03-xftp-web-browser-tests.md diff --git a/rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-browser-transport.md b/rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-browser-transport.md similarity index 99% rename from rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-browser-transport.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-browser-transport.md index 41915bf64..784627a5a 100644 --- a/rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-browser-transport.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-browser-transport.md @@ -1,3 +1,4 @@ + # Browser Transport & Web Worker Architecture ## TOC diff --git a/rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-page.md b/rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-page.md similarity index 99% rename from rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-page.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-page.md index b69234de8..496b3f73d 100644 --- a/rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-page.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-page.md @@ -20,7 +20,7 @@ Build a static web page for browser-based XFTP file transfer (Phase 5 of master Two build variants: - **Local**: single test server at `localhost:7000` (development/testing) -- **Production**: 12 preset XFTP servers (6 SimpleX + 6 Flux) +- **Production**: 12 preset XFTP routers (6 SimpleX + 6 Flux) Uses Vite for bundling (already a dependency via vitest). No CSS framework — plain CSS per RFC spec. @@ -258,7 +258,7 @@ export function pickRandomServer(servers: XFTPServer[]): XFTPServer { ### 4.3 Assumption -Production XFTP servers must have `[WEB]` section configured with a CA-signed certificate for browser TLS. Without this, browsers will reject the self-signed XFTP identity cert. The local test server uses `tests/fixtures/` certs which Chromium accepts via `ignoreHTTPSErrors`. +Production XFTP routers must have `[WEB]` section configured with a CA-signed certificate for browser TLS. Without this, browsers will reject the self-signed XFTP identity cert. The local test router uses `tests/fixtures/` certs which Chromium accepts via `ignoreHTTPSErrors`. ## 5. Page Structure & UI @@ -293,7 +293,7 @@ Both upload-complete and download-ready states display a brief non-technical sec ### 5.5 File expiry -Display on upload-complete state: "Files are typically available for 48 hours." This is an approximation — actual expiry depends on each XFTP server's `[STORE_LOG]` retention configuration. The 48-hour figure matches the current preset server defaults. +Display on upload-complete state: "Files are typically available for 48 hours." This is an approximation — actual expiry depends on each XFTP router's `[STORE_LOG]` retention configuration. The 48-hour figure matches the current preset router defaults. ### 5.6 Styling diff --git a/rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-persistent-connections.md b/rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-persistent-connections.md similarity index 100% rename from rfcs/2026-01-30-send-file-page/2026-02-04-xftp-web-persistent-connections.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-04-xftp-web-persistent-connections.md diff --git a/rfcs/2026-01-30-send-file-page/2026-02-05-xftp-web-e2e-tests.md b/rfcs/done/2026-01-30-send-file-page/2026-02-05-xftp-web-e2e-tests.md similarity index 98% rename from rfcs/2026-01-30-send-file-page/2026-02-05-xftp-web-e2e-tests.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-05-xftp-web-e2e-tests.md index 2dda76aee..c48e5250b 100644 --- a/rfcs/2026-01-30-send-file-page/2026-02-05-xftp-web-e2e-tests.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-02-05-xftp-web-e2e-tests.md @@ -19,10 +19,10 @@ This document specifies comprehensive Playwright E2E tests for the XFTP web page - **Upload flow**: File selection (picker + drag-drop), validation, progress, cancellation, link sharing, error handling - **Download flow**: Invalid link handling, download button, progress, file save, error states -- **Edge cases**: Boundary file sizes, special characters, network failures, multi-chunk files with redirect, UI information display +- **Edge cases**: Boundary file sizes, special characters, network failures, multi-packet files with redirect, UI information display **Key constraints**: -- Tests run against a local XFTP server (started via `globalSetup.ts`) +- Tests run against a local XFTP router (started via `globalSetup.ts`) - Server port is dynamic (read from `/tmp/xftp-test-server.port`) - Browser uses `--ignore-certificate-errors` for self-signed certs - OPFS and Web Workers are required (Chromium supports both) @@ -50,7 +50,7 @@ xftp-web/ ### 2.2 Prerequisites -- `globalSetup.ts` starts the XFTP server and writes port to `PORT_FILE` +- `globalSetup.ts` starts the XFTP router and writes port to `PORT_FILE` - Tests must read the port dynamically: `readFileSync(PORT_FILE, 'utf-8').trim()` - Vite builds and serves the page at `http://localhost:4173` @@ -699,7 +699,7 @@ test('concurrent downloads from same link', async ({browser}) => { }) ``` -### 6.7 Redirect File Handling (Multi-chunk) +### 6.7 Redirect File Handling (Multi-packet) **Test ID**: `edge-redirect-file` @@ -786,7 +786,7 @@ test('download page shows file size and security note', async ({uploadPage, down ### Phase 7: Error Recovery and Advanced (Priority: Low) 22. `upload-error-retry` - Retry after error 23. `edge-concurrent-downloads` - Concurrent access -24. `edge-redirect-file` - Multi-chunk file with redirect (slow) +24. `edge-redirect-file` - Multi-packet file with redirect (slow) 25. `edge-ui-info` - Expiry message, security notes --- diff --git a/rfcs/2026-01-30-send-file-page/2026-02-08-xftp-web-hello-header.md b/rfcs/done/2026-01-30-send-file-page/2026-02-08-xftp-web-hello-header.md similarity index 94% rename from rfcs/2026-01-30-send-file-page/2026-02-08-xftp-web-hello-header.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-08-xftp-web-hello-header.md index c46f38a46..e9b8c1885 100644 --- a/rfcs/2026-01-30-send-file-page/2026-02-08-xftp-web-hello-header.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-02-08-xftp-web-hello-header.md @@ -2,27 +2,27 @@ ## 1. Problem Statement -Browser HTTP/2 connection pooling reuses TLS connections across page navigations (same origin = same connection pool). The XFTP server maintains per-TLS-connection session state in `TMap SessionId Handshake` keyed by `tlsUniq tls`. When a browser navigates from the upload page to the download page (or reloads), the new page sends a fresh ClientHello on the reused HTTP/2 connection. The server is already in `HandshakeAccepted` state for that connection, so it routes the request to `processRequest`, which expects a 16384-byte command block but receives a 34-byte ClientHello → `ERR BLOCK`. +Browser HTTP/2 connection pooling reuses TLS connections across page navigations (same origin = same connection pool). The XFTP router maintains per-TLS-connection session state in `TMap SessionId Handshake` keyed by `tlsUniq tls`. When a browser navigates from the upload page to the download page (or reloads), the new page sends a fresh ClientHello on the reused HTTP/2 connection. The server is already in `HandshakeAccepted` state for that connection, so it routes the request to `processRequest`, which expects a 16384-byte command block but receives a 34-byte ClientHello → `ERR BLOCK`. -**Root cause**: The server cannot distinguish a ClientHello from a command on an already-handshaked connection because both arrive on the same HTTP/2 connection (same `tlsUniq`), and there is no content-level discriminator (ClientHello is unpadded, but the server never gets to parse it — the size check in `processRequest` rejects it first). +**Root cause**: The router cannot distinguish a ClientHello from a command on an already-handshaked connection because both arrive on the same HTTP/2 connection (same `tlsUniq`), and there is no content-level discriminator (ClientHello is unpadded, but the router never gets to parse it — the size check in `processRequest` rejects it first). **Browser limitation**: `fetch()` provides zero control over HTTP/2 connection pooling. There is no browser API to force a new connection or detect connection reuse before a request is sent. ## 2. Solution Summary -Add an HTTP header `xftp-web-hello` to web ClientHello requests. When the server sees this header on an already-handshaked connection (`HandshakeAccepted` state), it re-runs `processHello` **reusing the existing session keys** (same X25519 key pair from the original handshake). The client then completes the normal handshake flow (sends ClientHandshake, receives ack) and proceeds with commands. +Add an HTTP header `xftp-web-hello` to web ClientHello requests. When the router sees this header on an already-handshaked connection (`HandshakeAccepted` state), it re-runs `processHello` **reusing the existing session keys** (same X25519 key pair from the original handshake). The client then completes the normal handshake flow (sends ClientHandshake, receives ack) and proceeds with commands. Key properties: -- Server reuses existing `serverPrivKey` — no new key material generated on re-handshake, so `thAuth` remains consistent with any in-flight commands on concurrent HTTP/2 streams. +- Router reuses existing `serverPrivKey` — no new key material generated on re-handshake, so `thAuth` remains consistent with any in-flight commands on concurrent HTTP/2 streams. - Header is only checked when `sniUsed` is true (web/browser connections). Native XFTP clients are unaffected. - CORS preflight already allows all headers (`Access-Control-Allow-Headers: *`). - Web clients always send this header on ClientHello — it's harmless on first connection (`Nothing` state) and enables re-handshake on reused connections (`HandshakeAccepted` state). ## 3. Detailed Technical Design -### 3.1 Server change: parameterize `processHello` (`src/Simplex/FileTransfer/Server.hs`) +### 3.1 Router change: parameterize `processHello` (`src/Simplex/FileTransfer/Server.hs`) -The entire server change is parameterizing the existing `processHello` with `Maybe C.PrivateKeyX25519`. Zero new functions. +The entire router change is parameterizing the existing `processHello` with `Maybe C.PrivateKeyX25519`. Zero new functions. #### Current code (lines 165-191): @@ -125,7 +125,7 @@ Add optional `headers?` parameter to `Transport.post()`, thread it through `fetc ### 3.5 Haskell test (`tests/XFTPServerTests.hs`) -Add `testWebReHandshake` next to the existing `testWebHandshake` (line 504). It reuses the same SNI + HTTP/2 setup pattern, performs a full handshake, then sends a second ClientHello with the `xftp-web-hello` header on the same connection and verifies the server responds with a valid ServerHandshake (same `sessionId`), then completes the second handshake. +Add `testWebReHandshake` next to the existing `testWebHandshake` (line 504). It reuses the same SNI + HTTP/2 setup pattern, performs a full handshake, then sends a second ClientHello with the `xftp-web-hello` header on the same connection and verifies the router responds with a valid ServerHandshake (same `sessionId`), then completes the second handshake. ```haskell -- Register in xftpServerTests (after line 86): @@ -170,7 +170,7 @@ The only difference from `testWebHandshake`: the second `helloReq2` passes `[("x ## 4. Implementation Plan -### Step 1: Server — parameterize `processHello` +### Step 1: Router — parameterize `processHello` Apply the diff from Section 3.1 to `src/Simplex/FileTransfer/Server.hs`. @@ -216,6 +216,6 @@ Tab A (upload) and Tab B (download) share the same HTTP/2 connection. ## 6. Security Considerations - **No new key material**: Re-handshake reuses existing `serverPrivKey`. No opportunity for key confusion or downgrade. -- **Identity re-verification**: Server re-signs the web challenge with its long-term signing key. Client verifies identity again. -- **Header cannot escalate privileges**: The header only triggers re-handshake (which the server was already capable of doing on first connection). It does not bypass any authentication. +- **Identity re-verification**: Router re-signs the web challenge with its long-term signing key. Client verifies identity again. +- **Header cannot escalate privileges**: The header only triggers re-handshake (which the router was already capable of doing on first connection). It does not bypass any authentication. - **Timing**: Re-handshake takes the same code path as initial handshake, so timing side-channels are unchanged. diff --git a/rfcs/2026-01-30-send-file-page/2026-02-11-xftp-web-error-handling.md b/rfcs/done/2026-01-30-send-file-page/2026-02-11-xftp-web-error-handling.md similarity index 81% rename from rfcs/2026-01-30-send-file-page/2026-02-11-xftp-web-error-handling.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-11-xftp-web-error-handling.md index 2802f16a5..522eb070e 100644 --- a/rfcs/2026-01-30-send-file-page/2026-02-11-xftp-web-error-handling.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-02-11-xftp-web-error-handling.md @@ -2,13 +2,13 @@ ## 1. Problem Statement -The XFTP web client is fundamentally fragile: any transient error (browser opening a new HTTP/2 connection, network hiccup, server restart) causes an unrecoverable failure with a cryptic error message. There is no retry logic, no fetch timeout, no error categorization, and the upload uses a single server instead of distributing chunks across preset servers. This makes the app frustrating — it works most of the time but fails unpredictably, which is worse than being completely broken. +The XFTP web client is fundamentally fragile: any transient error (browser opening a new HTTP/2 connection, network hiccup, router restart) causes an unrecoverable failure with a cryptic error message. There is no retry logic, no fetch timeout, no error categorization, and the upload uses a single router instead of distributing data packets across preset routers. This makes the app frustrating — it works most of the time but fails unpredictably, which is worse than being completely broken. ### Confirmed root cause (from diagnostic logs) -When the browser opens a new HTTP/2 connection mid-operation, the new connection has a different TLS SessionId with no handshake state in the server's `TMap SessionId Handshake`. The server's `Nothing` branch in `xftpServerHandshakeV1` (Server.hs:169) unconditionally calls `processHello`, which tries to decode the command body as `XFTPClientHello`, fails, and sends a raw padded "HANDSHAKE" error string. The client cannot parse this as a proper transmission (first byte 'H' = 72 is read as batch count), producing `"expected batch count 1, got 72"`. +When the browser opens a new HTTP/2 connection mid-operation, the new connection has a different TLS SessionId with no handshake state in the router's `TMap SessionId Handshake`. The router's `Nothing` branch in `xftpServerHandshakeV1` (Server.hs:169) unconditionally calls `processHello`, which tries to decode the command body as `XFTPClientHello`, fails, and sends a raw padded "HANDSHAKE" error string. The client cannot parse this as a proper transmission (first byte 'H' = 72 is read as batch count), producing `"expected batch count 1, got 72"`. -Server log confirming the SessionId change: +Router log confirming the SessionId change: ``` DEBUG dispatch: Accepted+command sessId="ZSo1GGETgIvjbB7CWHbvGPpbMjx_b2IlC1eTI6aKfqc=" ...20 successful commands... @@ -17,32 +17,32 @@ DEBUG dispatch: Nothing sessId="mJC7Sck9xxW5UsXoPGoUWduuHghSVgf6CnD6ZC6SBhU=" we ### Why re-handshake is required (cannot be made optional) -1. **SessionId is baked into signed command data.** `encodeAuthTransmission` signs `concat(encode(sessionId), tInner)` with Ed25519. Server's `tDecodeServer` (Protocol.hs:2242) verifies `sessId == sessionId`. New connection = different sessionId = signature mismatch. -2. **Server generates per-session DH keys.** `processHello` creates fresh X25519 keypair stored in `HandshakeSent`. For SMP browser clients (future), `verifyCmdAuth` (Protocol.hs:1322) requires the matching `serverPrivKey` from `thAuth`. +1. **SessionId is baked into signed command data.** `encodeAuthTransmission` signs `concat(encode(sessionId), tInner)` with Ed25519. Router's `tDecodeServer` (Protocol.hs:2242) verifies `sessId == sessionId`. New connection = different sessionId = signature mismatch. +2. **Router generates per-session DH keys.** `processHello` creates fresh X25519 keypair stored in `HandshakeSent`. For SMP browser clients (future), `verifyCmdAuth` (Protocol.hs:1322) requires the matching `serverPrivKey` from `thAuth`. 3. **This applies to both XFTP and future SMP browser clients** — the session management approach is the same. -### Why multiple preset servers cannot work +### Why multiple preset routers cannot work -Upload (`agent.ts:105-157`) takes a single `server: XFTPServer` parameter and uploads ALL chunks to it. `web/upload.ts:133` calls `pickRandomServer(servers)` which selects ONE random server from all presets. The multi-server preset configuration is pointless — only one server is ever used per upload. The design intent (RFC section 11.6: "upload in parallel to 8 randomly selected servers") is not implemented. This must be fixed in Phase 2 (section 3.7). +Upload (`agent.ts:105-157`) takes a single `server: XFTPServer` parameter and uploads ALL data packets to it. `web/upload.ts:133` calls `pickRandomServer(servers)` which selects ONE random router from all presets. The multi-router preset configuration is pointless — only one router is ever used per upload. The design intent (RFC section 11.6: "upload in parallel to 8 randomly selected routers") is not implemented. This must be fixed in Phase 2 (section 3.7). ## 2. Solution Summary ### Phase 1: Error handling and connection resilience -1. **Server: strict dispatch for allowed protocol combinations** — reject all invalid combinations +1. **Router: strict dispatch for allowed protocol combinations** — reject all invalid combinations 2. **Client: automatic retry with re-handshake** on SESSION/HANDSHAKE errors 3. **Client: fetch timeout** with configurable duration 4. **UI: error categorization and retry** — auto-retry temporary, human-readable permanent -5. **Client: connection state with Promise-based lock and per-server queues** — `ServerConnection` with `client: Promise` + `queue: Promise` +5. **Client: connection state with Promise-based lock and per-router queues** — `ServerConnection` with `client: Promise` + `queue: Promise` 6. **Client: fix cache key** — include keyHash -### Phase 2: Multi-server upload (after Phase 1) +### Phase 2: Multi-router upload (after Phase 1) -7. **Multi-server upload with server selection and failover** — distribute chunks across servers, retry FNEW on different server if one fails +7. **Multi-router upload with router selection and failover** — distribute data packets across routers, retry FNEW on different router if one fails ## 3. Detailed Technical Design -### 3.1 Server: strict dispatch for allowed protocol combinations +### 3.1 Router: strict dispatch for allowed protocol combinations **Principle:** Everything not explicitly done by existing Haskell/TS clients is prohibited. It is better to fail on impossible combinations than to be permissive — permissiveness complicates debugging and creates attack vectors via unexpected behaviors. @@ -88,14 +88,14 @@ Nothing | `FRErr SESSION` | Temporary | Yes (auto) | "Session expired, reconnecting..." | | `FRErr HANDSHAKE` | Temporary | Yes (auto) | "Connection interrupted, reconnecting..." | | `fetch()` TypeError | Temporary | Yes (auto) | "Network error, retrying..." | -| AbortError (timeout) | Temporary | Yes (auto) | "Server timeout, retrying..." | +| AbortError (timeout) | Temporary | Yes (auto) | "Router timeout, retrying..." | | `FRErr AUTH` | Permanent | No | "File is invalid, expired, or has been removed" | | `FRErr NO_FILE` | Permanent | No | "File not found — it may have expired" | -| `FRErr SIZE` | Permanent | No | "File size exceeds server limit" | -| `FRErr QUOTA` | Permanent | No | "Server storage quota exceeded" | -| `FRErr BLOCKED` | Permanent | No | "File has been blocked by server" | +| `FRErr SIZE` | Permanent | No | "File size exceeds router limit" | +| `FRErr QUOTA` | Permanent | No | "Router storage quota exceeded" | +| `FRErr BLOCKED` | Permanent | No | "File has been blocked by router" | | `FRErr DIGEST` | Permanent | No | "File integrity check failed" | -| `FRErr INTERNAL` | Permanent | No | "Server internal error" | +| `FRErr INTERNAL` | Permanent | No | "Router internal error" | | `CMD *` | Permanent | No | "Protocol error" | **Retry behavior:** @@ -156,7 +156,7 @@ if (raw.length < 20) { 2. **FRErr classification** (replaces current unconditional throw): ```typescript -// After decodeResponse, instead of throw new Error("Server error: " + err.type): +// After decodeResponse, instead of throw new Error("Router error: " + err.type): if (response.type === "FRErr") { const err = response.err if (err.type === "SESSION" || err.type === "HANDSHAKE") { @@ -206,30 +206,30 @@ Default: 30s for production, 5s for tests. Threaded through `connectXFTP` → `c **Behavior (Option D):** -- **Temporary errors:** Auto-retry loop (3 attempts). After 3 failures, show human-readable diagnosis with manual retry button. Diagnosis examples: "Server timeout — the server may be temporarily unavailable", "Connection interrupted — your network may be unstable". +- **Temporary errors:** Auto-retry loop (3 attempts). After 3 failures, show human-readable diagnosis with manual retry button. Diagnosis examples: "Router timeout — the router may be temporarily unavailable", "Connection interrupted — your network may be unstable". - **Permanent errors:** Show human-readable error immediately, NO retry button. User can reload page if they want to retry. Examples: "File is invalid, expired, or has been removed" (AUTH), "File not found" (NO_FILE). **Current UI retry buttons:** - `upload.ts:73-75` — retry calls `startUpload(pendingFile)` from scratch - `download.ts:60` — retry calls `startDownload()` from scratch -**Improvement:** Track uploaded/downloaded chunk indices. On manual retry, skip completed chunks: +**Improvement:** Track uploaded/downloaded data packet indices. On manual retry, skip completed data packets: ```typescript -// Upload: track which chunks completed +// Upload: track which data packets completed const completedChunks: Set = new Set() for (let i = 0; i < specs.length; i++) { if (completedChunks.has(i)) continue - // ... create + upload chunk + // ... create + upload data packet completedChunks.add(i) } -// Download: already naturally resumable — each chunk is independent +// Download: already naturally resumable — each data packet is independent ``` -### 3.5 Client: connection state with Promise-based lock and per-server queues +### 3.5 Client: connection state with Promise-based lock and per-router queues -**Design:** Each server gets a `ServerConnection` record containing a `Promise` (the connection lock) and a `Promise` (the sequential command queue). The `XFTPClientAgent` maps server keys to these records. +**Design:** Each router gets a `ServerConnection` record containing a `Promise` (the connection lock) and a `Promise` (the sequential command queue). The `XFTPClientAgent` maps router keys to these records. The promise IS the lock — every consumer awaits the same promise. When reconnect is needed, the promise is replaced atomically. @@ -325,7 +325,7 @@ function removeStaleConnection( } ``` -**Per-server sequential queue:** `queue` is a `Promise` — the tail of the sequential operation chain. Each new operation `.then()`s onto it. It's `void` because callers hold their own typed promises; the queue only tracks completion order: +**Per-router sequential queue:** `queue` is a `Promise` — the tail of the sequential operation chain. Each new operation `.then()`s onto it. It's `void` because callers hold their own typed promises; the queue only tracks completion order: ```typescript async function enqueueCommand( @@ -348,9 +348,9 @@ async function enqueueCommand( } ``` -Commands to the same server execute one at a time via the queue. Commands to different servers execute concurrently because each has its own queue. `enqueueCommand` provides sequencing; `sendXFTPCommand` (called inside `fn` via command wrappers) provides retry. They compose as: `enqueueCommand` sequences calls to wrappers that internally use `sendXFTPCommand`. +Commands to the same router execute one at a time via the queue. Commands to different routers execute concurrently because each has its own queue. `enqueueCommand` provides sequencing; `sendXFTPCommand` (called inside `fn` via command wrappers) provides retry. They compose as: `enqueueCommand` sequences calls to wrappers that internally use `sendXFTPCommand`. -**Download change:** Group chunks by server, process each server's chunks sequentially, servers in parallel. Uses `for` loop for per-server sequencing (same pattern as Stage 2 upload). `enqueueCommand` is available for cases where different callers target the same server. +**Download change:** Group data packets by router, process each router's data packets sequentially, routers in parallel. Uses `for` loop for per-router sequencing (same pattern as Stage 2 upload). `enqueueCommand` is available for cases where different callers target the same router. ```typescript const byServer = new Map() @@ -374,7 +374,7 @@ await Promise.all([...byServer.entries()].map(async ([srv, chunks]) => { ### 3.6 Fix cache key -**Bug:** `getXFTPServerClient` (client.ts:110) uses `"https://" + server.host + ":" + server.port` as cache key, ignoring `keyHash`. Two servers with same host:port but different keyHash share a cached connection, bypassing identity verification. +**Bug:** `getXFTPServerClient` (client.ts:110) uses `"https://" + server.host + ":" + server.port` as cache key, ignoring `keyHash`. Two routers with same host:port but different keyHash share a cached connection, bypassing identity verification. **Fix:** Use `formatXFTPServer(server)` as cache key (includes keyHash). Already available in `protocol/address.ts:52-54`. @@ -388,11 +388,11 @@ const key = formatXFTPServer(server) Note: With the redesign in 3.5, the cache key fix is inherent — the `connections` Map uses `formatXFTPServer(server)` everywhere. -### 3.7 Phase 2: Multi-server upload with server selection and failover +### 3.7 Phase 2: Multi-router upload with router selection and failover -**Problem:** Current upload (`agent.ts:105-157`) takes a single `server: XFTPServer` and uploads ALL chunks to it. The 12 preset servers (6 SimpleX + 6 Flux) are pointless — only one is ever used. +**Problem:** Current upload (`agent.ts:105-157`) takes a single `server: XFTPServer` and uploads ALL data packets to it. The 12 preset routers (6 SimpleX + 6 Flux) are pointless — only one is ever used. -**Design goal:** Distribute chunks across servers. Retry FNEW on a different server if one fails. Once working servers are found, prefer them (heuristic: server unlikely to fail mid-process, more likely to be broken initially due to maintenance/downtime). +**Design goal:** Distribute data packets across routers. Retry FNEW on a different router if one fails. Once working routers are found, prefer them (heuristic: router unlikely to fail mid-process, more likely to be broken initially due to maintenance/downtime). **Reference implementation:** Haskell `Agent.hs:457-486` (`createChunk` / `createWithNextSrv`) + `Client.hs:2335-2385` (`getNextServer_` / `withNextSrv`). @@ -400,13 +400,13 @@ Note: With the redesign in 3.5, the cache key fix is inherent — the `connectio Two-stage architecture: -1. **Allocate stage (serial per file in Haskell):** For each chunk, call FNEW on a randomly-selected server. If FNEW fails, pick a different server and retry. Track tried hosts to avoid retrying the same server. After all chunks are assigned to servers, spawn one upload worker per server. +1. **Allocate stage (serial per file in Haskell):** For each data packet, call FNEW on a randomly-selected router. If FNEW fails, pick a different router and retry. Track tried hosts to avoid retrying the same router. After all data packets are assigned to routers, spawn one upload worker per router. -2. **Upload stage (parallel per server):** Each server worker uploads its assigned chunks sequentially (FPUT). On FPUT failure, retry on the same server with backoff (because the chunk replica already exists on that server). No server failover for FPUT. +2. **Upload stage (parallel per router):** Each router worker uploads its assigned data packets sequentially (FPUT). On FPUT failure, retry on the same router with backoff (because the data packet replica already exists on that router). No router failover for FPUT. -Server selection constraints (hierarchical, `getNextServer_` Client.hs:2335-2350): -1. Prefer servers from unused operators (operator diversity) -2. Prefer servers with unused hosts (host diversity) +Router selection constraints (hierarchical, `getNextServer_` Client.hs:2335-2350): +1. Prefer routers from unused operators (operator diversity) +2. Prefer routers with unused hosts (host diversity) 3. Random pick from the most-constrained candidate set 4. If all exhausted, reset tried set and start over @@ -414,17 +414,17 @@ Server selection constraints (hierarchical, `getNextServer_` Client.hs:2335-2350 The web client doesn't have operators or a database. Simplified algorithm with two stages: -**Stage 1 — Allocate:** Create chunk records on servers (FNEW). Unlike Haskell which is serial here, web FNEW runs concurrently within a concurrency limit. FNEW is a small command — concurrent FNEW on the same connection is not a problem, and concurrent FNEW across servers improves upload startup time. +**Stage 1 — Allocate:** Create data packet records on routers (FNEW). Unlike Haskell which is serial here, web FNEW runs concurrently within a concurrency limit. FNEW is a small command — concurrent FNEW on the same connection is not a problem, and concurrent FNEW across routers improves upload startup time. -**Stage 2 — Upload:** Upload chunk data (FPUT). Parallel across servers, sequential per server (reuses per-server queues from 3.5). FPUT retries on the same server with backoff — no server rotation because the chunk replica already exists on that server. Stage 2 reads chunk data by offset (via `readChunk`), so `SentChunk` must be extended with `chunkOffset: number` (from ChunkSpec). +**Stage 2 — Upload:** Upload data packet content (FPUT). Parallel across routers, sequential per router (reuses per-router queues from 3.5). FPUT retries on the same router with backoff — no router rotation because the data packet replica already exists on that router. Stage 2 reads data packet content by offset (via `readChunk`), so `SentChunk` must be extended with `chunkOffset: number` (from ChunkSpec). ```typescript interface UploadState { - untriedServers: XFTPServer[] // servers not yet attempted — initially all servers - workingServers: XFTPServer[] // servers that succeeded FNEW + untriedServers: XFTPServer[] // routers not yet attempted — initially all routers + workingServers: XFTPServer[] // routers that succeeded FNEW } -const MAX_FNEW_ATTEMPTS = 5 // per chunk: try up to 5 different servers +const MAX_FNEW_ATTEMPTS = 5 // per data packet: try up to 5 different routers async function uploadFile( agent: XFTPClientAgent, @@ -455,7 +455,7 @@ async function uploadFile( ) await Promise.all(allocateWorkers) - // Stage 2: Upload — parallel across servers, sequential per server + // Stage 2: Upload — parallel across routers, sequential per router // readChunk reads from the encrypted file by offset (same as Phase 1 uploadFile) let uploaded = 0 const total = encrypted.chunkSizes.reduce((a, b) => a + b, 0) @@ -473,7 +473,7 @@ async function uploadFile( } ``` -**`createChunkWithFailover`** — server selection with per-chunk retry limit: +**`createChunkWithFailover`** — router selection with per-data-packet retry limit: ```typescript async function createChunkWithFailover( @@ -515,7 +515,7 @@ function pickServer( state: UploadState, concurrency: number ): XFTPServer { - // Once enough working servers found, only use those + // Once enough working routers found, only use those if (state.workingServers.length >= concurrency) { return randomPick(state.workingServers) } @@ -524,7 +524,7 @@ function pickServer( const idx = Math.floor(Math.random() * state.untriedServers.length) return state.untriedServers.splice(idx, 1)[0] // remove from untried } - // All tried — reset untried to non-working servers and retry + // All tried — reset untried to non-working routers and retry state.untriedServers = allServers.filter( s => !state.workingServers.some(w => formatXFTPServer(w) === formatXFTPServer(s)) ) @@ -532,22 +532,22 @@ function pickServer( const idx = Math.floor(Math.random() * state.untriedServers.length) return state.untriedServers.splice(idx, 1)[0] } - // Every server is working — pick any working + // Every router is working — pick any working return randomPick(state.workingServers) } ``` -**Algorithm:** Two lists — `untriedServers` (initially all) and `workingServers` (initially empty). When `workingServers.length < concurrency`, pick from `untriedServers` (removing on pick). On FNEW success, add to `workingServers`. On FNEW failure, server is already removed from `untriedServers`; remove from `workingServers` if present. When `untriedServers` is empty, reset it to all non-working servers. Once `workingServers.length >= concurrency`, pick randomly only from `workingServers`. +**Algorithm:** Two lists — `untriedServers` (initially all) and `workingServers` (initially empty). When `workingServers.length < concurrency`, pick from `untriedServers` (removing on pick). On FNEW success, add to `workingServers`. On FNEW failure, router is already removed from `untriedServers`; remove from `workingServers` if present. When `untriedServers` is empty, reset it to all non-working routers. Once `workingServers.length >= concurrency`, pick randomly only from `workingServers`. -**Termination condition:** Each chunk tries at most `min(serverCount, 5)` different servers. If all attempts fail, the chunk fails and the upload fails with the last error. Rationale: if 5 out of 12 servers are down, something systemic is wrong and continuing is unlikely to help. Timeouts count as failures — the timed-out server is removed from working and a different server is picked next. +**Termination condition:** Each data packet tries at most `min(routerCount, 5)` different routers. If all attempts fail, the data packet fails and the upload fails with the last error. Rationale: if 5 out of 12 routers are down, something systemic is wrong and continuing is unlikely to help. Timeouts count as failures — the timed-out router is removed from working and a different router is picked next. **Key differences from Haskell:** - No operator concept — just host diversity via random selection - No database — state tracked in-memory during upload - FNEW runs concurrently (Haskell is serial) — improves startup time -- FNEW is cheap and retried with server rotation; FPUT retries on same server +- FNEW is cheap and retried with router rotation; FPUT retries on same router -**Download changes (also Phase 2):** Default concurrency should be 4 (matching Haskell). Download already groups by server in 3.5. If `replicas[0]` download fails, try `replicas[1]`, `replicas[2]`, etc. (fallback across replicas). +**Download changes (also Phase 2):** Default concurrency should be 4 (matching Haskell). Download already groups by router in 3.5. If `replicas[0]` download fails, try `replicas[1]`, `replicas[2]`, etc. (fallback across replicas). ## 4. Implementation Plan @@ -560,7 +560,7 @@ Steps are ordered by dependency and should be implemented one by one. - Add import for `formatXFTPServer` - Run existing tests to verify no regression -#### Step 2: Typed error detection for padded server errors (3.2 client-side) +#### Step 2: Typed error detection for padded router errors (3.2 client-side) - Add `XFTPRetriableError` class - In `sendXFTPCommand`, detect padded error strings before `decodeTransmission` - Classify `FRErr` responses as retriable or permanent with human-readable messages @@ -573,16 +573,16 @@ Steps are ordered by dependency and should be implemented one by one. - Add vitest test: timeout triggers after configured duration - Run existing tests -#### Step 4: Connection state with Promise-based lock and per-server queues (3.5) +#### Step 4: Connection state with Promise-based lock and per-router queues (3.5) - Introduce `ServerConnection` record: `{client: Promise, queue: Promise}` - Replace `XFTPClientAgent.clients: Map` with `connections: Map` - Implement `reconnectClient` — replaces `conn.client` with new promise, preserves queue -- Implement `enqueueCommand` — chains operation onto server's queue +- Implement `enqueueCommand` — chains operation onto router's queue - Implement `removeStaleConnection` — removes entry only if current promise is the failed one - Auto-cleanup: `p.catch(() => delete)` removes failed connections so next caller starts fresh - Adapt `closeXFTPServerClient` and `closeXFTPAgent` - Add vitest tests: - - Concurrent calls to same server produce single connection + - Concurrent calls to same router produce single connection - Failed promise is cleaned up, next caller gets fresh connection #### Step 5: Automatic retry in sendXFTPCommand (3.2) @@ -594,61 +594,61 @@ Steps are ordered by dependency and should be implemented one by one. - Max 3 retries for retriable errors, immediate throw for permanent - On retriable error: call `reconnectClient` and retry. On retriable error exhausted: call `removeStaleConnection` to clean up. On permanent error: throw immediately without touching connection - Add vitest tests: - - Server started with delay → first attempt fails, retry succeeds + - Router started with delay → first attempt fails, retry succeeds - 3 retries exhausted → error propagates with human-readable message - Non-retriable error (AUTH) → no retry, immediate failure -#### Step 6: Server-side stale session handling (3.1) +#### Step 6: Router-side stale session handling (3.1) - Add one guard to `Nothing` branch: `sniUsed && not webHello -> throwE SESSION` - Remove debug `hPutStrLn stderr` lines (all 6 occurrences in dispatch) - All other branches unchanged - Run Haskell tests + Playwright tests -#### Step 7: Download with per-server grouping -- Modify `downloadFileRaw` to group chunks by server, sequential within each server (`for` loop), parallel across servers (`Promise.all`) -- Add vitest test: concurrent downloads from different servers run in parallel +#### Step 7: Download with per-router grouping +- Modify `downloadFileRaw` to group data packets by router, sequential within each router (`for` loop), parallel across routers (`Promise.all`) +- Add vitest test: concurrent downloads from different routers run in parallel #### Step 8: UI error improvements (3.4) - Temporary errors: auto-retry loop (3 attempts), then show human-readable diagnosis + manual retry button - Permanent errors: show human-readable error, NO retry button -- Manual retry resumes from last successful chunk (not full restart) +- Manual retry resumes from last successful data packet (not full restart) #### Step 9: Remove debug logging - Remove all `console.log('[DEBUG ...]')` and `hPutStrLn stderr "DEBUG ..."` lines - Keep `console.error('[XFTP] ...')` error logging -### Phase 2: Multi-server upload +### Phase 2: Multi-router upload Implement after Phase 1 is complete and tested. -#### Step 10: Multi-server upload with failover (3.7) -- Extend `SentChunk` with `chunkOffset: number` (from ChunkSpec) and `server: XFTPServer` (assigned during allocate) — Stage 2 reads data by offset and groups chunks by server +#### Step 10: Multi-router upload with failover (3.7) +- Extend `SentChunk` with `chunkOffset: number` (from ChunkSpec) and `server: XFTPServer` (assigned during allocate) — Stage 2 reads data by offset and groups data packets by router - Change `uploadFile` signature: takes `allServers: XFTPServer[]` instead of single `server` - Implement `UploadState` with `untriedServers` and `workingServers` -- Implement `createChunkWithFailover` and `pickServer`: two-list selection (untried → working once enough found), max `min(serverCount, 5)` attempts per chunk +- Implement `createChunkWithFailover` and `pickServer`: two-list selection (untried → working once enough found), max `min(routerCount, 5)` attempts per data packet - Allocate stage: concurrent FNEW within concurrency limit (default 4) -- Upload stage: parallel across servers, sequential per server (reuse queue from Step 7) +- Upload stage: parallel across routers, sequential per router (reuse queue from Step 7) - Update `web/upload.ts`: pass `getServers()` instead of `pickRandomServer(getServers())` -- Update description building: each chunk references its actual server +- Update description building: each data packet references its actual router - Add vitest tests: - - File split across N servers (verify different servers in description) - - One server down → chunks redistributed to others - - All servers down → error after exhausting 5 attempts per chunk + - File split across N routers (verify different routers in description) + - One router down → data packets redistributed to others + - All routers down → error after exhausting 5 attempts per data packet #### Step 11: Download concurrency and replica fallback - Change default download concurrency from 1 to 4 - If `replicas[0]` download fails, try `replicas[1]`, `replicas[2]`, etc. -- Uses per-server queues from Step 7 +- Uses per-router queues from Step 7 ## 5. Testing Plan ### Principle -Prefer low-level vitest tests over Playwright E2E. Each new function gets one focused test. Pure functions tested without mocks; connection management tested with mock `connectXFTP`; server behavior tested with real server. Total: 13 tests across 4 files. +Prefer low-level vitest tests over Playwright E2E. Each new function gets one focused test. Pure functions tested without mocks; connection management tested with mock `connectXFTP`; router behavior tested with real router. Total: 13 tests across 4 files. -Tests A-C run in browser context (`@vitest/browser` with Chromium headless), configured in `vitest.config.ts`. Test D (integration) requires a separate Node.js vitest config since it uses `node:http2`. Existing `globalSetup.ts` provides a real XFTP server for integration tests. +Tests A-C run in browser context (`@vitest/browser` with Chromium headless), configured in `vitest.config.ts`. Test D (integration) requires a separate Node.js vitest config since it uses `node:http2`. Existing `globalSetup.ts` provides a real XFTP router for integration tests. -### Test file A: `test/errors.test.ts` — pure, no server +### Test file A: `test/errors.test.ts` — pure, no router Tests error classification and padded error detection (Steps 2, 5). @@ -682,7 +682,7 @@ expect(re.message).toContain("expired") // "Session expired, reconnecting..." **T3. Padded error detection extracts error string from padded block** ```typescript import {blockPad, blockUnpad} from '../src/protocol/transmission.js' -// Simulate server sending padded "SESSION" +// Simulate router sending padded "SESSION" const padded = blockPad(new TextEncoder().encode("SESSION")) const raw = blockUnpad(padded) expect(raw.length).toBeLessThan(20) @@ -694,7 +694,7 @@ const normalRaw = blockUnpad(normalBlock) expect(normalRaw.length).toBeGreaterThan(20) // not mistaken for padded error ``` -### Test file B: `test/connection.test.ts` — mock connectXFTP, no server +### Test file B: `test/connection.test.ts` — mock connectXFTP, no router Tests connection management functions (Steps 4, 5). Uses `vi.mock` to replace `connectXFTP` with a controllable promise factory. @@ -800,7 +800,7 @@ await expect(sendXFTPCommand(agent3, server, dummyKey, dummyId, encodePING())) expect(vi.mocked(connectXFTP)).toHaveBeenCalledTimes(1) // initial only, no reconnect ``` -### Test file C: `test/server-selection.test.ts` — pure, no server +### Test file C: `test/server-selection.test.ts` — pure, no router Tests `pickServer` state machine (Step 10). Determinism: seed `Math.random` or test invariants not specific picks. @@ -833,12 +833,12 @@ const state: UploadState = { workingServers: [s1, s2] // only 2 working, concurrency=4 } const picked = pickServer(servers, state, 4) -// Should have reset untried to non-working servers and picked from them +// Should have reset untried to non-working routers and picked from them expect([s3, s4, s5]).toContainEqual(picked) expect(state.untriedServers.length).toBe(2) // 3 non-working minus 1 picked ``` -### Test file D: `test/integration.test.ts` — real server, Node.js mode +### Test file D: `test/integration.test.ts` — real router, Node.js mode Requires separate vitest config with `browser: {enabled: false}` since these tests use `node:http2` directly. Alternatively, add `test/vitest.node.config.ts` that includes only `test/integration.test.ts` and runs in Node.js. @@ -847,10 +847,10 @@ Requires separate vitest config with `browser: {enabled: false}` since these tes import http2 from 'node:http2' // Connect and handshake normally via the client const client = await connectXFTP(server) -// Create a raw HTTP/2 session (new TLS SessionId, no handshake state on server) +// Create a raw HTTP/2 session (new TLS SessionId, no handshake state on router) const session = http2.connect(client.baseUrl, {rejectUnauthorized: false}) // Build a dummy command block using the old client's sessionId. -// Content doesn't matter — server detects stale session before parsing command. +// Content doesn't matter — router detects stale session before parsing command. const dummyKey = new Uint8Array(64) // Ed25519 private key (dummy) const dummyId = new Uint8Array(24) // entity ID (dummy) const cmdBlock = encodeAuthTransmission(client.sessionId, new Uint8Array(0), dummyId, encodePING(), dummyKey) @@ -862,7 +862,7 @@ const resp = await new Promise((resolve, reject) => { req.on("error", reject) req.end(Buffer.from(cmdBlock)) }) -// Server should return padded "SESSION" (not crash, not "HANDSHAKE") +// Router should return padded "SESSION" (not crash, not "HANDSHAKE") const raw = blockUnpad(resp.subarray(0, XFTP_BLOCK_SIZE)) expect(new TextDecoder().decode(raw)).toBe("SESSION") session.close() @@ -885,7 +885,7 @@ await expect( | Cache key fix (Step 1) | Existing round-trip test — uses `formatXFTPServer` after refactor | | Basic upload/download | 24 Playwright tests + 1 vitest browser test | | File size limits, unicode filenames | Playwright edge case tests | -| Server startup/teardown | `globalSetup.ts` / `globalTeardown.ts` | +| Router startup/teardown | `globalSetup.ts` / `globalTeardown.ts` | | Handshake + identity verification | `connectXFTP` in existing round-trip test | ### Test ordering @@ -895,7 +895,7 @@ Tests must be added alongside their implementation step: - **Step 3**: Add T13 (test/integration.test.ts) — requires Node.js vitest config - **Step 4**: Add T4, T5, T6, T7 (test/connection.test.ts) - **Step 5**: Add T8 (test/connection.test.ts) -- **Step 6**: Add T12 (test/integration.test.ts) — requires server change + Node.js vitest config +- **Step 6**: Add T12 (test/integration.test.ts) — requires router change + Node.js vitest config - **Step 10**: Add T9, T10, T11 (test/server-selection.test.ts) ## 6. Context for Implementation Sessions @@ -914,30 +914,30 @@ Tests must be added alongside their implementation step: - `web/servers.ts` — `getServers`, `pickRandomServer` **TypeScript (xftp-web/test/):** -- `browser.test.ts` — vitest Node.js test template (uses real Haskell server) -- `globalSetup.ts` — server startup, config generation, port file +- `browser.test.ts` — vitest Node.js test template (uses real Haskell router) +- `globalSetup.ts` — router startup, config generation, port file - `page.spec.ts` — Playwright page tests -**Haskell (reference for multi-server):** -- `src/Simplex/FileTransfer/Agent.hs` — `createChunk` (lines 457-486, allocate stage), `runXFTPSndPrepareWorker` (lines 391-430, serial allocate in Haskell), `runXFTPSndWorker` (lines 494-548, per-server upload worker) +**Haskell (reference for multi-router):** +- `src/Simplex/FileTransfer/Agent.hs` — `createChunk` (lines 457-486, allocate stage), `runXFTPSndPrepareWorker` (lines 391-430, serial allocate in Haskell), `runXFTPSndWorker` (lines 494-548, per-router upload worker) - `src/Simplex/Messaging/Agent/Client.hs` — `getNextServer_` (lines 2335-2350), `withNextSrv` (lines 2366-2385), `pickServer` (lines 2309-2314) -**Haskell (server):** +**Haskell (router):** - `src/Simplex/FileTransfer/Server.hs` — `xftpServerHandshakeV1` (lines 165-244), `processRequest` (lines 403-435) - `src/Simplex/Messaging/Protocol.hs` — `tDecodeServer` (lines 2239-2265) — sessionId verification at line 2242 ### Key design constraints 1. `tDecodeServer` (Protocol.hs:2242) verifies `sessId == sessionId` — commands signed with old sessionId WILL fail on new connection -2. Server generates per-session DH key in `processHello` (Server.hs:207) — cannot be shared across sessions +2. Router generates per-session DH key in `processHello` (Server.hs:207) — cannot be shared across sessions 3. `fetch()` provides zero control over HTTP/2 connection reuse — browser decides 4. `xftp-web-hello` header is only checked in dispatch (Server.hs:192), NOT inside `processHello` 5. Handshake-phase errors are raw padded strings; command-phase errors are proper ERR transmissions 6. Ed25519 signature verification (`TASignature` path, Protocol.hs:1314) does NOT use `thAuth` — but SMP will -7. Reconnect must re-handshake to get new sessionId AND new server DH key +7. Reconnect must re-handshake to get new sessionId AND new router DH key 8. The new `throwE SESSION` guard (Step 6) sends a raw padded "SESSION" string — no sessionId framing. Client detects this via padded error heuristic (section 3.2), not via sessionId mismatch -9. FNEW is cheap (creates chunk record on server) — retry with different server on failure -10. FPUT retries on same server (chunk replica already exists there) — close connection + backoff +9. FNEW is cheap (creates data packet record on router) — retry with different router on failure +10. FPUT retries on same router (data packet replica already exists there) — close connection + backoff ## 7. Plan Maintenance diff --git a/rfcs/2026-01-30-send-file-page/2026-02-12-xftp-cli-web-link-compat.md b/rfcs/done/2026-01-30-send-file-page/2026-02-12-xftp-cli-web-link-compat.md similarity index 87% rename from rfcs/2026-01-30-send-file-page/2026-02-12-xftp-cli-web-link-compat.md rename to rfcs/done/2026-01-30-send-file-page/2026-02-12-xftp-cli-web-link-compat.md index 14b7187e9..2fa4ca791 100644 --- a/rfcs/2026-01-30-send-file-page/2026-02-12-xftp-cli-web-link-compat.md +++ b/rfcs/done/2026-01-30-send-file-page/2026-02-12-xftp-cli-web-link-compat.md @@ -12,8 +12,8 @@ Make CLI produce and consume web-compatible links so that: - CLI `recv` accepts a web link URL as input (alternative to `.xftp` file path) - Browser can download files uploaded by CLI and vice versa -The web page host is derived from the XFTP server address - the server that hosts the file -also hosts the download page. Making XFTP servers actually serve the web page is a separate +The web page host is derived from the XFTP router address - the router that hosts the file +also hosts the download page. Making XFTP routers actually serve the web page is a separate concern (not covered here), but the link format anticipates it. The YAML file description format is already identical between CLI and web. @@ -33,7 +33,7 @@ Encoding chain (agent.ts:64-68): 3. `pako.deflateRaw(bytes)` -> compressed 4. `base64urlEncode(compressed)` -> URI fragment (no `#`) -For multi-chunk files exceeding ~400 chars in URI, a redirect description is uploaded: +For multi-packet files exceeding ~400 chars in URI, a redirect description is uploaded: the real file description is encrypted, uploaded as a separate XFTP file, and a smaller "redirect" description (pointing to it) is put in the URI. @@ -111,7 +111,7 @@ Extracts the actual filename from the path and embeds it in the encrypted header #### CLI download: uses filename from header (ok) -`Crypto.hs:62-66` (single chunk) / `Crypto.hs:72-74` (multi-chunk): +`Crypto.hs:62-66` (single data packet) / `Crypto.hs:72-74` (multi-packet): ```haskell (FileHeader {fileName}, rest) <- parseFileHeader decryptedContent destFile <- withExceptT FTCEFileIOError $ getDestFile fileName @@ -163,19 +163,19 @@ The CLI should consider adding filename sanitization similar to the web client f ### 2. Web Link Host Derivation -The web page URL domain comes from the XFTP server address, not from a CLI flag: +The web page URL domain comes from the XFTP router address, not from a CLI flag: -- **Non-redirected description**: use the server host of the first chunk's first replica. +- **Non-redirected description**: use the router host of the first data packet's first replica. E.g., `xftp://abc=@xftp1.simplex.im` -> `https://xftp1.simplex.im/#` -- **Redirected description**: use the server host of the redirect chunk (the outer description's - chunk that stores the encrypted inner description). +- **Redirected description**: use the router host of the redirect data packet (the outer description's + data packet that stores the encrypted inner description). -The server address format is `xftp://@[,,...][:]`. +The router address format is `xftp://@[,,...][:]`. The web link uses `https://` (port 443 implied). -This means the CLI does not need a `--web-url` flag - the server address fully determines -the link. The XFTP server serving the web page is a separate deployment concern. +This means the CLI does not need a `--web-url` flag - the router address fully determines +the link. The XFTP router serving the web page is a separate deployment concern. ### 3. Web URI Encoding/Decoding in Haskell @@ -196,7 +196,7 @@ decodeWebURI :: ByteString -> Either String (ValidFileDescription 'FRecipient) -- 4. validateFileDescription -- Build full web link from file description --- Extracts server host from first chunk replica (or redirect chunk) +-- Extracts router host from first data packet replica (or redirect data packet) fileWebLink :: FileDescription 'FRecipient -> (String, ByteString) -- Returns (webHost, uriFragment) -- Caller assembles: "https://" <> webHost <> "/#" <> uriFragment @@ -210,20 +210,20 @@ The `zlib` Haskell package provides `Codec.Compression.Zlib.Raw` for raw DEFLATE ### 4. Redirect Description Support -The CLI currently does NOT create redirect descriptions. For single-server single-recipient -uploads, most file descriptions fit in a reasonable URI even for multi-chunk files. But for -large files (many chunks x long server hostnames), the URI can exceed practical limits. +The CLI currently does NOT create redirect descriptions. For single-router single-recipient +uploads, most file descriptions fit in a reasonable URI even for multi-packet files. But for +large files (many data packets x long router hostnames), the URI can exceed practical limits. **Approach**: Match the web client threshold. -- After encoding the URI, if `length > 400` and chunks > 1, upload a redirect description. +- After encoding the URI, if `length > 400` and data packets > 1, upload a redirect description. - The redirect upload uses the same XFTP upload flow: encrypt YAML -> upload as file -> create outer description pointing to it. - This matches `agent.ts:152-155` exactly. -- The redirect chunk's server becomes the web link host. +- The redirect data packet's router becomes the web link host. For CLI download from a redirect URI, the existing `cliReceiveFile` needs extension: - After decoding the file description, check `redirect` field. -- If present: download and decrypt the redirect chunks first to get the inner description, +- If present: download and decrypt the redirect data packets first to get the inner description, then download the actual file using the inner description. - The web client already does this (`resolveRedirect` in agent.ts:320-346). @@ -281,16 +281,16 @@ Already identical. The web `description.ts` explicitly matches Haskell `Data.Yam Adding a cross-client test (CLI upload -> web download, or web upload -> CLI download) would validate interop end-to-end. -### 7. Server Compatibility +### 7. Router Compatibility -No server changes needed. Both clients use the same XFTP protocol (FGET, FPUT, FNEW, FACK, FDEL). +No router changes needed. Both clients use the same XFTP protocol (FGET, FPUT, FNEW, FACK, FDEL). The web client adds `xftp-web-hello: 1` header for the hello handshake, but the actual file operations are identical wire-format. The only consideration: CLI uses native HTTP/2 (via `http2` Haskell package), web uses browser `fetch()` API over HTTP/2. Both produce identical XFTP protocol frames. -**Note**: Making XFTP servers actually serve the web download page at `https:///` is a +**Note**: Making XFTP routers actually serve the web download page at `https:///` is a separate deployment/infrastructure task. This plan only establishes the link format convention so that links are ready to work once servers serve the page. @@ -301,7 +301,7 @@ so that links are ready to work once servers serve the page. 1. Add `zlib` dependency to `simplexmq.cabal` 2. Add `encodeWebURI` / `decodeWebURI` / `fileWebLink` to `Simplex.FileTransfer.Description` (or a new `Simplex.FileTransfer.Description.WebURI` module) -3. `fileWebLink` extracts host from first chunk's first replica server address +3. `fileWebLink` extracts host from first data packet's first replica router address 4. Add unit tests: encode a known FileDescription, verify output matches web client encoding 5. Add round-trip test: encode -> decode -> compare @@ -309,7 +309,7 @@ so that links are ready to work once servers serve the page. 1. Modify `ReceiveOptions` to accept `Either FilePath WebURL` for `fileDescription` 2. In `cliReceiveFile`: if URL, extract fragment after `#`, call `decodeWebURI` -3. Add redirect resolution: if `redirect /= Nothing`, download redirect chunks, +3. Add redirect resolution: if `redirect /= Nothing`, download redirect data packets, decrypt, parse inner description, then proceed with download 4. Test: upload via web page -> copy link -> `xftp recv ` diff --git a/rfcs/done/2022-07-22-access-via-tor.md b/rfcs/standard/2026-03-09-access-via-tor.md similarity index 94% rename from rfcs/done/2022-07-22-access-via-tor.md rename to rfcs/standard/2026-03-09-access-via-tor.md index b4517d403..16ed1a072 100644 --- a/rfcs/done/2022-07-22-access-via-tor.md +++ b/rfcs/standard/2026-03-09-access-via-tor.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-07-22 +Implemented: ~2022-08 +Standardized: 2026-03-09 +Protocol: simplex-messaging +--- + # Accessing SMP servers via Tor ## Problem diff --git a/rfcs/done/2021-01-26-crypto.md b/rfcs/standard/2026-03-09-crypto.md similarity index 96% rename from rfcs/done/2021-01-26-crypto.md rename to rfcs/standard/2026-03-09-crypto.md index 39ca6eb70..9bf06c0d2 100644 --- a/rfcs/done/2021-01-26-crypto.md +++ b/rfcs/standard/2026-03-09-crypto.md @@ -1,3 +1,12 @@ +--- +Proposed: 2021-01-26 +Implemented: ~2022 +Standardized: 2026-03-09 +Protocol: simplex-messaging v1, evolved through v7 +--- + +> **Implementation note:** All cryptographic primitives changed from this proposal. Transport: TLS 1.2/1.3 replaced the custom RSA handshake. E2E: Double ratchet with AES-GCM replaced per-message RSA-OAEP encryption. Auth: Ed25519/X25519 DH-based authenticated encryption (SMP v7) replaced RSA-PSS signatures. The transmission format (signature CRLF signed) was implemented as proposed. + # SMP agent: cryptography 3 main directions of work to enable basic level of security for communication via SMP agents and servers at the current stage of the project: diff --git a/rfcs/done/2022-06-13-db-sync.md b/rfcs/standard/2026-03-09-db-sync.md similarity index 98% rename from rfcs/done/2022-06-13-db-sync.md rename to rfcs/standard/2026-03-09-db-sync.md index dc375e20f..a7e32ad8d 100644 --- a/rfcs/done/2022-06-13-db-sync.md +++ b/rfcs/standard/2026-03-09-db-sync.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-06-13 +Implemented: ~2022-06 +Standardized: 2026-03-09 +Protocol: agent-protocol +--- + # DB access and processing messages for iOS notification service extension ## Problem diff --git a/rfcs/done/2022-06-13-db-sync.mmd b/rfcs/standard/2026-03-09-db-sync.mmd similarity index 95% rename from rfcs/done/2022-06-13-db-sync.mmd rename to rfcs/standard/2026-03-09-db-sync.mmd index 022f57d65..8eefc0ead 100644 --- a/rfcs/done/2022-06-13-db-sync.mmd +++ b/rfcs/standard/2026-03-09-db-sync.mmd @@ -1,3 +1,10 @@ +--- +Proposed: 2022-06-13 +Implemented: ~2022-06 +Standardized: 2026-03-09 +Protocol: agent-protocol +--- + sequenceDiagram participant M as iOS message
notification participant S as iOS system diff --git a/rfcs/done/2023-05-03-delivery-receipts.md b/rfcs/standard/2026-03-09-delivery-receipts.md similarity index 99% rename from rfcs/done/2023-05-03-delivery-receipts.md rename to rfcs/standard/2026-03-09-delivery-receipts.md index bc5658f96..e3887f863 100644 --- a/rfcs/done/2023-05-03-delivery-receipts.md +++ b/rfcs/standard/2026-03-09-delivery-receipts.md @@ -1,3 +1,10 @@ +--- +Proposed: 2023-05-03 +Implemented: 2023-07-13 +Standardized: 2026-03-09 +Protocol: agent-protocol v4 +--- + # Delivery receipts ## Problems diff --git a/rfcs/done/2024-02-03-deniability.md b/rfcs/standard/2026-03-09-deniability.md similarity index 98% rename from rfcs/done/2024-02-03-deniability.md rename to rfcs/standard/2026-03-09-deniability.md index b7bd3f7c5..22993263d 100644 --- a/rfcs/done/2024-02-03-deniability.md +++ b/rfcs/standard/2026-03-09-deniability.md @@ -1,3 +1,10 @@ +--- +Proposed: 2024-02-03 +Implemented: 2024-04-30 +Standardized: 2026-03-09 +Protocol: simplex-messaging v7 +--- + # Repudiation for message senders ## Problem diff --git a/rfcs/done/2024-06-14-fast-connection.md b/rfcs/standard/2026-03-09-fast-connection.md similarity index 96% rename from rfcs/done/2024-06-14-fast-connection.md rename to rfcs/standard/2026-03-09-fast-connection.md index 000f0ef10..81bd04cb2 100644 --- a/rfcs/done/2024-06-14-fast-connection.md +++ b/rfcs/standard/2026-03-09-fast-connection.md @@ -1,3 +1,10 @@ +--- +Proposed: 2024-06-14 +Implemented: 2024-06-30 +Standardized: 2026-03-09 +Protocol: simplex-messaging v9, agent-protocol v6 +--- + # Faster connection establishment ## Problem diff --git a/rfcs/done/2024-01-26-file-links.md b/rfcs/standard/2026-03-09-file-links.md similarity index 98% rename from rfcs/done/2024-01-26-file-links.md rename to rfcs/standard/2026-03-09-file-links.md index 3ff2f430e..8de727b35 100644 --- a/rfcs/done/2024-01-26-file-links.md +++ b/rfcs/standard/2026-03-09-file-links.md @@ -1,3 +1,10 @@ +--- +Proposed: 2024-01-26 +Implemented: ~2024-01 +Standardized: 2026-03-09 +Protocol: xftp +--- + # Sending large file descriptions It is desirable to provide a QR code/URI from which a file can be downloaded. This way files may be addressed outside a chat client. diff --git a/rfcs/done/2021-01-20-logging.md b/rfcs/standard/2026-03-09-logging.md similarity index 85% rename from rfcs/done/2021-01-20-logging.md rename to rfcs/standard/2026-03-09-logging.md index ae84f8e69..9d2341997 100644 --- a/rfcs/done/2021-01-20-logging.md +++ b/rfcs/standard/2026-03-09-logging.md @@ -1,3 +1,12 @@ +--- +Proposed: 2021-01-20 +Implemented: ~2021 +Standardized: 2026-03-09 +Protocol: agent-protocol +--- + +> **Implementation note:** Logging infrastructure exists but the format evolved from the proposed ASCII art format to structured server statistics, TLS error logging, and Prometheus metrics. + # SMP agent logging ## Problem and proposed solution. diff --git a/rfcs/done/2021-01-26-messages.md b/rfcs/standard/2026-03-09-messages.md similarity index 89% rename from rfcs/done/2021-01-26-messages.md rename to rfcs/standard/2026-03-09-messages.md index 71db02408..bcfb43c9a 100644 --- a/rfcs/done/2021-01-26-messages.md +++ b/rfcs/standard/2026-03-09-messages.md @@ -1,3 +1,12 @@ +--- +Proposed: 2021-01-26 +Implemented: ~2022 +Standardized: 2026-03-09 +Protocol: agent-protocol, simplex-messaging v2 +--- + +> **Implementation note:** Phase 1 (agent auto-ACK, store in DB, forward to client on SUB) is implemented. The GET command was added in SMP v2 for iOS NSE message retrieval. Phases 2 and 3 (fine-grained MGET/MDEL/MACK commands and autonomous agent with background polling) were not implemented. + # SMP Agent: message management The proposal is to change the way SMP agent manages the messages from the SMP servers. diff --git a/rfcs/done/2022-03-22-nofication-server.md b/rfcs/standard/2026-03-09-nofication-server.md similarity index 96% rename from rfcs/done/2022-03-22-nofication-server.md rename to rfcs/standard/2026-03-09-nofication-server.md index eebb94861..eee122ca0 100644 --- a/rfcs/done/2022-03-22-nofication-server.md +++ b/rfcs/standard/2026-03-09-nofication-server.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-03-22 +Implemented: ~2022 +Standardized: 2026-03-09 +Protocol: push-notifications v1 +--- + # Notification server ## Background and motivation diff --git a/rfcs/done/2021-05-17-open-connection.md b/rfcs/standard/2026-03-09-open-connection.md similarity index 95% rename from rfcs/done/2021-05-17-open-connection.md rename to rfcs/standard/2026-03-09-open-connection.md index 02eec21f4..090a5c17b 100644 --- a/rfcs/done/2021-05-17-open-connection.md +++ b/rfcs/standard/2026-03-09-open-connection.md @@ -1,3 +1,10 @@ +--- +Proposed: 2021-05-17 +Implemented: ~2021 +Standardized: 2026-03-09 +Protocol: agent-protocol v1 +--- + # Open connections ## Problem diff --git a/rfcs/done/2024-03-03-pqdr-version.md b/rfcs/standard/2026-03-09-pqdr-version.md similarity index 94% rename from rfcs/done/2024-03-03-pqdr-version.md rename to rfcs/standard/2026-03-09-pqdr-version.md index 5db9f23a5..c05051c7a 100644 --- a/rfcs/done/2024-03-03-pqdr-version.md +++ b/rfcs/standard/2026-03-09-pqdr-version.md @@ -1,3 +1,12 @@ +--- +Proposed: 2024-03-03 +Implemented: 2024-03-14 +Standardized: 2026-03-09 +Protocol: agent-protocol v5 +--- + +> **Implementation note:** PQ version negotiation and per-connection PQ mode are implemented. The proposed `RatchetVR` and `EncodingV` type class names were not adopted; the functionality was integrated through existing version range types, PQ-dependent size constants (`e2eEncConnInfoLength`, `e2eEncAgentMsgLength`), and the `pqdrSMPAgentVersion` constant. + # Migrating existing connections to post-quantum double ratchet algorithm ## Problem diff --git a/rfcs/done/2023-12-29-pqdr.md b/rfcs/standard/2026-03-09-pqdr.md similarity index 96% rename from rfcs/done/2023-12-29-pqdr.md rename to rfcs/standard/2026-03-09-pqdr.md index 7fd88ffbb..4c478da2f 100644 --- a/rfcs/done/2023-12-29-pqdr.md +++ b/rfcs/standard/2026-03-09-pqdr.md @@ -1,3 +1,10 @@ +--- +Proposed: 2023-12-29 +Implemented: 2024-03-14 +Standardized: 2026-03-09 +Protocol: pqdr v1, agent-protocol v5 +--- + # Post-quantum double ratchet implementation See [the previous doc](https://github.com/simplex-chat/simplex-chat/blob/stable/docs/rfcs/2023-09-30-pq-double-ratchet.md). diff --git a/rfcs/done/2022-12-27-queue-quota.md b/rfcs/standard/2026-03-09-queue-quota.md similarity index 94% rename from rfcs/done/2022-12-27-queue-quota.md rename to rfcs/standard/2026-03-09-queue-quota.md index 2337f84f2..b2e666d43 100644 --- a/rfcs/done/2022-12-27-queue-quota.md +++ b/rfcs/standard/2026-03-09-queue-quota.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-12-27 +Implemented: ~2023 +Standardized: 2026-03-09 +Protocol: simplex-messaging, agent-protocol +--- + # SMP and SMP agent protocol extensions to manage queue quotas ## Problem diff --git a/rfcs/done/2022-08-14-queue-rotation.md b/rfcs/standard/2026-03-09-queue-rotation.md similarity index 96% rename from rfcs/done/2022-08-14-queue-rotation.md rename to rfcs/standard/2026-03-09-queue-rotation.md index fa40fda7e..18fabc560 100644 --- a/rfcs/done/2022-08-14-queue-rotation.md +++ b/rfcs/standard/2026-03-09-queue-rotation.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-08-14 +Implemented: ~2022 +Standardized: 2026-03-09 +Protocol: agent-protocol v2 +--- + # SMP queue rotation and redundancy ## Problem diff --git a/rfcs/done/2023-10-25-remote-control.md b/rfcs/standard/2026-03-09-remote-control.md similarity index 99% rename from rfcs/done/2023-10-25-remote-control.md rename to rfcs/standard/2026-03-09-remote-control.md index 8507dba49..854b70866 100644 --- a/rfcs/done/2023-10-25-remote-control.md +++ b/rfcs/standard/2026-03-09-remote-control.md @@ -1,3 +1,10 @@ +--- +Proposed: 2023-10-25 +Implemented: ~2024 +Standardized: 2026-03-09 +Protocol: xrcp v1 +--- + # SimpleX Remote Control protocol Using profiles in SimpleX Chat mobile app from desktop app with minimal risk to the security/threat model of SimpleX protocols. diff --git a/rfcs/done/2023-05-02-resync-ratchets.md b/rfcs/standard/2026-03-09-resync-ratchets-design.md similarity index 92% rename from rfcs/done/2023-05-02-resync-ratchets.md rename to rfcs/standard/2026-03-09-resync-ratchets-design.md index a80a48136..6f8a4066d 100644 --- a/rfcs/done/2023-05-02-resync-ratchets.md +++ b/rfcs/standard/2026-03-09-resync-ratchets-design.md @@ -1,3 +1,12 @@ +--- +Proposed: 2023-05-02 +Implemented: 2023-06-30 +Standardized: 2026-03-09 +Protocol: agent-protocol v3 +--- + +> **Implementation note:** Early brainstorm document. The implementation followed the more detailed RFC 2023-06-08-resync-ratchets, which refined the state machine to use a single RatchetSyncState (RSOk/RSAllowed/RSRequired/RSStarted/RSAgreed) and defined the AgentRatchetKey envelope type. + # Re-sync encryption ratchets, queue rotation, message delivery receipts This is very unfocussed doc outlining several problems that seem somewhat related, and some possible solution approaches. diff --git a/rfcs/done/2023-06-08-resync-ratchets.md b/rfcs/standard/2026-03-09-resync-ratchets.md similarity index 99% rename from rfcs/done/2023-06-08-resync-ratchets.md rename to rfcs/standard/2026-03-09-resync-ratchets.md index fc4572eec..2c9a1d78d 100644 --- a/rfcs/done/2023-06-08-resync-ratchets.md +++ b/rfcs/standard/2026-03-09-resync-ratchets.md @@ -1,3 +1,10 @@ +--- +Proposed: 2023-06-08 +Implemented: 2023-06-30 +Standardized: 2026-03-09 +Protocol: agent-protocol v3 +--- + # Re-sync encryption ratchets ## Problem diff --git a/rfcs/done/2023-09-12-second-relays.md b/rfcs/standard/2026-03-09-second-relays.md similarity index 99% rename from rfcs/done/2023-09-12-second-relays.md rename to rfcs/standard/2026-03-09-second-relays.md index cad6c4a92..94137df91 100644 --- a/rfcs/done/2023-09-12-second-relays.md +++ b/rfcs/standard/2026-03-09-second-relays.md @@ -1,3 +1,10 @@ +--- +Proposed: 2023-09-12 +Implemented: 2024-06-21 +Standardized: 2026-03-09 +Protocol: simplex-messaging v8 +--- + # Protecting IP addresses of the users from their contacts ## Problem diff --git a/rfcs/done/2022-12-26-simplex-file-transfer.md b/rfcs/standard/2026-03-09-simplex-file-transfer.md similarity index 98% rename from rfcs/done/2022-12-26-simplex-file-transfer.md rename to rfcs/standard/2026-03-09-simplex-file-transfer.md index a833505f1..e0c4258ee 100644 --- a/rfcs/done/2022-12-26-simplex-file-transfer.md +++ b/rfcs/standard/2026-03-09-simplex-file-transfer.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-12-26 +Implemented: ~2023 +Standardized: 2026-03-09 +Protocol: xftp v1 +--- + # SimpleX File Transfer protocol ## Problem diff --git a/rfcs/done/2022-11-11-smp-basic-auth.md b/rfcs/standard/2026-03-09-smp-basic-auth.md similarity index 95% rename from rfcs/done/2022-11-11-smp-basic-auth.md rename to rfcs/standard/2026-03-09-smp-basic-auth.md index f0bfbd97a..ee4d9cea8 100644 --- a/rfcs/done/2022-11-11-smp-basic-auth.md +++ b/rfcs/standard/2026-03-09-smp-basic-auth.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-11-11 +Implemented: 2022-11-12 +Standardized: 2026-03-09 +Protocol: simplex-messaging v5 +--- + # SMP Basic Auth ## Problem diff --git a/rfcs/done/2023-05-24-smp-delivery-proxy.md b/rfcs/standard/2026-03-09-smp-delivery-proxy.md similarity index 86% rename from rfcs/done/2023-05-24-smp-delivery-proxy.md rename to rfcs/standard/2026-03-09-smp-delivery-proxy.md index 73bdd034b..dad0624b4 100644 --- a/rfcs/done/2023-05-24-smp-delivery-proxy.md +++ b/rfcs/standard/2026-03-09-smp-delivery-proxy.md @@ -1,3 +1,12 @@ +--- +Proposed: 2023-05-24 +Implemented: 2024-06-21 +Standardized: 2026-03-09 +Protocol: simplex-messaging v8 +--- + +> **Implementation note:** Short conceptual proposal. The full design evolved into the two-hop onion routing architecture described in RFC 2023-09-12-second-relays, implemented as SMP v8 with PRXY/PKEY/PFWD/RFWD/RRES/PRES commands. + # SMP and XFTP delivery relays ## Problem diff --git a/rfcs/done/2022-06-05-smp-notifications.md b/rfcs/standard/2026-03-09-smp-notifications.md similarity index 97% rename from rfcs/done/2022-06-05-smp-notifications.md rename to rfcs/standard/2026-03-09-smp-notifications.md index 4e58189b7..8c1808552 100644 --- a/rfcs/done/2022-06-05-smp-notifications.md +++ b/rfcs/standard/2026-03-09-smp-notifications.md @@ -1,3 +1,10 @@ +--- +Proposed: 2022-06-05 +Implemented: 2022-06-06 +Standardized: 2026-03-09 +Protocol: simplex-messaging v2 +--- + # SMP protocol changes to support push notifications on iOS ## Problem diff --git a/rfcs/done/2024-03-28-xftp-version.md b/rfcs/standard/2026-03-09-xftp-version.md similarity index 98% rename from rfcs/done/2024-03-28-xftp-version.md rename to rfcs/standard/2026-03-09-xftp-version.md index c46810bb9..785b62340 100644 --- a/rfcs/done/2024-03-28-xftp-version.md +++ b/rfcs/standard/2026-03-09-xftp-version.md @@ -1,3 +1,10 @@ +--- +Proposed: 2024-03-28 +Implemented: ~2024 +Standardized: 2026-03-09 +Protocol: xftp v2 +--- + # XFTP version agreement ## Problem diff --git a/rfcs/2024-06-01-agent-protocol.md b/rfcs/standard/2026-03-10-agent-protocol.md similarity index 63% rename from rfcs/2024-06-01-agent-protocol.md rename to rfcs/standard/2026-03-10-agent-protocol.md index 616aed33f..896ae825a 100644 --- a/rfcs/2024-06-01-agent-protocol.md +++ b/rfcs/standard/2026-03-10-agent-protocol.md @@ -1,3 +1,12 @@ +--- +Proposed: 2024-06-01 +Implemented: ~2024 +Standardized: 2026-03-10 +Protocol: agent-protocol +--- + +> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior. + # Evolving agent API ## Problem diff --git a/rfcs/2025-05-05-client-certificates.md b/rfcs/standard/2026-03-10-client-certificates.md similarity index 75% rename from rfcs/2025-05-05-client-certificates.md rename to rfcs/standard/2026-03-10-client-certificates.md index 00de2f9b3..68321256d 100644 --- a/rfcs/2025-05-05-client-certificates.md +++ b/rfcs/standard/2026-03-10-client-certificates.md @@ -1,12 +1,21 @@ -# Service certificates for high volume servers and services connecting to SMP servers +--- +Proposed: 2025-05-05 +Implemented: ~2025 (SMP v16) +Standardized: 2026-03-10 +Protocol: simplex-messaging +--- + +> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior. + +# Service certificates for high volume routers and services connecting to SMP routers ## Problem -The absense of user and client identification benefits privacy, but it requires separately authorizing subscription for each messaging queue, that doesn't scale when a high volume server or service acts as a client for SMP server even for the current traffic and network size. +The absence of user and client identification benefits privacy, but it requires separately authorizing subscription for each messaging queue, that doesn't scale when a high volume router or service acts as a client for SMP router even for the current traffic and network size. -These servers/services include: +These routers/services include: - operators' chat relays (aka super-peers), -- notification servers, +- notification routers, - high-traffic service chat bots, - high-traffic business support clients. @@ -16,31 +25,31 @@ Self-hosted chat relays may want to retain privacy, so they will not use client Even today, directory service subscribing to all queues may take 15-20 minutes, which is experienced as downtime by the end users. -Notification servers also acting as clients to messaging servers also take 15-20 minutes to subscribe to all notifications, during which time notifications are not delivered. +Notification routers also acting as clients to messaging routers also take 15-20 minutes to subscribe to all notifications, during which time notifications are not delivered. -Not only these subscription take a lot of time, they also consume a large amount of memory both in the clients and in the servers, as association between clients and queues is currently session-scoped and not persisted anywhere (and it should not be, because end-users' clients do need privacy). +Not only these subscriptions take a lot of time, they also consume a large amount of memory both in the clients and in the routers, as association between clients and queues is currently session-scoped and not persisted anywhere (and it should not be, because end-users' clients do need privacy). ## Solution -High volume "clients" (operators' chat relays, directory service, SimpleX Chat team support client, SimpleX Status bot, etc.) that don't need privacy will identify themselves to the messaging servers at a point of connection by providing client sertificate, both in TLS handshake and in SMP handshake (the same certificate must be provided). +High volume "clients" (operators' chat relays, directory service, SimpleX Chat team support client, SimpleX Status bot, etc.) that don't need privacy will identify themselves to the messaging routers at a point of connection by providing client certificate, both in TLS handshake and in SMP handshake (the same certificate must be provided). All the new queues and subscriptions made in this session will be creating a permanent association of the messaging queue with the client, and on subsequent reconnections the client can "subscribe" to all their queues with a single client subscription command. -This will save a lot of time subscribing and resubscribing on server and client restarts, servers' bandwidth, servers' traffic spikes, and memory of both clients and servers. +This will save a lot of time subscribing and resubscribing on router and client restarts, routers' bandwidth, routers' traffic spikes, and memory of both clients and routers. ## Protocol -An ephemeral per-session signature key signed by long-term client certificate is used for client authorization – this session signature key will be passed in SMP handshake. +An ephemeral per-session signature key signed by long-term client certificate is used for client authorization -- this session signature key will be passed in SMP handshake. To transition existing queues, the subscription command will have to be double-signed - by the queue key, and then by client key. -When server receives such "hand-over" subscription it would create a permanent association between the client certificate and the queue, and on subsequent re-connections the client can subscribe to all the existing queues still associated with the client with one command. +When router receives such "hand-over" subscription it would create a permanent association between the client certificate and the queue, and on subsequent re-connections the client can subscribe to all the existing queues still associated with the client with one command. -The server will respond to the client with the number of queues it was subscribed to - it would both inform the client that it has to re-connect in case of interruption, and can be used for client and server statistics. +The router will respond to the client with the number of queues it was subscribed to - it would both inform the client that it has to re-connect in case of interruption, and can be used for client and router statistics. When client creates a new queue, it would also sign the request with both keys, per-queue and client's. Other queue operations (e.g., deletion, or changing associated queue data for short links) would still require two signatures, both the queue key and the client key. -The open question is whether there is any value in allowing to remove the association between the client and the queue. Probably not, as threat model should assume that the server would retain this information, and the use-case for users controlling their servers is narrow. +The open question is whether there is any value in allowing to remove the association between the client and the queue. Probably not, as threat model should assume that the router would retain this information, and the use-case for users controlling their routers is narrow. ## Protocol connection handshake @@ -69,7 +78,7 @@ data ClientHandshake = ClientHandshake } ``` -`ServerHandshake` already contains `authPubKey` with the server certificate chain and the signed key for connection encryption and creating a shared secret for denable authorization (with client entity key) and session encryption layer. +`ServerHandshake` already contains `authPubKey` with the router certificate chain and the signed key for connection encryption and creating a shared secret for deniable authorization (with client entity key) and session encryption layer. `ClientHandshake` contains only ephemeral `authPubKey` to compute a shared secret for session encryption layer, so we need an additional field for an optional client certificate: @@ -77,9 +86,9 @@ data ClientHandshake = ClientHandshake serviceCertKey :: Maybe (X.CertificateChain, X.SignedExact X.PubKey) ``` -Certificate here defines client identity. The actual key to be used to sign commands is session-scoped, and is signed by the certificate key. In case of notification server it MUST be the same certificate that is used for server TLS connections. +Certificate here defines client identity. The actual key to be used to sign commands is session-scoped, and is signed by the certificate key. In case of notification router it MUST be the same certificate that is used for router TLS connections. -For operators' clients we may optionally include operators' certificate in the chain, and that would allow servers to identify operators if either wants to. This would improve end-user security, as not only the server would validate that its certificate matches the address, but it would also validate that it is operated by SimpleX Chat or by Flux, preventing any server impersonation (e.g., via DNS manipulations) - the client could then report that the files are hosted on SimpleX Chat servers, but then can stop and show additional warning in case certificate does not match the domain - same as the browsers do with CA stores in the client. +For operators' clients we may optionally include operators' certificate in the chain, and that would allow routers to identify operators if either wants to. This would improve end-user security, as not only the router would validate that its certificate matches the address, but it would also validate that it is operated by SimpleX Chat or by Flux, preventing any router impersonation (e.g., via DNS manipulations) - the client could then report that the files are hosted on SimpleX Chat routers, but then can stop and show additional warning in case certificate does not match the domain - same as the browsers do with CA stores in the client. ## Protocol transmissions @@ -104,9 +113,9 @@ authenticator = queue_authenticator ("0" / "1" service_authenticator) In case service_authenticator is present, queue_authenticator should authorize over `fingerprint authorized` (concatenation of service identity certificate fingerprint and the rest of the transmission). -All queues created with client key will have to be double-authorized with both the queue key and the client key - both the client and the server would have to maintain this knowledge, whether the queue is associated with the client or not. +All queues created with client key will have to be double-authorized with both the queue key and the client key - both the client and the router would have to maintain this knowledge, whether the queue is associated with the client or not. -Asymmetric retries have to be supported - the first request creating this association may succeed on the server and timeout on the client. +Asymmetric retries have to be supported - the first request creating this association may succeed on the router and timeout on the client. ## Subscription @@ -118,7 +127,7 @@ The command and response: SUBS :: Command Recipient -- to enable all client subscriptions, empty entity ID in the transmission, signed by client key - it must be the same as was used in handover subscription signature. NSUBS :: Command Recipient -- notification subscription SOK :: Maybe ServiceId -- new subscription response -SOKS :: Int64 -> BrokerMsg -- response from the server, includes the number of subscribed queues +SOKS :: Int64 -> BrokerMsg -- response from the router, includes the number of subscribed queues ENDS :: Int64 -> BrokerMsg -- when another session subscribes with the same certificate ``` @@ -133,7 +142,7 @@ This was considered to reduce costs for the usual clients to re-subscribe. Curre For some very busy end-user clients it may help. -Given that server has access to an ephemeral association between recipient client session and queues anyway (even with clients connecting via Tor, unless per-connection transport isolation is used), introducing `sessionPubKey` to allow resubscription to the previously subscribed queues may reduce the traffic. This won't change threat model as the server would only keep this association in memory, and not persist it. Clients on another hand may safely persist this association for fast resubscription on client restarts. +Given that router has access to an ephemeral association between recipient client session and queues anyway (even with clients connecting via Tor, unless per-connection transport isolation is used), introducing `sessionPubKey` to allow resubscription to the previously subscribed queues may reduce the traffic. This won't change threat model as the router would only keep this association in memory, and not persist it. Clients on another hand may safely persist this association for fast resubscription on client restarts. This is not planned for the forseable future, as migrating to chat relays would solve most of the problem. diff --git a/rfcs/2024-02-12-encryption.md b/rfcs/standard/2026-03-10-encryption.md similarity index 77% rename from rfcs/2024-02-12-encryption.md rename to rfcs/standard/2026-03-10-encryption.md index 37a936ae4..483da6db8 100644 --- a/rfcs/2024-02-12-encryption.md +++ b/rfcs/standard/2026-03-10-encryption.md @@ -1,10 +1,19 @@ +--- +Proposed: 2024-02-12 +Implemented: ~2024 (SMP v11) +Standardized: 2026-03-10 +Protocol: simplex-messaging +--- + +> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior. + # Transmission encryption ## Problems ### Protection of meta-data from sending proxy -The SEND commands and message queue IDs need to be encrypted so that sending proxy cannot see how many queues exist on each server. +The SEND commands and message queue IDs need to be encrypted so that sending proxy cannot see how many queues exist on each router. Correlation IDs need to be random and can be re-used as nonces so that the destination relay cannot use the increasing correlation IDs that are sent in v6 of the protocol to track the sender. @@ -24,10 +33,10 @@ encRespTransmission = replyNonce encrypted(respTransmission) respTransmission = entityId command ``` -The keys to encrypt and decrypt both the command and responses would be computed as curve25519 from the key sent together with command and server session key. For the requests, the nonce has to be random and sent outside of the encrypted envelopt, but for the response respNonce would be taken from inside of the encrypted envelope and it would also be used for correlating commands and responses. This way the attacker who could compromise TLS would not be able to correlate the commands and responses, and also observe entity IDs. +The keys to encrypt and decrypt both the command and responses would be computed as curve25519 from the key sent together with command and router session key. For the requests, the nonce has to be random and sent outside of the encrypted envelope, but for the response respNonce would be taken from inside of the encrypted envelope and it would also be used for correlating commands and responses. This way the attacker who could compromise TLS would not be able to correlate the commands and responses, and also observe entity IDs. 2. The remaining question is to how encrypt and decrypt messages delivered not in response to the commands. The possible options are: - restore client session key only for that purpose, but do not forward this key to the destination proxy for sent messages. Then the messages can be sent with a random replyNonce and the key would be computed from session keys. The advantage here is that we won't need to parameterize handles as both client and server would have session keys. The downside that we would have to either somehow differentiate messages and responses, either by some flag that would allow some correlation or just by the absense of replyNonce in the lookup map - that is if the client can find replyNonce, it would use the associated key to decrypt, and if not it would use session key. -- use the same key that was sent with SUB or ACK command. This is much more complex, and would only have some upside if we were to introduce receiving proxies (to conceal transport sessions from the receiving relays for the recipients). +- use the same key that was sent with SUB or ACK command. This is much more complex, and would only have some upside if we were to introduce receiving proxies (to conceal transport sessions from the receiving routers for the recipients). diff --git a/rfcs/2024-03-20-server-metadata.md b/rfcs/standard/2026-03-10-server-metadata.md similarity index 83% rename from rfcs/2024-03-20-server-metadata.md rename to rfcs/standard/2026-03-10-server-metadata.md index 22b163c05..0506c01bd 100644 --- a/rfcs/2024-03-20-server-metadata.md +++ b/rfcs/standard/2026-03-10-server-metadata.md @@ -1,8 +1,17 @@ +--- +Proposed: 2024-03-20 +Implemented: ~2024 +Standardized: 2026-03-10 +Protocol: simplex-messaging +--- + +> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior. + # Relay metadata and SimpleX network decentralization ## Problem -Currently, the clients configure/choose which servers to use, but they cannot see who operates them, in which geography and hosting provider, what is the server source code (in case it was modified from the reference implementation we provide) and also any administrative and feedback contacts. +Currently, the clients configure/choose which routers to use, but they cannot see who operates them, in which geography and hosting provider, what is the router source code (in case it was modified from the reference implementation we provide) and also any administrative and feedback contacts. Further, we currently use simplex.chat domain to host group links, and as diversity of the groups grows it is beginning to require managing feedback from the users about groups. It is important that this feedback is directed to relay owners and not to us, in case they are not our relays, as we are simply providing software here. @@ -21,28 +30,28 @@ While this document is not the end of the journey to decentralize the network, i The proposed solution consists of two parts: -- communicate server metadata via protocol, so it can be observed by the clients. +- communicate router metadata via protocol, so it can be observed by the clients. - create home page for the relays, with all the same metadata. - create invitation and address links in the same domain name as the relay. -The latter point is important so it is clear to the users who operates and owns the relay and where the access point to the content or group is hosted. Even though simplex.chat domain is never accessed by the app, and the meaningful part of the address is never sent to the page hosting server, it creates an impression of centralization, and some dependency on simplex.chat domain for anything other that showing the link QR code. +The latter point is important so it is clear to the users who operates and owns the relay and where the access point to the content or group is hosted. Even though simplex.chat domain is never accessed by the app, and the meaningful part of the address is never sent to the page hosting router, it creates an impression of centralization, and some dependency on simplex.chat domain for anything other that showing the link QR code. Moving invitation links to the domain of the relay (primary relay, in case the link has redundancy) will both clarify relay ownership, solve the incorrect mis-perception of centralization, remove the dependency on simplex-chat domain without any user effort, and provides the means to submit content complaints to the relay operators (should they wish to receive them, which seems reasonable for large public relays, but may be unnecessary for private relays where unidentified parties cannot create links). ## Solution details -Extend server INI file with information section: +Extend router INI file with information section: ``` [INFORMATION] # Please note that under AGPLv3 license conditions you MUST make -# any source code modifications available to the end users of the server. +# any source code modifications available to the end users of the router. # LICENSE: https://github.com/simplex-chat/simplexmq/blob/stable/LICENSE # Not doing so would constitute a license violation. # Declaring an incorrect information here amounts to a fraud. # The license holders reserve the right to prosecute missing or incorrect # information about the server source code to the fullest extent permitted by the law. -# The server will show warning on start if this field is absent +# The router will show warning on start if this field is absent # and will not launch from v6.0 until this field is added. # If any other information field is present, source code property also MUST be present. source_code: https://github.com/simplex-chat/simplexmq @@ -69,13 +78,13 @@ hosting: Linode / Akamai Inc. hosting_country: US ``` -Server home page would show whether queue creation is allowed and/or password protected, server retention policy (e.g., preserve messages on restart or not, and persist connections or not). +Router home page would show whether queue creation is allowed and/or password protected, router retention policy (e.g., preserve messages on restart or not, and persist connections or not). -Server queue address/contact pages will optionally, provide the UI to submit feedback, comments and complaints directly from the web page (not an MVP, initially we would simply show addresses for feedback, and, probably, create link that opens in the app with pre-populated message, and we could also use this addresses defined in server meta-data to submit feedback from inside of the app - it's also out of MVP scope). +Router queue address/contact pages will optionally, provide the UI to submit feedback, comments and complaints directly from the web page (not an MVP, initially we would simply show addresses for feedback, and, probably, create link that opens in the app with pre-populated message, and we could also use this addresses defined in router meta-data to submit feedback from inside of the app - it's also out of MVP scope). -If server is available on .onion address, the web pages would show "open via .onion" in Tor browser. +If router is available on .onion address, the web pages would show "open via .onion" in Tor browser. -Extend server handshake header with these information fields: +Extend router handshake header with these information fields: ```haskell data ServerHandshake = ServerHandshake @@ -93,13 +102,13 @@ data ServerInformation = ServerInformation info :: ServerPublicInfo } --- based on server configuration +-- based on router configuration data ServerPublicConfig = ServerPublicConfig { persistence :: SMPServerPersistenceMode, messageExpiration :: Int, statsEnabled :: Bool, newQueuesAllowed :: Bool, - basicAuthEnabled :: Bool -- server is private if enabled + basicAuthEnabled :: Bool -- router is private if enabled } -- based on INFORMATION section of INI file @@ -127,4 +136,4 @@ data ServerContactAddress = ServerContactAddress } ``` -This extended server information will be stored in the chat database every time it changes and shown in the UI of the server configuration. +This extended router information will be stored in the chat database every time it changes and shown in the UI of the router configuration. diff --git a/rfcs/2024-06-21-short-links.md b/rfcs/standard/2026-03-10-short-links.md similarity index 79% rename from rfcs/2024-06-21-short-links.md rename to rfcs/standard/2026-03-10-short-links.md index df028a8ff..3e3e7bf8c 100644 --- a/rfcs/2024-06-21-short-links.md +++ b/rfcs/standard/2026-03-10-short-links.md @@ -1,3 +1,12 @@ +--- +Proposed: 2024-06-21 +Implemented: ~2025 (SMP v15) +Standardized: 2026-03-10 +Protocol: simplex-messaging + agent-protocol +--- + +> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior. + # Short invitation links ## Problem @@ -14,7 +23,7 @@ Additionally, if we store short links, they can also include chat preferences an MITM-resistant link shortening. -Instead of generating the random address that would resolve into the link - doing so would create the possibility of MITM by the server hosting this link - we can use private key as the link ID that will be passed to the accepting party, and the hash of the public key as ID for the server - the accepting party would present this key itself as ID and it will also be used for server to client encryption (see Protocol below). HKDF will be used to derive symmetric key from private key and used in secret_box together with random nonce (to allow replacing data with the same key but with a different nonce - nonce will be sent to the server too). secret_box construction is authenticated encryption, so it would protect from MITM. +Instead of generating the random address that would resolve into the link - doing so would create the possibility of MITM by the router hosting this link - we can use private key as the link ID that will be passed to the accepting party, and the hash of the public key as ID for the router - the accepting party would present this key itself as ID and it will also be used for router to client encryption (see Protocol below). HKDF will be used to derive symmetric key from private key and used in secret_box together with random nonce (to allow replacing data with the same key but with a different nonce - nonce will be sent to the router too). secret_box construction is authenticated encryption, so it would protect from MITM. The proposed syntax: @@ -29,7 +38,7 @@ srvHosts = ["," srvHosts] ; RFC1123, RFC5891 linkHash = ``` -If SMP server supports pages, its name can be used as clientAppServer, without repeating it after #, for a shorter link. +If SMP router supports pages, its name can be used as clientAppServer, without repeating it after #, for a shorter link. Example link: @@ -40,12 +49,12 @@ https://simplex.chat/contact/#0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU=@smp8. This link has the length of ~136 characters (256 bits), which is shorter than the full contact address (~310 characters) and much shorter than invitation links (~528 characters) even without post-quantum keys added to them. This size can be further reduced by -- use server domain in the link. -- do not include onion address, as the connection happens via proxy anyway, if it's untrusted server. -- not pinning server TLS certificate - the downside here is that while the attack that compromises TLS will not be able to substitute the link (because it's hash will not match), it will be able to intercept and to block it. +- use router domain in the link. +- do not include onion address, as the connection happens via proxy anyway, if it's untrusted router. +- not pinning router TLS certificate - the downside here is that while the attack that compromises TLS will not be able to substitute the link (because it's hash will not match), it will be able to intercept and to block it. - using shorter hash, e.g. SHA128 - reducing the collision resistance. -If the server is known, the client could use it's hash and onion address, otherwise it could trust the proxy to use any existing session with the same hostname or to accept the risk of interception - given that there is no risk of substitution. +If the router is known, the client could use its hash and onion address, otherwise it could trust the proxy to use any existing session with the same hostname or to accept the risk of interception - given that there is no risk of substitution. With the first two of these "improvements" the link could be ~122 characters: @@ -59,13 +68,13 @@ If onion address is preserved the link will be ~184 characters (won't fit in Twi https://smp8.simplex.im/contact/#0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU@beccx4yfxxbvyhqypaavemqurytl6hozr47wfc7uuecacjqdvwpw2xid.onion/abcdefghij0123456789abcdefghij0123456789abc ``` -If we implement it, the request to resolve the link would be made via proxied SMP command (to avoid the direct connection between the client and the recipient's server). +If we implement it, the request to resolve the link would be made via proxied SMP command (to avoid the direct connection between the client and the recipient's router). Pros: - a bit shorter link. - possibility to include post-quantum keys into the full link keeping the same shortened link size. - possibility to include chat profile of contact or group, and preferences, for a much better connection experience, and to show this information when the link sent in the conversation (clients can resolve them automatically, without connecting - it can be resolved by the sending clients). -- server will not have access to the link. +- router will not have access to the link. Cons: - protocol complexity. @@ -75,7 +84,7 @@ Pros are a huge improvement of UX of connecting both within and from outside of ## Protocol -To support short links, the SMP servers would provide a simple key-value store enabled by three additional commands: `WRT`, `CLR` and `READ` +To support short links, the SMP routers would provide a simple key-value store enabled by three additional commands: `WRT`, `CLR` and `READ` `WRT` command is used to store and to update values in the store. The size of the value is limited by the same size as sent messages (or, possibly, smaller - as connection information size used in confirmation messages) - the clients would use this fixed size irrespective of the content. `WRT` command will be sent with the data blob ID in the transaction entityId field, public authorization key used to authorize `WRT` and `CLR` commands (subsequent WRT commands to the existing key must use the same key), and the data blob. @@ -89,22 +98,22 @@ To support short links, the SMP servers would provide a simple key-value store e - the data blob owner generates X25519 key pair: `(k, pk)`. - private key `pk` will be included in the short link shared with the other party (only base64url encoded key bytes, not X509 encoding). -- `HKDF(pk)` will be used to encrypt the link data with secret_box before storing it on the server. +- `HKDF(pk)` will be used to encrypt the link data with secret_box before storing it on the router. - the hash of public key `sha256(k)` will be used as ID by the owner to store and to remove the data blob (`WRT` and `CLR` commands). **Retrieve data blob** -- the sender uses the public key `k` derived from the private key `pk` included in the link as entity ID to retrieve data blob (the server will compute the ID used by the owner as `sha256(k)` and will be able to look it up). This provides the quality that the traffic of the parties has no shared IDs inside TLS. It also means that unlike message queue creation, the ID to retrieve the blob was never sent to the blob creator, and also is not known to the server in advance (the second part is only an observation, in itself it does not increase security, as server has access to an encrypted blob anyway). +- the sender uses the public key `k` derived from the private key `pk` included in the link as entity ID to retrieve data blob (the router will compute the ID used by the owner as `sha256(k)` and will be able to look it up). This provides the quality that the traffic of the parties has no shared IDs inside TLS. It also means that unlike message queue creation, the ID to retrieve the blob was never sent to the blob creator, and also is not known to the router in advance (the second part is only an observation, in itself it does not increase security, as router has access to an encrypted blob anyway). - note that the sender does not authorize the request to retrieve the blob, as it would not increase security unless a different key is used to authorize, and adding a key would increase link size. -- server session keys with the sender will be `(sk, spk)`, where `sk` is public key shared with the sender during session handshake, and `spk` is the private key known only to the server. -- this public key `k` will also be combined with server session key `spk` using `dh(k, spk)` to encrypt the response, so that there is no ciphertext in common in sent and received traffic for these blobs. Correlation ID will be used as a nonce for this encryption. +- router session keys with the sender will be `(sk, spk)`, where `sk` is public key shared with the sender during session handshake, and `spk` is the private key known only to the router. +- this public key `k` will also be combined with router session key `spk` using `dh(k, spk)` to encrypt the response, so that there is no ciphertext in common in sent and received traffic for these blobs. Correlation ID will be used as a nonce for this encryption. - having received the blob, the client can now decrypt it using secret_box with `HKDF(pk)`. Using the same key as ID for the request, and also to additionally encrypt the response allows to use a single key in the link, without increasing the link size. ## Threat model -**Compromised SMP server** +**Compromised SMP router** can: - delete link data. diff --git a/rfcs/2024-09-09-smp-blobs.md b/rfcs/standard/2026-03-10-smp-blobs.md similarity index 81% rename from rfcs/2024-09-09-smp-blobs.md rename to rfcs/standard/2026-03-10-smp-blobs.md index be990f09c..5d81c1ade 100644 --- a/rfcs/2024-09-09-smp-blobs.md +++ b/rfcs/standard/2026-03-10-smp-blobs.md @@ -1,3 +1,12 @@ +--- +Proposed: 2024-09-09 +Implemented: ~2025 (SMP v15) +Standardized: 2026-03-10 +Protocol: simplex-messaging + agent-protocol +--- + +> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior. + # Blob extensions for SMP queues Evolution of the design for short links, see [here](./2024-06-21-short-links.md) and [here](./2024-09-05-queue-storage.md). @@ -11,13 +20,13 @@ Allow storing extended information with SMP queues to improve UX and security of ## Design -1. Queue creation/update date is already added to server persistence, allowing to expire queues and blobs, depending on their usage. +1. Queue creation/update date is already added to router persistence, allowing to expire queues and blobs, depending on their usage. 2. Add "queue type" metadata to NEW command to indicate whether messaging queue is used as public address or as messaging queue (see previous docs on why it doesn't change threat model). While at the moment it would match sndSecure flag there may be future scenarios when they diverge. Initially only "invitation" and "contact" types will be supported. 3. Prohibit sndSecure flag for "contact" queues, prohibit securing contact queues. 4. Add "queue blobs" to NEW command: - - blob0: ratchetKeys up to N0 bytes - priority 0, can't be removed by the server, only in "invitation" - - blob1: PQ key up to N1 bytes - priority 1, can be removed by the server, only used in "invitation" - - blob2: Application data up to N2 bytes - priority 2, can be removed by the server. + - blob0: ratchetKeys up to N0 bytes - priority 0, can't be removed by the router, only in "invitation" + - blob1: PQ key up to N1 bytes - priority 1, can be removed by the router, only used in "invitation" + - blob2: Application data up to N2 bytes - priority 2, can be removed by the router. 5. Add linkId to NEW command 6. linkId and blobs will be removed when queue is secured. 7. Add recipient command to remove/upsert blob2 for contact queues. @@ -28,7 +37,7 @@ Allow storing extended information with SMP queues to improve UX and security of ### Creating a queue: The queue owner: -- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the server, same as now. `sk` and `dhk` will be sent in NEW command. +- generates Ed25529 key pair `(sk, spk)` and X25519 key pair `(dhk, dhpk)` to use with the router, same as now. `sk` and `dhk` will be sent in NEW command. - generates X25519 key pair `(k, pk)` to use with the accepting party to encrypt queue messages. - derives from `k` using HKDF: - symmetric key `bk` for authenticated encryption of blobs. @@ -73,7 +82,7 @@ Response to GET: blobs = %s"BLOB" senderId [ "0" blob0 ] [ "1" blob1 ] [ "2" blob2 ] ``` -As blobs are retrieved using a separate linkId, once blobs are removed it will be impossible to find senderId from short link - it is a threat model improvement. Once server storage is compacted, it will be impossible to find queue related to the link even with the access to server data (unless server preserves the data). +As blobs are retrieved using a separate linkId, once blobs are removed it will be impossible to find senderId from short link - it is a threat model improvement. Once router storage is compacted, it will be impossible to find queue related to the link even with the access to router data (unless router preserves the data). ### Possible privacy improvement diff --git a/rfcs/2025-03-16-smp-queues.md b/rfcs/standard/2026-03-10-smp-queues.md similarity index 83% rename from rfcs/2025-03-16-smp-queues.md rename to rfcs/standard/2026-03-10-smp-queues.md index d79e6f419..a9afa23d4 100644 --- a/rfcs/2025-03-16-smp-queues.md +++ b/rfcs/standard/2026-03-10-smp-queues.md @@ -1,3 +1,12 @@ +--- +Proposed: 2025-03-16 +Implemented: ~2025 (SMP v15) +Standardized: 2026-03-10 +Protocol: simplex-messaging + agent-protocol +--- + +> **Implementation note:** This RFC was promoted from done/ to standard/ based on verification that the described feature exists in the codebase. The RFC text reflects the original proposal and may not match the actual implementation in all details. The consolidated protocol specifications in `protocol/` are the authoritative reference for current behavior. + # Protocol changes for creating and connecting to SMP queues ## Problems @@ -19,18 +28,18 @@ Simply designating queue types would allow to use this information to decide for We want to achieve these objectives for short links and associated queue data: 1. no possibility to provide incorrect SenderId inside link data (e.g. from another queue). -2. link data cannot be accessed by the server unless it has the link. -3. prevent MITM attack by the server, including the server that obtained the link. +2. link data cannot be accessed by the router unless it has the link. +3. prevent MITM attack by the router, including the router that obtained the link. 4. prevent changing of connection request by the user (to prevent MITM via break-in attack in the originating client). -5. for one-time links, prevent accessing link data by link observers who did not compromise the server. +5. for one-time links, prevent accessing link data by link observers who did not compromise the router. 6. allow changing the user-defined part of link data. -7. avoid changing the link when user-defined part of link data changes, while preventing MITM attack by the server on user-defined part, even if it has the link. -8. retain the quality that it is impossible to check the existence of secured queue from having any of its temporary visible IDs (sender ID and link ID in 1-time invitations) - it requires that these IDs remain server-generated (contrary to the previous RFCs). +7. avoid changing the link when user-defined part of link data changes, while preventing MITM attack by the router on user-defined part, even if it has the link. +8. retain the quality that it is impossible to check the existence of secured queue from having any of its temporary visible IDs (sender ID and link ID in 1-time invitations) - it requires that these IDs remain router-generated (contrary to the previous RFCs). To achieve these objectives the queue data will include fixed (immutable) and user-defined (mutable) parts. Fixed part would include: -- full connection request (the current long link with all keys, including PQ keys). This includes SenderId that must match server response. +- full connection request (the current long link with all keys, including PQ keys). This includes SenderId that must match router response. - public signature key to verify mutable part of link data. Signed mutable part would include: @@ -41,7 +50,7 @@ The link itself should include both the key and auth tag from the encryption of ## Solution -Current NEW and NKEY commands: +Current NEW and NKEY commands (code identifiers like `QueueIdsKeys` are Haskell type names): ```haskell NEW :: RcvPublicAuthKey -> RcvPublicDhKey -> Maybe BasicAuth -> SubscriptionMode -> SenderCanSecure -> Command Recipient @@ -76,8 +85,8 @@ data QueueReqData | QRContact (Maybe (LinkId, (SenderId, QueueLinkData))) -- SenderId should be computed client-side as the first 24 bytes of sha3-384(correlation_id), --- The server must verify it and reject if it is not. --- It allows to include sender ID inside encrypted associated link data as part of full connection URI without requesting it from the server, but prevents checking if a given sender ID exists (queue creation would fail for a duplicate sender ID), as sha3-384 derivation is not reversible. +-- The router must verify it and reject if it is not. +-- It allows to include sender ID inside encrypted associated link data as part of full connection URI without requesting it from the router, but prevents checking if a given sender ID exists (queue creation would fail for a duplicate sender ID), as sha3-384 derivation is not reversible. type QueueLinkData = (EncFixedLinkData, EncUserDataBytes) type EncFixedLinkData = ByteString @@ -86,7 +95,7 @@ type EncUserDataBytes = ByteString -- We need to use binary encoding for ConnectionRequestUri to reduce its size -- The clients would reject changed immutable data and --- ConnectionRequestUri where server or SenderId of the queue do not match. +-- ConnectionRequestUri where router or SenderId of the queue do not match. data FixedLinkData c = FixedLinkData { agentVRange :: VersionRangeSMPA, rootKey :: C.PublicKeyEd25519, @@ -110,11 +119,11 @@ newtype UserLinkData = UserLinkData ByteString -- | Updated queue IDs and keys, returned in IDS response data QueueIdsKeys = QIK - { rcvId :: RecipientId, -- server-generated - sndId :: SenderId, -- server-generated + { rcvId :: RecipientId, -- router-generated + sndId :: SenderId, -- router-generated rcvPublicDhKey :: RcvPublicDhKey, sndSecure :: SenderCanSecure, -- possibly, can be removed? or implied? - linkId :: Maybe LinkId -- server-generated + linkId :: Maybe LinkId -- router-generated } ``` @@ -149,31 +158,31 @@ LGET :: Command Sender LNK :: SenderId -> QueueLinkData -> BrokerMsg ``` -To both include sender_id into the full link before the server response, and to prevent "oracle attack" when a failure to create the queue with the supplied `sender_id` can be used as a proof of queue existence, it is proposed that `sender_id` is computed client-side as the first 24 bytes of 48 in `sha3-384(correlation_id)` and validated server-side, where `corelation_id` is the transmission correlation ID. +To both include sender_id into the full link before the router response, and to prevent "oracle attack" when a failure to create the queue with the supplied `sender_id` can be used as a proof of queue existence, it is proposed that `sender_id` is computed client-side as the first 24 bytes of 48 in `sha3-384(correlation_id)` and validated router-side, where `corelation_id` is the transmission correlation ID. -To allow retries, every time the command is sent a new random `correlation_id` and new `sender_id` (and for contact queue, also `link_id`, which would be random as it is derived from hash of fixed link data that includes a random signature key) should be used on each attempt, because other IDs would be generated randomly on the server, and in case the previous command succeeded on the server but failed to be communicated to the client, the retry will fail if the same ID is used. +To allow retries, every time the command is sent a new random `correlation_id` and new `sender_id` (and for contact queue, also `link_id`, which would be random as it is derived from hash of fixed link data that includes a random signature key) should be used on each attempt, because other IDs would be generated randomly on the router, and in case the previous command succeeded on the router but failed to be communicated to the client, the retry will fail if the same ID is used. Alternative solutions that would allow retries that were considered and rejected: -- additional request to save queue data, after `sender_id` is returned by the server. The scenarios that require short links are interactive - creating user addresses and 1-time invitations - so making two requests instead of one would make the UX worse. -- include empty sender_id in the immutable data and have it replaced by the accepting party with `sender_id` received in `LINK` response - both a weird design, and might create possibility for some attacks via server, especially for contact addresses. +- additional request to save queue data, after `sender_id` is returned by the router. The scenarios that require short links are interactive - creating user addresses and 1-time invitations - so making two requests instead of one would make the UX worse. +- include empty sender_id in the immutable data and have it replaced by the accepting party with `sender_id` received in `LINK` response - both a weird design, and might create possibility for some attacks via router, especially for contact addresses. - making NEW commands idempotent. Doing it would require generating all IDs client-side, not only `sender_id`. It increases complexity, and it is not really necessary as the only scenarios when retries are needed are async NEW commands, that do not require short links. For future short links of chat relays the retries are much less likely, as chat relays will have good network connections. ## Algorithm to prepare and to interpret queue link data. -For contact addresses this approach follows the design proposed in [Short links](./2024-06-21-short-links.md) RFC - when link id is derived from the same random binary as key. For 1-time invitations link ID is independent and server-generated, to prevent existence checks (oracle attack). +For contact addresses this approach follows the design proposed in [Short links](./2024-06-21-short-links.md) RFC - when link id is derived from the same random binary as key. For 1-time invitations link ID is independent and router-generated, to prevent existence checks (oracle attack). This scheme results in 32 byte binary size for contact addresses and 56 bytes for 1-time invitation links. For fixed link data. -1. Generate random `nonce` (also used as a correlation ID for server command) and signature key (public `rootKey` included in fixed data). +1. Generate random `nonce` (also used as a correlation ID for router command) and signature key (public `rootKey` included in fixed data). 2. Compute sender ID from `nonce` as the first 24 bytes of sha3-384 of `nonce`. 3. Generate other keys for queue address, including queue e2e encryption keys and double ratchet connection e2e encryption keys. 4. Construct the full connection address to be included in fixed data. 5. `link_key = SHA3-256(fixed_data)` - used as part of the link, and to derive the key to encrypt content. 6. HKDF: 1) contact address: `(link_id, key) = HKDF(link_key, 56 bytes)`. - 2) 1-time invitation: `key = HKDF(link_key, 32 bytes)`, `link-id` - server-generated. + 2) 1-time invitation: `key = HKDF(link_key, 32 bytes)`, `link-id` - router-generated. 7. Encrypt: `(ct1, tag1) = secret_box(fixed_data, key, nonce1)`, where `nonce1` is a random nonce 5. Store: `(nonce1, ct1, tag1)` stored as fixed link data. @@ -202,7 +211,7 @@ While using content hash as encryption key is unconventional, it is not complete ## Threat model -**Compromised SMP server** +**Compromised SMP router** can: - delete link data. @@ -223,22 +232,22 @@ cannot: - undetectably check the existence of messaging queue or 1-time link (objective 8). - replace or delete the link data. -**Queue owner who did not compromise the server**: +**Queue owner who did not compromise the router**: cannot: -- redirect connecting user to another queue, on the same or on another server (objective 1). +- redirect connecting user to another queue, on the same or on another router (objective 1). - replace connection request in the link (objective 4). ## Correlation of design objectives with design elements -1. The presence of `SenderId` in `LNK` response from the server. +1. The presence of `SenderId` in `LNK` response from the router. 2. Encryption of link data with crypto_box. -3. Deriving encryption key from the hash of fixed data prevents it being modified by the server - any change would be detected and rejected by the client, as the hash of fixed data won't match the link. Signature verification with the key from fixed data, and signing of mutable data prevents server modification of mutable data. -4. No server command to change fixed data once it's set. Also, changing fixed data would require changing the link. +3. Deriving encryption key from the hash of fixed data prevents it being modified by the router - any change would be detected and rejected by the client, as the hash of fixed data won't match the link. Signature verification with the key from fixed data, and signing of mutable data prevents router modification of mutable data. +4. No router command to change fixed data once it's set. Also, changing fixed data would require changing the link. 5. 1-time link data can only be accessed with `LKEY` command, that while allows retries to mitigate network failures, will require the same key for retries. 6. `LSET` command. -7. The link is derived from fixed data only, so it does not change when mutable link data changes. Mutable part is signed preventing server MITM attacks. -8. SenderId is derived from request correlation ID, so it cannot be arbitrary defined to check existence of some known queue. LinkId for 1-time invitation is generated server-side, so it cannot be provided by the client when creating the queues to check if these IDs are used. +7. The link is derived from fixed data only, so it does not change when mutable link data changes. Mutable part is signed preventing router MITM attacks. +8. SenderId is derived from request correlation ID, so it cannot be arbitrary defined to check existence of some known queue. LinkId for 1-time invitation is generated router-side, so it cannot be provided by the client when creating the queues to check if these IDs are used. ## Syntax for short links @@ -257,34 +266,34 @@ contactLink = ; 32 bytes / 43 base64 encoded characters param = hostsParam / portParam / certHashParam hostsParam = %s"h=" host *("," host) ; additional hostnames, e.g. onion -portParam = %s"p=" 1*DIGIT ; server port -certHashParam = %s"c=" +portParam = %s"p=" 1*DIGIT ; router port +certHashParam = %s"c=" ``` -To have shorter links fingerprint and additional server hostnames do not need to be specified for pre-configured servers, even if they are disabled - they can be used from the client code. Any user defined servers will require including additional hosts and server fingerprint. +To have shorter links fingerprint and additional router hostnames do not need to be specified for pre-configured routers, even if they are disabled - they can be used from the client code. Any user defined routers will require including additional hosts and router fingerprint. -Example one-time link for preset server (104 characters): +Example one-time link for preset router (104 characters): ``` https://smp12.simplex.im/i#abcdefghij0123456789abcdefghij01/23456789abcdefghij0123456789abcdefghij01234 ``` -Example contact link for preset server (71 characters): +Example contact link for preset router (71 characters): ``` https://smp12.simplex.im/c#abcdefghij0123456789abcdefghij0123456789abc ``` -Example contact link for user-defined server (with fingerprint, but without onion hostname - 117 characters): +Example contact link for user-defined router (with fingerprint, but without onion hostname - 117 characters): ``` https://smp1.example.com/c#abcdefghij0123456789abcdefghij0123456789abc?c=0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU ``` -Example contact link for user-defined server (with fingerprint ant onion hostname - 182 characters): +Example contact link for user-defined router (with fingerprint and onion hostname - 182 characters): ``` https://smp1.example.com/c#abcdefghij0123456789abcdefghij0123456789abc?c=0YuTwO05YJWS8rkjn9eLJDjQhFKvIYd8d4xG8X1blIU&h=beccx4yfxxbvyhqypaavemqurytl6hozr47wfc7uuecacjqdvwpw2xid.onion ``` -For the links to work in the browser the servers must provide server pages. +For the links to work in the browser the routers must provide router pages. diff --git a/simplexmq.cabal b/simplexmq.cabal index 7f856543d..d1c1e03bc 100644 --- a/simplexmq.cabal +++ b/simplexmq.cabal @@ -168,6 +168,7 @@ library Simplex.Messaging.Agent.Store.Postgres.Migrations.M20251009_queue_to_subscribe Simplex.Messaging.Agent.Store.Postgres.Migrations.M20251010_client_notices Simplex.Messaging.Agent.Store.Postgres.Migrations.M20251230_strict_tables + Simplex.Messaging.Agent.Store.Postgres.Migrations.M20260115_service_certs else exposed-modules: Simplex.Messaging.Agent.Store.SQLite @@ -218,6 +219,7 @@ library Simplex.Messaging.Agent.Store.SQLite.Migrations.M20251009_queue_to_subscribe Simplex.Messaging.Agent.Store.SQLite.Migrations.M20251010_client_notices Simplex.Messaging.Agent.Store.SQLite.Migrations.M20251230_strict_tables + Simplex.Messaging.Agent.Store.SQLite.Migrations.M20260115_service_certs Simplex.Messaging.Agent.Store.SQLite.Util if flag(client_postgres) || flag(server_postgres) exposed-modules: @@ -225,6 +227,7 @@ library Simplex.Messaging.Agent.Store.Postgres.Common Simplex.Messaging.Agent.Store.Postgres.DB Simplex.Messaging.Agent.Store.Postgres.Migrations + Simplex.Messaging.Agent.Store.Postgres.Migrations.Util Simplex.Messaging.Agent.Store.Postgres.Util if !flag(client_library) exposed-modules: @@ -275,7 +278,6 @@ library Simplex.Messaging.Notifications.Server.Store.Migrations Simplex.Messaging.Notifications.Server.Store.Postgres Simplex.Messaging.Notifications.Server.Store.Types - Simplex.Messaging.Notifications.Server.StoreLog Simplex.Messaging.Server.MsgStore.Postgres Simplex.Messaging.Server.QueueStore.Postgres Simplex.Messaging.Server.QueueStore.Postgres.Migrations @@ -518,6 +520,8 @@ test-suite simplexmq-test AgentTests.NotificationTests NtfClient NtfServerTests + if flag(client_postgres) || flag(server_postgres) + other-modules: PostgresSchemaDump hs-source-dirs: tests @@ -566,6 +570,7 @@ test-suite simplexmq-test , text , time , timeit ==2.0.* + , tls >=1.9.0 && <1.10 , transformers , unliftio , unliftio-core diff --git a/src/Simplex/FileTransfer/Client.hs b/src/Simplex/FileTransfer/Client.hs index 30dc5b12a..9d19d0492 100644 --- a/src/Simplex/FileTransfer/Client.hs +++ b/src/Simplex/FileTransfer/Client.hs @@ -32,6 +32,7 @@ module Simplex.FileTransfer.Client getChunkDigest, ) where +import qualified Control.Exception as E import Control.Logger.Simple import Control.Monad import Control.Monad.Except @@ -67,6 +68,7 @@ import Simplex.Messaging.Client netTimeoutInt, transportClientConfig, unexpectedResponse, + clientHandlers, useWebPort, ) import qualified Simplex.Messaging.Crypto as C @@ -80,7 +82,6 @@ import Simplex.Messaging.Protocol ProtocolServer (..), RecipientId, SenderId, - toNetworkError, pattern NoEntity, ) import Simplex.Messaging.Transport (ALPN, CertChainPubKey (..), HandshakeError (..), THandleAuth (..), THandleParams (..), TransportError (..), TransportPeer (..), defaultSupportedParams) @@ -90,8 +91,10 @@ import Simplex.Messaging.Transport.HTTP2.Client import Simplex.Messaging.Transport.HTTP2.File import Simplex.Messaging.Util (liftEitherWith, liftError', tshow, whenM) import Simplex.Messaging.Version -import UnliftIO +import System.IO (IOMode (..), SeekMode (..), hSeek, withFile) +import System.Timeout (timeout) import UnliftIO.Directory +import UnliftIO.STM data XFTPClient = XFTPClient { http2Client :: HTTP2Client, @@ -282,13 +285,11 @@ downloadXFTPChunk g c@XFTPClient {config} rpKey fId chunkSpec@XFTPRcvChunkSpec { let dhSecret = C.dh' sDhKey rpDhKey cbState <- liftEither . first PCECryptoError $ LC.cbInit dhSecret cbNonce let t = chunkTimeout config chunkSize - ExceptT (sequence <$> (t `timeout` (download cbState `catches` errors))) >>= maybe (throwE PCEResponseTimeout) pure + ExceptT (sequence <$> (t `timeout` (download cbState `E.catches` handlers))) >>= maybe (throwE PCEResponseTimeout) pure where - errors = - [ Handler $ \(e :: H.HTTP2Error) -> pure $ Left $ PCENetworkError $ NEConnectError $ displayException e, - Handler $ \(e :: IOException) -> pure $ Left $ PCEIOError e, - Handler $ \(e :: SomeException) -> pure $ Left $ PCENetworkError $ toNetworkError e - ] + handlers = + E.Handler (\(e :: H.HTTP2Error) -> pure $ Left $ PCENetworkError $ NEConnectError $ E.displayException e) + : clientHandlers download cbState = runExceptT . withExceptT PCEResponseError $ receiveEncFile chunkPart cbState chunkSpec `catchError` \e -> diff --git a/src/Simplex/Messaging/Agent.hs b/src/Simplex/Messaging/Agent.hs index e8eb22fcf..9e637ca96 100644 --- a/src/Simplex/Messaging/Agent.hs +++ b/src/Simplex/Messaging/Agent.hs @@ -47,6 +47,7 @@ module Simplex.Messaging.Agent withInvLock, createUser, deleteUser, + setUserService, connRequestPQSupport, createConnectionAsync, setConnShortLinkAsync, @@ -82,7 +83,7 @@ module Simplex.Messaging.Agent getNotificationConns, resubscribeConnection, resubscribeConnections, - subscribeClientService, + subscribeClientServices, sendMessage, sendMessages, sendMessagesB, @@ -156,7 +157,7 @@ import Data.Bifunctor (bimap, first) import Data.ByteString.Char8 (ByteString) import qualified Data.ByteString.Char8 as B import Data.Composition -import Data.Either (isRight, partitionEithers, rights) +import Data.Either (fromRight, isRight, partitionEithers, rights) import Data.Foldable (foldl', toList) import Data.Functor (($>)) import Data.Functor.Identity @@ -197,7 +198,7 @@ import Simplex.Messaging.Agent.Store.Entity import Simplex.Messaging.Agent.Store.Interface (closeDBStore, execSQL, getCurrentMigrations) import Simplex.Messaging.Agent.Store.Shared (UpMigration (..), upMigration) import qualified Simplex.Messaging.Agent.TSessionSubs as SS -import Simplex.Messaging.Client (NetworkRequestMode (..), SMPClientError, ServerTransmission (..), ServerTransmissionBatch, nonBlockingWriteTBQueue, smpErrorClientNotice, temporaryClientError, unexpectedResponse) +import Simplex.Messaging.Client (NetworkRequestMode (..), ProtocolClientError (..), SMPClientError, ServerTransmission (..), ServerTransmissionBatch, TransportSessionMode (..), nonBlockingWriteTBQueue, smpErrorClientNotice, temporaryClientError, unexpectedResponse) import qualified Simplex.Messaging.Crypto as C import Simplex.Messaging.Crypto.File (CryptoFile, CryptoFileArgs) import Simplex.Messaging.Crypto.Ratchet (PQEncryption, PQSupport (..), pattern PQEncOff, pattern PQEncOn, pattern PQSupportOff, pattern PQSupportOn) @@ -224,6 +225,9 @@ import Simplex.Messaging.Protocol SMPMsgMeta, SParty (..), SProtocolType (..), + ServiceSub (..), + ServiceSubError (..), + ServiceSubResult (..), SndPublicAuthKey, SubscriptionMode (..), UserProtocol, @@ -234,7 +238,7 @@ import qualified Simplex.Messaging.Protocol as SMP import Simplex.Messaging.ServiceScheme (ServiceScheme (..)) import Simplex.Messaging.SystemTime import qualified Simplex.Messaging.TMap as TM -import Simplex.Messaging.Transport (SMPVersion) +import Simplex.Messaging.Transport (SMPVersion, THClientService' (..), THandleAuth (..), THandleParams (..)) import Simplex.Messaging.Util import Simplex.Messaging.Version import Simplex.RemoteControl.Client @@ -251,13 +255,15 @@ import UnliftIO.STM type AE a = ExceptT AgentErrorType IO a -- | Creates an SMP agent client instance -getSMPAgentClient :: AgentConfig -> InitialAgentServers -> DBStore -> Bool -> IO AgentClient +getSMPAgentClient :: AgentConfig -> InitialAgentServers -> DBStore -> Bool -> AE AgentClient getSMPAgentClient = getSMPAgentClient_ 1 {-# INLINE getSMPAgentClient #-} -getSMPAgentClient_ :: Int -> AgentConfig -> InitialAgentServers -> DBStore -> Bool -> IO AgentClient -getSMPAgentClient_ clientId cfg initServers@InitialAgentServers {smp, xftp, presetServers} store backgroundMode = - newSMPAgentEnv cfg store >>= runReaderT runAgent +getSMPAgentClient_ :: Int -> AgentConfig -> InitialAgentServers -> DBStore -> Bool -> AE AgentClient +getSMPAgentClient_ clientId cfg initServers@InitialAgentServers {smp, xftp, netCfg, useServices, presetServers} store backgroundMode = do + -- This error should be prevented in the app + when (any id useServices && sessionMode netCfg == TSMEntity) $ throwE $ CMD PROHIBITED "newAgentClient" + liftIO $ newSMPAgentEnv cfg store >>= runReaderT runAgent where runAgent = do liftIO $ checkServers "SMP" smp >> checkServers "XFTP" xftp @@ -335,8 +341,8 @@ resumeAgentClient :: AgentClient -> IO () resumeAgentClient c = atomically $ writeTVar (active c) True {-# INLINE resumeAgentClient #-} -createUser :: AgentClient -> NonEmpty (ServerCfg 'PSMP) -> NonEmpty (ServerCfg 'PXFTP) -> AE UserId -createUser c = withAgentEnv c .: createUser' c +createUser :: AgentClient -> Bool -> NonEmpty (ServerCfg 'PSMP) -> NonEmpty (ServerCfg 'PXFTP) -> AE UserId +createUser c = withAgentEnv c .:. createUser' c {-# INLINE createUser #-} -- | Delete user record optionally deleting all user's connections on SMP servers @@ -344,6 +350,11 @@ deleteUser :: AgentClient -> UserId -> Bool -> AE () deleteUser c = withAgentEnv c .: deleteUser' c {-# INLINE deleteUser #-} +-- | Enable using service certificate for this user +setUserService :: AgentClient -> UserId -> Bool -> AE () +setUserService c = withAgentEnv c .: setUserService' c +{-# INLINE setUserService #-} + -- | Create SMP agent connection (NEW command) asynchronously, synchronous response is new connection id createConnectionAsync :: ConnectionModeI c => AgentClient -> UserId -> ACorrId -> Bool -> SConnectionMode c -> CR.InitialKeys -> SubscriptionMode -> AE ConnId createConnectionAsync c userId aCorrId enableNtfs = withAgentEnv c .:. newConnAsync c userId aCorrId enableNtfs @@ -396,7 +407,7 @@ deleteConnectionsAsync c waitDelivery = withAgentEnv c . deleteConnectionsAsync' {-# INLINE deleteConnectionsAsync #-} -- | Create SMP agent connection (NEW command) -createConnection :: ConnectionModeI c => AgentClient -> NetworkRequestMode -> UserId -> Bool -> Bool -> SConnectionMode c -> Maybe (UserConnLinkData c) -> Maybe CRClientData -> CR.InitialKeys -> SubscriptionMode -> AE (ConnId, (CreatedConnLink c, Maybe ClientServiceId)) +createConnection :: ConnectionModeI c => AgentClient -> NetworkRequestMode -> UserId -> Bool -> Bool -> SConnectionMode c -> Maybe (UserConnLinkData c) -> Maybe CRClientData -> CR.InitialKeys -> SubscriptionMode -> AE (ConnId, CreatedConnLink c) createConnection c nm userId enableNtfs checkNotices = withAgentEnv c .::. newConn c nm userId enableNtfs checkNotices {-# INLINE createConnection #-} @@ -452,7 +463,7 @@ prepareConnectionToAccept c userId enableNtfs = withAgentEnv c .: newConnToAccep {-# INLINE prepareConnectionToAccept #-} -- | Join SMP agent connection (JOIN command). -joinConnection :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> AE (SndQueueSecured, Maybe ClientServiceId) +joinConnection :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> AE SndQueueSecured joinConnection c nm userId connId enableNtfs = withAgentEnv c .:: joinConn c nm userId connId enableNtfs {-# INLINE joinConnection #-} @@ -462,7 +473,7 @@ allowConnection c = withAgentEnv c .:. allowConnection' c {-# INLINE allowConnection #-} -- | Accept contact after REQ notification (ACPT command) -acceptContact :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConfirmationId -> ConnInfo -> PQSupport -> SubscriptionMode -> AE (SndQueueSecured, Maybe ClientServiceId) +acceptContact :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConfirmationId -> ConnInfo -> PQSupport -> SubscriptionMode -> AE SndQueueSecured acceptContact c userId connId enableNtfs = withAgentEnv c .::. acceptContact' c userId connId enableNtfs {-# INLINE acceptContact #-} @@ -490,12 +501,12 @@ syncConnections c = withAgentEnv c .: syncConnections' c {-# INLINE syncConnections #-} -- | Subscribe to receive connection messages (SUB command) -subscribeConnection :: AgentClient -> ConnId -> AE (Maybe ClientServiceId) +subscribeConnection :: AgentClient -> ConnId -> AE () subscribeConnection c = withAgentEnv c . subscribeConnection' c {-# INLINE subscribeConnection #-} -- | Subscribe to receive connection messages from multiple connections, batching commands when possible -subscribeConnections :: AgentClient -> [ConnId] -> AE (Map ConnId (Either AgentErrorType (Maybe ClientServiceId))) +subscribeConnections :: AgentClient -> [ConnId] -> AE (Map ConnId (Either AgentErrorType ())) subscribeConnections c = withAgentEnv c . subscribeConnections' c {-# INLINE subscribeConnections #-} @@ -513,18 +524,17 @@ getNotificationConns :: AgentClient -> C.CbNonce -> ByteString -> AE (NonEmpty N getNotificationConns c = withAgentEnv c .: getNotificationConns' c {-# INLINE getNotificationConns #-} -resubscribeConnection :: AgentClient -> ConnId -> AE (Maybe ClientServiceId) +resubscribeConnection :: AgentClient -> ConnId -> AE () resubscribeConnection c = withAgentEnv c . resubscribeConnection' c {-# INLINE resubscribeConnection #-} -resubscribeConnections :: AgentClient -> [ConnId] -> AE (Map ConnId (Either AgentErrorType (Maybe ClientServiceId))) +resubscribeConnections :: AgentClient -> [ConnId] -> AE (Map ConnId (Either AgentErrorType ())) resubscribeConnections c = withAgentEnv c . resubscribeConnections' c {-# INLINE resubscribeConnections #-} --- TODO [certs rcv] how to communicate that service ID changed - as error or as result? -subscribeClientService :: AgentClient -> ClientServiceId -> AE Int -subscribeClientService c = withAgentEnv c . subscribeClientService' c -{-# INLINE subscribeClientService #-} +subscribeClientServices :: AgentClient -> UserId -> AE (Map SMPServer (Either AgentErrorType ServiceSubResult)) +subscribeClientServices c = withAgentEnv c . subscribeClientServices' c +{-# INLINE subscribeClientServices #-} -- | Send message to the connection (SEND command) sendMessage :: AgentClient -> ConnId -> PQEncryption -> MsgFlags -> MsgBody -> AE (AgentMsgId, PQEncryption) @@ -616,17 +626,22 @@ testProtocolServer c nm userId srv = withAgentEnv' c $ case protocolTypeI @p of SPNTF -> runNTFServerTest c nm userId srv -- | set SOCKS5 proxy on/off and optionally set TCP timeouts for fast network -setNetworkConfig :: AgentClient -> NetworkConfig -> IO () +setNetworkConfig :: AgentClient -> NetworkConfig -> AE () setNetworkConfig c@AgentClient {useNetworkConfig, proxySessTs} cfg' = do - ts <- getCurrentTime - changed <- atomically $ do - (_, cfg) <- readTVar useNetworkConfig - let changed = cfg /= cfg' - !cfgSlow = slowNetworkConfig cfg' - when changed $ writeTVar useNetworkConfig (cfgSlow, cfg') - when (socksProxy cfg /= socksProxy cfg') $ writeTVar proxySessTs ts - pure changed - when changed $ reconnectAllServers c + ts <- liftIO getCurrentTime + (ok, changed) <- atomically $ do + useServices <- readTVar $ useClientServices c + if any id useServices && sessionMode cfg' == TSMEntity + then pure (False, False) + else do + (_, cfg) <- readTVar useNetworkConfig + let changed = cfg /= cfg' + !cfgSlow = slowNetworkConfig cfg' + when changed $ writeTVar useNetworkConfig (cfgSlow, cfg') + when (socksProxy cfg /= socksProxy cfg') $ writeTVar proxySessTs ts + pure (True, changed) + unless ok $ throwE $ CMD PROHIBITED "setNetworkConfig" + when changed $ liftIO $ reconnectAllServers c setUserNetworkInfo :: AgentClient -> UserNetworkInfo -> IO () setUserNetworkInfo c@AgentClient {userNetworkInfo, userNetworkUpdated} ni = withAgentEnv' c $ do @@ -767,13 +782,23 @@ logConnection c connected = let event = if connected then "connected to" else "disconnected from" in logInfo $ T.unwords ["client", tshow (clientId c), event, "Agent"] -createUser' :: AgentClient -> NonEmpty (ServerCfg 'PSMP) -> NonEmpty (ServerCfg 'PXFTP) -> AM UserId -createUser' c smp xftp = do +createUser' :: AgentClient -> Bool -> NonEmpty (ServerCfg 'PSMP) -> NonEmpty (ServerCfg 'PXFTP) -> AM UserId +createUser' c useService smp xftp = do liftIO $ checkUserServers "createUser SMP" smp liftIO $ checkUserServers "createUser XFTP" xftp userId <- withStore' c createUserRecord - atomically $ TM.insert userId (mkUserServers smp) $ smpServers c - atomically $ TM.insert userId (mkUserServers xftp) $ xftpServers c + ok <- atomically $ do + (cfg, _) <- readTVar $ useNetworkConfig c + if useService && sessionMode cfg == TSMEntity + then pure False + else do + TM.insert userId (mkUserServers smp) $ smpServers c + TM.insert userId (mkUserServers xftp) $ xftpServers c + TM.insert userId useService $ useClientServices c + pure True + unless ok $ do + withStore c (`deleteUserRecord` userId) + throwE $ CMD PROHIBITED "createUser'" pure userId deleteUser' :: AgentClient -> UserId -> Bool -> AM () @@ -783,6 +808,7 @@ deleteUser' c@AgentClient {smpServersStats, xftpServersStats} userId delSMPQueue else withStore c (`deleteUserRecord` userId) atomically $ TM.delete userId $ smpServers c atomically $ TM.delete userId $ xftpServers c + atomically $ TM.delete userId $ useClientServices c atomically $ modifyTVar' smpServersStats $ M.filterWithKey (\(userId', _) _ -> userId' /= userId) atomically $ modifyTVar' xftpServersStats $ M.filterWithKey (\(userId', _) _ -> userId' /= userId) lift $ saveServersStats c @@ -791,6 +817,20 @@ deleteUser' c@AgentClient {smpServersStats, xftpServersStats} userId delSMPQueue whenM (withStore' c (`deleteUserWithoutConns` userId)) . atomically $ writeTBQueue (subQ c) ("", "", AEvt SAENone $ DEL_USER userId) +setUserService' :: AgentClient -> UserId -> Bool -> AM () +setUserService' c userId enable = do + (ok, changed) <- atomically $ do + (cfg, _) <- readTVar $ useNetworkConfig c + if enable && sessionMode cfg == TSMEntity + then pure (False, False) + else do + wasEnabled <- fromMaybe False <$> TM.lookup userId (useClientServices c) + let changed = enable /= wasEnabled + when changed $ TM.insert userId enable $ useClientServices c + pure (True, changed) + unless ok $ throwE $ CMD PROHIBITED "setNetworkConfig" + when (changed && not enable) $ withStore' c (`deleteClientServices` userId) + newConnAsync :: ConnectionModeI c => AgentClient -> UserId -> ACorrId -> Bool -> SConnectionMode c -> CR.InitialKeys -> SubscriptionMode -> AM ConnId newConnAsync c userId corrId enableNtfs cMode pqInitKeys subMode = do connId <- newConnNoQueues c userId enableNtfs cMode (CR.connPQEncryption pqInitKeys) @@ -908,7 +948,7 @@ switchConnectionAsync' c corrId connId = connectionStats c $ DuplexConnection cData rqs' sqs _ -> throwE $ CMD PROHIBITED "switchConnectionAsync: not duplex" -newConn :: ConnectionModeI c => AgentClient -> NetworkRequestMode -> UserId -> Bool -> Bool -> SConnectionMode c -> Maybe (UserConnLinkData c) -> Maybe CRClientData -> CR.InitialKeys -> SubscriptionMode -> AM (ConnId, (CreatedConnLink c, Maybe ClientServiceId)) +newConn :: ConnectionModeI c => AgentClient -> NetworkRequestMode -> UserId -> Bool -> Bool -> SConnectionMode c -> Maybe (UserConnLinkData c) -> Maybe CRClientData -> CR.InitialKeys -> SubscriptionMode -> AM (ConnId, CreatedConnLink c) newConn c nm userId enableNtfs checkNotices cMode linkData_ clientData pqInitKeys subMode = do srv <- getSMPServer c userId when (checkNotices && connMode cMode == CMContact) $ checkClientNotices c srv @@ -968,12 +1008,12 @@ createRcvQueue :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> SMPSe createRcvQueue c nm userId connId srvWithAuth@(ProtoServerWithAuth srv _) enableNtfs subMode nonce_ qd e2eKeys = do AgentConfig {smpClientVRange = vr} <- asks config ntfServer_ <- if enableNtfs then newQueueNtfServer else pure Nothing - (rq, qUri, tSess, sessId) <- + (rq, qUri, tSess, sessId, serviceId_) <- newRcvQueue_ c nm userId connId srvWithAuth vr qd (isJust ntfServer_) subMode nonce_ e2eKeys `catchAllErrors` \e -> liftIO (print e) >> throwE e atomically $ incSMPServerStat c userId srv connCreated rq' <- withStore c $ \db -> updateNewConnRcv db connId rq subMode - lift . when (subMode == SMSubscribe) $ addNewQueueSubscription c rq' tSess sessId + lift . when (subMode == SMSubscribe) $ addNewQueueSubscription c rq' tSess sessId serviceId_ mapM_ (newQueueNtfSubscription c rq') ntfServer_ pure (rq', qUri) @@ -1131,7 +1171,7 @@ changeConnectionUser' c oldUserId connId newUserId = do where updateConn = withStore' c $ \db -> setConnUserId db oldUserId connId newUserId -newRcvConnSrv :: forall c. ConnectionModeI c => AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> SConnectionMode c -> Maybe (UserConnLinkData c) -> Maybe CRClientData -> CR.InitialKeys -> SubscriptionMode -> SMPServerWithAuth -> AM (CreatedConnLink c, Maybe ClientServiceId) +newRcvConnSrv :: forall c. ConnectionModeI c => AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> SConnectionMode c -> Maybe (UserConnLinkData c) -> Maybe CRClientData -> CR.InitialKeys -> SubscriptionMode -> SMPServerWithAuth -> AM (CreatedConnLink c) newRcvConnSrv c nm userId connId enableNtfs cMode userLinkData_ clientData pqInitKeys subMode srvWithAuth@(ProtoServerWithAuth srv _) = do case (cMode, pqInitKeys) of (SCMContact, CR.IKUsePQ) -> throwE $ CMD PROHIBITED "newRcvConnSrv" @@ -1142,12 +1182,12 @@ newRcvConnSrv c nm userId connId enableNtfs cMode userLinkData_ clientData pqIni (nonce, qUri, cReq, qd) <- prepareLinkData d $ fst e2eKeys (rq, qUri') <- createRcvQueue c nm userId connId srvWithAuth enableNtfs subMode (Just nonce) qd e2eKeys ccLink <- connReqWithShortLink qUri cReq qUri' (shortLink rq) - pure (ccLink, clientServiceId rq) + pure ccLink Nothing -> do let qd = case cMode of SCMContact -> CQRContact Nothing; SCMInvitation -> CQRMessaging Nothing - (rq, qUri) <- createRcvQueue c nm userId connId srvWithAuth enableNtfs subMode Nothing qd e2eKeys + (_rq, qUri) <- createRcvQueue c nm userId connId srvWithAuth enableNtfs subMode Nothing qd e2eKeys cReq <- createConnReq qUri - pure (CCLink cReq Nothing, clientServiceId rq) + pure $ CCLink cReq Nothing where createConnReq :: SMPQueueUri -> AM (ConnectionRequestUri c) createConnReq qUri = do @@ -1236,7 +1276,7 @@ newConnToAccept c userId connId enableNtfs invId pqSup = do Invitation {connReq} <- withStore c $ \db -> getInvitation db "newConnToAccept" invId newConnToJoin c userId connId enableNtfs connReq pqSup -joinConn :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> AM (SndQueueSecured, Maybe ClientServiceId) +joinConn :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> AM SndQueueSecured joinConn c nm userId connId enableNtfs cReq cInfo pqSupport subMode = do srv <- getNextSMPServer c userId [qServer $ connReqQueue cReq] joinConnSrv c nm userId connId enableNtfs cReq cInfo pqSupport subMode srv @@ -1318,7 +1358,7 @@ versionPQSupport_ :: VersionSMPA -> Maybe CR.VersionE2E -> PQSupport versionPQSupport_ agentV e2eV_ = PQSupport $ agentV >= pqdrSMPAgentVersion && maybe True (>= CR.pqRatchetE2EEncryptVersion) e2eV_ {-# INLINE versionPQSupport_ #-} -joinConnSrv :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> SMPServerWithAuth -> AM (SndQueueSecured, Maybe ClientServiceId) +joinConnSrv :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> SMPServerWithAuth -> AM SndQueueSecured joinConnSrv c nm userId connId enableNtfs inv@CRInvitationUri {} cInfo pqSup subMode srv = withInvLock c (strEncode inv) "joinConnSrv" $ do SomeConn cType conn <- withStore c (`getConn` connId) @@ -1329,7 +1369,7 @@ joinConnSrv c nm userId connId enableNtfs inv@CRInvitationUri {} cInfo pqSup sub | sqStatus == New || sqStatus == Secured -> doJoin (Just rq) (Just sq) _ -> throwE $ CMD PROHIBITED $ "joinConnSrv: bad connection " <> show cType where - doJoin :: Maybe RcvQueue -> Maybe SndQueue -> AM (SndQueueSecured, Maybe ClientServiceId) + doJoin :: Maybe RcvQueue -> Maybe SndQueue -> AM SndQueueSecured doJoin rq_ sq_ = do (cData, sq, e2eSndParams, lnkId_) <- startJoinInvitation c userId connId sq_ enableNtfs inv pqSup secureConfirmQueue c nm cData rq_ sq srv cInfo (Just e2eSndParams) subMode @@ -1340,14 +1380,14 @@ joinConnSrv c nm userId connId enableNtfs cReqUri@CRContactUri {} cInfo pqSup su withInvLock c (strEncode cReqUri) "joinConnSrv" $ do SomeConn cType conn <- withStore c (`getConn` connId) let pqInitKeys = CR.joinContactInitialKeys (v >= pqdrSMPAgentVersion) pqSup - (CCLink cReq _, service) <- case conn of + CCLink cReq _ <- case conn of NewConnection _ -> newRcvConnSrv c NRMBackground userId connId enableNtfs SCMInvitation Nothing Nothing pqInitKeys subMode srv RcvConnection _ rq -> mkJoinInvitation rq pqInitKeys _ -> throwE $ CMD PROHIBITED $ "joinConnSrv: bad connection " <> show cType void $ sendInvitation c nm userId connId qInfo vrsn cReq cInfo - pure (False, service) + pure False where - mkJoinInvitation rq@RcvQueue {clientService} pqInitKeys = do + mkJoinInvitation rq pqInitKeys = do g <- asks random AgentConfig {smpClientVRange = vr, smpAgentVRange, e2eEncryptVRange = e2eVR} <- asks config let qUri = SMPQueueUri vr $ (rcvSMPQueueAddress rq) {queueMode = Just QMMessaging} @@ -1363,7 +1403,7 @@ joinConnSrv c nm userId connId enableNtfs cReqUri@CRContactUri {} cInfo pqSup su createRatchetX3dhKeys db connId pk1 pk2 pKem pure e2eRcvParams let cReq = CRInvitationUri crData $ toVersionRangeT e2eRcvParams e2eVR - pure (CCLink cReq Nothing, dbServiceId <$> clientService) + pure $ CCLink cReq Nothing Nothing -> throwE $ AGENT A_VERSION delInvSL :: AgentClient -> ConnId -> SMPServerWithAuth -> SMP.LinkId -> AM () @@ -1371,7 +1411,7 @@ delInvSL c connId srv lnkId = withStore' c (\db -> deleteInvShortLink db (protoServer srv) lnkId) `catchE` \e -> liftIO $ nonBlockingWriteTBQueue (subQ c) ("", connId, AEvt SAEConn (ERR $ INTERNAL $ "error deleting short link " <> show e)) -joinConnSrvAsync :: AgentClient -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> SMPServerWithAuth -> AM (SndQueueSecured, Maybe ClientServiceId) +joinConnSrvAsync :: AgentClient -> UserId -> ConnId -> Bool -> ConnectionRequestUri c -> ConnInfo -> PQSupport -> SubscriptionMode -> SMPServerWithAuth -> AM SndQueueSecured joinConnSrvAsync c userId connId enableNtfs inv@CRInvitationUri {} cInfo pqSupport subMode srv = do SomeConn cType conn <- withStore c (`getConn` connId) case conn of @@ -1383,7 +1423,7 @@ joinConnSrvAsync c userId connId enableNtfs inv@CRInvitationUri {} cInfo pqSuppo | sqStatus == New || sqStatus == Secured -> doJoin (Just rq) (Just sq) _ -> throwE $ CMD PROHIBITED $ "joinConnSrvAsync: bad connection " <> show cType where - doJoin :: Maybe RcvQueue -> Maybe SndQueue -> AM (SndQueueSecured, Maybe ClientServiceId) + doJoin :: Maybe RcvQueue -> Maybe SndQueue -> AM SndQueueSecured doJoin rq_ sq_ = do (cData, sq, e2eSndParams, lnkId_) <- startJoinInvitation c userId connId sq_ enableNtfs inv pqSupport secureConfirmQueueAsync c cData rq_ sq srv cInfo (Just e2eSndParams) subMode @@ -1391,16 +1431,16 @@ joinConnSrvAsync c userId connId enableNtfs inv@CRInvitationUri {} cInfo pqSuppo joinConnSrvAsync _c _userId _connId _enableNtfs (CRContactUri _) _cInfo _subMode _pqSupport _srv = do throwE $ CMD PROHIBITED "joinConnSrvAsync" -createReplyQueue :: AgentClient -> NetworkRequestMode -> ConnData -> SndQueue -> SubscriptionMode -> SMPServerWithAuth -> AM (SMPQueueInfo, Maybe ClientServiceId) +createReplyQueue :: AgentClient -> NetworkRequestMode -> ConnData -> SndQueue -> SubscriptionMode -> SMPServerWithAuth -> AM SMPQueueInfo createReplyQueue c nm ConnData {userId, connId, enableNtfs} SndQueue {smpClientVersion} subMode srv = do ntfServer_ <- if enableNtfs then newQueueNtfServer else pure Nothing - (rq, qUri, tSess, sessId) <- newRcvQueue c nm userId connId srv (versionToRange smpClientVersion) SCMInvitation (isJust ntfServer_) subMode + (rq, qUri, tSess, sessId, serviceId_) <- newRcvQueue c nm userId connId srv (versionToRange smpClientVersion) SCMInvitation (isJust ntfServer_) subMode atomically $ incSMPServerStat c userId (qServer rq) connCreated let qInfo = toVersionT qUri smpClientVersion rq' <- withStore c $ \db -> upgradeSndConnToDuplex db connId rq subMode - lift . when (subMode == SMSubscribe) $ addNewQueueSubscription c rq' tSess sessId + lift . when (subMode == SMSubscribe) $ addNewQueueSubscription c rq' tSess sessId serviceId_ mapM_ (newQueueNtfSubscription c rq') ntfServer_ - pure (qInfo, clientServiceId rq') + pure qInfo -- | Approve confirmation (LET command) in Reader monad allowConnection' :: AgentClient -> ConnId -> ConfirmationId -> ConnInfo -> AM () @@ -1413,7 +1453,7 @@ allowConnection' c connId confId ownConnInfo = withConnLock c connId "allowConne _ -> throwE $ CMD PROHIBITED "allowConnection" -- | Accept contact (ACPT command) in Reader monad -acceptContact' :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> InvitationId -> ConnInfo -> PQSupport -> SubscriptionMode -> AM (SndQueueSecured, Maybe ClientServiceId) +acceptContact' :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> Bool -> InvitationId -> ConnInfo -> PQSupport -> SubscriptionMode -> AM SndQueueSecured acceptContact' c nm userId connId enableNtfs invId ownConnInfo pqSupport subMode = withConnLock c connId "acceptContact" $ do Invitation {connReq} <- withStore c $ \db -> getInvitation db "acceptContact'" invId r <- joinConn c nm userId connId enableNtfs connReq ownConnInfo pqSupport subMode @@ -1448,7 +1488,7 @@ databaseDiff passed known = in DatabaseDiff {missingIds, extraIds} -- | Subscribe to receive connection messages (SUB command) in Reader monad -subscribeConnection' :: AgentClient -> ConnId -> AM (Maybe ClientServiceId) +subscribeConnection' :: AgentClient -> ConnId -> AM () subscribeConnection' c connId = toConnResult connId =<< subscribeConnections' c [connId] {-# INLINE subscribeConnection' #-} @@ -1458,34 +1498,30 @@ toConnResult connId rs = case M.lookup connId rs of Just (Left e) -> throwE e _ -> throwE $ INTERNAL $ "no result for connection " <> B.unpack connId -type QCmdResult a = (QueueStatus, Either AgentErrorType a) - -type QDelResult = QCmdResult () +type QCmdResult = (QueueStatus, Either AgentErrorType ()) -type QSubResult = QCmdResult (Maybe SMP.ServiceId) - -subscribeConnections' :: AgentClient -> [ConnId] -> AM (Map ConnId (Either AgentErrorType (Maybe ClientServiceId))) +subscribeConnections' :: AgentClient -> [ConnId] -> AM (Map ConnId (Either AgentErrorType ())) subscribeConnections' _ [] = pure M.empty subscribeConnections' c connIds = subscribeConnections_ c . zip connIds =<< withStore' c (`getConnSubs` connIds) -subscribeConnections_ :: AgentClient -> [(ConnId, Either StoreError SomeConnSub)] -> AM (Map ConnId (Either AgentErrorType (Maybe ClientServiceId))) +subscribeConnections_ :: AgentClient -> [(ConnId, Either StoreError SomeConnSub)] -> AM (Map ConnId (Either AgentErrorType ())) subscribeConnections_ c conns = do let (subRs, cs) = foldr partitionResultsConns ([], []) conns resumeDelivery cs resumeConnCmds c $ map fst cs + -- queue/service association is handled in the client rcvRs <- lift $ connResults <$> subscribeQueues c False (concatMap rcvQueues cs) - rcvRs' <- storeClientServiceAssocs rcvRs ns <- asks ntfSupervisor - lift $ whenM (liftIO $ hasInstantNotifications ns) . void . forkIO . void $ sendNtfCreate ns rcvRs' cs + lift $ whenM (liftIO $ hasInstantNotifications ns) . void . forkIO . void $ sendNtfCreate ns rcvRs cs -- union is left-biased - let rs = rcvRs' `M.union` subRs + let rs = rcvRs `M.union` subRs notifyResultError rs pure rs where partitionResultsConns :: (ConnId, Either StoreError SomeConnSub) -> - (Map ConnId (Either AgentErrorType (Maybe ClientServiceId)), [(ConnId, SomeConnSub)]) -> - (Map ConnId (Either AgentErrorType (Maybe ClientServiceId)), [(ConnId, SomeConnSub)]) + (Map ConnId (Either AgentErrorType ()), [(ConnId, SomeConnSub)]) -> + (Map ConnId (Either AgentErrorType ()), [(ConnId, SomeConnSub)]) partitionResultsConns (connId, conn_) (rs, cs) = case conn_ of Left e -> (M.insert connId (Left $ storeError e) rs, cs) Right c'@(SomeConn _ conn) -> case conn of @@ -1493,35 +1529,32 @@ subscribeConnections_ c conns = do SndConnection _ sq -> (M.insert connId (sndSubResult sq) rs, cs') RcvConnection _ _ -> (rs, cs') ContactConnection _ _ -> (rs, cs') - NewConnection _ -> (M.insert connId (Right Nothing) rs, cs') + NewConnection _ -> (M.insert connId (Right ()) rs, cs') where cs' = (connId, c') : cs - sndSubResult :: SndQueue -> Either AgentErrorType (Maybe ClientServiceId) + sndSubResult :: SndQueue -> Either AgentErrorType () sndSubResult SndQueue {status} = case status of - Confirmed -> Right Nothing + Confirmed -> Right () Active -> Left $ CONN SIMPLEX "subscribeConnections" _ -> Left $ INTERNAL "unexpected queue status" rcvQueues :: (ConnId, SomeConnSub) -> [RcvQueueSub] rcvQueues (_, SomeConn _ conn) = connRcvQueues conn - connResults :: [(RcvQueueSub, Either AgentErrorType (Maybe SMP.ServiceId))] -> Map ConnId (Either AgentErrorType (Maybe SMP.ServiceId)) + connResults :: [(RcvQueueSub, Either AgentErrorType (Maybe SMP.ServiceId))] -> Map ConnId (Either AgentErrorType ()) connResults = M.map snd . foldl' addResult M.empty where -- collects results by connection ID - addResult :: Map ConnId QSubResult -> (RcvQueueSub, Either AgentErrorType (Maybe SMP.ServiceId)) -> Map ConnId QSubResult - addResult rs (RcvQueueSub {connId, status}, r) = M.alter (combineRes (status, r)) connId rs + addResult :: Map ConnId QCmdResult -> (RcvQueueSub, Either AgentErrorType (Maybe SMP.ServiceId)) -> Map ConnId QCmdResult + addResult rs (RcvQueueSub {connId, status}, r) = M.alter (combineRes (status, () <$ r)) connId rs -- combines two results for one connection, by using only Active queues (if there is at least one Active queue) - combineRes :: QSubResult -> Maybe QSubResult -> Maybe QSubResult + combineRes :: QCmdResult -> Maybe QCmdResult -> Maybe QCmdResult combineRes r' (Just r) = Just $ if order r <= order r' then r else r' combineRes r' _ = Just r' - order :: QSubResult -> Int + order :: QCmdResult -> Int order (Active, Right _) = 1 order (Active, _) = 2 order (_, Right _) = 3 order _ = 4 - -- TODO [certs rcv] store associations of queues with client service ID - storeClientServiceAssocs :: Map ConnId (Either AgentErrorType (Maybe SMP.ServiceId)) -> AM (Map ConnId (Either AgentErrorType (Maybe ClientServiceId))) - storeClientServiceAssocs = pure . M.map (Nothing <$) - sendNtfCreate :: NtfSupervisor -> Map ConnId (Either AgentErrorType (Maybe ClientServiceId)) -> [(ConnId, SomeConnSub)] -> AM' () + sendNtfCreate :: NtfSupervisor -> Map ConnId (Either AgentErrorType ()) -> [(ConnId, SomeConnSub)] -> AM' () sendNtfCreate ns rcvRs cs = do let oks = M.keysSet $ M.filter (either temporaryAgentError $ const True) rcvRs (csCreate, csDelete) = foldr (groupConnIds oks) ([], []) cs @@ -1545,7 +1578,7 @@ subscribeConnections_ c conns = do DuplexConnection _ _ sqs -> L.toList sqs SndConnection _ sq -> [sq] _ -> [] - notifyResultError :: Map ConnId (Either AgentErrorType (Maybe ClientServiceId)) -> AM () + notifyResultError :: Map ConnId (Either AgentErrorType ()) -> AM () notifyResultError rs = do let actual = M.size rs expected = length conns @@ -1561,7 +1594,15 @@ subscribeAllConnections' c onlyNeeded activeUserId_ = handleErr $ do let userSrvs' = case activeUserId_ of Just activeUserId -> sortOn (\(uId, _) -> if uId == activeUserId then 0 else 1 :: Int) userSrvs Nothing -> userSrvs - rs <- lift $ mapConcurrently (subscribeUserServer maxPending currPending) userSrvs' + useServices <- readTVarIO $ useClientServices c + -- Service will be loaded for all user/server combinations: + -- a) service is enabled for user ID and service record exists: subscription will be attempted, + -- b) service is disabled and record exists: service record and all associations will be removed, + -- c) service is disabled or no record: no subscription attempt. + -- On successful service subscription, only unassociated queues will be subscribed. + userSrvs2 <- withStore' c $ \db -> mapM (getService db useServices) userSrvs' + userSrvs3 <- lift $ mapConcurrently subscribeService userSrvs2 + rs <- lift $ mapConcurrently (subscribeUserServer maxPending currPending) userSrvs3 let (errs, oks) = partitionEithers rs logInfo $ "subscribed " <> tshow (sum oks) <> " queues" forM_ (L.nonEmpty errs) $ notifySub c . ERRS . L.map ("",) @@ -1570,12 +1611,40 @@ subscribeAllConnections' c onlyNeeded activeUserId_ = handleErr $ do resumeAllCommands c where handleErr = (`catchAllErrors` \e -> notifySub' c "" (ERR e) >> throwE e) - subscribeUserServer :: Int -> TVar Int -> (UserId, SMPServer) -> AM' (Either AgentErrorType Int) - subscribeUserServer maxPending currPending (userId, srv) = do + getService :: DB.Connection -> Map UserId Bool -> (UserId, SMPServer) -> IO ((UserId, SMPServer), Maybe ServiceSub) + getService db useServices us@(userId, srv) = + fmap (us,) $ + getSubscriptionService db userId srv >>= \case + Just serviceSub -> case M.lookup userId useServices of + Just True -> pure $ Just serviceSub + _ -> Nothing <$ unassocUserServerRcvQueueSubs' db userId srv + _ -> pure Nothing + subscribeService :: ((UserId, SMPServer), Maybe ServiceSub) -> AM' ((UserId, SMPServer), ServiceAssoc) + subscribeService (us@(userId, srv), serviceSub_) = fmap ((us,) . fromRight False) $ + tryAllErrors' $ + case serviceSub_ of + Just serviceSub -> + tryAllErrors (subscribeClientService c True userId srv serviceSub) >>= \case + Right (ServiceSubResult e _) -> case e of + Just SSErrorServiceId {} -> unassocQueues + -- Possibly, we should always resubscribe all when expected is greater than subscribed + Just SSErrorQueueCount {expectedQueueCount = n, subscribedQueueCount = n'} | n > 0 && n' == 0 -> unassocQueues + _ -> pure True + Left e -> do + atomically $ writeTBQueue (subQ c) ("", "", AEvt SAEConn $ ERR e) + if clientServiceError e + then unassocQueues + else pure True + where + unassocQueues :: AM Bool + unassocQueues = False <$ withStore' c (\db -> unassocUserServerRcvQueueSubs' db userId srv) + _ -> pure False + subscribeUserServer :: Int -> TVar Int -> ((UserId, SMPServer), ServiceAssoc) -> AM' (Either AgentErrorType Int) + subscribeUserServer maxPending currPending ((userId, srv), hasService) = do atomically $ whenM ((maxPending <=) <$> readTVar currPending) retry tryAllErrors' $ do qs <- withStore' c $ \db -> do - qs <- getUserServerRcvQueueSubs db userId srv onlyNeeded + qs <- getUserServerRcvQueueSubs db userId srv onlyNeeded hasService unless (null qs) $ atomically $ modifyTVar' currPending (+ length qs) -- update before leaving transaction pure qs let n = length qs @@ -1584,7 +1653,6 @@ subscribeAllConnections' c onlyNeeded activeUserId_ = handleErr $ do where subscribe qs = do rs <- subscribeUserServerQueues c userId srv qs - -- TODO [certs rcv] storeClientServiceAssocs store associations of queues with client service ID ns <- asks ntfSupervisor whenM (liftIO $ hasInstantNotifications ns) $ sendNtfCreate ns rs sendNtfCreate :: NtfSupervisor -> [(RcvQueueSub, Either AgentErrorType (Maybe SMP.ServiceId))] -> AM' () @@ -1605,15 +1673,15 @@ subscribeAllConnections' c onlyNeeded activeUserId_ = handleErr $ do sqs <- withStore' c getAllSndQueuesForDelivery lift $ mapM_ (resumeMsgDelivery c) sqs -resubscribeConnection' :: AgentClient -> ConnId -> AM (Maybe ClientServiceId) +resubscribeConnection' :: AgentClient -> ConnId -> AM () resubscribeConnection' c connId = toConnResult connId =<< resubscribeConnections' c [connId] {-# INLINE resubscribeConnection' #-} -resubscribeConnections' :: AgentClient -> [ConnId] -> AM (Map ConnId (Either AgentErrorType (Maybe ClientServiceId))) +resubscribeConnections' :: AgentClient -> [ConnId] -> AM (Map ConnId (Either AgentErrorType ())) resubscribeConnections' _ [] = pure M.empty resubscribeConnections' c connIds = do conns <- zip connIds <$> withStore' c (`getConnSubs` connIds) - let r = M.fromList $ map (,Right Nothing) connIds -- TODO [certs rcv] + let r = M.fromList $ map (,Right ()) connIds conns' <- filterM (fmap not . isActiveConn . snd) conns -- union is left-biased, so results returned by subscribeConnections' take precedence (`M.union` r) <$> subscribeConnections_ c conns' @@ -1624,9 +1692,14 @@ resubscribeConnections' c connIds = do [] -> pure True rqs' -> anyM $ map (atomically . hasActiveSubscription c) rqs' --- TODO [certs rcv] -subscribeClientService' :: AgentClient -> ClientServiceId -> AM Int -subscribeClientService' = undefined +subscribeClientServices' :: AgentClient -> UserId -> AM (Map SMPServer (Either AgentErrorType ServiceSubResult)) +subscribeClientServices' c userId = + ifM useService subscribe $ throwError $ CMD PROHIBITED "no user service allowed" + where + useService = liftIO $ (Just True ==) <$> TM.lookupIO userId (useClientServices c) + subscribe = do + srvs <- withStore' c (`getClientServiceServers` userId) + lift $ M.fromList <$> mapConcurrently (\(srv, serviceSub) -> fmap (srv,) $ tryAllErrors' $ subscribeClientService c False userId srv serviceSub) srvs -- requesting messages sequentially, to reduce memory usage getConnectionMessages' :: AgentClient -> NonEmpty ConnMsgReq -> AM' (NonEmpty (Either AgentErrorType (Maybe SMPMsgMeta))) @@ -1788,8 +1861,8 @@ runCommandProcessing c@AgentClient {subQ} connId server_ Worker {doWork} = do NEW enableNtfs (ACM cMode) pqEnc subMode -> noServer $ do triedHosts <- newTVarIO S.empty tryCommand . withNextSrv c userId storageSrvs triedHosts [] $ \srv -> do - (CCLink cReq _, service) <- newRcvConnSrv c NRMBackground userId connId enableNtfs cMode Nothing Nothing pqEnc subMode srv - notify $ INV (ACR cMode cReq) service + CCLink cReq _ <- newRcvConnSrv c NRMBackground userId connId enableNtfs cMode Nothing Nothing pqEnc subMode srv + notify $ INV (ACR cMode cReq) LSET userLinkData clientData -> withServer' . tryCommand $ do link <- setConnShortLink' c NRMBackground connId SCMContact userLinkData clientData @@ -1801,15 +1874,15 @@ runCommandProcessing c@AgentClient {subQ} connId server_ Worker {doWork} = do JOIN enableNtfs (ACR _ cReq@(CRInvitationUri ConnReqUriData {crSmpQueues = q :| _} _)) pqEnc subMode connInfo -> noServer $ do triedHosts <- newTVarIO S.empty tryCommand . withNextSrv c userId storageSrvs triedHosts [qServer q] $ \srv -> do - (sqSecured, service) <- joinConnSrvAsync c userId connId enableNtfs cReq connInfo pqEnc subMode srv - notify $ JOINED sqSecured service + sqSecured <- joinConnSrvAsync c userId connId enableNtfs cReq connInfo pqEnc subMode srv + notify $ JOINED sqSecured -- TODO TBC using joinConnSrvAsync for contact URIs, with receive queue created asynchronously. -- Currently joinConnSrv is used because even joinConnSrvAsync for invitation URIs creates receive queue synchronously. JOIN enableNtfs (ACR _ cReq@(CRContactUri ConnReqUriData {crSmpQueues = q :| _})) pqEnc subMode connInfo -> noServer $ do triedHosts <- newTVarIO S.empty tryCommand . withNextSrv c userId storageSrvs triedHosts [qServer q] $ \srv -> do - (sqSecured, service) <- joinConnSrv c NRMBackground userId connId enableNtfs cReq connInfo pqEnc subMode srv - notify $ JOINED sqSecured service + sqSecured <- joinConnSrv c NRMBackground userId connId enableNtfs cReq connInfo pqEnc subMode srv + notify $ JOINED sqSecured LET confId ownCInfo -> withServer' . tryCommand $ allowConnection' c connId confId ownCInfo >> notify OK ACK msgId rcptInfo_ -> withServer' . tryCommand $ ackMessage' c connId msgId rcptInfo_ >> notify OK SWCH -> @@ -2335,10 +2408,10 @@ switchDuplexConnection c nm (DuplexConnection cData@ConnData {connId, userId} rq srv' <- if srv == server then getNextSMPServer c userId [server] else pure srvAuth -- TODO [notications] possible improvement would be to create ntf credentials here, to avoid creating them after rotation completes. -- The problem is that currently subscription already exists, and we do not support queues with credentials but without subscriptions. - (q, qUri, tSess, sessId) <- newRcvQueue c nm userId connId srv' clientVRange SCMInvitation False SMSubscribe + (q, qUri, tSess, sessId, serviceId_) <- newRcvQueue c nm userId connId srv' clientVRange SCMInvitation False SMSubscribe let rq' = (q :: NewRcvQueue) {primary = True, dbReplaceQueueId = Just dbQueueId} rq'' <- withStore c $ \db -> addConnRcvQueue db connId rq' SMSubscribe - lift $ addNewQueueSubscription c rq'' tSess sessId + lift $ addNewQueueSubscription c rq'' tSess sessId serviceId_ void . enqueueMessages c cData sqs SMP.noMsgFlags $ QADD [(qUri, Just (server, sndId))] rq1 <- withStore' c $ \db -> setRcvSwitchStatus db rq $ Just RSSendingQADD let rqs' = updatedQs rq1 rqs <> [rq''] @@ -2518,13 +2591,13 @@ deleteConnQueues c nm waitDelivery ntf rqs = do connResults = M.map snd . foldl' addResult M.empty where -- collects results by connection ID - addResult :: Map ConnId QDelResult -> (RcvQueue, Either AgentErrorType ()) -> Map ConnId QDelResult + addResult :: Map ConnId QCmdResult -> (RcvQueue, Either AgentErrorType ()) -> Map ConnId QCmdResult addResult rs (RcvQueue {connId, status}, r) = M.alter (combineRes (status, r)) connId rs -- combines two results for one connection, by prioritizing errors in Active queues - combineRes :: QDelResult -> Maybe QDelResult -> Maybe QDelResult + combineRes :: QCmdResult -> Maybe QCmdResult -> Maybe QCmdResult combineRes r' (Just r) = Just $ if order r <= order r' then r else r' combineRes r' _ = Just r' - order :: QDelResult -> Int + order :: QCmdResult -> Int order (Active, Left _) = 1 order (_, Left _) = 2 order _ = 3 @@ -2977,23 +3050,29 @@ data ACKd = ACKd | ACKPending -- It cannot be finally, as sometimes it needs to be ACK+DEL, -- and sometimes ACK has to be sent from the consumer. processSMPTransmissions :: AgentClient -> ServerTransmissionBatch SMPVersion ErrorType BrokerMsg -> AM' () -processSMPTransmissions c@AgentClient {subQ} (tSess@(userId, srv, _), _v, sessId, ts) = do +processSMPTransmissions c@AgentClient {subQ} (tSess@(userId, srv, _), THandleParams {thAuth, sessionId = sessId}, ts) = do upConnIds <- newTVarIO [] + serviceRQs <- newTVarIO ([] :: [RcvQueue]) forM_ ts $ \(entId, t) -> case t of - STEvent msgOrErr -> - withRcvConn entId $ \rq@RcvQueue {connId} conn -> case msgOrErr of - Right msg -> runProcessSMP rq conn (toConnData conn) msg - Left e -> lift $ do - processClientNotice rq e - notifyErr connId e + STEvent msgOrErr + | entId == SMP.NoEntity -> case msgOrErr of + Right msg -> case msg of + SMP.ALLS -> notifySub c $ SERVICE_ALL srv + SMP.ERR e -> notifyErr "" $ PCEProtocolError e + _ -> logError $ "unexpected event: " <> tshow msg + Left e -> notifyErr "" e + | otherwise -> withRcvConn entId $ \rq@RcvQueue {connId} conn -> case msgOrErr of + Right msg -> runProcessSMP rq conn (toConnData conn) msg + Left e -> lift $ do + processClientNotice rq e + notifyErr connId e STResponse (Cmd SRecipient cmd) respOrErr -> withRcvConn entId $ \rq conn -> case cmd of SMP.SUB -> case respOrErr of - Right SMP.OK -> liftIO $ processSubOk rq upConnIds - -- TODO [certs rcv] associate queue with the service - Right (SMP.SOK serviceId_) -> liftIO $ processSubOk rq upConnIds + Right SMP.OK -> liftIO $ processSubOk rq upConnIds serviceRQs Nothing + Right (SMP.SOK serviceId_) -> liftIO $ processSubOk rq upConnIds serviceRQs serviceId_ Right msg@SMP.MSG {} -> do - liftIO $ processSubOk rq upConnIds -- the connection is UP even when processing this particular message fails + liftIO $ processSubOk rq upConnIds serviceRQs Nothing -- the connection is UP even when processing this particular message fails runProcessSMP rq conn (toConnData conn) msg Right r -> lift $ processSubErr rq $ unexpectedResponse r Left e -> lift $ unless (temporaryClientError e) $ processSubErr rq e -- timeout/network was already reported @@ -3009,6 +3088,7 @@ processSMPTransmissions c@AgentClient {subQ} (tSess@(userId, srv, _), _v, sessId unless (null connIds) $ do notify' "" $ UP srv connIds atomically $ incSMPServerStat' c userId srv connSubscribed $ length connIds + readTVarIO serviceRQs >>= processRcvServiceAssocs c where withRcvConn :: SMP.RecipientId -> (forall c. RcvQueue -> Connection c -> AM ()) -> AM' () withRcvConn rId a = do @@ -3018,11 +3098,13 @@ processSMPTransmissions c@AgentClient {subQ} (tSess@(userId, srv, _), _v, sessId tryAllErrors' (a rq conn) >>= \case Left e -> notify' connId (ERR e) Right () -> pure () - processSubOk :: RcvQueue -> TVar [ConnId] -> IO () - processSubOk rq@RcvQueue {connId} upConnIds = + processSubOk :: RcvQueue -> TVar [ConnId] -> TVar [RcvQueue] -> Maybe SMP.ServiceId -> IO () + processSubOk rq@RcvQueue {connId} upConnIds serviceRQs serviceId_ = atomically . whenM (isPendingSub rq) $ do - SS.addActiveSub tSess sessId (rcvQueueSub rq) $ currentSubs c + SS.addActiveSub tSess sessId serviceId_ rq $ currentSubs c modifyTVar' upConnIds (connId :) + when (isJust serviceId_ && serviceId_ == clientServiceId_) $ modifyTVar' serviceRQs (rq :) + clientServiceId_ = (\THClientService {serviceId} -> serviceId) <$> (clientService =<< thAuth) processSubErr :: RcvQueue -> SMPClientError -> AM' () processSubErr rq@RcvQueue {connId} e = do atomically . whenM (isPendingSub rq) $ @@ -3227,14 +3309,26 @@ processSMPTransmissions c@AgentClient {subQ} (tSess@(userId, srv, _), _v, sessId notifyEnd removed | removed = notify END >> logServer "<--" c srv rId "END" | otherwise = logServer "<--" c srv rId "END from disconnected client - ignored" - -- Possibly, we need to add some flag to connection that it was deleted + SMP.ENDS n idsHash -> + atomically (ifM (activeClientSession c tSess sessId) (SS.deleteServiceSub tSess (currentSubs c) $> True) (pure False)) + >>= notifyEnd + where + notifyEnd removed + | removed = do + forM_ clientServiceId_ $ \serviceId -> + notify_ B.empty $ SERVICE_END srv $ ServiceSub serviceId n idsHash + logServer "<--" c srv rId "ENDS" + | otherwise = logServer "<--" c srv rId "ENDS from disconnected client - ignored" + -- TODO [certs rcv] Possibly, we need to add some flag to connection that it was deleted SMP.DELD -> atomically (removeSubscription c tSess connId rq) >> notify DELD SMP.ERR e -> notify $ ERR $ SMP (B.unpack $ strEncode srv) e r -> unexpected r where notify :: forall e m. (AEntityI e, MonadIO m) => AEvent e -> m () - notify msg = - let t = ("", connId, AEvt (sAEntity @e) msg) + notify = notify_ connId + notify_ :: forall e m. (AEntityI e, MonadIO m) => ConnId -> AEvent e -> m () + notify_ connId' msg = + let t = ("", connId', AEvt (sAEntity @e) msg) in atomically $ ifM (isFullTBQueue subQ) (modifyTVar' pendingMsgs (t :)) (writeTBQueue subQ t) prohibited :: Text -> AM () @@ -3614,22 +3708,22 @@ connectReplyQueues c cData@ConnData {userId, connId} ownConnInfo sq_ (qInfo :| _ (sq, _) <- lift $ newSndQueue userId connId qInfo' Nothing withStore c $ \db -> upgradeRcvConnToDuplex db connId sq -secureConfirmQueueAsync :: AgentClient -> ConnData -> Maybe RcvQueue -> SndQueue -> SMPServerWithAuth -> ConnInfo -> Maybe (CR.SndE2ERatchetParams 'C.X448) -> SubscriptionMode -> AM (SndQueueSecured, Maybe ClientServiceId) +secureConfirmQueueAsync :: AgentClient -> ConnData -> Maybe RcvQueue -> SndQueue -> SMPServerWithAuth -> ConnInfo -> Maybe (CR.SndE2ERatchetParams 'C.X448) -> SubscriptionMode -> AM SndQueueSecured secureConfirmQueueAsync c cData rq_ sq srv connInfo e2eEncryption_ subMode = do sqSecured <- agentSecureSndQueue c NRMBackground cData sq - (qInfo, service) <- mkAgentConfirmation c NRMBackground cData rq_ sq srv connInfo subMode + qInfo <- mkAgentConfirmation c NRMBackground cData rq_ sq srv connInfo subMode storeConfirmation c cData sq e2eEncryption_ qInfo lift $ submitPendingMsg c sq - pure (sqSecured, service) + pure sqSecured -secureConfirmQueue :: AgentClient -> NetworkRequestMode -> ConnData -> Maybe RcvQueue -> SndQueue -> SMPServerWithAuth -> ConnInfo -> Maybe (CR.SndE2ERatchetParams 'C.X448) -> SubscriptionMode -> AM (SndQueueSecured, Maybe ClientServiceId) +secureConfirmQueue :: AgentClient -> NetworkRequestMode -> ConnData -> Maybe RcvQueue -> SndQueue -> SMPServerWithAuth -> ConnInfo -> Maybe (CR.SndE2ERatchetParams 'C.X448) -> SubscriptionMode -> AM SndQueueSecured secureConfirmQueue c nm cData@ConnData {connId, connAgentVersion, pqSupport} rq_ sq srv connInfo e2eEncryption_ subMode = do sqSecured <- agentSecureSndQueue c nm cData sq - (qInfo, service) <- mkAgentConfirmation c nm cData rq_ sq srv connInfo subMode + qInfo <- mkAgentConfirmation c nm cData rq_ sq srv connInfo subMode msg <- mkConfirmation qInfo void $ sendConfirmation c nm sq msg withStore' c $ \db -> setSndQueueStatus db sq Confirmed - pure (sqSecured, service) + pure sqSecured where mkConfirmation :: AgentMessage -> AM MsgBody mkConfirmation aMessage = do @@ -3655,12 +3749,12 @@ agentSecureSndQueue c nm ConnData {connAgentVersion} sq@SndQueue {queueMode, sta sndSecure = senderCanSecure queueMode initiatorRatchetOnConf = connAgentVersion >= ratchetOnConfSMPAgentVersion -mkAgentConfirmation :: AgentClient -> NetworkRequestMode -> ConnData -> Maybe RcvQueue -> SndQueue -> SMPServerWithAuth -> ConnInfo -> SubscriptionMode -> AM (AgentMessage, Maybe ClientServiceId) +mkAgentConfirmation :: AgentClient -> NetworkRequestMode -> ConnData -> Maybe RcvQueue -> SndQueue -> SMPServerWithAuth -> ConnInfo -> SubscriptionMode -> AM AgentMessage mkAgentConfirmation c nm cData rq_ sq srv connInfo subMode = do - (qInfo, service) <- case rq_ of + qInfo <- case rq_ of Nothing -> createReplyQueue c nm cData sq subMode srv - Just rq@RcvQueue {smpClientVersion = v, clientService} -> pure (SMPQueueInfo v $ rcvSMPQueueAddress rq, dbServiceId <$> clientService) - pure (AgentConnInfoReply (qInfo :| []) connInfo, service) + Just rq@RcvQueue {smpClientVersion = v} -> pure $ SMPQueueInfo v $ rcvSMPQueueAddress rq + pure $ AgentConnInfoReply (qInfo :| []) connInfo enqueueConfirmation :: AgentClient -> ConnData -> SndQueue -> ConnInfo -> Maybe (CR.SndE2ERatchetParams 'C.X448) -> AM () enqueueConfirmation c cData sq connInfo e2eEncryption_ = do diff --git a/src/Simplex/Messaging/Agent/Client.hs b/src/Simplex/Messaging/Agent/Client.hs index 00a228330..46a441aaf 100644 --- a/src/Simplex/Messaging/Agent/Client.hs +++ b/src/Simplex/Messaging/Agent/Client.hs @@ -49,6 +49,8 @@ module Simplex.Messaging.Agent.Client newRcvQueue_, subscribeQueues, subscribeUserServerQueues, + subscribeClientService, + processRcvServiceAssocs, processClientNotices, getQueueMessage, decryptSMPMessage, @@ -118,6 +120,7 @@ module Simplex.Messaging.Agent.Client getAgentSubscriptions, slowNetworkConfig, protocolClientError, + clientServiceError, Worker (..), SessionVar (..), SubscriptionsInfo (..), @@ -223,6 +226,7 @@ import Data.Text.Encoding import Data.Time (UTCTime, addUTCTime, defaultTimeLocale, formatTime, getCurrentTime) import Data.Time.Clock.System (getSystemTime) import Data.Word (Word16) +import qualified Data.X509.Validation as XV import Network.Socket (HostName) import Simplex.FileTransfer.Client (XFTPChunkSpec (..), XFTPClient, XFTPClientConfig (..), XFTPClientError) import qualified Simplex.FileTransfer.Client as X @@ -238,8 +242,8 @@ import Simplex.Messaging.Agent.Protocol import Simplex.Messaging.Agent.RetryInterval import Simplex.Messaging.Agent.Stats import Simplex.Messaging.Agent.Store -import Simplex.Messaging.Agent.Store.AgentStore (getClientNotices, updateClientNotices) -import Simplex.Messaging.Agent.Store.Common (DBStore, withTransaction) +import Simplex.Messaging.Agent.Store.AgentStore +import Simplex.Messaging.Agent.Store.Common (DBStore) import Simplex.Messaging.Agent.Store.DB (SQLError) import qualified Simplex.Messaging.Agent.Store.DB as DB import Simplex.Messaging.Agent.Store.Entity @@ -277,6 +281,9 @@ import Simplex.Messaging.Protocol RcvNtfPublicDhKey, SMPMsgMeta (..), SProtocolType (..), + ServiceSub (..), + ServiceSubResult (..), + ServiceSubError (..), SndPublicAuthKey, SubscriptionMode (..), NewNtfCreds (..), @@ -289,6 +296,7 @@ import Simplex.Messaging.Protocol XFTPServerWithAuth, pattern NoEntity, senderCanSecure, + serviceSubResult, ) import qualified Simplex.Messaging.Protocol as SMP import Simplex.Messaging.Protocol.Types @@ -297,8 +305,9 @@ import Simplex.Messaging.Session import Simplex.Messaging.SystemTime import Simplex.Messaging.TMap (TMap) import qualified Simplex.Messaging.TMap as TM -import Simplex.Messaging.Transport (SMPVersion, SessionId, THandleParams (sessionId, thVersion), TransportError (..), TransportPeer (..), sndAuthKeySMPVersion, shortLinksSMPVersion, newNtfCredsSMPVersion) +import Simplex.Messaging.Transport (HandshakeError (..), SMPServiceRole (..), SMPVersion, ServiceCredentials (..), SessionId, THClientService' (..), THandleAuth (..), THandleParams (sessionId, thAuth, thVersion), TransportError (..), TransportPeer (..), sndAuthKeySMPVersion, shortLinksSMPVersion, newNtfCredsSMPVersion) import Simplex.Messaging.Transport.Client (TransportHost (..)) +import Simplex.Messaging.Transport.Credentials import Simplex.Messaging.Util import Simplex.Messaging.Version import System.Mem.Weak (Weak, deRefWeak) @@ -332,6 +341,7 @@ data AgentClient = AgentClient msgQ :: TBQueue (ServerTransmissionBatch SMPVersion ErrorType BrokerMsg), smpServers :: TMap UserId (UserServers 'PSMP), smpClients :: TMap SMPTransportSession SMPClientVar, + useClientServices :: TMap UserId Bool, -- smpProxiedRelays: -- SMPTransportSession defines connection from proxy to relay, -- SMPServerWithAuth defines client connected to SMP proxy (with the same userId and entityId in TransportSession) @@ -496,7 +506,7 @@ data UserNetworkType = UNNone | UNCellular | UNWifi | UNEthernet | UNOther -- | Creates an SMP agent client instance that receives commands and sends responses via 'TBQueue's. newAgentClient :: Int -> InitialAgentServers -> UTCTime -> Map (Maybe SMPServer) (Maybe SystemSeconds) -> Env -> IO AgentClient -newAgentClient clientId InitialAgentServers {smp, ntf, xftp, netCfg, presetDomains, presetServers} currentTs notices agentEnv = do +newAgentClient clientId InitialAgentServers {smp, ntf, xftp, netCfg, useServices, presetDomains, presetServers} currentTs notices agentEnv = do let cfg = config agentEnv qSize = tbqSize cfg proxySessTs <- newTVarIO =<< getCurrentTime @@ -506,6 +516,7 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, netCfg, presetDomai msgQ <- newTBQueueIO qSize smpServers <- newTVarIO $ M.map mkUserServers smp smpClients <- TM.emptyIO + useClientServices <- newTVarIO useServices smpProxiedRelays <- TM.emptyIO ntfServers <- newTVarIO ntf ntfClients <- TM.emptyIO @@ -545,6 +556,7 @@ newAgentClient clientId InitialAgentServers {smp, ntf, xftp, netCfg, presetDomai msgQ, smpServers, smpClients, + useClientServices, smpProxiedRelays, ntfServers, ntfClients, @@ -599,6 +611,27 @@ agentDRG :: AgentClient -> TVar ChaChaDRG agentDRG AgentClient {agentEnv = Env {random}} = random {-# INLINE agentDRG #-} +getServiceCredentials :: AgentClient -> UserId -> SMPServer -> AM (Maybe (ServiceCredentials, Maybe ServiceId)) +getServiceCredentials c userId srv = + liftIO (TM.lookupIO userId $ useClientServices c) + $>>= \useService -> if useService then Just <$> getService else pure Nothing + where + getService :: AM (ServiceCredentials, Maybe ServiceId) + getService = do + let g = agentDRG c + ((C.KeyHash kh, serviceCreds), serviceId_) <- + withStore' c $ \db -> + getClientServiceCredentials db userId srv >>= \case + Just service -> pure service + Nothing -> do + cred <- genCredentials g Nothing (25, 24 * 999999) "simplex" + let tlsCreds = tlsCredentials [cred] + createClientService db userId srv tlsCreds + pure (tlsCreds, Nothing) + serviceSignKey <- liftEitherWith INTERNAL $ C.x509ToPrivate' $ snd serviceCreds + let creds = ServiceCredentials {serviceRole = SRMessaging, serviceCreds, serviceCertHash = XV.Fingerprint kh, serviceSignKey} + pure (creds, serviceId_) + class (Encoding err, Show err) => ProtocolServerClient v err msg | msg -> v, msg -> err where type Client msg = c | c -> msg getProtocolServerClient :: AgentClient -> NetworkRequestMode -> TransportSession msg -> AM (Client msg) @@ -702,7 +735,7 @@ getSMPProxyClient c@AgentClient {active, smpClients, smpProxiedRelays, workerSeq Nothing -> Left $ BROKER (B.unpack $ strEncode srv) TIMEOUT smpConnectClient :: AgentClient -> NetworkRequestMode -> SMPTransportSession -> TMap SMPServer ProxiedRelayVar -> SMPClientVar -> AM SMPConnectedClient -smpConnectClient c@AgentClient {smpClients, msgQ, proxySessTs, presetDomains} nm tSess@(_, srv, _) prs v = +smpConnectClient c@AgentClient {smpClients, msgQ, proxySessTs, presetDomains} nm tSess@(userId, srv, _) prs v = newProtocolClient c tSess smpClients connectClient v `catchAllErrors` \e -> lift (resubscribeSMPSession c tSess) >> throwE e where @@ -710,12 +743,22 @@ smpConnectClient c@AgentClient {smpClients, msgQ, proxySessTs, presetDomains} nm connectClient v' = do cfg <- lift $ getClientConfig c smpCfg g <- asks random + service <- getServiceCredentials c userId srv + let cfg' = cfg {serviceCredentials = fst <$> service} env <- ask - liftError (protocolClientError SMP $ B.unpack $ strEncode srv) $ do + smp <- liftError (protocolClientError SMP $ B.unpack $ strEncode srv) $ do ts <- readTVarIO proxySessTs - smp <- ExceptT $ getProtocolClient g nm tSess cfg presetDomains (Just msgQ) ts $ smpClientDisconnected c tSess env v' prs - atomically $ SS.setSessionId tSess (sessionId $ thParams smp) $ currentSubs c - pure SMPConnectedClient {connectedClient = smp, proxiedRelays = prs} + ExceptT $ getProtocolClient g nm tSess cfg' presetDomains (Just msgQ) ts $ smpClientDisconnected c tSess env v' prs + atomically $ SS.setSessionId tSess (sessionId $ thParams smp) $ currentSubs c + updateClientService service smp + pure SMPConnectedClient {connectedClient = smp, proxiedRelays = prs} + updateClientService service smp = case (service, smpClientServiceId smp) of + (Just (_, serviceId_), Just serviceId) -> withStore' c $ \db -> do + setClientServiceId db userId srv serviceId + forM_ serviceId_ $ \sId -> when (sId /= serviceId) $ removeRcvServiceAssocs db userId srv + (Just _, Nothing) -> withStore' c $ \db -> deleteClientService db userId srv -- e.g., server version downgrade + (Nothing, Just _) -> logError "server returned serviceId without service credentials in request" + (Nothing, Nothing) -> pure () smpClientDisconnected :: AgentClient -> SMPTransportSession -> Env -> SMPClientVar -> TMap SMPServer ProxiedRelayVar -> SMPClient -> IO () smpClientDisconnected c@AgentClient {active, smpClients, smpProxiedRelays} tSess@(userId, srv, cId) env v prs client = do @@ -725,32 +768,35 @@ smpClientDisconnected c@AgentClient {active, smpClients, smpProxiedRelays} tSess -- we make active subscriptions pending only if the client for tSess was current (in the map) and active, -- because we can have a race condition when a new current client could have already -- made subscriptions active, and the old client would be processing diconnection later. - removeClientAndSubs :: IO ([RcvQueueSub], [ConnId]) + removeClientAndSubs :: IO ([RcvQueueSub], [ConnId], Maybe ServiceSub) removeClientAndSubs = atomically $ do removeSessVar v tSess smpClients - ifM (readTVar active) removeSubs (pure ([], [])) + ifM (readTVar active) removeSubs (pure ([], [], Nothing)) where sessId = sessionId $ thParams client removeSubs = do mode <- getSessionMode c - subs <- SS.setSubsPending mode tSess sessId $ currentSubs c + (subs, serviceSub_) <- SS.setSubsPending mode tSess sessId $ currentSubs c let qs = M.elems subs cs = nubOrd $ map qConnId qs -- this removes proxied relays that this client created sessions to destSrvs <- M.keys <$> readTVar prs forM_ destSrvs $ \destSrv -> TM.delete (userId, destSrv, cId) smpProxiedRelays - pure (qs, cs) + pure (qs, cs, serviceSub_) - serverDown :: ([RcvQueueSub], [ConnId]) -> IO () - serverDown (qs, conns) = whenM (readTVarIO active) $ do + serverDown :: ([RcvQueueSub], [ConnId], Maybe ServiceSub) -> IO () + serverDown (qs, conns, serviceSub_) = whenM (readTVarIO active) $ do notifySub c $ hostEvent' DISCONNECT client unless (null conns) $ notifySub c $ DOWN srv conns - unless (null qs) $ do + mapM_ (notifySub c . SERVICE_DOWN srv) serviceSub_ + unless (null qs && isNothing serviceSub_) $ do releaseGetLocksIO c qs mode <- getSessionModeIO c let resubscribe | (mode == TSMEntity) == isJust cId = resubscribeSMPSession c tSess - | otherwise = void $ subscribeQueues c True qs + | otherwise = do + mapM_ (runExceptT . resubscribeClientService c tSess) serviceSub_ + unless (null qs) $ void $ subscribeQueues c True qs runReaderT resubscribe env resubscribeSMPSession :: AgentClient -> SMPTransportSession -> AM' () @@ -769,11 +815,12 @@ resubscribeSMPSession c@AgentClient {smpSubWorkers, workerSeq} tSess = do runSubWorker = do ri <- asks $ reconnectInterval . config withRetryForeground ri isForeground (isNetworkOnline c) $ \_ loop -> do - pending <- atomically $ SS.getPendingSubs tSess $ currentSubs c - unless (M.null pending) $ do + (pendingSubs, pendingSS) <- atomically $ SS.getPendingSubs tSess $ currentSubs c + unless (M.null pendingSubs && isNothing pendingSS) $ do liftIO $ waitUntilForeground c liftIO $ waitForUserNetwork c - handleNotify $ resubscribeSessQueues c tSess $ M.elems pending + mapM_ (handleNotify . void . runExceptT . resubscribeClientService c tSess) pendingSS + unless (M.null pendingSubs) $ handleNotify $ resubscribeSessQueues c tSess $ M.elems pendingSubs loop isForeground = (ASForeground ==) <$> readTVar (agentState c) cleanup :: SessionVar (Async ()) -> STM () @@ -863,7 +910,6 @@ waitForProtocolClient c nm tSess@(_, srv, _) clients v = do (throwE e) Nothing -> throwE $ BROKER (B.unpack $ strEncode srv) TIMEOUT --- clientConnected arg is only passed for SMP server newProtocolClient :: forall v err msg. (ProtocolTypeI (ProtoType msg), ProtocolServerClient v err msg) => @@ -1210,7 +1256,15 @@ protocolClientError protocolError_ host = \case PCETransportError e -> BROKER host $ TRANSPORT e e@PCECryptoError {} -> INTERNAL $ show e PCEServiceUnavailable {} -> BROKER host NO_SERVICE - PCEIOError e -> BROKER host $ NETWORK $ NEConnectError $ E.displayException e + PCEIOError e -> BROKER host $ NETWORK $ NEConnectError e + +-- it is consistent with smpClientServiceError +clientServiceError :: AgentErrorType -> Bool +clientServiceError = \case + BROKER _ NO_SERVICE -> True + SMP _ SMP.SERVICE -> True + SMP _ (SMP.PROXY (SMP.BROKER NO_SERVICE)) -> True -- for completeness, it cannot happen. + _ -> False data ProtocolTestStep = TSConnect @@ -1367,7 +1421,7 @@ getSessionMode :: AgentClient -> STM TransportSessionMode getSessionMode = fmap (sessionMode . snd) . readTVar . useNetworkConfig {-# INLINE getSessionMode #-} -newRcvQueue :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> SMPServerWithAuth -> VersionRangeSMPC -> SConnectionMode c -> Bool -> SubscriptionMode -> AM (NewRcvQueue, SMPQueueUri, SMPTransportSession, SessionId) +newRcvQueue :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> SMPServerWithAuth -> VersionRangeSMPC -> SConnectionMode c -> Bool -> SubscriptionMode -> AM (NewRcvQueue, SMPQueueUri, SMPTransportSession, SessionId, Maybe ServiceId) newRcvQueue c nm userId connId srv vRange cMode enableNtfs subMode = do let qrd = case cMode of SCMInvitation -> CQRMessaging Nothing; SCMContact -> CQRContact Nothing e2eKeys <- atomically . C.generateKeyPair =<< asks random @@ -1388,7 +1442,7 @@ queueReqData = \case CQRMessaging d -> QRMessaging $ srvReq <$> d CQRContact d -> QRContact $ srvReq <$> d -newRcvQueue_ :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> SMPServerWithAuth -> VersionRangeSMPC -> ClntQueueReqData -> Bool -> SubscriptionMode -> Maybe C.CbNonce -> C.KeyPairX25519 -> AM (NewRcvQueue, SMPQueueUri, SMPTransportSession, SessionId) +newRcvQueue_ :: AgentClient -> NetworkRequestMode -> UserId -> ConnId -> SMPServerWithAuth -> VersionRangeSMPC -> ClntQueueReqData -> Bool -> SubscriptionMode -> Maybe C.CbNonce -> C.KeyPairX25519 -> AM (NewRcvQueue, SMPQueueUri, SMPTransportSession, SessionId, Maybe ServiceId) newRcvQueue_ c nm userId connId (ProtoServerWithAuth srv auth) vRange cqrd enableNtfs subMode nonce_ (e2eDhKey, e2ePrivKey) = do C.AuthAlg a <- asks (rcvAuthAlg . config) g <- asks random @@ -1400,7 +1454,8 @@ newRcvQueue_ c nm userId connId (ProtoServerWithAuth srv auth) vRange cqrd enabl withClient c nm tSess $ \(SMPConnectedClient smp _) -> do (ntfKeys, ntfCreds) <- liftIO $ mkNtfCreds a g smp (thParams smp,ntfKeys,) <$> createSMPQueue smp nm nonce_ rKeys dhKey auth subMode (queueReqData cqrd) ntfCreds - -- TODO [certs rcv] validate that serviceId is the same as in the client session + let sessServiceId = (\THClientService {serviceId = sId} -> sId) <$> (clientService =<< thAuth thParams') + when (isJust serviceId && serviceId /= sessServiceId) $ logError "incorrect service ID in NEW response" liftIO . logServer "<--" c srv NoEntity $ B.unwords ["IDS", logSecret rcvId, logSecret sndId] shortLink <- mkShortLinkCreds thParams' qik let rq = @@ -1416,7 +1471,7 @@ newRcvQueue_ c nm userId connId (ProtoServerWithAuth srv auth) vRange cqrd enabl sndId, queueMode, shortLink, - clientService = ClientService DBNewEntity <$> serviceId, + rcvServiceAssoc = isJust serviceId && serviceId == sessServiceId, status = New, enableNtfs, clientNoticeId = Nothing, @@ -1429,7 +1484,7 @@ newRcvQueue_ c nm userId connId (ProtoServerWithAuth srv auth) vRange cqrd enabl deleteErrors = 0 } qUri = SMPQueueUri vRange $ SMPQueueAddress srv sndId e2eDhKey queueMode - pure (rq, qUri, tSess, sessionId thParams') + pure (rq, qUri, tSess, sessionId thParams', sessServiceId) where mkNtfCreds :: (C.AlgorithmI a, C.AuthAlgorithm a) => C.SAlgorithm a -> TVar ChaChaDRG -> SMPClient -> IO (Maybe (C.AAuthKeyPair, C.PrivateKeyX25519), Maybe NewNtfCreds) mkNtfCreds a g smp @@ -1470,27 +1525,27 @@ newRcvQueue_ c nm userId connId (ProtoServerWithAuth srv auth) vRange cqrd enabl newErr :: String -> AM (Maybe ShortLinkCreds) newErr = throwE . BROKER (B.unpack $ strEncode srv) . UNEXPECTED . ("Create queue: " <>) -processSubResults :: AgentClient -> SMPTransportSession -> SessionId -> NonEmpty (RcvQueueSub, Either SMPClientError (Maybe ServiceId)) -> STM [(RcvQueueSub, Maybe ClientNotice)] -processSubResults c tSess@(userId, srv, _) sessId rs = do - pendingSubs <- SS.getPendingSubs tSess $ currentSubs c - let (failed, subscribed, notices, ignored) = foldr (partitionResults pendingSubs) (M.empty, [], [], 0) rs +processSubResults :: AgentClient -> SMPTransportSession -> SessionId -> Maybe ServiceId -> NonEmpty (RcvQueueSub, Either SMPClientError (Maybe ServiceId)) -> STM ([RcvQueueSub], [(RcvQueueSub, Maybe ClientNotice)]) +processSubResults c tSess@(userId, srv, _) sessId serviceId_ rs = do + pendingSubs <- SS.getPendingQueueSubs tSess $ currentSubs c + let (failed, subscribed@(qs, sQs), notices, ignored) = foldr (partitionResults pendingSubs) (M.empty, ([], []), [], 0) rs unless (M.null failed) $ do incSMPServerStat' c userId srv connSubErrs $ M.size failed failSubscriptions c tSess failed - unless (null subscribed) $ do - incSMPServerStat' c userId srv connSubscribed $ length subscribed - SS.batchAddActiveSubs tSess sessId subscribed $ currentSubs c + unless (null qs && null sQs) $ do + incSMPServerStat' c userId srv connSubscribed $ length qs + length sQs + SS.batchAddActiveSubs tSess sessId serviceId_ subscribed $ currentSubs c unless (ignored == 0) $ incSMPServerStat' c userId srv connSubIgnored ignored - pure notices + pure (sQs, notices) where partitionResults :: Map SMP.RecipientId RcvQueueSub -> (RcvQueueSub, Either SMPClientError (Maybe ServiceId)) -> - (Map SMP.RecipientId SMPClientError, [RcvQueueSub], [(RcvQueueSub, Maybe ClientNotice)], Int) -> - (Map SMP.RecipientId SMPClientError, [RcvQueueSub], [(RcvQueueSub, Maybe ClientNotice)], Int) - partitionResults pendingSubs (rq@RcvQueueSub {rcvId, clientNoticeId}, r) acc@(failed, subscribed, notices, ignored) = case r of + (Map SMP.RecipientId SMPClientError, ([RcvQueueSub], [RcvQueueSub]), [(RcvQueueSub, Maybe ClientNotice)], Int) -> + (Map SMP.RecipientId SMPClientError, ([RcvQueueSub], [RcvQueueSub]), [(RcvQueueSub, Maybe ClientNotice)], Int) + partitionResults pendingSubs (rq@RcvQueueSub {rcvId, clientNoticeId}, r) acc@(failed, subscribed@(qs, sQs), notices, ignored) = case r of Left e -> case smpErrorClientNotice e of - Just notice_ -> (failed', subscribed, (rq, notice_) : notices, ignored) + Just notice_ -> (failed', subscribed, notices', ignored) where notices' = if isJust notice_ || isJust clientNoticeId then (rq, notice_) : notices else notices Nothing @@ -1498,8 +1553,12 @@ processSubResults c tSess@(userId, srv, _) sessId rs = do | otherwise -> (failed', subscribed, notices, ignored) where failed' = M.insert rcvId e failed - Right _serviceId -- TODO [certs rcv] store association with the service - | rcvId `M.member` pendingSubs -> (failed, rq : subscribed, notices', ignored) + Right serviceId_' + | rcvId `M.member` pendingSubs -> + let subscribed' = case (serviceId_, serviceId_') of + (Just sId, Just sId') | sId == sId' -> (qs, rq : sQs) + _ -> (rq : qs, sQs) + in (failed, subscribed', notices', ignored) | otherwise -> (failed, subscribed, notices', ignored + 1) where notices' = if isJust clientNoticeId then (rq, Nothing) : notices else notices @@ -1508,6 +1567,8 @@ temporaryAgentError :: AgentErrorType -> Bool temporaryAgentError = \case BROKER _ e -> tempBrokerError e SMP _ (SMP.PROXY (SMP.BROKER e)) -> tempBrokerError e + SMP _ (SMP.STORE _) -> True + NTF _ (SMP.STORE _) -> True XFTP _ XFTP.TIMEOUT -> True PROXY _ _ (ProxyProtocolError (SMP.PROXY (SMP.BROKER e))) -> tempBrokerError e PROXY _ _ (ProxyProtocolError (SMP.PROXY SMP.NO_SESSION)) -> True @@ -1518,6 +1579,7 @@ temporaryAgentError = \case tempBrokerError = \case NETWORK _ -> True TIMEOUT -> True + TRANSPORT (TEHandshake BAD_SERVICE) -> True -- this error is considered temporary because it is DB error _ -> False temporaryOrHostError :: AgentErrorType -> Bool @@ -1538,6 +1600,7 @@ serverHostError = \case -- | Batch by transport session and subscribe queues. The list of results can have a different order. subscribeQueues :: AgentClient -> Bool -> [RcvQueueSub] -> AM' [(RcvQueueSub, Either AgentErrorType (Maybe ServiceId))] +subscribeQueues _ _ [] = pure [] subscribeQueues c withEvents qs = do (errs, qs') <- checkQueues c qs atomically $ modifyTVar' (subscrConns c) (`S.union` S.fromList (map qConnId qs')) @@ -1594,6 +1657,7 @@ checkQueues c = fmap partitionEithers . mapM checkQueue -- This function expects that all queues belong to one transport session, -- and that they are already added to pending subscriptions. resubscribeSessQueues :: AgentClient -> SMPTransportSession -> [RcvQueueSub] -> AM' () +resubscribeSessQueues _ _ [] = pure () resubscribeSessQueues c tSess qs = do (errs, qs_) <- checkQueues c qs forM_ (L.nonEmpty qs_) $ \qs' -> void $ subscribeSessQueues_ c True (tSess, qs') @@ -1612,13 +1676,15 @@ subscribeSessQueues_ c withEvents qs = sendClientBatch_ "SUB" False subscribe_ c then Just . S.fromList . map qConnId . M.elems <$> atomically (SS.getActiveSubs tSess $ currentSubs c) else pure Nothing active <- E.uninterruptibleMask_ $ do - (active, notices) <- atomically $ do - r@(_, notices) <- ifM + (active, (serviceQs, notices)) <- atomically $ do + r@(_, (_, notices)) <- ifM (activeClientSession c tSess sessId) - ((True,) <$> processSubResults c tSess sessId rs) - ((False, []) <$ incSMPServerStat' c userId srv connSubIgnored (length rs)) + ((True,) <$> processSubResults c tSess sessId (smpClientServiceId smp) rs) + ((False, ([], [])) <$ incSMPServerStat' c userId srv connSubIgnored (length rs)) unless (null notices) $ takeTMVar $ clientNoticesLock c pure r + unless (null serviceQs) $ void $ + processRcvServiceAssocs c serviceQs `runReaderT` agentEnv c unless (null notices) $ void $ (processClientNotices c tSess notices `runReaderT` agentEnv c) `E.finally` atomically (putTMVar (clientNoticesLock c) ()) @@ -1640,6 +1706,13 @@ subscribeSessQueues_ c withEvents qs = sendClientBatch_ "SUB" False subscribe_ c tSess = transportSession' smp sessId = sessionId $ thParams smp +processRcvServiceAssocs :: SMPQueue q => AgentClient -> [q] -> AM' () +processRcvServiceAssocs _ [] = pure () +processRcvServiceAssocs c serviceQs = + withStore' c (`setRcvServiceAssocs` serviceQs) `catchAllErrors'` \e -> do + logError $ "processClientNotices error: " <> tshow e + notifySub' c "" $ ERR e + processClientNotices :: AgentClient -> SMPTransportSession -> [(RcvQueueSub, Maybe ClientNotice)] -> AM' () processClientNotices c@AgentClient {presetServers} tSess notices = do now <- liftIO getSystemSeconds @@ -1651,6 +1724,49 @@ processClientNotices c@AgentClient {presetServers} tSess notices = do logError $ "processClientNotices error: " <> tshow e notifySub' c "" $ ERR e +resubscribeClientService :: AgentClient -> SMPTransportSession -> ServiceSub -> AM ServiceSubResult +resubscribeClientService c tSess@(userId, srv, _) serviceSub = + tryAllErrors (withServiceClient c tSess $ \smp _ -> subscribeClientService_ c True tSess smp serviceSub) >>= \case + Right r@(ServiceSubResult e _) -> case e of + Just SSErrorServiceId {} -> unassocSubscribeQueues $> r + _ -> pure r + Left e -> do + when (clientServiceError e) $ unassocSubscribeQueues + atomically $ writeTBQueue (subQ c) ("", "", AEvt SAEConn $ ERR e) + throwE e + where + unassocSubscribeQueues = do + qs <- withStore' c $ \db -> unassocUserServerRcvQueueSubs db userId srv + void $ lift $ subscribeUserServerQueues c userId srv qs + +-- TODO [certs rcv] update service in the database if it has different ID and re-associate queues, and send event +subscribeClientService :: AgentClient -> Bool -> UserId -> SMPServer -> ServiceSub -> AM ServiceSubResult +subscribeClientService c withEvent userId srv (ServiceSub _ n idsHash) = + withServiceClient c tSess $ \smp smpServiceId -> do + let serviceSub = ServiceSub smpServiceId n idsHash + atomically $ SS.setPendingServiceSub tSess serviceSub $ currentSubs c + subscribeClientService_ c withEvent tSess smp serviceSub + where + tSess = (userId, srv, Nothing) + +withServiceClient :: AgentClient -> SMPTransportSession -> (SMPClient -> ServiceId -> ExceptT SMPClientError IO a) -> AM a +withServiceClient c tSess subscribe = + withLogClient c NRMBackground tSess B.empty "SUBS" $ \(SMPConnectedClient smp _) -> + case smpClientServiceId smp of + Just smpServiceId -> subscribe smp smpServiceId + Nothing -> throwE PCEServiceUnavailable + +-- TODO [certs rcv] send subscription error event? +subscribeClientService_ :: AgentClient -> Bool -> SMPTransportSession -> SMPClient -> ServiceSub -> ExceptT SMPClientError IO ServiceSubResult +subscribeClientService_ c withEvent tSess@(_, srv, _) smp expected@(ServiceSub _ n idsHash) = do + subscribed <- subscribeService smp SMP.SRecipientService n idsHash + let sessId = sessionId $ thParams smp + r = serviceSubResult expected subscribed + atomically $ whenM (activeClientSession c tSess sessId) $ + SS.setActiveServiceSub tSess sessId subscribed $ currentSubs c + when withEvent $ notifySub c $ SERVICE_UP srv r + pure r + activeClientSession :: AgentClient -> SMPTransportSession -> SessionId -> STM Bool activeClientSession c tSess sessId = sameSess <$> tryReadSessVar tSess (smpClients c) where @@ -1712,14 +1828,14 @@ getRemovedSubs AgentClient {removedSubs} k = TM.lookup k removedSubs >>= maybe n TM.insert k s removedSubs pure s -addNewQueueSubscription :: AgentClient -> RcvQueue -> SMPTransportSession -> SessionId -> AM' () -addNewQueueSubscription c rq' tSess sessId = do +addNewQueueSubscription :: AgentClient -> RcvQueue -> SMPTransportSession -> SessionId -> Maybe ServiceId -> AM' () +addNewQueueSubscription c rq' tSess sessId serviceId_ = do let rq = rcvQueueSub rq' same <- atomically $ do modifyTVar' (subscrConns c) $ S.insert $ qConnId rq active <- activeClientSession c tSess sessId if active - then SS.addActiveSub tSess sessId rq $ currentSubs c + then SS.addActiveSub tSess sessId serviceId_ rq' $ currentSubs c else SS.addPendingSub tSess rq $ currentSubs c pure active unless same $ resubscribeSMPSession c tSess @@ -1908,6 +2024,7 @@ releaseGetLock c rq = {-# INLINE releaseGetLock #-} releaseGetLocksIO :: SomeRcvQueue q => AgentClient -> [q] -> IO () +releaseGetLocksIO _ [] = pure () releaseGetLocksIO c rqs = do locks <- readTVarIO $ getMsgLocks c forM_ rqs $ \rq -> diff --git a/src/Simplex/Messaging/Agent/Env/SQLite.hs b/src/Simplex/Messaging/Agent/Env/SQLite.hs index e81240cc5..900323ede 100644 --- a/src/Simplex/Messaging/Agent/Env/SQLite.hs +++ b/src/Simplex/Messaging/Agent/Env/SQLite.hs @@ -90,6 +90,7 @@ data InitialAgentServers = InitialAgentServers ntf :: [NtfServer], xftp :: Map UserId (NonEmpty (ServerCfg 'PXFTP)), netCfg :: NetworkConfig, + useServices :: Map UserId Bool, presetDomains :: [HostName], presetServers :: [SMPServer] } diff --git a/src/Simplex/Messaging/Agent/NtfSubSupervisor.hs b/src/Simplex/Messaging/Agent/NtfSubSupervisor.hs index c89ad3d90..06c9b4ca4 100644 --- a/src/Simplex/Messaging/Agent/NtfSubSupervisor.hs +++ b/src/Simplex/Messaging/Agent/NtfSubSupervisor.hs @@ -311,7 +311,7 @@ runNtfWorker c srv Worker {doWork} = _ -> ((ntfSubConnId sub, INTERNAL "NSACheck - no subscription ID") : errs, subs, subIds) updateSub :: DB.Connection -> NtfServer -> UTCTime -> UTCTime -> (NtfSubscription, NtfSubStatus) -> IO (Maybe SMPServer) updateSub db ntfServer ts nextCheckTs (sub, status) - | ntfShouldSubscribe status = + | status `elem` subscribeNtfStatuses = let sub' = sub {ntfSubStatus = NASCreated status} in Nothing <$ updateNtfSubscription db sub' (NSANtf NSACheck) nextCheckTs -- ntf server stopped subscribing to this queue diff --git a/src/Simplex/Messaging/Agent/Protocol.hs b/src/Simplex/Messaging/Agent/Protocol.hs index a848d9aff..557a92a73 100644 --- a/src/Simplex/Messaging/Agent/Protocol.hs +++ b/src/Simplex/Messaging/Agent/Protocol.hs @@ -130,9 +130,6 @@ module Simplex.Messaging.Agent.Protocol ShortLinkScheme (..), LinkKey (..), PreparedLinkParams (..), - StoredClientService (..), - ClientService, - ClientServiceId, validateOwners, validateLinkOwners, sameConnReqContact, @@ -218,7 +215,6 @@ import Simplex.FileTransfer.Transport (XFTPErrorType) import Simplex.FileTransfer.Types (FileErrorType) import Simplex.Messaging.Agent.QueryString import Simplex.Messaging.Agent.Store.DB (Binary (..), FromField (..), ToField (..), blobFieldDecoder, fromTextField_) -import Simplex.Messaging.Agent.Store.Entity import Simplex.Messaging.Client (ProxyClientError) import qualified Simplex.Messaging.Crypto as C import Simplex.Messaging.Crypto.Ratchet @@ -247,6 +243,8 @@ import Simplex.Messaging.Protocol SMPClientVersion, SMPServer, SMPServerWithAuth, + ServiceSub, + ServiceSubResult, SndPublicAuthKey, SubscriptionMode, VersionRangeSMPC, @@ -387,7 +385,7 @@ type SndQueueSecured = Bool -- | Parameterized type for SMP agent events data AEvent (e :: AEntity) where - INV :: AConnectionRequestUri -> Maybe ClientServiceId -> AEvent AEConn + INV :: AConnectionRequestUri -> AEvent AEConn LINK :: ConnShortLink 'CMContact -> UserConnLinkData 'CMContact -> AEvent AEConn LDATA :: FixedLinkData 'CMContact -> ConnLinkData 'CMContact -> AEvent AEConn CONF :: ConfirmationId -> PQSupport -> [SMPServer] -> ConnInfo -> AEvent AEConn -- ConnInfo is from sender, [SMPServer] will be empty only in v1 handshake @@ -400,6 +398,10 @@ data AEvent (e :: AEntity) where DISCONNECT :: AProtocolType -> TransportHost -> AEvent AENone DOWN :: SMPServer -> [ConnId] -> AEvent AENone UP :: SMPServer -> [ConnId] -> AEvent AENone + SERVICE_ALL :: SMPServer -> AEvent AENone -- all service messages are delivered + SERVICE_DOWN :: SMPServer -> ServiceSub -> AEvent AENone + SERVICE_UP :: SMPServer -> ServiceSubResult -> AEvent AENone + SERVICE_END :: SMPServer -> ServiceSub -> AEvent AENone SWITCH :: QueueDirection -> SwitchPhase -> ConnectionStats -> AEvent AEConn RSYNC :: RatchetSyncState -> Maybe AgentCryptoError -> ConnectionStats -> AEvent AEConn SENT :: AgentMsgId -> Maybe SMPServer -> AEvent AEConn @@ -415,7 +417,7 @@ data AEvent (e :: AEntity) where DEL_USER :: Int64 -> AEvent AENone STAT :: ConnectionStats -> AEvent AEConn OK :: AEvent AEConn - JOINED :: SndQueueSecured -> Maybe ClientServiceId -> AEvent AEConn + JOINED :: SndQueueSecured -> AEvent AEConn ERR :: AgentErrorType -> AEvent AEConn ERRS :: NonEmpty (ConnId, AgentErrorType) -> AEvent AENone SUSPENDED :: AEvent AENone @@ -477,6 +479,10 @@ data AEventTag (e :: AEntity) where DISCONNECT_ :: AEventTag AENone DOWN_ :: AEventTag AENone UP_ :: AEventTag AENone + SERVICE_ALL_ :: AEventTag AENone + SERVICE_DOWN_ :: AEventTag AENone + SERVICE_UP_ :: AEventTag AENone + SERVICE_END_ :: AEventTag AENone SWITCH_ :: AEventTag AEConn RSYNC_ :: AEventTag AEConn SENT_ :: AEventTag AEConn @@ -536,6 +542,10 @@ aEventTag = \case DISCONNECT {} -> DISCONNECT_ DOWN {} -> DOWN_ UP {} -> UP_ + SERVICE_ALL _ -> SERVICE_ALL_ + SERVICE_DOWN {} -> SERVICE_DOWN_ + SERVICE_UP {} -> SERVICE_UP_ + SERVICE_END {} -> SERVICE_END_ SWITCH {} -> SWITCH_ RSYNC {} -> RSYNC_ SENT {} -> SENT_ @@ -1894,16 +1904,6 @@ instance Encoding UserLinkData where smpP = UserLinkData <$> ((A.char '\255' *> (unLarge <$> smpP)) <|> smpP) {-# INLINE smpP #-} -data StoredClientService (s :: DBStored) = ClientService - { dbServiceId :: DBEntityId' s, - serviceId :: SMP.ServiceId - } - deriving (Eq, Show) - -type ClientService = StoredClientService 'DBStored - -type ClientServiceId = DBEntityId - -- | SMP queue status. data QueueStatus = -- | queue is created diff --git a/src/Simplex/Messaging/Agent/Store.hs b/src/Simplex/Messaging/Agent/Store.hs index 7e246da5c..605a07e78 100644 --- a/src/Simplex/Messaging/Agent/Store.hs +++ b/src/Simplex/Messaging/Agent/Store.hs @@ -67,9 +67,9 @@ module Simplex.Messaging.Agent.Store AsyncCmdId, StoreError (..), AnyStoreError (..), + ServiceAssoc, createStore, rcvQueueSub, - clientServiceId, rcvSMPQueueAddress, canAbortRcvSwitch, findQ, @@ -101,9 +101,9 @@ import Data.Time (UTCTime) import Data.Type.Equality import Simplex.Messaging.Agent.Protocol import Simplex.Messaging.Agent.RetryInterval (RI2State) -import Simplex.Messaging.Agent.Store.Entity import Simplex.Messaging.Agent.Store.Common import Simplex.Messaging.Agent.Store.DB (SQLError) +import Simplex.Messaging.Agent.Store.Entity import Simplex.Messaging.Agent.Store.Interface (createDBStore) import Simplex.Messaging.Agent.Store.Migrations.App (appMigrations) import Simplex.Messaging.Agent.Store.Shared (MigrationConfig (..), MigrationError (..)) @@ -158,7 +158,7 @@ data StoredRcvQueue (q :: DBStored) = RcvQueue -- | short link ID and credentials shortLink :: Maybe ShortLinkCreds, -- | associated client service - clientService :: Maybe (StoredClientService q), + rcvServiceAssoc :: ServiceAssoc, -- | queue status status :: QueueStatus, -- | to enable notifications for this queue - this field is duplicated from ConnData @@ -199,9 +199,7 @@ rcvQueueSub :: RcvQueue -> RcvQueueSub rcvQueueSub RcvQueue {userId, connId, server, rcvId, rcvPrivateKey, status, enableNtfs, clientNoticeId, dbQueueId = DBEntityId dbQueueId, primary, dbReplaceQueueId} = RcvQueueSub {userId, connId, server, rcvId, rcvPrivateKey, status, enableNtfs, clientNoticeId, dbQueueId, primary, dbReplaceQueueId} -clientServiceId :: RcvQueue -> Maybe ClientServiceId -clientServiceId = fmap dbServiceId . clientService -{-# INLINE clientServiceId #-} +type ServiceAssoc = Bool rcvSMPQueueAddress :: RcvQueue -> SMPQueueAddress rcvSMPQueueAddress RcvQueue {server, sndId, e2ePrivKey, queueMode} = diff --git a/src/Simplex/Messaging/Agent/Store/AgentStore.hs b/src/Simplex/Messaging/Agent/Store/AgentStore.hs index 711f6f05b..32720dd85 100644 --- a/src/Simplex/Messaging/Agent/Store/AgentStore.hs +++ b/src/Simplex/Messaging/Agent/Store/AgentStore.hs @@ -35,6 +35,15 @@ module Simplex.Messaging.Agent.Store.AgentStore deleteUsersWithoutConns, checkUser, + -- * Client services + createClientService, + getClientServiceCredentials, + getSubscriptionService, + getClientServiceServers, + setClientServiceId, + deleteClientService, + deleteClientServices, + -- * Queues and connections createServer, createNewConn, @@ -45,7 +54,11 @@ module Simplex.Messaging.Agent.Store.AgentStore updateClientNotices, getSubscriptionServers, getUserServerRcvQueueSubs, + unassocUserServerRcvQueueSubs, + unassocUserServerRcvQueueSubs', unsetQueuesToSubscribe, + setRcvServiceAssocs, + removeRcvServiceAssocs, getConnIds, getConn, getDeletedConn, @@ -280,7 +293,9 @@ import qualified Data.Set as S import Data.Text.Encoding (decodeLatin1, encodeUtf8) import Data.Time.Clock (NominalDiffTime, UTCTime, addUTCTime, getCurrentTime) import Data.Word (Word32) +import qualified Data.X509 as X import Network.Socket (ServiceName) +import qualified Network.TLS as TLS import Simplex.FileTransfer.Client (XFTPChunkSpec (..)) import Simplex.FileTransfer.Description import Simplex.FileTransfer.Protocol (FileParty (..), SFileParty (..)) @@ -334,7 +349,7 @@ handleSQLError err e = case constraintViolation e of #else handleSQLError err e | SQL.sqlError e == SQL.ErrorConstraint = err - | otherwise = SEInternal $ bshow e + | otherwise = SEInternal $ encodeUtf8 $ tshow e <> ": " <> SQL.sqlErrorDetails e <> ", " <> SQL.sqlErrorContext e #endif createUserRecord :: DB.Connection -> IO UserId @@ -395,6 +410,110 @@ deleteUsersWithoutConns db = do forM_ userIds $ DB.execute db "DELETE FROM users WHERE user_id = ?" . Only pure userIds +createClientService :: DB.Connection -> UserId -> SMPServer -> (C.KeyHash, TLS.Credential) -> IO () +createClientService db userId srv (kh, (cert, pk)) = do + serverKeyHash_ <- createServer db srv + DB.execute + db + [sql| + INSERT INTO client_services + (user_id, host, port, server_key_hash, service_cert_hash, service_cert, service_priv_key) + VALUES (?,?,?,?,?,?,?) + ON CONFLICT (user_id, host, port, server_key_hash) + DO UPDATE SET + service_cert_hash = EXCLUDED.service_cert_hash, + service_cert = EXCLUDED.service_cert, + service_priv_key = EXCLUDED.service_priv_key, + service_id = NULL + |] + (userId, host srv, port srv, serverKeyHash_, kh, cert, pk) + +getClientServiceCredentials :: DB.Connection -> UserId -> SMPServer -> IO (Maybe ((C.KeyHash, TLS.Credential), Maybe ServiceId)) +getClientServiceCredentials db userId srv = + maybeFirstRow toService $ + DB.query + db + [sql| + SELECT c.service_cert_hash, c.service_cert, c.service_priv_key, c.service_id + FROM client_services c + JOIN servers s ON c.host = s.host AND c.port = s.port + WHERE c.user_id = ? AND c.host = ? AND c.port = ? + AND COALESCE(c.server_key_hash, s.key_hash) = ? + |] + (userId, host srv, port srv, keyHash srv) + where + toService (kh, cert, pk, serviceId_) = ((kh, (cert, pk)), serviceId_) + +getSubscriptionService :: DB.Connection -> UserId -> SMPServer -> IO (Maybe ServiceSub) +getSubscriptionService db userId (SMPServer h p kh) = + maybeFirstRow toService $ + DB.query + db + [sql| + SELECT c.service_id, c.service_queue_count, c.service_queue_ids_hash + FROM client_services c + JOIN servers s ON s.host = c.host AND s.port = c.port + WHERE c.user_id = ? AND c.host = ? AND c.port = ? AND COALESCE(c.server_key_hash, s.key_hash) = ? AND service_id IS NOT NULL + |] + (userId, h, p, kh) + where + toService (serviceId, qCnt, idsHash) = ServiceSub serviceId qCnt idsHash + +getClientServiceServers :: DB.Connection -> UserId -> IO [(SMPServer, ServiceSub)] +getClientServiceServers db userId = + map toServerService <$> + DB.query + db + [sql| + SELECT c.host, c.port, COALESCE(c.server_key_hash, s.key_hash), c.service_id, c.service_queue_count, c.service_queue_ids_hash + FROM client_services c + JOIN servers s ON s.host = c.host AND s.port = c.port + WHERE c.user_id = ? AND service_id IS NOT NULL + |] + (Only userId) + +toServerService :: (NonEmpty TransportHost, ServiceName, C.KeyHash, ServiceId, Int64, Binary ByteString) -> (ProtocolServer 'PSMP, ServiceSub) +toServerService (host, port, kh, serviceId, n, Binary idsHash) = + (SMPServer host port kh, ServiceSub serviceId n (IdsHash idsHash)) + +setClientServiceId :: DB.Connection -> UserId -> SMPServer -> ServiceId -> IO () +setClientServiceId db userId (SMPServer h p kh) serviceId = + DB.execute + db + [sql| + UPDATE client_services + SET service_id = ? + FROM servers s + WHERE client_services.user_id = ? + AND client_services.host = ? + AND client_services.port = ? + AND s.host = client_services.host + AND s.port = client_services.port + AND COALESCE(client_services.server_key_hash, s.key_hash) = ? + |] + (serviceId, userId, h, p, kh) + +deleteClientService :: DB.Connection -> UserId -> SMPServer -> IO () +deleteClientService db userId (SMPServer h p kh) = + DB.execute + db + [sql| + DELETE FROM client_services + WHERE user_id = ? AND host = ? AND port = ? + AND EXISTS ( + SELECT 1 FROM servers s + WHERE s.host = client_services.host + AND s.port = client_services.port + AND COALESCE(client_services.server_key_hash, s.key_hash) = ? + ); + |] + (userId, h, p, Just kh) + +deleteClientServices :: DB.Connection -> UserId -> IO () +deleteClientServices db userId = do + DB.execute db "DELETE FROM client_services WHERE user_id = ?" (Only userId) + removeUserRcvServiceAssocs db userId + createConn_ :: DB.Connection -> TVar ChaChaDRG -> @@ -409,7 +528,6 @@ createNewConn :: DB.Connection -> TVar ChaChaDRG -> ConnData -> SConnectionMode createNewConn db gVar cData cMode = do fst <$$> createConn_ db gVar cData (\connId -> createConnRecord db connId cData cMode) --- TODO [certs rcv] store clientServiceId from NewRcvQueue updateNewConnRcv :: DB.Connection -> ConnId -> NewRcvQueue -> SubscriptionMode -> IO (Either StoreError RcvQueue) updateNewConnRcv db connId rq subMode = getConnForUpdate db connId $>>= \case @@ -503,7 +621,6 @@ upgradeRcvConnToDuplex db connId sq = (SomeConn _ RcvConnection {}) -> Right <$> addConnSndQueue_ db connId sq (SomeConn c _) -> pure . Left . SEBadConnType "upgradeRcvConnToDuplex" $ connType c --- TODO [certs rcv] store clientServiceId from NewRcvQueue upgradeSndConnToDuplex :: DB.Connection -> ConnId -> NewRcvQueue -> SubscriptionMode -> IO (Either StoreError RcvQueue) upgradeSndConnToDuplex db connId rq subMode = getConnForUpdate db connId >>= \case @@ -511,7 +628,6 @@ upgradeSndConnToDuplex db connId rq subMode = Right (SomeConn c _) -> pure . Left . SEBadConnType "upgradeSndConnToDuplex" $ connType c _ -> pure $ Left SEConnNotFound --- TODO [certs rcv] store clientServiceId from NewRcvQueue addConnRcvQueue :: DB.Connection -> ConnId -> NewRcvQueue -> SubscriptionMode -> IO (Either StoreError RcvQueue) addConnRcvQueue db connId rq subMode = getConnForUpdate db connId >>= \case @@ -1989,6 +2105,15 @@ deriving newtype instance ToField ChunkReplicaId deriving newtype instance FromField ChunkReplicaId +instance ToField X.CertificateChain where toField = toField . Binary . smpEncode . C.encodeCertChain + +instance FromField X.CertificateChain where fromField = blobFieldDecoder (parseAll C.certChainP) + +instance ToField X.PrivKey where toField = toField . Binary . C.encodeASNObj + +instance FromField X.PrivKey where + fromField = blobFieldDecoder $ C.decodeASNKey >=> \case (pk, []) -> Right pk; r -> C.asnKeyError r + fromOnlyBI :: Only BoolInt -> Bool fromOnlyBI (Only (BI b)) = b {-# INLINE fromOnlyBI #-} @@ -2070,19 +2195,18 @@ insertRcvQueue_ db connId' rq@RcvQueue {..} subMode serverKeyHash_ = do db [sql| INSERT INTO rcv_queues - ( host, port, rcv_id, conn_id, rcv_private_key, rcv_dh_secret, e2e_priv_key, e2e_dh_secret, + ( host, port, rcv_id, rcv_service_assoc, conn_id, rcv_private_key, rcv_dh_secret, e2e_priv_key, e2e_dh_secret, snd_id, queue_mode, status, to_subscribe, rcv_queue_id, rcv_primary, replace_rcv_queue_id, smp_client_version, server_key_hash, link_id, link_key, link_priv_sig_key, link_enc_fixed_data, ntf_public_key, ntf_private_key, ntf_id, rcv_ntf_dh_secret - ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); |] - ( (host server, port server, rcvId, connId', rcvPrivateKey, rcvDhSecret, e2ePrivKey, e2eDhSecret) + ( (host server, port server, rcvId, BI rcvServiceAssoc, connId', rcvPrivateKey, rcvDhSecret, e2ePrivKey, e2eDhSecret) :. (sndId, queueMode, status, BI toSubscribe, qId, BI primary, dbReplaceQueueId, smpClientVersion, serverKeyHash_) :. (shortLinkId <$> shortLink, shortLinkKey <$> shortLink, linkPrivSigKey <$> shortLink, linkEncFixedData <$> shortLink) :. ntfCredsFields ) - -- TODO [certs rcv] save client service - pure (rq :: NewRcvQueue) {connId = connId', dbQueueId = qId, clientService = Nothing} + pure (rq :: NewRcvQueue) {connId = connId', dbQueueId = qId} where toSubscribe = subMode == SMOnlyCreate ntfCredsFields = case clientNtfCreds of @@ -2211,21 +2335,84 @@ getSubscriptionServers db onlyNeeded = toUserServer :: (UserId, NonEmpty TransportHost, ServiceName, C.KeyHash) -> (UserId, SMPServer) toUserServer (userId, host, port, keyHash) = (userId, SMPServer host port keyHash) -getUserServerRcvQueueSubs :: DB.Connection -> UserId -> SMPServer -> Bool -> IO [RcvQueueSub] -getUserServerRcvQueueSubs db userId (SMPServer h p kh) onlyNeeded = +-- TODO [certs rcv] check index for getting queues with service present +getUserServerRcvQueueSubs :: DB.Connection -> UserId -> SMPServer -> Bool -> ServiceAssoc -> IO [RcvQueueSub] +getUserServerRcvQueueSubs db userId (SMPServer h p kh) onlyNeeded hasService = map toRcvQueueSub <$> DB.query db - (rcvQueueSubQuery <> toSubscribe <> " c.deleted = 0 AND q.deleted = 0 AND c.user_id = ? AND q.host = ? AND q.port = ? AND COALESCE(q.server_key_hash, s.key_hash) = ?") + (rcvQueueSubQuery <> toSubscribe <> " c.deleted = 0 AND q.deleted = 0 AND c.user_id = ? AND q.host = ? AND q.port = ? AND COALESCE(q.server_key_hash, s.key_hash) = ?" <> serviceCond) (userId, h, p, kh) where toSubscribe | onlyNeeded = " WHERE q.to_subscribe = 1 AND " | otherwise = " WHERE " + serviceCond + | hasService = " AND q.rcv_service_assoc = 0" + | otherwise = "" + +unassocUserServerRcvQueueSubs :: DB.Connection -> UserId -> SMPServer -> IO [RcvQueueSub] +unassocUserServerRcvQueueSubs db userId srv@(SMPServer h p kh) = do + deleteClientService db userId srv + map toRcvQueueSub + <$> DB.query + db + (removeRcvAssocsQuery <> " " <> returningColums) + (h, p, userId, kh) + where + returningColums = + [sql| + RETURNING c.user_id, rcv_queues.conn_id, rcv_queues.host, rcv_queues.port, COALESCE(rcv_queues.server_key_hash, s.key_hash), + rcv_queues.rcv_id, rcv_queues.rcv_private_key, rcv_queues.status, c.enable_ntfs, rcv_queues.client_notice_id, + rcv_queues.rcv_queue_id, rcv_queues.rcv_primary, rcv_queues.replace_rcv_queue_id + |] + +unassocUserServerRcvQueueSubs' :: DB.Connection -> UserId -> SMPServer -> IO () +unassocUserServerRcvQueueSubs' db userId srv@(SMPServer h p kh) = do + deleteClientService db userId srv + DB.execute db removeRcvAssocsQuery (h, p, userId, kh) unsetQueuesToSubscribe :: DB.Connection -> IO () unsetQueuesToSubscribe db = DB.execute_ db "UPDATE rcv_queues SET to_subscribe = 0 WHERE to_subscribe = 1" +setRcvServiceAssocs :: SMPQueue q => DB.Connection -> [q] -> IO () +setRcvServiceAssocs db rqs = +#if defined(dbPostgres) + DB.execute db "UPDATE rcv_queues SET rcv_service_assoc = 1 WHERE rcv_id IN ?" $ Only $ In (map queueId rqs) +#else + DB.executeMany db "UPDATE rcv_queues SET rcv_service_assoc = 1 WHERE rcv_id = ?" $ map (Only . queueId) rqs +#endif + +removeRcvServiceAssocs :: DB.Connection -> UserId -> SMPServer -> IO () +removeRcvServiceAssocs db userId (SMPServer h p kh) = DB.execute db removeRcvAssocsQuery (h, p, userId, kh) + +removeRcvAssocsQuery :: Query +removeRcvAssocsQuery = + [sql| + UPDATE rcv_queues + SET rcv_service_assoc = 0 + FROM connections c, servers s + WHERE rcv_queues.host = ? + AND rcv_queues.port = ? + AND c.conn_id = rcv_queues.conn_id + AND c.user_id = ? + AND s.host = rcv_queues.host + AND s.port = rcv_queues.port + AND COALESCE(rcv_queues.server_key_hash, s.key_hash) = ? + |] + +removeUserRcvServiceAssocs :: DB.Connection -> UserId -> IO () +removeUserRcvServiceAssocs db userId = + DB.execute + db + [sql| + UPDATE rcv_queues + SET rcv_service_assoc = 0 + FROM connections c + WHERE c.conn_id = rcv_queues.conn_id AND c.user_id = ? + |] + (Only userId) + -- * getConn helpers getConnIds :: DB.Connection -> IO [ConnId] @@ -2471,7 +2658,7 @@ rcvQueueQuery = [sql| SELECT c.user_id, COALESCE(q.server_key_hash, s.key_hash), q.conn_id, q.host, q.port, q.rcv_id, q.rcv_private_key, q.rcv_dh_secret, q.e2e_priv_key, q.e2e_dh_secret, q.snd_id, q.queue_mode, q.status, c.enable_ntfs, q.client_notice_id, - q.rcv_queue_id, q.rcv_primary, q.replace_rcv_queue_id, q.switch_status, q.smp_client_version, q.delete_errors, + q.rcv_queue_id, q.rcv_primary, q.replace_rcv_queue_id, q.switch_status, q.smp_client_version, q.delete_errors, q.rcv_service_assoc, q.ntf_public_key, q.ntf_private_key, q.ntf_id, q.rcv_ntf_dh_secret, q.link_id, q.link_key, q.link_priv_sig_key, q.link_enc_fixed_data FROM rcv_queues q @@ -2481,13 +2668,13 @@ rcvQueueQuery = toRcvQueue :: (UserId, C.KeyHash, ConnId, NonEmpty TransportHost, ServiceName, SMP.RecipientId, SMP.RcvPrivateAuthKey, SMP.RcvDhSecret, C.PrivateKeyX25519, Maybe C.DhSecretX25519, SMP.SenderId, Maybe QueueMode) - :. (QueueStatus, Maybe BoolInt, Maybe NoticeId, DBEntityId, BoolInt, Maybe Int64, Maybe RcvSwitchStatus, Maybe VersionSMPC, Int) + :. (QueueStatus, Maybe BoolInt, Maybe NoticeId, DBEntityId, BoolInt, Maybe Int64, Maybe RcvSwitchStatus, Maybe VersionSMPC, Int, BoolInt) :. (Maybe SMP.NtfPublicAuthKey, Maybe SMP.NtfPrivateAuthKey, Maybe SMP.NotifierId, Maybe RcvNtfDhSecret) :. (Maybe SMP.LinkId, Maybe LinkKey, Maybe C.PrivateKeyEd25519, Maybe EncDataBytes) -> RcvQueue toRcvQueue ( (userId, keyHash, connId, host, port, rcvId, rcvPrivateKey, rcvDhSecret, e2ePrivKey, e2eDhSecret, sndId, queueMode) - :. (status, enableNtfs_, clientNoticeId, dbQueueId, BI primary, dbReplaceQueueId, rcvSwchStatus, smpClientVersion_, deleteErrors) + :. (status, enableNtfs_, clientNoticeId, dbQueueId, BI primary, dbReplaceQueueId, rcvSwchStatus, smpClientVersion_, deleteErrors, BI rcvServiceAssoc) :. (ntfPublicKey_, ntfPrivateKey_, notifierId_, rcvNtfDhSecret_) :. (shortLinkId_, shortLinkKey_, linkPrivSigKey_, linkEncFixedData_) ) = @@ -2500,8 +2687,7 @@ toRcvQueue (Just shortLinkId, Just shortLinkKey, Just linkPrivSigKey, Just linkEncFixedData) -> Just ShortLinkCreds {shortLinkId, shortLinkKey, linkPrivSigKey, linkRootSigKey = Nothing, linkEncFixedData} -- TODO linkRootSigKey should be stored in a separate field _ -> Nothing enableNtfs = maybe True unBI enableNtfs_ - -- TODO [certs rcv] read client service - in RcvQueue {userId, connId, server, rcvId, rcvPrivateKey, rcvDhSecret, e2ePrivKey, e2eDhSecret, sndId, queueMode, shortLink, clientService = Nothing, status, enableNtfs, clientNoticeId, dbQueueId, primary, dbReplaceQueueId, rcvSwchStatus, smpClientVersion, clientNtfCreds, deleteErrors} + in RcvQueue {userId, connId, server, rcvId, rcvPrivateKey, rcvDhSecret, e2ePrivKey, e2eDhSecret, sndId, queueMode, shortLink, rcvServiceAssoc, status, enableNtfs, clientNoticeId, dbQueueId, primary, dbReplaceQueueId, rcvSwchStatus, smpClientVersion, clientNtfCreds, deleteErrors} -- | returns all connection queue credentials, the first queue is the primary one getRcvQueueSubsByConnId_ :: DB.Connection -> ConnId -> IO (Maybe (NonEmpty RcvQueueSub)) diff --git a/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/App.hs b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/App.hs index 6f1d68bcb..862ce7a1c 100644 --- a/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/App.hs +++ b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/App.hs @@ -11,6 +11,7 @@ import Simplex.Messaging.Agent.Store.Postgres.Migrations.M20250702_conn_invitati import Simplex.Messaging.Agent.Store.Postgres.Migrations.M20251009_queue_to_subscribe import Simplex.Messaging.Agent.Store.Postgres.Migrations.M20251010_client_notices import Simplex.Messaging.Agent.Store.Postgres.Migrations.M20251230_strict_tables +import Simplex.Messaging.Agent.Store.Postgres.Migrations.M20260115_service_certs import Simplex.Messaging.Agent.Store.Shared (Migration (..)) schemaMigrations :: [(String, Text, Maybe Text)] @@ -21,7 +22,8 @@ schemaMigrations = ("20250702_conn_invitations_remove_cascade_delete", m20250702_conn_invitations_remove_cascade_delete, Just down_m20250702_conn_invitations_remove_cascade_delete), ("20251009_queue_to_subscribe", m20251009_queue_to_subscribe, Just down_m20251009_queue_to_subscribe), ("20251010_client_notices", m20251010_client_notices, Just down_m20251010_client_notices), - ("20251230_strict_tables", m20251230_strict_tables, Just down_m20251230_strict_tables) + ("20251230_strict_tables", m20251230_strict_tables, Just down_m20251230_strict_tables), + ("20260115_service_certs", m20260115_service_certs, Just down_m20260115_service_certs) ] -- | The list of migrations in ascending order by date diff --git a/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/M20260115_service_certs.hs b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/M20260115_service_certs.hs new file mode 100644 index 000000000..a4e774d3e --- /dev/null +++ b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/M20260115_service_certs.hs @@ -0,0 +1,114 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE QuasiQuotes #-} + +module Simplex.Messaging.Agent.Store.Postgres.Migrations.M20260115_service_certs where + +import Data.Text (Text) +import Simplex.Messaging.Agent.Store.Postgres.Migrations.Util +import Text.RawString.QQ (r) + +m20260115_service_certs :: Text +m20260115_service_certs = + createXorHashFuncs <> [r| +CREATE TABLE client_services( + user_id BIGINT NOT NULL REFERENCES users ON UPDATE RESTRICT ON DELETE CASCADE, + host TEXT NOT NULL, + port TEXT NOT NULL, + server_key_hash BYTEA, + service_cert BYTEA NOT NULL, + service_cert_hash BYTEA NOT NULL, + service_priv_key BYTEA NOT NULL, + service_id BYTEA, + service_queue_count BIGINT NOT NULL DEFAULT 0, + service_queue_ids_hash BYTEA NOT NULL DEFAULT '\x00000000000000000000000000000000', + FOREIGN KEY(host, port) REFERENCES servers ON DELETE RESTRICT +); + +CREATE UNIQUE INDEX idx_server_certs_user_id_host_port ON client_services(user_id, host, port, server_key_hash); +CREATE INDEX idx_server_certs_host_port ON client_services(host, port); + +ALTER TABLE rcv_queues ADD COLUMN rcv_service_assoc SMALLINT NOT NULL DEFAULT 0; + +CREATE FUNCTION update_aggregates(p_conn_id BYTEA, p_host TEXT, p_port TEXT, p_change BIGINT, p_rcv_id BYTEA) RETURNS VOID +LANGUAGE plpgsql +AS $$ +DECLARE q_user_id BIGINT; +BEGIN + SELECT user_id INTO q_user_id FROM connections WHERE conn_id = p_conn_id; + UPDATE client_services + SET service_queue_count = service_queue_count + p_change, + service_queue_ids_hash = xor_combine(service_queue_ids_hash, public.digest(p_rcv_id, 'md5')) + WHERE user_id = q_user_id AND host = p_host AND port = p_port; +END; +$$; + +CREATE FUNCTION on_rcv_queue_insert() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 THEN + PERFORM update_aggregates(NEW.conn_id, NEW.host, NEW.port, 1, NEW.rcv_id); + END IF; + RETURN NEW; +END; +$$; + +CREATE FUNCTION on_rcv_queue_delete() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 THEN + PERFORM update_aggregates(OLD.conn_id, OLD.host, OLD.port, -1, OLD.rcv_id); + END IF; + RETURN OLD; +END; +$$; + +CREATE FUNCTION on_rcv_queue_update() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 THEN + IF NOT (NEW.rcv_service_assoc != 0 AND NEW.deleted = 0) THEN + PERFORM update_aggregates(OLD.conn_id, OLD.host, OLD.port, -1, OLD.rcv_id); + END IF; + ELSIF NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 THEN + PERFORM update_aggregates(NEW.conn_id, NEW.host, NEW.port, 1, NEW.rcv_id); + END IF; + RETURN NEW; +END; +$$; + +CREATE TRIGGER tr_rcv_queue_insert +AFTER INSERT ON rcv_queues +FOR EACH ROW EXECUTE PROCEDURE on_rcv_queue_insert(); + +CREATE TRIGGER tr_rcv_queue_delete +AFTER DELETE ON rcv_queues +FOR EACH ROW EXECUTE PROCEDURE on_rcv_queue_delete(); + +CREATE TRIGGER tr_rcv_queue_update +AFTER UPDATE ON rcv_queues +FOR EACH ROW EXECUTE PROCEDURE on_rcv_queue_update(); + |] + +down_m20260115_service_certs :: Text +down_m20260115_service_certs = + [r| +DROP TRIGGER tr_rcv_queue_insert ON rcv_queues; +DROP TRIGGER tr_rcv_queue_delete ON rcv_queues; +DROP TRIGGER tr_rcv_queue_update ON rcv_queues; + +DROP FUNCTION on_rcv_queue_insert; +DROP FUNCTION on_rcv_queue_delete; +DROP FUNCTION on_rcv_queue_update; + +DROP FUNCTION update_aggregates; + +ALTER TABLE rcv_queues DROP COLUMN rcv_service_assoc; + +DROP INDEX idx_server_certs_host_port; +DROP INDEX idx_server_certs_user_id_host_port; +DROP TABLE client_services; + |] + <> dropXorHashFuncs diff --git a/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/Util.hs b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/Util.hs new file mode 100644 index 000000000..b51d487e4 --- /dev/null +++ b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/Util.hs @@ -0,0 +1,46 @@ +{-# LANGUAGE QuasiQuotes #-} + +module Simplex.Messaging.Agent.Store.Postgres.Migrations.Util where + +import Data.Text (Text) +import qualified Data.Text as T +import Text.RawString.QQ (r) + +-- xor_combine is only applied to locally computed md5 hashes (128 bits/16 bytes), +-- so it is safe to require that all values are of the same length. +createXorHashFuncs :: Text +createXorHashFuncs = + T.pack + [r| +CREATE OR REPLACE FUNCTION xor_combine(state BYTEA, value BYTEA) RETURNS BYTEA +LANGUAGE plpgsql IMMUTABLE STRICT +AS $$ +DECLARE + result BYTEA := state; + i INTEGER; + len INTEGER := octet_length(value); +BEGIN + IF octet_length(state) != len THEN + RAISE EXCEPTION 'Inputs must be equal length (% != %)', octet_length(state), len; + END IF; + FOR i IN 0..len-1 LOOP + result := set_byte(result, i, get_byte(state, i) # get_byte(value, i)); + END LOOP; + RETURN result; +END; +$$; + +CREATE OR REPLACE AGGREGATE xor_aggregate(BYTEA) ( + SFUNC = xor_combine, + STYPE = BYTEA, + INITCOND = '\x00000000000000000000000000000000' -- 16 bytes +); + |] + +dropXorHashFuncs :: Text +dropXorHashFuncs = + T.pack + [r| +DROP AGGREGATE xor_aggregate(BYTEA); +DROP FUNCTION xor_combine; + |] diff --git a/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/agent_postgres_schema.sql b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/agent_postgres_schema.sql new file mode 100644 index 000000000..e6c695808 --- /dev/null +++ b/src/Simplex/Messaging/Agent/Store/Postgres/Migrations/agent_postgres_schema.sql @@ -0,0 +1,1469 @@ + + +SET statement_timeout = 0; +SET lock_timeout = 0; +SET idle_in_transaction_session_timeout = 0; +SET client_encoding = 'UTF8'; +SET standard_conforming_strings = on; +SELECT pg_catalog.set_config('search_path', '', false); +SET check_function_bodies = false; +SET xmloption = content; +SET client_min_messages = warning; +SET row_security = off; + + +CREATE SCHEMA smp_agent_test_protocol_schema; + + + +CREATE FUNCTION smp_agent_test_protocol_schema.on_rcv_queue_delete() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 THEN + PERFORM update_aggregates(OLD.conn_id, OLD.host, OLD.port, -1, OLD.rcv_id); + END IF; + RETURN OLD; +END; +$$; + + + +CREATE FUNCTION smp_agent_test_protocol_schema.on_rcv_queue_insert() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 THEN + PERFORM update_aggregates(NEW.conn_id, NEW.host, NEW.port, 1, NEW.rcv_id); + END IF; + RETURN NEW; +END; +$$; + + + +CREATE FUNCTION smp_agent_test_protocol_schema.on_rcv_queue_update() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 THEN + IF NOT (NEW.rcv_service_assoc != 0 AND NEW.deleted = 0) THEN + PERFORM update_aggregates(OLD.conn_id, OLD.host, OLD.port, -1, OLD.rcv_id); + END IF; + ELSIF NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 THEN + PERFORM update_aggregates(NEW.conn_id, NEW.host, NEW.port, 1, NEW.rcv_id); + END IF; + RETURN NEW; +END; +$$; + + + +CREATE FUNCTION smp_agent_test_protocol_schema.update_aggregates(p_conn_id bytea, p_host text, p_port text, p_change bigint, p_rcv_id bytea) RETURNS void + LANGUAGE plpgsql + AS $$ +DECLARE q_user_id BIGINT; +BEGIN + SELECT user_id INTO q_user_id FROM connections WHERE conn_id = p_conn_id; + UPDATE client_services + SET service_queue_count = service_queue_count + p_change, + service_queue_ids_hash = xor_combine(service_queue_ids_hash, public.digest(p_rcv_id, 'md5')) + WHERE user_id = q_user_id AND host = p_host AND port = p_port; +END; +$$; + + + +CREATE FUNCTION smp_agent_test_protocol_schema.xor_combine(state bytea, value bytea) RETURNS bytea + LANGUAGE plpgsql IMMUTABLE STRICT + AS $$ +DECLARE + result BYTEA := state; + i INTEGER; + len INTEGER := octet_length(value); +BEGIN + IF octet_length(state) != len THEN + RAISE EXCEPTION 'Inputs must be equal length (% != %)', octet_length(state), len; + END IF; + FOR i IN 0..len-1 LOOP + result := set_byte(result, i, get_byte(state, i) # get_byte(value, i)); + END LOOP; + RETURN result; +END; +$$; + + + +CREATE AGGREGATE smp_agent_test_protocol_schema.xor_aggregate(bytea) ( + SFUNC = smp_agent_test_protocol_schema.xor_combine, + STYPE = bytea, + INITCOND = '\x00000000000000000000000000000000' +); + + +SET default_table_access_method = heap; + + +CREATE TABLE smp_agent_test_protocol_schema.client_notices ( + client_notice_id bigint NOT NULL, + protocol text NOT NULL, + host text NOT NULL, + port text NOT NULL, + entity_id bytea NOT NULL, + server_key_hash bytea, + notice_ttl bigint, + created_at bigint NOT NULL, + updated_at bigint NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.client_notices ALTER COLUMN client_notice_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.client_notices_client_notice_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.client_services ( + user_id bigint NOT NULL, + host text NOT NULL, + port text NOT NULL, + server_key_hash bytea, + service_cert bytea NOT NULL, + service_cert_hash bytea NOT NULL, + service_priv_key bytea NOT NULL, + service_id bytea, + service_queue_count bigint DEFAULT 0 NOT NULL, + service_queue_ids_hash bytea DEFAULT '\x00000000000000000000000000000000'::bytea NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.commands ( + command_id bigint NOT NULL, + conn_id bytea NOT NULL, + host text, + port text, + corr_id bytea NOT NULL, + command_tag bytea NOT NULL, + command bytea NOT NULL, + agent_version integer DEFAULT 1 NOT NULL, + server_key_hash bytea, + created_at timestamp with time zone DEFAULT '1970-01-01 00:00:00+01'::timestamp with time zone NOT NULL, + failed smallint DEFAULT 0 +); + + + +ALTER TABLE smp_agent_test_protocol_schema.commands ALTER COLUMN command_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.commands_command_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.conn_confirmations ( + confirmation_id bytea NOT NULL, + conn_id bytea NOT NULL, + e2e_snd_pub_key bytea NOT NULL, + sender_key bytea, + ratchet_state bytea NOT NULL, + sender_conn_info bytea NOT NULL, + accepted smallint NOT NULL, + own_conn_info bytea, + created_at timestamp with time zone DEFAULT now() NOT NULL, + smp_reply_queues bytea, + smp_client_version integer +); + + + +CREATE TABLE smp_agent_test_protocol_schema.conn_invitations ( + invitation_id bytea NOT NULL, + contact_conn_id bytea, + cr_invitation bytea NOT NULL, + recipient_conn_info bytea NOT NULL, + accepted smallint DEFAULT 0 NOT NULL, + own_conn_info bytea, + created_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.connections ( + conn_id bytea NOT NULL, + conn_mode text NOT NULL, + last_internal_msg_id bigint DEFAULT 0 NOT NULL, + last_internal_rcv_msg_id bigint DEFAULT 0 NOT NULL, + last_internal_snd_msg_id bigint DEFAULT 0 NOT NULL, + last_external_snd_msg_id bigint DEFAULT 0 NOT NULL, + last_rcv_msg_hash bytea DEFAULT '\x'::bytea NOT NULL, + last_snd_msg_hash bytea DEFAULT '\x'::bytea NOT NULL, + smp_agent_version integer DEFAULT 1 NOT NULL, + duplex_handshake smallint DEFAULT 0, + enable_ntfs smallint, + deleted smallint DEFAULT 0 NOT NULL, + user_id bigint NOT NULL, + ratchet_sync_state text DEFAULT 'ok'::text NOT NULL, + deleted_at_wait_delivery timestamp with time zone, + pq_support smallint DEFAULT 0 NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.deleted_snd_chunk_replicas ( + deleted_snd_chunk_replica_id bigint NOT NULL, + user_id bigint NOT NULL, + xftp_server_id bigint NOT NULL, + replica_id bytea NOT NULL, + replica_key bytea NOT NULL, + chunk_digest bytea NOT NULL, + delay bigint, + retries bigint DEFAULT 0 NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + failed smallint DEFAULT 0 +); + + + +ALTER TABLE smp_agent_test_protocol_schema.deleted_snd_chunk_replicas ALTER COLUMN deleted_snd_chunk_replica_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.deleted_snd_chunk_replicas_deleted_snd_chunk_replica_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.encrypted_rcv_message_hashes ( + encrypted_rcv_message_hash_id bigint NOT NULL, + conn_id bytea NOT NULL, + hash bytea NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.encrypted_rcv_message_hashes ALTER COLUMN encrypted_rcv_message_hash_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.encrypted_rcv_message_hashes_encrypted_rcv_message_hash_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.inv_short_links ( + inv_short_link_id bigint NOT NULL, + host text NOT NULL, + port text NOT NULL, + server_key_hash bytea, + link_id bytea NOT NULL, + link_key bytea NOT NULL, + snd_private_key bytea NOT NULL, + snd_id bytea +); + + + +ALTER TABLE smp_agent_test_protocol_schema.inv_short_links ALTER COLUMN inv_short_link_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.inv_short_links_inv_short_link_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.messages ( + conn_id bytea NOT NULL, + internal_id bigint NOT NULL, + internal_ts timestamp with time zone NOT NULL, + internal_rcv_id bigint, + internal_snd_id bigint, + msg_type bytea NOT NULL, + msg_body bytea DEFAULT '\x'::bytea NOT NULL, + msg_flags text, + pq_encryption smallint DEFAULT 0 NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.migrations ( + name text NOT NULL, + ts timestamp without time zone NOT NULL, + down text +); + + + +CREATE TABLE smp_agent_test_protocol_schema.ntf_servers ( + ntf_host text NOT NULL, + ntf_port text NOT NULL, + ntf_key_hash bytea NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.ntf_subscriptions ( + conn_id bytea NOT NULL, + smp_host text, + smp_port text, + smp_ntf_id bytea, + ntf_host text NOT NULL, + ntf_port text NOT NULL, + ntf_sub_id bytea, + ntf_sub_status text NOT NULL, + ntf_sub_action text, + ntf_sub_smp_action text, + ntf_sub_action_ts timestamp with time zone, + updated_by_supervisor smallint DEFAULT 0 NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + smp_server_key_hash bytea, + ntf_failed smallint DEFAULT 0, + smp_failed smallint DEFAULT 0 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.ntf_tokens ( + provider text NOT NULL, + device_token bytea NOT NULL, + ntf_host text NOT NULL, + ntf_port text NOT NULL, + tkn_id bytea, + tkn_pub_key bytea NOT NULL, + tkn_priv_key bytea NOT NULL, + tkn_pub_dh_key bytea NOT NULL, + tkn_priv_dh_key bytea NOT NULL, + tkn_dh_secret bytea, + tkn_status text NOT NULL, + tkn_action bytea, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + ntf_mode text +); + + + +CREATE TABLE smp_agent_test_protocol_schema.ntf_tokens_to_delete ( + ntf_token_to_delete_id bigint NOT NULL, + ntf_host text NOT NULL, + ntf_port text NOT NULL, + ntf_key_hash bytea NOT NULL, + tkn_id bytea NOT NULL, + tkn_priv_key bytea NOT NULL, + del_failed smallint DEFAULT 0, + created_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.ntf_tokens_to_delete ALTER COLUMN ntf_token_to_delete_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.ntf_tokens_to_delete_ntf_token_to_delete_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.processed_ratchet_key_hashes ( + processed_ratchet_key_hash_id bigint NOT NULL, + conn_id bytea NOT NULL, + hash bytea NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.processed_ratchet_key_hashes ALTER COLUMN processed_ratchet_key_hash_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.processed_ratchet_key_hashes_processed_ratchet_key_hash_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.ratchets ( + conn_id bytea NOT NULL, + x3dh_priv_key_1 bytea, + x3dh_priv_key_2 bytea, + ratchet_state bytea, + e2e_version integer DEFAULT 1 NOT NULL, + x3dh_pub_key_1 bytea, + x3dh_pub_key_2 bytea, + pq_priv_kem bytea, + pq_pub_kem bytea +); + + + +CREATE TABLE smp_agent_test_protocol_schema.rcv_file_chunk_replicas ( + rcv_file_chunk_replica_id bigint NOT NULL, + rcv_file_chunk_id bigint NOT NULL, + replica_number bigint NOT NULL, + xftp_server_id bigint NOT NULL, + replica_id bytea NOT NULL, + replica_key bytea NOT NULL, + received smallint DEFAULT 0 NOT NULL, + delay bigint, + retries bigint DEFAULT 0 NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.rcv_file_chunk_replicas ALTER COLUMN rcv_file_chunk_replica_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.rcv_file_chunk_replicas_rcv_file_chunk_replica_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.rcv_file_chunks ( + rcv_file_chunk_id bigint NOT NULL, + rcv_file_id bigint NOT NULL, + chunk_no bigint NOT NULL, + chunk_size bigint NOT NULL, + digest bytea NOT NULL, + tmp_path text, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.rcv_file_chunks ALTER COLUMN rcv_file_chunk_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.rcv_file_chunks_rcv_file_chunk_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.rcv_files ( + rcv_file_id bigint NOT NULL, + rcv_file_entity_id bytea NOT NULL, + user_id bigint NOT NULL, + size bigint NOT NULL, + digest bytea NOT NULL, + key bytea NOT NULL, + nonce bytea NOT NULL, + chunk_size bigint NOT NULL, + prefix_path text NOT NULL, + tmp_path text, + save_path text NOT NULL, + status text NOT NULL, + deleted smallint DEFAULT 0 NOT NULL, + error text, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + save_file_key bytea, + save_file_nonce bytea, + failed smallint DEFAULT 0, + redirect_id bigint, + redirect_entity_id bytea, + redirect_size bigint, + redirect_digest bytea, + approved_relays smallint DEFAULT 0 NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.rcv_files ALTER COLUMN rcv_file_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.rcv_files_rcv_file_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.rcv_messages ( + conn_id bytea NOT NULL, + internal_rcv_id bigint NOT NULL, + internal_id bigint NOT NULL, + external_snd_id bigint NOT NULL, + broker_id bytea NOT NULL, + broker_ts timestamp with time zone NOT NULL, + internal_hash bytea NOT NULL, + external_prev_snd_hash bytea NOT NULL, + integrity bytea NOT NULL, + user_ack smallint DEFAULT 0, + rcv_queue_id bigint NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.rcv_queues ( + host text NOT NULL, + port text NOT NULL, + rcv_id bytea NOT NULL, + conn_id bytea NOT NULL, + rcv_private_key bytea NOT NULL, + rcv_dh_secret bytea NOT NULL, + e2e_priv_key bytea NOT NULL, + e2e_dh_secret bytea, + snd_id bytea NOT NULL, + snd_key bytea, + status text NOT NULL, + smp_server_version integer DEFAULT 1 NOT NULL, + smp_client_version integer, + ntf_public_key bytea, + ntf_private_key bytea, + ntf_id bytea, + rcv_ntf_dh_secret bytea, + rcv_queue_id bigint NOT NULL, + rcv_primary smallint NOT NULL, + replace_rcv_queue_id bigint, + delete_errors bigint DEFAULT 0 NOT NULL, + server_key_hash bytea, + switch_status text, + deleted smallint DEFAULT 0 NOT NULL, + last_broker_ts timestamp with time zone, + link_id bytea, + link_key bytea, + link_priv_sig_key bytea, + link_enc_fixed_data bytea, + queue_mode text, + to_subscribe smallint DEFAULT 0 NOT NULL, + client_notice_id bigint, + rcv_service_assoc smallint DEFAULT 0 NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.servers ( + host text NOT NULL, + port text NOT NULL, + key_hash bytea NOT NULL +); + + + +CREATE TABLE smp_agent_test_protocol_schema.servers_stats ( + servers_stats_id bigint NOT NULL, + servers_stats text, + started_at timestamp with time zone DEFAULT now() NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.servers_stats ALTER COLUMN servers_stats_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.servers_stats_servers_stats_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.skipped_messages ( + skipped_message_id bigint NOT NULL, + conn_id bytea NOT NULL, + header_key bytea NOT NULL, + msg_n bigint NOT NULL, + msg_key bytea NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.skipped_messages ALTER COLUMN skipped_message_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.skipped_messages_skipped_message_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_file_chunk_replica_recipients ( + snd_file_chunk_replica_recipient_id bigint NOT NULL, + snd_file_chunk_replica_id bigint NOT NULL, + rcv_replica_id bytea NOT NULL, + rcv_replica_key bytea NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.snd_file_chunk_replica_recipients ALTER COLUMN snd_file_chunk_replica_recipient_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.snd_file_chunk_replica_recipi_snd_file_chunk_replica_recipi_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_file_chunk_replicas ( + snd_file_chunk_replica_id bigint NOT NULL, + snd_file_chunk_id bigint NOT NULL, + replica_number bigint NOT NULL, + xftp_server_id bigint NOT NULL, + replica_id bytea NOT NULL, + replica_key bytea NOT NULL, + replica_status text NOT NULL, + delay bigint, + retries bigint DEFAULT 0 NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.snd_file_chunk_replicas ALTER COLUMN snd_file_chunk_replica_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.snd_file_chunk_replicas_snd_file_chunk_replica_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_file_chunks ( + snd_file_chunk_id bigint NOT NULL, + snd_file_id bigint NOT NULL, + chunk_no bigint NOT NULL, + chunk_offset bigint NOT NULL, + chunk_size bigint NOT NULL, + digest bytea NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.snd_file_chunks ALTER COLUMN snd_file_chunk_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.snd_file_chunks_snd_file_chunk_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_files ( + snd_file_id bigint NOT NULL, + snd_file_entity_id bytea NOT NULL, + user_id bigint NOT NULL, + num_recipients bigint NOT NULL, + digest bytea, + key bytea NOT NULL, + nonce bytea NOT NULL, + path text NOT NULL, + prefix_path text, + status text NOT NULL, + deleted smallint DEFAULT 0 NOT NULL, + error text, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL, + src_file_key bytea, + src_file_nonce bytea, + failed smallint DEFAULT 0, + redirect_size bigint, + redirect_digest bytea +); + + + +ALTER TABLE smp_agent_test_protocol_schema.snd_files ALTER COLUMN snd_file_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.snd_files_snd_file_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_message_bodies ( + snd_message_body_id bigint NOT NULL, + agent_msg bytea DEFAULT '\x'::bytea NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.snd_message_bodies ALTER COLUMN snd_message_body_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.snd_message_bodies_snd_message_body_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_message_deliveries ( + snd_message_delivery_id bigint NOT NULL, + conn_id bytea NOT NULL, + snd_queue_id bigint NOT NULL, + internal_id bigint NOT NULL, + failed smallint DEFAULT 0 +); + + + +ALTER TABLE smp_agent_test_protocol_schema.snd_message_deliveries ALTER COLUMN snd_message_delivery_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.snd_message_deliveries_snd_message_delivery_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_messages ( + conn_id bytea NOT NULL, + internal_snd_id bigint NOT NULL, + internal_id bigint NOT NULL, + internal_hash bytea NOT NULL, + previous_msg_hash bytea DEFAULT '\x'::bytea NOT NULL, + retry_int_slow bigint, + retry_int_fast bigint, + rcpt_internal_id bigint, + rcpt_status text, + msg_encrypt_key bytea, + padded_msg_len bigint, + snd_message_body_id bigint +); + + + +CREATE TABLE smp_agent_test_protocol_schema.snd_queues ( + host text NOT NULL, + port text NOT NULL, + snd_id bytea NOT NULL, + conn_id bytea NOT NULL, + snd_private_key bytea NOT NULL, + e2e_dh_secret bytea NOT NULL, + status text NOT NULL, + smp_server_version integer DEFAULT 1 NOT NULL, + smp_client_version integer DEFAULT 1 NOT NULL, + snd_public_key bytea, + e2e_pub_key bytea, + snd_queue_id bigint NOT NULL, + snd_primary smallint NOT NULL, + replace_snd_queue_id bigint, + server_key_hash bytea, + switch_status text, + queue_mode text +); + + + +CREATE TABLE smp_agent_test_protocol_schema.users ( + user_id bigint NOT NULL, + deleted smallint DEFAULT 0 NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.users ALTER COLUMN user_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.users_user_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +CREATE TABLE smp_agent_test_protocol_schema.xftp_servers ( + xftp_server_id bigint NOT NULL, + xftp_host text NOT NULL, + xftp_port text NOT NULL, + xftp_key_hash bytea NOT NULL, + created_at timestamp with time zone DEFAULT now() NOT NULL, + updated_at timestamp with time zone DEFAULT now() NOT NULL +); + + + +ALTER TABLE smp_agent_test_protocol_schema.xftp_servers ALTER COLUMN xftp_server_id ADD GENERATED ALWAYS AS IDENTITY ( + SEQUENCE NAME smp_agent_test_protocol_schema.xftp_servers_xftp_server_id_seq + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1 +); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.client_notices + ADD CONSTRAINT client_notices_pkey PRIMARY KEY (client_notice_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.commands + ADD CONSTRAINT commands_pkey PRIMARY KEY (command_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.conn_confirmations + ADD CONSTRAINT conn_confirmations_pkey PRIMARY KEY (confirmation_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.conn_invitations + ADD CONSTRAINT conn_invitations_pkey PRIMARY KEY (invitation_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.connections + ADD CONSTRAINT connections_pkey PRIMARY KEY (conn_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.deleted_snd_chunk_replicas + ADD CONSTRAINT deleted_snd_chunk_replicas_pkey PRIMARY KEY (deleted_snd_chunk_replica_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.encrypted_rcv_message_hashes + ADD CONSTRAINT encrypted_rcv_message_hashes_pkey PRIMARY KEY (encrypted_rcv_message_hash_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.inv_short_links + ADD CONSTRAINT inv_short_links_pkey PRIMARY KEY (inv_short_link_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.messages + ADD CONSTRAINT messages_pkey PRIMARY KEY (conn_id, internal_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.migrations + ADD CONSTRAINT migrations_pkey PRIMARY KEY (name); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ntf_servers + ADD CONSTRAINT ntf_servers_pkey PRIMARY KEY (ntf_host, ntf_port); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ntf_subscriptions + ADD CONSTRAINT ntf_subscriptions_pkey PRIMARY KEY (conn_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ntf_tokens + ADD CONSTRAINT ntf_tokens_pkey PRIMARY KEY (provider, device_token, ntf_host, ntf_port); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ntf_tokens_to_delete + ADD CONSTRAINT ntf_tokens_to_delete_pkey PRIMARY KEY (ntf_token_to_delete_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.processed_ratchet_key_hashes + ADD CONSTRAINT processed_ratchet_key_hashes_pkey PRIMARY KEY (processed_ratchet_key_hash_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ratchets + ADD CONSTRAINT ratchets_pkey PRIMARY KEY (conn_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_file_chunk_replicas + ADD CONSTRAINT rcv_file_chunk_replicas_pkey PRIMARY KEY (rcv_file_chunk_replica_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_file_chunks + ADD CONSTRAINT rcv_file_chunks_pkey PRIMARY KEY (rcv_file_chunk_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_files + ADD CONSTRAINT rcv_files_pkey PRIMARY KEY (rcv_file_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_files + ADD CONSTRAINT rcv_files_rcv_file_entity_id_key UNIQUE (rcv_file_entity_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_messages + ADD CONSTRAINT rcv_messages_pkey PRIMARY KEY (conn_id, internal_rcv_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_queues + ADD CONSTRAINT rcv_queues_host_port_snd_id_key UNIQUE (host, port, snd_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_queues + ADD CONSTRAINT rcv_queues_pkey PRIMARY KEY (host, port, rcv_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.servers + ADD CONSTRAINT servers_pkey PRIMARY KEY (host, port); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.servers_stats + ADD CONSTRAINT servers_stats_pkey PRIMARY KEY (servers_stats_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.skipped_messages + ADD CONSTRAINT skipped_messages_pkey PRIMARY KEY (skipped_message_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_file_chunk_replica_recipients + ADD CONSTRAINT snd_file_chunk_replica_recipients_pkey PRIMARY KEY (snd_file_chunk_replica_recipient_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_file_chunk_replicas + ADD CONSTRAINT snd_file_chunk_replicas_pkey PRIMARY KEY (snd_file_chunk_replica_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_file_chunks + ADD CONSTRAINT snd_file_chunks_pkey PRIMARY KEY (snd_file_chunk_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_files + ADD CONSTRAINT snd_files_pkey PRIMARY KEY (snd_file_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_message_bodies + ADD CONSTRAINT snd_message_bodies_pkey PRIMARY KEY (snd_message_body_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_message_deliveries + ADD CONSTRAINT snd_message_deliveries_pkey PRIMARY KEY (snd_message_delivery_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_messages + ADD CONSTRAINT snd_messages_pkey PRIMARY KEY (conn_id, internal_snd_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_queues + ADD CONSTRAINT snd_queues_pkey PRIMARY KEY (host, port, snd_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.users + ADD CONSTRAINT users_pkey PRIMARY KEY (user_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.xftp_servers + ADD CONSTRAINT xftp_servers_pkey PRIMARY KEY (xftp_server_id); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.xftp_servers + ADD CONSTRAINT xftp_servers_xftp_host_xftp_port_xftp_key_hash_key UNIQUE (xftp_host, xftp_port, xftp_key_hash); + + + +CREATE UNIQUE INDEX idx_client_notices_entity ON smp_agent_test_protocol_schema.client_notices USING btree (protocol, host, port, entity_id); + + + +CREATE INDEX idx_commands_conn_id ON smp_agent_test_protocol_schema.commands USING btree (conn_id); + + + +CREATE INDEX idx_commands_host_port ON smp_agent_test_protocol_schema.commands USING btree (host, port); + + + +CREATE INDEX idx_commands_server_commands ON smp_agent_test_protocol_schema.commands USING btree (host, port, created_at, command_id); + + + +CREATE INDEX idx_conn_confirmations_conn_id ON smp_agent_test_protocol_schema.conn_confirmations USING btree (conn_id); + + + +CREATE INDEX idx_conn_invitations_contact_conn_id ON smp_agent_test_protocol_schema.conn_invitations USING btree (contact_conn_id); + + + +CREATE INDEX idx_connections_user ON smp_agent_test_protocol_schema.connections USING btree (user_id); + + + +CREATE INDEX idx_deleted_snd_chunk_replicas_pending ON smp_agent_test_protocol_schema.deleted_snd_chunk_replicas USING btree (created_at); + + + +CREATE INDEX idx_deleted_snd_chunk_replicas_user_id ON smp_agent_test_protocol_schema.deleted_snd_chunk_replicas USING btree (user_id); + + + +CREATE INDEX idx_deleted_snd_chunk_replicas_xftp_server_id ON smp_agent_test_protocol_schema.deleted_snd_chunk_replicas USING btree (xftp_server_id); + + + +CREATE INDEX idx_encrypted_rcv_message_hashes_created_at ON smp_agent_test_protocol_schema.encrypted_rcv_message_hashes USING btree (created_at); + + + +CREATE INDEX idx_encrypted_rcv_message_hashes_hash ON smp_agent_test_protocol_schema.encrypted_rcv_message_hashes USING btree (conn_id, hash); + + + +CREATE UNIQUE INDEX idx_inv_short_links_link_id ON smp_agent_test_protocol_schema.inv_short_links USING btree (host, port, link_id); + + + +CREATE INDEX idx_messages_conn_id ON smp_agent_test_protocol_schema.messages USING btree (conn_id); + + + +CREATE INDEX idx_messages_conn_id_internal_rcv_id ON smp_agent_test_protocol_schema.messages USING btree (conn_id, internal_rcv_id); + + + +CREATE INDEX idx_messages_conn_id_internal_snd_id ON smp_agent_test_protocol_schema.messages USING btree (conn_id, internal_snd_id); + + + +CREATE INDEX idx_messages_internal_ts ON smp_agent_test_protocol_schema.messages USING btree (internal_ts); + + + +CREATE INDEX idx_messages_snd_expired ON smp_agent_test_protocol_schema.messages USING btree (conn_id, internal_snd_id, internal_ts); + + + +CREATE INDEX idx_ntf_subscriptions_ntf_host_ntf_port ON smp_agent_test_protocol_schema.ntf_subscriptions USING btree (ntf_host, ntf_port); + + + +CREATE INDEX idx_ntf_subscriptions_smp_host_smp_port ON smp_agent_test_protocol_schema.ntf_subscriptions USING btree (smp_host, smp_port); + + + +CREATE INDEX idx_ntf_tokens_ntf_host_ntf_port ON smp_agent_test_protocol_schema.ntf_tokens USING btree (ntf_host, ntf_port); + + + +CREATE INDEX idx_processed_ratchet_key_hashes_created_at ON smp_agent_test_protocol_schema.processed_ratchet_key_hashes USING btree (created_at); + + + +CREATE INDEX idx_processed_ratchet_key_hashes_hash ON smp_agent_test_protocol_schema.processed_ratchet_key_hashes USING btree (conn_id, hash); + + + +CREATE INDEX idx_ratchets_conn_id ON smp_agent_test_protocol_schema.ratchets USING btree (conn_id); + + + +CREATE INDEX idx_rcv_file_chunk_replicas_pending ON smp_agent_test_protocol_schema.rcv_file_chunk_replicas USING btree (received, replica_number); + + + +CREATE INDEX idx_rcv_file_chunk_replicas_rcv_file_chunk_id ON smp_agent_test_protocol_schema.rcv_file_chunk_replicas USING btree (rcv_file_chunk_id); + + + +CREATE INDEX idx_rcv_file_chunk_replicas_xftp_server_id ON smp_agent_test_protocol_schema.rcv_file_chunk_replicas USING btree (xftp_server_id); + + + +CREATE INDEX idx_rcv_file_chunks_rcv_file_id ON smp_agent_test_protocol_schema.rcv_file_chunks USING btree (rcv_file_id); + + + +CREATE INDEX idx_rcv_files_redirect_id ON smp_agent_test_protocol_schema.rcv_files USING btree (redirect_id); + + + +CREATE INDEX idx_rcv_files_status_created_at ON smp_agent_test_protocol_schema.rcv_files USING btree (status, created_at); + + + +CREATE INDEX idx_rcv_files_user_id ON smp_agent_test_protocol_schema.rcv_files USING btree (user_id); + + + +CREATE INDEX idx_rcv_messages_conn_id_internal_id ON smp_agent_test_protocol_schema.rcv_messages USING btree (conn_id, internal_id); + + + +CREATE UNIQUE INDEX idx_rcv_queue_id ON smp_agent_test_protocol_schema.rcv_queues USING btree (conn_id, rcv_queue_id); + + + +CREATE INDEX idx_rcv_queues_client_notice_id ON smp_agent_test_protocol_schema.rcv_queues USING btree (client_notice_id); + + + +CREATE UNIQUE INDEX idx_rcv_queues_link_id ON smp_agent_test_protocol_schema.rcv_queues USING btree (host, port, link_id); + + + +CREATE UNIQUE INDEX idx_rcv_queues_ntf ON smp_agent_test_protocol_schema.rcv_queues USING btree (host, port, ntf_id); + + + +CREATE INDEX idx_rcv_queues_to_subscribe ON smp_agent_test_protocol_schema.rcv_queues USING btree (to_subscribe); + + + +CREATE INDEX idx_server_certs_host_port ON smp_agent_test_protocol_schema.client_services USING btree (host, port); + + + +CREATE UNIQUE INDEX idx_server_certs_user_id_host_port ON smp_agent_test_protocol_schema.client_services USING btree (user_id, host, port, server_key_hash); + + + +CREATE INDEX idx_skipped_messages_conn_id ON smp_agent_test_protocol_schema.skipped_messages USING btree (conn_id); + + + +CREATE INDEX idx_snd_file_chunk_replica_recipients_snd_file_chunk_replica_id ON smp_agent_test_protocol_schema.snd_file_chunk_replica_recipients USING btree (snd_file_chunk_replica_id); + + + +CREATE INDEX idx_snd_file_chunk_replicas_pending ON smp_agent_test_protocol_schema.snd_file_chunk_replicas USING btree (replica_status, replica_number); + + + +CREATE INDEX idx_snd_file_chunk_replicas_snd_file_chunk_id ON smp_agent_test_protocol_schema.snd_file_chunk_replicas USING btree (snd_file_chunk_id); + + + +CREATE INDEX idx_snd_file_chunk_replicas_xftp_server_id ON smp_agent_test_protocol_schema.snd_file_chunk_replicas USING btree (xftp_server_id); + + + +CREATE INDEX idx_snd_file_chunks_snd_file_id ON smp_agent_test_protocol_schema.snd_file_chunks USING btree (snd_file_id); + + + +CREATE INDEX idx_snd_files_snd_file_entity_id ON smp_agent_test_protocol_schema.snd_files USING btree (snd_file_entity_id); + + + +CREATE INDEX idx_snd_files_status_created_at ON smp_agent_test_protocol_schema.snd_files USING btree (status, created_at); + + + +CREATE INDEX idx_snd_files_user_id ON smp_agent_test_protocol_schema.snd_files USING btree (user_id); + + + +CREATE INDEX idx_snd_message_deliveries ON smp_agent_test_protocol_schema.snd_message_deliveries USING btree (conn_id, snd_queue_id); + + + +CREATE INDEX idx_snd_message_deliveries_conn_id_internal_id ON smp_agent_test_protocol_schema.snd_message_deliveries USING btree (conn_id, internal_id); + + + +CREATE INDEX idx_snd_message_deliveries_expired ON smp_agent_test_protocol_schema.snd_message_deliveries USING btree (conn_id, snd_queue_id, failed, internal_id); + + + +CREATE INDEX idx_snd_messages_conn_id_internal_id ON smp_agent_test_protocol_schema.snd_messages USING btree (conn_id, internal_id); + + + +CREATE INDEX idx_snd_messages_rcpt_internal_id ON smp_agent_test_protocol_schema.snd_messages USING btree (conn_id, rcpt_internal_id); + + + +CREATE INDEX idx_snd_messages_snd_message_body_id ON smp_agent_test_protocol_schema.snd_messages USING btree (snd_message_body_id); + + + +CREATE UNIQUE INDEX idx_snd_queue_id ON smp_agent_test_protocol_schema.snd_queues USING btree (conn_id, snd_queue_id); + + + +CREATE INDEX idx_snd_queues_host_port ON smp_agent_test_protocol_schema.snd_queues USING btree (host, port); + + + +CREATE TRIGGER tr_rcv_queue_delete AFTER DELETE ON smp_agent_test_protocol_schema.rcv_queues FOR EACH ROW EXECUTE FUNCTION smp_agent_test_protocol_schema.on_rcv_queue_delete(); + + + +CREATE TRIGGER tr_rcv_queue_insert AFTER INSERT ON smp_agent_test_protocol_schema.rcv_queues FOR EACH ROW EXECUTE FUNCTION smp_agent_test_protocol_schema.on_rcv_queue_insert(); + + + +CREATE TRIGGER tr_rcv_queue_update AFTER UPDATE ON smp_agent_test_protocol_schema.rcv_queues FOR EACH ROW EXECUTE FUNCTION smp_agent_test_protocol_schema.on_rcv_queue_update(); + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.client_services + ADD CONSTRAINT client_services_host_port_fkey FOREIGN KEY (host, port) REFERENCES smp_agent_test_protocol_schema.servers(host, port) ON DELETE RESTRICT; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.client_services + ADD CONSTRAINT client_services_user_id_fkey FOREIGN KEY (user_id) REFERENCES smp_agent_test_protocol_schema.users(user_id) ON UPDATE RESTRICT ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.commands + ADD CONSTRAINT commands_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.commands + ADD CONSTRAINT commands_host_port_fkey FOREIGN KEY (host, port) REFERENCES smp_agent_test_protocol_schema.servers(host, port) ON UPDATE CASCADE ON DELETE RESTRICT; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.conn_confirmations + ADD CONSTRAINT conn_confirmations_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.conn_invitations + ADD CONSTRAINT conn_invitations_contact_conn_id_fkey FOREIGN KEY (contact_conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE SET NULL; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.connections + ADD CONSTRAINT connections_user_id_fkey FOREIGN KEY (user_id) REFERENCES smp_agent_test_protocol_schema.users(user_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.deleted_snd_chunk_replicas + ADD CONSTRAINT deleted_snd_chunk_replicas_user_id_fkey FOREIGN KEY (user_id) REFERENCES smp_agent_test_protocol_schema.users(user_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.deleted_snd_chunk_replicas + ADD CONSTRAINT deleted_snd_chunk_replicas_xftp_server_id_fkey FOREIGN KEY (xftp_server_id) REFERENCES smp_agent_test_protocol_schema.xftp_servers(xftp_server_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.encrypted_rcv_message_hashes + ADD CONSTRAINT encrypted_rcv_message_hashes_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.messages + ADD CONSTRAINT fk_messages_rcv_messages FOREIGN KEY (conn_id, internal_rcv_id) REFERENCES smp_agent_test_protocol_schema.rcv_messages(conn_id, internal_rcv_id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.messages + ADD CONSTRAINT fk_messages_snd_messages FOREIGN KEY (conn_id, internal_snd_id) REFERENCES smp_agent_test_protocol_schema.snd_messages(conn_id, internal_snd_id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.inv_short_links + ADD CONSTRAINT inv_short_links_host_port_fkey FOREIGN KEY (host, port) REFERENCES smp_agent_test_protocol_schema.servers(host, port) ON UPDATE CASCADE ON DELETE RESTRICT; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.messages + ADD CONSTRAINT messages_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ntf_subscriptions + ADD CONSTRAINT ntf_subscriptions_ntf_host_ntf_port_fkey FOREIGN KEY (ntf_host, ntf_port) REFERENCES smp_agent_test_protocol_schema.ntf_servers(ntf_host, ntf_port) ON UPDATE CASCADE ON DELETE RESTRICT; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ntf_subscriptions + ADD CONSTRAINT ntf_subscriptions_smp_host_smp_port_fkey FOREIGN KEY (smp_host, smp_port) REFERENCES smp_agent_test_protocol_schema.servers(host, port) ON UPDATE CASCADE ON DELETE SET NULL; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ntf_tokens + ADD CONSTRAINT ntf_tokens_ntf_host_ntf_port_fkey FOREIGN KEY (ntf_host, ntf_port) REFERENCES smp_agent_test_protocol_schema.ntf_servers(ntf_host, ntf_port) ON UPDATE CASCADE ON DELETE RESTRICT; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.processed_ratchet_key_hashes + ADD CONSTRAINT processed_ratchet_key_hashes_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.ratchets + ADD CONSTRAINT ratchets_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_file_chunk_replicas + ADD CONSTRAINT rcv_file_chunk_replicas_rcv_file_chunk_id_fkey FOREIGN KEY (rcv_file_chunk_id) REFERENCES smp_agent_test_protocol_schema.rcv_file_chunks(rcv_file_chunk_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_file_chunk_replicas + ADD CONSTRAINT rcv_file_chunk_replicas_xftp_server_id_fkey FOREIGN KEY (xftp_server_id) REFERENCES smp_agent_test_protocol_schema.xftp_servers(xftp_server_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_file_chunks + ADD CONSTRAINT rcv_file_chunks_rcv_file_id_fkey FOREIGN KEY (rcv_file_id) REFERENCES smp_agent_test_protocol_schema.rcv_files(rcv_file_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_files + ADD CONSTRAINT rcv_files_redirect_id_fkey FOREIGN KEY (redirect_id) REFERENCES smp_agent_test_protocol_schema.rcv_files(rcv_file_id) ON DELETE SET NULL; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_files + ADD CONSTRAINT rcv_files_user_id_fkey FOREIGN KEY (user_id) REFERENCES smp_agent_test_protocol_schema.users(user_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_messages + ADD CONSTRAINT rcv_messages_conn_id_internal_id_fkey FOREIGN KEY (conn_id, internal_id) REFERENCES smp_agent_test_protocol_schema.messages(conn_id, internal_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_queues + ADD CONSTRAINT rcv_queues_client_notice_id_fkey FOREIGN KEY (client_notice_id) REFERENCES smp_agent_test_protocol_schema.client_notices(client_notice_id) ON UPDATE RESTRICT ON DELETE SET NULL; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_queues + ADD CONSTRAINT rcv_queues_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.rcv_queues + ADD CONSTRAINT rcv_queues_host_port_fkey FOREIGN KEY (host, port) REFERENCES smp_agent_test_protocol_schema.servers(host, port) ON UPDATE CASCADE ON DELETE RESTRICT; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.skipped_messages + ADD CONSTRAINT skipped_messages_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.ratchets(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_file_chunk_replica_recipients + ADD CONSTRAINT snd_file_chunk_replica_recipient_snd_file_chunk_replica_id_fkey FOREIGN KEY (snd_file_chunk_replica_id) REFERENCES smp_agent_test_protocol_schema.snd_file_chunk_replicas(snd_file_chunk_replica_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_file_chunk_replicas + ADD CONSTRAINT snd_file_chunk_replicas_snd_file_chunk_id_fkey FOREIGN KEY (snd_file_chunk_id) REFERENCES smp_agent_test_protocol_schema.snd_file_chunks(snd_file_chunk_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_file_chunk_replicas + ADD CONSTRAINT snd_file_chunk_replicas_xftp_server_id_fkey FOREIGN KEY (xftp_server_id) REFERENCES smp_agent_test_protocol_schema.xftp_servers(xftp_server_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_file_chunks + ADD CONSTRAINT snd_file_chunks_snd_file_id_fkey FOREIGN KEY (snd_file_id) REFERENCES smp_agent_test_protocol_schema.snd_files(snd_file_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_files + ADD CONSTRAINT snd_files_user_id_fkey FOREIGN KEY (user_id) REFERENCES smp_agent_test_protocol_schema.users(user_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_message_deliveries + ADD CONSTRAINT snd_message_deliveries_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_message_deliveries + ADD CONSTRAINT snd_message_deliveries_conn_id_internal_id_fkey FOREIGN KEY (conn_id, internal_id) REFERENCES smp_agent_test_protocol_schema.messages(conn_id, internal_id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_messages + ADD CONSTRAINT snd_messages_conn_id_internal_id_fkey FOREIGN KEY (conn_id, internal_id) REFERENCES smp_agent_test_protocol_schema.messages(conn_id, internal_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_messages + ADD CONSTRAINT snd_messages_snd_message_body_id_fkey FOREIGN KEY (snd_message_body_id) REFERENCES smp_agent_test_protocol_schema.snd_message_bodies(snd_message_body_id) ON DELETE SET NULL; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_queues + ADD CONSTRAINT snd_queues_conn_id_fkey FOREIGN KEY (conn_id) REFERENCES smp_agent_test_protocol_schema.connections(conn_id) ON DELETE CASCADE; + + + +ALTER TABLE ONLY smp_agent_test_protocol_schema.snd_queues + ADD CONSTRAINT snd_queues_host_port_fkey FOREIGN KEY (host, port) REFERENCES smp_agent_test_protocol_schema.servers(host, port) ON UPDATE CASCADE ON DELETE RESTRICT; + + + diff --git a/src/Simplex/Messaging/Agent/Store/Postgres/Util.hs b/src/Simplex/Messaging/Agent/Store/Postgres/Util.hs index 0913c76e3..bcbb0e281 100644 --- a/src/Simplex/Messaging/Agent/Store/Postgres/Util.hs +++ b/src/Simplex/Messaging/Agent/Store/Postgres/Util.hs @@ -21,30 +21,32 @@ import Database.PostgreSQL.Simple.SqlQQ (sql) createDBAndUserIfNotExists :: ConnectInfo -> IO () createDBAndUserIfNotExists ConnectInfo {connectUser = user, connectDatabase = dbName} = do -- connect to the default "postgres" maintenance database - bracket (PSQL.connect defaultConnectInfo {connectUser = "postgres", connectDatabase = "postgres"}) PSQL.close $ - \postgresDB -> do - void $ PSQL.execute_ postgresDB "SET client_min_messages TO WARNING" - -- check if the user exists, create if not - [Only userExists] <- - PSQL.query - postgresDB - [sql| - SELECT EXISTS ( - SELECT 1 FROM pg_catalog.pg_roles - WHERE rolname = ? - ) - |] - (Only user) - unless userExists $ void $ PSQL.execute_ postgresDB (fromString $ "CREATE USER " <> user) - -- check if the database exists, create if not - dbExists <- checkDBExists postgresDB dbName - unless dbExists $ void $ PSQL.execute_ postgresDB (fromString $ "CREATE DATABASE " <> dbName <> " OWNER " <> user) + bracket (PSQL.connect defaultConnectInfo {connectUser = "postgres", connectDatabase = "postgres"}) PSQL.close $ \db -> do + execSQL db "SET client_min_messages TO WARNING" + -- check if the user exists, create if not + [Only userExists] <- + PSQL.query + db + [sql| + SELECT EXISTS ( + SELECT 1 FROM pg_catalog.pg_roles + WHERE rolname = ? + ) + |] + (Only user) + unless userExists $ execSQL db $ "CREATE USER " <> user + -- check if the database exists, create if not + dbExists <- checkDBExists db dbName + unless dbExists $ do + execSQL db $ "CREATE DATABASE " <> dbName <> " OWNER " <> user + bracket (PSQL.connect defaultConnectInfo {connectUser = "postgres", connectDatabase = dbName}) PSQL.close $ + (`execSQL` "CREATE EXTENSION IF NOT EXISTS pgcrypto") checkDBExists :: PSQL.Connection -> String -> IO Bool -checkDBExists postgresDB dbName = do +checkDBExists db dbName = do [Only dbExists] <- PSQL.query - postgresDB + db [sql| SELECT EXISTS ( SELECT 1 FROM pg_catalog.pg_database @@ -56,45 +58,45 @@ checkDBExists postgresDB dbName = do dropSchema :: ConnectInfo -> String -> IO () dropSchema connectInfo schema = - bracket (PSQL.connect connectInfo) PSQL.close $ - \db -> do - void $ PSQL.execute_ db "SET client_min_messages TO WARNING" - void $ PSQL.execute_ db (fromString $ "DROP SCHEMA IF EXISTS " <> schema <> " CASCADE") + bracket (PSQL.connect connectInfo) PSQL.close $ \db -> do + execSQL db "SET client_min_messages TO WARNING" + execSQL db $ "DROP SCHEMA IF EXISTS " <> schema <> " CASCADE" dropAllSchemasExceptSystem :: ConnectInfo -> IO () dropAllSchemasExceptSystem connectInfo = - bracket (PSQL.connect connectInfo) PSQL.close $ - \db -> do - void $ PSQL.execute_ db "SET client_min_messages TO WARNING" - schemaNames :: [Only String] <- - PSQL.query_ + bracket (PSQL.connect connectInfo) PSQL.close $ \db -> do + execSQL db "SET client_min_messages TO WARNING" + schemaNames :: [Only String] <- + PSQL.query_ + db + [sql| + SELECT schema_name + FROM information_schema.schemata + WHERE schema_name NOT IN ('public', 'pg_catalog', 'information_schema') + |] + forM_ schemaNames $ \(Only schema) -> + execSQL db $ "DROP SCHEMA " <> schema <> " CASCADE" + +dropDatabaseAndUser :: ConnectInfo -> IO () +dropDatabaseAndUser ConnectInfo {connectUser = user, connectDatabase = dbName} = + bracket (PSQL.connect defaultConnectInfo {connectUser = "postgres", connectDatabase = "postgres"}) PSQL.close $ \db -> do + execSQL db "SET client_min_messages TO WARNING" + dbExists <- checkDBExists db dbName + when dbExists $ do + execSQL db $ "ALTER DATABASE " <> dbName <> " WITH ALLOW_CONNECTIONS false" + -- terminate all connections to the database + _r :: [Only Bool] <- + PSQL.query db [sql| - SELECT schema_name - FROM information_schema.schemata - WHERE schema_name NOT IN ('public', 'pg_catalog', 'information_schema') + SELECT pg_terminate_backend(pg_stat_activity.pid) + FROM pg_stat_activity + WHERE datname = ? + AND pid <> pg_backend_pid() |] - forM_ schemaNames $ \(Only schema) -> - PSQL.execute_ db (fromString $ "DROP SCHEMA " <> schema <> " CASCADE") + (Only dbName) + execSQL db $ "DROP DATABASE " <> dbName + execSQL db $ "DROP USER IF EXISTS " <> user -dropDatabaseAndUser :: ConnectInfo -> IO () -dropDatabaseAndUser ConnectInfo {connectUser = user, connectDatabase = dbName} = - bracket (PSQL.connect defaultConnectInfo {connectUser = "postgres", connectDatabase = "postgres"}) PSQL.close $ - \postgresDB -> do - void $ PSQL.execute_ postgresDB "SET client_min_messages TO WARNING" - dbExists <- checkDBExists postgresDB dbName - when dbExists $ do - void $ PSQL.execute_ postgresDB (fromString $ "ALTER DATABASE " <> dbName <> " WITH ALLOW_CONNECTIONS false") - -- terminate all connections to the database - _r :: [Only Bool] <- - PSQL.query - postgresDB - [sql| - SELECT pg_terminate_backend(pg_stat_activity.pid) - FROM pg_stat_activity - WHERE datname = ? - AND pid <> pg_backend_pid() - |] - (Only dbName) - void $ PSQL.execute_ postgresDB (fromString $ "DROP DATABASE " <> dbName) - void $ PSQL.execute_ postgresDB (fromString $ "DROP USER IF EXISTS " <> user) +execSQL :: PSQL.Connection -> String -> IO () +execSQL db = void . PSQL.execute_ db . fromString diff --git a/src/Simplex/Messaging/Agent/Store/SQLite.hs b/src/Simplex/Messaging/Agent/Store/SQLite.hs index 6cc63c066..a670dd3e2 100644 --- a/src/Simplex/Messaging/Agent/Store/SQLite.hs +++ b/src/Simplex/Messaging/Agent/Store/SQLite.hs @@ -46,8 +46,11 @@ import Control.Concurrent.MVar import Control.Concurrent.STM import Control.Exception (bracketOnError, onException, throwIO) import Control.Monad +import Data.Bits (xor) import Data.ByteArray (ScrubbedBytes) import qualified Data.ByteArray as BA +import Data.ByteString (ByteString) +import qualified Data.ByteString as B import Data.Functor (($>)) import Data.IORef import Data.Maybe (fromMaybe) @@ -57,13 +60,17 @@ import Database.SQLite.Simple (Query (..)) import qualified Database.SQLite.Simple as SQL import Database.SQLite.Simple.QQ (sql) import qualified Database.SQLite3 as SQLite3 +import Database.SQLite3.Bindings +import Foreign.C.Types +import Foreign.Ptr import Simplex.Messaging.Agent.Store.Migrations (DBMigrate (..), sharedMigrateSchema) import qualified Simplex.Messaging.Agent.Store.SQLite.Migrations as Migrations import Simplex.Messaging.Agent.Store.SQLite.Common import qualified Simplex.Messaging.Agent.Store.SQLite.DB as DB import Simplex.Messaging.Agent.Store.SQLite.Util import Simplex.Messaging.Agent.Store.Shared (Migration (..), MigrationConfig (..), MigrationError (..)) -import Simplex.Messaging.Util (ifM, safeDecodeUtf8) +import qualified Simplex.Messaging.Crypto as C +import Simplex.Messaging.Util (ifM, packZipWith, safeDecodeUtf8) import System.Directory (copyFile, createDirectoryIfMissing, doesFileExist) import System.FilePath (takeDirectory, takeFileName, ()) @@ -119,14 +126,29 @@ connectDB path functions key track = do PRAGMA secure_delete = ON; PRAGMA auto_vacuum = FULL; |] - mapM_ addFunction functions + mapM_ addFunction functions' where db' = SQL.connectionHandle $ DB.conn db + functions' = SQLiteFuncDef "simplex_xor_md5_combine" 2 (SQLiteFuncPtr True sqliteXorMd5CombinePtr) : functions addFunction SQLiteFuncDef {funcName, argCount, funcPtrs} = either (throwIO . userError . show) pure =<< case funcPtrs of SQLiteFuncPtr isDet funcPtr -> createStaticFunction db' funcName argCount isDet funcPtr SQLiteAggrPtrs stepPtr finalPtr -> createStaticAggregate db' funcName argCount stepPtr finalPtr +foreign export ccall "simplex_xor_md5_combine" sqliteXorMd5Combine :: SQLiteFunc + +foreign import ccall "&simplex_xor_md5_combine" sqliteXorMd5CombinePtr :: FunPtr SQLiteFunc + +sqliteXorMd5Combine :: SQLiteFunc +sqliteXorMd5Combine = mkSQLiteFunc $ \cxt args -> do + idsHash <- SQLite3.funcArgBlob args 0 + rId <- SQLite3.funcArgBlob args 1 + SQLite3.funcResultBlob cxt $ xorMd5Combine idsHash rId + +xorMd5Combine :: ByteString -> ByteString -> ByteString +xorMd5Combine idsHash rId = packZipWith xor idsHash $ C.md5Hash rId +{-# INLINE xorMd5Combine #-} + closeDBStore :: DBStore -> IO () closeDBStore st@DBStore {dbClosed} = ifM (readTVarIO dbClosed) (putStrLn "closeDBStore: already closed") $ diff --git a/src/Simplex/Messaging/Agent/Store/SQLite/Common.hs b/src/Simplex/Messaging/Agent/Store/SQLite/Common.hs index f01c87b6e..2f9350a37 100644 --- a/src/Simplex/Messaging/Agent/Store/SQLite/Common.hs +++ b/src/Simplex/Messaging/Agent/Store/SQLite/Common.hs @@ -74,6 +74,12 @@ withConnectionPriority DBStore {dbSem, dbConnection} priority action | priority = E.bracket_ signal release $ withMVar dbConnection action | otherwise = lowPriority where + -- To debug FK errors, set foreign_keys = OFF in Simplex.Messaging.Agent.Store.SQLite and use action' instead of action + -- action' conn = do + -- r <- action conn + -- violations <- DB.query_ conn "PRAGMA foreign_key_check" :: IO [ (String, Int, String, Int)] + -- unless (null violations) $ print violations + -- pure r lowPriority = wait >> withMVar dbConnection (\db -> ifM free (Just <$> action db) (pure Nothing)) >>= maybe lowPriority pure signal = atomically $ modifyTVar' dbSem (+ 1) release = atomically $ modifyTVar' dbSem $ \sem -> if sem > 0 then sem - 1 else 0 diff --git a/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/App.hs b/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/App.hs index 26df43bc8..b2738269c 100644 --- a/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/App.hs +++ b/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/App.hs @@ -47,6 +47,7 @@ import Simplex.Messaging.Agent.Store.SQLite.Migrations.M20250702_conn_invitation import Simplex.Messaging.Agent.Store.SQLite.Migrations.M20251009_queue_to_subscribe import Simplex.Messaging.Agent.Store.SQLite.Migrations.M20251010_client_notices import Simplex.Messaging.Agent.Store.SQLite.Migrations.M20251230_strict_tables +import Simplex.Messaging.Agent.Store.SQLite.Migrations.M20260115_service_certs import Simplex.Messaging.Agent.Store.Shared (Migration (..)) schemaMigrations :: [(String, Query, Maybe Query)] @@ -93,7 +94,8 @@ schemaMigrations = ("m20250702_conn_invitations_remove_cascade_delete", m20250702_conn_invitations_remove_cascade_delete, Just down_m20250702_conn_invitations_remove_cascade_delete), ("m20251009_queue_to_subscribe", m20251009_queue_to_subscribe, Just down_m20251009_queue_to_subscribe), ("m20251010_client_notices", m20251010_client_notices, Just down_m20251010_client_notices), - ("m20251230_strict_tables", m20251230_strict_tables, Just down_m20251230_strict_tables) + ("m20251230_strict_tables", m20251230_strict_tables, Just down_m20251230_strict_tables), + ("m20260115_service_certs", m20260115_service_certs, Just down_m20260115_service_certs) ] -- | The list of migrations in ascending order by date diff --git a/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/M20250517_service_certs.hs b/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/M20250517_service_certs.hs deleted file mode 100644 index 7708fd6d2..000000000 --- a/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/M20250517_service_certs.hs +++ /dev/null @@ -1,40 +0,0 @@ -{-# LANGUAGE QuasiQuotes #-} - -module Simplex.Messaging.Agent.Store.SQLite.Migrations.M20250517_service_certs where - -import Database.SQLite.Simple (Query) -import Database.SQLite.Simple.QQ (sql) - --- TODO move date forward, create migration for postgres -m20250517_service_certs :: Query -m20250517_service_certs = - [sql| -CREATE TABLE server_certs( - server_cert_id INTEGER PRIMARY KEY AUTOINCREMENT, - user_id INTEGER NOT NULL REFERENCES users ON UPDATE RESTRICT ON DELETE CASCADE, - host TEXT NOT NULL, - port TEXT NOT NULL, - certificate BLOB NOT NULL, - priv_key BLOB NOT NULL, - service_id BLOB, - FOREIGN KEY(host, port) REFERENCES servers ON UPDATE CASCADE ON DELETE RESTRICT, -); - -CREATE UNIQUE INDEX idx_server_certs_user_id_host_port ON server_certs(user_id, host, port); - -CREATE INDEX idx_server_certs_host_port ON server_certs(host, port); - -ALTER TABLE rcv_queues ADD COLUMN rcv_service_id BLOB; - |] - -down_m20250517_service_certs :: Query -down_m20250517_service_certs = - [sql| -ALTER TABLE rcv_queues DROP COLUMN rcv_service_id; - -DROP INDEX idx_server_certs_host_port; - -DROP INDEX idx_server_certs_user_id_host_port; - -DROP TABLE server_certs; - |] diff --git a/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/M20260115_service_certs.hs b/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/M20260115_service_certs.hs new file mode 100644 index 000000000..4572ba73a --- /dev/null +++ b/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/M20260115_service_certs.hs @@ -0,0 +1,93 @@ +{-# LANGUAGE QuasiQuotes #-} + +module Simplex.Messaging.Agent.Store.SQLite.Migrations.M20260115_service_certs where + +import Database.SQLite.Simple (Query) +import Database.SQLite.Simple.QQ (sql) + +m20260115_service_certs :: Query +m20260115_service_certs = + [sql| +CREATE TABLE client_services( + user_id INTEGER NOT NULL REFERENCES users ON DELETE CASCADE, + host TEXT NOT NULL, + port TEXT NOT NULL, + server_key_hash BLOB, + service_cert BLOB NOT NULL, + service_cert_hash BLOB NOT NULL, + service_priv_key BLOB NOT NULL, + service_id BLOB, + service_queue_count INTEGER NOT NULL DEFAULT 0, + service_queue_ids_hash BLOB NOT NULL DEFAULT x'00000000000000000000000000000000', + FOREIGN KEY(host, port) REFERENCES servers ON UPDATE CASCADE ON DELETE RESTRICT +) STRICT; + +CREATE UNIQUE INDEX idx_server_certs_user_id_host_port ON client_services(user_id, host, port, server_key_hash); +CREATE INDEX idx_server_certs_host_port ON client_services(host, port); + +ALTER TABLE rcv_queues ADD COLUMN rcv_service_assoc INTEGER NOT NULL DEFAULT 0; + +CREATE TRIGGER tr_rcv_queue_insert +AFTER INSERT ON rcv_queues +FOR EACH ROW +WHEN NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count + 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, NEW.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = NEW.conn_id) + AND host = NEW.host AND port = NEW.port; +END; + +CREATE TRIGGER tr_rcv_queue_delete +AFTER DELETE ON rcv_queues +FOR EACH ROW +WHEN OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count - 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, OLD.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = OLD.conn_id) + AND host = OLD.host AND port = OLD.port; +END; + +CREATE TRIGGER tr_rcv_queue_update_remove +AFTER UPDATE ON rcv_queues +FOR EACH ROW +WHEN OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 AND NOT (NEW.rcv_service_assoc != 0 AND NEW.deleted = 0) +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count - 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, OLD.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = OLD.conn_id) + AND host = OLD.host AND port = OLD.port; +END; + +CREATE TRIGGER tr_rcv_queue_update_add +AFTER UPDATE ON rcv_queues +FOR EACH ROW +WHEN NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 AND NOT (OLD.rcv_service_assoc != 0 AND OLD.deleted = 0) +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count + 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, NEW.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = NEW.conn_id) + AND host = NEW.host AND port = NEW.port; +END; + |] + +down_m20260115_service_certs :: Query +down_m20260115_service_certs = + [sql| +DROP TRIGGER tr_rcv_queue_insert; +DROP TRIGGER tr_rcv_queue_delete; +DROP TRIGGER tr_rcv_queue_update_remove; +DROP TRIGGER tr_rcv_queue_update_add; + +ALTER TABLE rcv_queues DROP COLUMN rcv_service_assoc; + +DROP INDEX idx_server_certs_host_port; +DROP INDEX idx_server_certs_user_id_host_port; + +DROP TABLE client_services; + |] diff --git a/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/agent_schema.sql b/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/agent_schema.sql index b98d3dbf4..2a6136d6b 100644 --- a/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/agent_schema.sql +++ b/src/Simplex/Messaging/Agent/Store/SQLite/Migrations/agent_schema.sql @@ -63,6 +63,7 @@ CREATE TABLE rcv_queues( to_subscribe INTEGER NOT NULL DEFAULT 0, client_notice_id INTEGER REFERENCES client_notices ON UPDATE RESTRICT ON DELETE SET NULL, + rcv_service_assoc INTEGER NOT NULL DEFAULT 0, PRIMARY KEY(host, port, rcv_id), FOREIGN KEY(host, port) REFERENCES servers ON DELETE RESTRICT ON UPDATE CASCADE, @@ -450,6 +451,19 @@ CREATE TABLE client_notices( created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL ) STRICT; +CREATE TABLE client_services( + user_id INTEGER NOT NULL REFERENCES users ON DELETE CASCADE, + host TEXT NOT NULL, + port TEXT NOT NULL, + server_key_hash BLOB, + service_cert BLOB NOT NULL, + service_cert_hash BLOB NOT NULL, + service_priv_key BLOB NOT NULL, + service_id BLOB, + service_queue_count INTEGER NOT NULL DEFAULT 0, + service_queue_ids_hash BLOB NOT NULL DEFAULT x'00000000000000000000000000000000', + FOREIGN KEY(host, port) REFERENCES servers ON UPDATE CASCADE ON DELETE RESTRICT +) STRICT; CREATE UNIQUE INDEX idx_rcv_queues_ntf ON rcv_queues(host, port, ntf_id); CREATE UNIQUE INDEX idx_rcv_queue_id ON rcv_queues(conn_id, rcv_queue_id); CREATE UNIQUE INDEX idx_snd_queue_id ON snd_queues(conn_id, snd_queue_id); @@ -593,3 +607,54 @@ CREATE UNIQUE INDEX idx_client_notices_entity ON client_notices( entity_id ); CREATE INDEX idx_rcv_queues_client_notice_id ON rcv_queues(client_notice_id); +CREATE UNIQUE INDEX idx_server_certs_user_id_host_port ON client_services( + user_id, + host, + port, + server_key_hash +); +CREATE INDEX idx_server_certs_host_port ON client_services(host, port); +CREATE TRIGGER tr_rcv_queue_insert +AFTER INSERT ON rcv_queues +FOR EACH ROW +WHEN NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count + 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, NEW.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = NEW.conn_id) + AND host = NEW.host AND port = NEW.port; +END; +CREATE TRIGGER tr_rcv_queue_delete +AFTER DELETE ON rcv_queues +FOR EACH ROW +WHEN OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count - 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, OLD.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = OLD.conn_id) + AND host = OLD.host AND port = OLD.port; +END; +CREATE TRIGGER tr_rcv_queue_update_remove +AFTER UPDATE ON rcv_queues +FOR EACH ROW +WHEN OLD.rcv_service_assoc != 0 AND OLD.deleted = 0 AND NOT (NEW.rcv_service_assoc != 0 AND NEW.deleted = 0) +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count - 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, OLD.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = OLD.conn_id) + AND host = OLD.host AND port = OLD.port; +END; +CREATE TRIGGER tr_rcv_queue_update_add +AFTER UPDATE ON rcv_queues +FOR EACH ROW +WHEN NEW.rcv_service_assoc != 0 AND NEW.deleted = 0 AND NOT (OLD.rcv_service_assoc != 0 AND OLD.deleted = 0) +BEGIN + UPDATE client_services + SET service_queue_count = service_queue_count + 1, + service_queue_ids_hash = simplex_xor_md5_combine(service_queue_ids_hash, NEW.rcv_id) + WHERE user_id = (SELECT user_id FROM connections WHERE conn_id = NEW.conn_id) + AND host = NEW.host AND port = NEW.port; +END; diff --git a/src/Simplex/Messaging/Agent/Store/SQLite/Util.hs b/src/Simplex/Messaging/Agent/Store/SQLite/Util.hs index 22c5dec9f..49eafffd2 100644 --- a/src/Simplex/Messaging/Agent/Store/SQLite/Util.hs +++ b/src/Simplex/Messaging/Agent/Store/SQLite/Util.hs @@ -6,6 +6,7 @@ module Simplex.Messaging.Agent.Store.SQLite.Util mkSQLiteAggFinal, createStaticFunction, createStaticAggregate, + mkSQLiteFunc, ) where import Control.Exception (SomeException, catch, mask_) diff --git a/src/Simplex/Messaging/Agent/TSessionSubs.hs b/src/Simplex/Messaging/Agent/TSessionSubs.hs index cce103fe6..a1db48c9e 100644 --- a/src/Simplex/Messaging/Agent/TSessionSubs.hs +++ b/src/Simplex/Messaging/Agent/TSessionSubs.hs @@ -2,6 +2,7 @@ {-# LANGUAGE LambdaCase #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TupleSections #-} module Simplex.Messaging.Agent.TSessionSubs ( TSessionSubs (sessionSubs), @@ -12,15 +13,20 @@ module Simplex.Messaging.Agent.TSessionSubs hasPendingSub, addPendingSub, setSessionId, + setPendingServiceSub, + setActiveServiceSub, addActiveSub, + addActiveSub', batchAddActiveSubs, batchAddPendingSubs, deletePendingSub, batchDeletePendingSubs, deleteSub, batchDeleteSubs, + deleteServiceSub, hasPendingSubs, getPendingSubs, + getPendingQueueSubs, getActiveSubs, setSubsPending, updateClientNotices, @@ -35,16 +41,16 @@ import Data.Int (Int64) import Data.List (foldl') import Data.Map.Strict (Map) import qualified Data.Map.Strict as M -import Data.Maybe (isJust) +import Data.Maybe (fromMaybe, isJust) import qualified Data.Set as S import Simplex.Messaging.Agent.Protocol (SMPQueue (..)) -import Simplex.Messaging.Agent.Store (RcvQueueSub (..), SomeRcvQueue) +import Simplex.Messaging.Agent.Store (RcvQueue, RcvQueueSub (..), ServiceAssoc, SomeRcvQueue, StoredRcvQueue (rcvServiceAssoc), rcvQueueSub) import Simplex.Messaging.Client (SMPTransportSession, TransportSessionMode (..)) -import Simplex.Messaging.Protocol (RecipientId) +import Simplex.Messaging.Protocol (IdsHash, RecipientId, ServiceSub (..), queueIdHash) import Simplex.Messaging.TMap (TMap) import qualified Simplex.Messaging.TMap as TM import Simplex.Messaging.Transport -import Simplex.Messaging.Util (($>>=)) +import Simplex.Messaging.Util (anyM, ($>>=)) data TSessionSubs = TSessionSubs { sessionSubs :: TMap SMPTransportSession SessSubs @@ -53,7 +59,9 @@ data TSessionSubs = TSessionSubs data SessSubs = SessSubs { subsSessId :: TVar (Maybe SessionId), activeSubs :: TMap RecipientId RcvQueueSub, - pendingSubs :: TMap RecipientId RcvQueueSub + pendingSubs :: TMap RecipientId RcvQueueSub, + activeServiceSub :: TVar (Maybe ServiceSub), + pendingServiceSub :: TVar (Maybe ServiceSub) } emptyIO :: IO TSessionSubs @@ -72,7 +80,7 @@ getSessSubs :: SMPTransportSession -> TSessionSubs -> STM SessSubs getSessSubs tSess ss = lookupSubs tSess ss >>= maybe new pure where new = do - s <- SessSubs <$> newTVar Nothing <*> newTVar M.empty <*> newTVar M.empty + s <- SessSubs <$> newTVar Nothing <*> newTVar M.empty <*> newTVar M.empty <*> newTVar Nothing <*> newTVar Nothing TM.insert tSess s $ sessionSubs ss pure s @@ -98,27 +106,63 @@ setSessionId tSess sessId ss = do Nothing -> writeTVar (subsSessId s) (Just sessId) Just sessId' -> unless (sessId == sessId') $ void $ setSubsPending_ s $ Just sessId -addActiveSub :: SMPTransportSession -> SessionId -> RcvQueueSub -> TSessionSubs -> STM () -addActiveSub tSess sessId rq ss = do +setPendingServiceSub :: SMPTransportSession -> ServiceSub -> TSessionSubs -> STM () +setPendingServiceSub tSess serviceSub ss = do + s <- getSessSubs tSess ss + writeTVar (pendingServiceSub s) $ Just serviceSub + +setActiveServiceSub :: SMPTransportSession -> SessionId -> ServiceSub -> TSessionSubs -> STM () +setActiveServiceSub tSess sessId serviceSub ss = do + s <- getSessSubs tSess ss + sessId' <- readTVar $ subsSessId s + if Just sessId == sessId' + then do + writeTVar (activeServiceSub s) $ Just serviceSub + writeTVar (pendingServiceSub s) Nothing + else writeTVar (pendingServiceSub s) $ Just serviceSub + +addActiveSub :: SMPTransportSession -> SessionId -> Maybe ServiceId -> RcvQueue -> TSessionSubs -> STM () +addActiveSub tSess sessId serviceId_ rq = addActiveSub' tSess sessId serviceId_ (rcvQueueSub rq) (rcvServiceAssoc rq) +{-# INLINE addActiveSub #-} + +addActiveSub' :: SMPTransportSession -> SessionId -> Maybe ServiceId -> RcvQueueSub -> ServiceAssoc -> TSessionSubs -> STM () +addActiveSub' tSess sessId serviceId_ rq serviceAssoc ss = do s <- getSessSubs tSess ss sessId' <- readTVar $ subsSessId s let rId = rcvId rq if Just sessId == sessId' then do - TM.insert rId rq $ activeSubs s TM.delete rId $ pendingSubs s + case serviceId_ of + Just serviceId | serviceAssoc -> updateActiveService s serviceId 1 (queueIdHash rId) + _ -> TM.insert rId rq $ activeSubs s else TM.insert rId rq $ pendingSubs s -batchAddActiveSubs :: SMPTransportSession -> SessionId -> [RcvQueueSub] -> TSessionSubs -> STM () -batchAddActiveSubs tSess sessId rqs ss = do +batchAddActiveSubs :: SMPTransportSession -> SessionId -> Maybe ServiceId -> ([RcvQueueSub], [RcvQueueSub]) -> TSessionSubs -> STM () +batchAddActiveSubs tSess sessId serviceId_ (rqs, serviceRQs) ss = do s <- getSessSubs tSess ss sessId' <- readTVar $ subsSessId s - let qs = M.fromList $ map (\rq -> (rcvId rq, rq)) rqs + let qs = queuesMap rqs + serviceQs = queuesMap serviceRQs if Just sessId == sessId' then do TM.union qs $ activeSubs s modifyTVar' (pendingSubs s) (`M.difference` qs) - else TM.union qs $ pendingSubs s + unless (null serviceRQs) $ forM_ serviceId_ $ \serviceId -> do + modifyTVar' (pendingSubs s) (`M.difference` serviceQs) + updateActiveService s serviceId (fromIntegral $ length serviceRQs) (mconcat $ map (queueIdHash . rcvId) serviceRQs) + else do + TM.union qs $ pendingSubs s + when (isJust serviceId_ && not (null serviceRQs)) $ TM.union serviceQs $ pendingSubs s + where + queuesMap = M.fromList . map (\rq -> (rcvId rq, rq)) + +updateActiveService :: SessSubs -> ServiceId -> Int64 -> IdsHash -> STM () +updateActiveService s serviceId addN addIdsHash = do + ServiceSub serviceId' n idsHash <- + fromMaybe (ServiceSub serviceId 0 mempty) <$> readTVar (activeServiceSub s) + when (serviceId == serviceId') $ + writeTVar (activeServiceSub s) $ Just $ ServiceSub serviceId (n + addN) (idsHash <> addIdsHash) batchAddPendingSubs :: SMPTransportSession -> [RcvQueueSub] -> TSessionSubs -> STM () batchAddPendingSubs tSess rqs ss = do @@ -142,12 +186,23 @@ batchDeleteSubs tSess rqs = lookupSubs tSess >=> mapM_ (\s -> delete (activeSubs rIds = S.fromList $ map queueId rqs delete = (`modifyTVar'` (`M.withoutKeys` rIds)) +deleteServiceSub :: SMPTransportSession -> TSessionSubs -> STM () +deleteServiceSub tSess = lookupSubs tSess >=> mapM_ (\s -> writeTVar (activeServiceSub s) Nothing >> writeTVar (pendingServiceSub s) Nothing) + hasPendingSubs :: SMPTransportSession -> TSessionSubs -> STM Bool -hasPendingSubs tSess = lookupSubs tSess >=> maybe (pure False) (fmap (not . null) . readTVar . pendingSubs) +hasPendingSubs tSess = lookupSubs tSess >=> maybe (pure False) (\s -> anyM [hasSubs s, hasServiceSub s]) + where + hasSubs = fmap (not . null) . readTVar . pendingSubs + hasServiceSub = fmap isJust . readTVar . pendingServiceSub -getPendingSubs :: SMPTransportSession -> TSessionSubs -> STM (Map RecipientId RcvQueueSub) -getPendingSubs = getSubs_ pendingSubs -{-# INLINE getPendingSubs #-} +getPendingSubs :: SMPTransportSession -> TSessionSubs -> STM (Map RecipientId RcvQueueSub, Maybe ServiceSub) +getPendingSubs tSess = lookupSubs tSess >=> maybe (pure (M.empty, Nothing)) get + where + get s = liftM2 (,) (readTVar $ pendingSubs s) (readTVar $ pendingServiceSub s) + +getPendingQueueSubs :: SMPTransportSession -> TSessionSubs -> STM (Map RecipientId RcvQueueSub) +getPendingQueueSubs = getSubs_ pendingSubs +{-# INLINE getPendingQueueSubs #-} getActiveSubs :: SMPTransportSession -> TSessionSubs -> STM (Map RecipientId RcvQueueSub) getActiveSubs = getSubs_ activeSubs @@ -156,7 +211,7 @@ getActiveSubs = getSubs_ activeSubs getSubs_ :: (SessSubs -> TMap RecipientId RcvQueueSub) -> SMPTransportSession -> TSessionSubs -> STM (Map RecipientId RcvQueueSub) getSubs_ subs tSess = lookupSubs tSess >=> maybe (pure M.empty) (readTVar . subs) -setSubsPending :: TransportSessionMode -> SMPTransportSession -> SessionId -> TSessionSubs -> STM (Map RecipientId RcvQueueSub) +setSubsPending :: TransportSessionMode -> SMPTransportSession -> SessionId -> TSessionSubs -> STM (Map RecipientId RcvQueueSub, Maybe ServiceSub) setSubsPending mode tSess@(uId, srv, connId_) sessId tss@(TSessionSubs ss) | entitySession == isJust connId_ = TM.lookup tSess ss >>= withSessSubs (`setSubsPending_` Nothing) @@ -166,17 +221,17 @@ setSubsPending mode tSess@(uId, srv, connId_) sessId tss@(TSessionSubs ss) entitySession = mode == TSMEntity sessEntId = if entitySession then Just else const Nothing withSessSubs run = \case - Nothing -> pure M.empty + Nothing -> pure (M.empty, Nothing) Just s -> do sessId' <- readTVar $ subsSessId s - if Just sessId == sessId' then run s else pure M.empty + if Just sessId == sessId' then run s else pure (M.empty, Nothing) setPendingChangeMode s = do subs <- M.union <$> readTVar (activeSubs s) <*> readTVar (pendingSubs s) unless (null subs) $ forM_ subs $ \rq -> addPendingSub (uId, srv, sessEntId (connId rq)) rq tss - pure subs + (subs,) <$> setServiceSubPending_ s -setSubsPending_ :: SessSubs -> Maybe SessionId -> STM (Map RecipientId RcvQueueSub) +setSubsPending_ :: SessSubs -> Maybe SessionId -> STM (Map RecipientId RcvQueueSub, Maybe ServiceSub) setSubsPending_ s sessId_ = do writeTVar (subsSessId s) sessId_ let as = activeSubs s @@ -184,7 +239,15 @@ setSubsPending_ s sessId_ = do unless (null subs) $ do writeTVar as M.empty modifyTVar' (pendingSubs s) $ M.union subs - pure subs + (subs,) <$> setServiceSubPending_ s + +setServiceSubPending_ :: SessSubs -> STM (Maybe ServiceSub) +setServiceSubPending_ s = do + serviceSub_ <- readTVar $ activeServiceSub s + forM_ serviceSub_ $ \serviceSub -> do + writeTVar (activeServiceSub s) Nothing + writeTVar (pendingServiceSub s) $ Just serviceSub + pure serviceSub_ updateClientNotices :: SMPTransportSession -> [(RecipientId, Maybe Int64)] -> TSessionSubs -> STM () updateClientNotices tSess noticeIds ss = do diff --git a/src/Simplex/Messaging/Client.hs b/src/Simplex/Messaging/Client.hs index 3ffdda145..67b31de18 100644 --- a/src/Simplex/Messaging/Client.hs +++ b/src/Simplex/Messaging/Client.hs @@ -52,6 +52,7 @@ module Simplex.Messaging.Client subscribeSMPQueuesNtfs, subscribeService, smpClientService, + smpClientServiceId, secureSMPQueue, secureSndSMPQueue, proxySecureSndSMPQueue, @@ -106,6 +107,7 @@ module Simplex.Messaging.Client smpProxyError, smpErrorClientNotice, textToHostMode, + clientHandlers, ServerTransmissionBatch, ServerTransmission (..), ClientCommand, @@ -128,7 +130,8 @@ import Control.Applicative ((<|>)) import Control.Concurrent (ThreadId, forkFinally, forkIO, killThread, mkWeakThreadId) import Control.Concurrent.Async import Control.Concurrent.STM -import Control.Exception +import Control.Exception (Exception, Handler (..), IOException, SomeAsyncException, SomeException) +import qualified Control.Exception as E import Control.Logger.Simple import Control.Monad import Control.Monad.Except @@ -252,7 +255,7 @@ type ClientCommand msg = (EntityId, Maybe C.APrivateAuthKey, ProtoCommand msg) -- | Type synonym for transmission from SPM servers. -- Batch response is presented as a single `ServerTransmissionBatch` tuple. -type ServerTransmissionBatch v err msg = (TransportSession msg, Version v, SessionId, NonEmpty (EntityId, ServerTransmission err msg)) +type ServerTransmissionBatch v err msg = (TransportSession msg, THandleParams v 'TClient, NonEmpty (EntityId, ServerTransmission err msg)) data ServerTransmission err msg = STEvent (Either (ProtocolClientError err) msg) @@ -566,7 +569,7 @@ getProtocolClient g nm transportSession@(_, srv, _) cfg@ProtocolClientConfig {qS case chooseTransportHost networkConfig (host srv) of Right useHost -> (getCurrentTime >>= mkProtocolClient useHost >>= runClient useTransport useHost) - `catch` \(e :: IOException) -> pure . Left $ PCEIOError e + `E.catches` clientHandlers Left e -> pure $ Left e where NetworkConfig {tcpConnectTimeout, tcpTimeout, smpPingInterval} = networkConfig @@ -639,7 +642,7 @@ getProtocolClient g nm transportSession@(_, srv, _) cfg@ProtocolClientConfig {qS writeTVar (connected c) True putTMVar cVar $ Right c' raceAny_ ([send c' th, process c', receive c' th] <> [monitor c' | smpPingInterval > 0]) - `finally` disconnected c' + `E.finally` disconnected c' send :: Transport c => ProtocolClient v err msg -> THandle v c 'TClient -> IO () send ProtocolClient {client_ = PClient {sndQ}} h = forever $ atomically (readTBQueue sndQ) >>= sendPending @@ -718,6 +721,13 @@ getProtocolClient g nm transportSession@(_, srv, _) cfg@ProtocolClientConfig {qS Left e -> logError $ "SMP client error: " <> tshow e Right _ -> logWarn "SMP client unprocessed event" +clientHandlers :: [Handler (Either (ProtocolClientError e) a)] +clientHandlers = + [ Handler $ \(e :: IOException) -> pure $ Left $ PCEIOError $ E.displayException e, + Handler $ \(e :: SomeAsyncException) -> E.throwIO e, + Handler $ \(e :: SomeException) -> pure $ Left $ PCENetworkError $ toNetworkError e + ] + useWebPort :: NetworkConfig -> [HostName] -> ProtocolServer p -> Bool useWebPort cfg presetDomains ProtocolServer {host = h :| _} = case smpWebPortServers cfg of SWPAll -> True @@ -766,7 +776,7 @@ data ProtocolClientError err | -- | Error when cryptographically "signing" the command or when initializing crypto_box. PCECryptoError C.CryptoError | -- | IO Error - PCEIOError IOException + PCEIOError String deriving (Eq, Show, Exception) type SMPClientError = ProtocolClientError ErrorType @@ -779,10 +789,10 @@ temporaryClientError = \case _ -> False {-# INLINE temporaryClientError #-} +-- it is consistent with clientServiceError smpClientServiceError :: SMPClientError -> Bool smpClientServiceError = \case PCEServiceUnavailable -> True - PCETransportError (TEHandshake BAD_SERVICE) -> True -- TODO [certs] this error may be temporary, so we should possibly resubscribe. PCEProtocolError SERVICE -> True PCEProtocolError (PROXY (BROKER NO_SERVICE)) -> True -- for completeness, it cannot happen. _ -> False @@ -865,8 +875,7 @@ writeSMPMessage :: SMPClient -> RecipientId -> BrokerMsg -> IO () writeSMPMessage c rId msg = atomically $ mapM_ (`writeTBQueue` serverTransmission c [(rId, STEvent (Right msg))]) (msgQ $ client_ c) serverTransmission :: ProtocolClient v err msg -> NonEmpty (RecipientId, ServerTransmission err msg) -> ServerTransmissionBatch v err msg -serverTransmission ProtocolClient {thParams = THandleParams {thVersion, sessionId}, client_ = PClient {transportSession}} ts = - (transportSession, thVersion, sessionId, ts) +serverTransmission ProtocolClient {thParams, client_ = PClient {transportSession}} ts = (transportSession, thParams, ts) -- | Get message from SMP queue. The server returns ERR PROHIBITED if a client uses SUB and GET via the same transport connection for the same queue -- @@ -910,24 +919,28 @@ nsubResponse_ = \case {-# INLINE nsubResponse_ #-} -- This command is always sent in background request mode -subscribeService :: forall p. (PartyI p, ServiceParty p) => SMPClient -> SParty p -> ExceptT SMPClientError IO Int64 -subscribeService c party = case smpClientService c of +subscribeService :: forall p. (PartyI p, ServiceParty p) => SMPClient -> SParty p -> Int64 -> IdsHash -> ExceptT SMPClientError IO ServiceSub +subscribeService c party n idsHash = case smpClientService c of Just THClientService {serviceId, serviceKey} -> do liftIO $ enablePings c sendSMPCommand c NRMBackground (Just (C.APrivateAuthKey C.SEd25519 serviceKey)) serviceId subCmd >>= \case - SOKS n -> pure n + SOKS n' idsHash' -> pure $ ServiceSub serviceId n' idsHash' r -> throwE $ unexpectedResponse r where subCmd :: Command p subCmd = case party of - SRecipientService -> SUBS - SNotifierService -> NSUBS + SRecipientService -> SUBS n idsHash + SNotifierService -> NSUBS n idsHash Nothing -> throwE PCEServiceUnavailable smpClientService :: SMPClient -> Maybe THClientService smpClientService = thAuth . thParams >=> clientService {-# INLINE smpClientService #-} +smpClientServiceId :: SMPClient -> Maybe ServiceId +smpClientServiceId = fmap (\THClientService {serviceId} -> serviceId) . smpClientService +{-# INLINE smpClientServiceId #-} + enablePings :: SMPClient -> IO () enablePings ProtocolClient {client_ = PClient {sendPings}} = atomically $ writeTVar sendPings True {-# INLINE enablePings #-} diff --git a/src/Simplex/Messaging/Client/Agent.hs b/src/Simplex/Messaging/Client/Agent.hs index 604960360..d302ba237 100644 --- a/src/Simplex/Messaging/Client/Agent.hs +++ b/src/Simplex/Messaging/Client/Agent.hs @@ -15,6 +15,7 @@ module Simplex.Messaging.Client.Agent ( SMPClientAgent (..), SMPClientAgentConfig (..), SMPClientAgentEvent (..), + DBService (..), OwnServer, defaultSMPClientAgentConfig, newSMPClientAgent, @@ -36,6 +37,7 @@ where import Control.Concurrent (forkIO) import Control.Concurrent.Async (Async, uninterruptibleCancel) import Control.Concurrent.STM (retry) +import qualified Control.Exception as E import Control.Logger.Simple import Control.Monad import Control.Monad.Except @@ -45,7 +47,6 @@ import Crypto.Random (ChaChaDRG) import Data.ByteString.Char8 (ByteString) import qualified Data.ByteString.Char8 as B import Data.Constraint (Dict (..)) -import Data.Int (Int64) import Data.List.NonEmpty (NonEmpty) import qualified Data.List.NonEmpty as L import Data.Map.Strict (Map) @@ -69,10 +70,12 @@ import Simplex.Messaging.Protocol ProtocolServer (..), QueueId, SMPServer, + ServiceSub (..), SParty (..), ServiceParty, serviceParty, - partyServiceRole + partyServiceRole, + queueIdsHash, ) import Simplex.Messaging.Session import Simplex.Messaging.TMap (TMap) @@ -81,7 +84,6 @@ import Simplex.Messaging.Transport import Simplex.Messaging.Util (catchAll_, ifM, safeDecodeUtf8, toChunks, tshow, whenM, ($>>=), (<$$>)) import System.Timeout (timeout) import UnliftIO (async) -import qualified UnliftIO.Exception as E import UnliftIO.STM type SMPClientVar = SessionVar (Either (SMPClientError, Maybe UTCTime) (OwnServer, SMPClient)) @@ -91,14 +93,14 @@ data SMPClientAgentEvent | CADisconnected SMPServer (NonEmpty QueueId) | CASubscribed SMPServer (Maybe ServiceId) (NonEmpty QueueId) | CASubError SMPServer (NonEmpty (QueueId, SMPClientError)) - | CAServiceDisconnected SMPServer (ServiceId, Int64) - | CAServiceSubscribed SMPServer (ServiceId, Int64) Int64 - | CAServiceSubError SMPServer (ServiceId, Int64) SMPClientError + | CAServiceDisconnected SMPServer ServiceSub + | CAServiceSubscribed {subServer :: SMPServer, expected :: ServiceSub, subscribed :: ServiceSub} + | CAServiceSubError SMPServer ServiceSub SMPClientError -- CAServiceUnavailable is used when service ID in pending subscription is different from the current service in connection. -- This will require resubscribing to all queues associated with this service ID individually, creating new associations. -- It may happen if, for example, SMP server deletes service information (e.g. via downgrade and upgrade) -- and assigns different service ID to the service certificate. - | CAServiceUnavailable SMPServer (ServiceId, Int64) + | CAServiceUnavailable SMPServer ServiceSub data SMPClientAgentConfig = SMPClientAgentConfig { smpCfg :: ProtocolClientConfig SMPVersion, @@ -132,6 +134,7 @@ defaultSMPClientAgentConfig = data SMPClientAgent p = SMPClientAgent { agentCfg :: SMPClientAgentConfig, agentParty :: SParty p, + dbService :: Maybe DBService, active :: TVar Bool, startedAt :: UTCTime, msgQ :: TBQueue (ServerTransmissionBatch SMPVersion ErrorType BrokerMsg), @@ -142,11 +145,11 @@ data SMPClientAgent p = SMPClientAgent -- Only one service subscription can exist per server with this agent. -- With correctly functioning SMP server, queue and service subscriptions can't be -- active at the same time. - activeServiceSubs :: TMap SMPServer (TVar (Maybe ((ServiceId, Int64), SessionId))), + activeServiceSubs :: TMap SMPServer (TVar (Maybe (ServiceSub, SessionId))), activeQueueSubs :: TMap SMPServer (TMap QueueId (SessionId, C.APrivateAuthKey)), -- Pending service subscriptions can co-exist with pending queue subscriptions -- on the same SMP server during subscriptions being transitioned from per-queue to service. - pendingServiceSubs :: TMap SMPServer (TVar (Maybe (ServiceId, Int64))), + pendingServiceSubs :: TMap SMPServer (TVar (Maybe ServiceSub)), pendingQueueSubs :: TMap SMPServer (TMap QueueId C.APrivateAuthKey), smpSubWorkers :: TMap SMPServer (SessionVar (Async ())), workerSeq :: TVar Int @@ -154,8 +157,8 @@ data SMPClientAgent p = SMPClientAgent type OwnServer = Bool -newSMPClientAgent :: SParty p -> SMPClientAgentConfig -> TVar ChaChaDRG -> IO (SMPClientAgent p) -newSMPClientAgent agentParty agentCfg@SMPClientAgentConfig {msgQSize, agentQSize} randomDrg = do +newSMPClientAgent :: SParty p -> SMPClientAgentConfig -> Maybe DBService -> TVar ChaChaDRG -> IO (SMPClientAgent p) +newSMPClientAgent agentParty agentCfg@SMPClientAgentConfig {msgQSize, agentQSize} dbService randomDrg = do active <- newTVarIO True startedAt <- getCurrentTime msgQ <- newTBQueueIO msgQSize @@ -172,6 +175,7 @@ newSMPClientAgent agentParty agentCfg@SMPClientAgentConfig {msgQSize, agentQSize SMPClientAgent { agentCfg, agentParty, + dbService, active, startedAt, msgQ, @@ -187,6 +191,11 @@ newSMPClientAgent agentParty agentCfg@SMPClientAgentConfig {msgQSize, agentQSize workerSeq } +data DBService = DBService + { getCredentials :: SMPServer -> IO (Either SMPClientError ServiceCredentials), + updateServiceId :: SMPServer -> Maybe ServiceId -> IO (Either SMPClientError ()) + } + -- | Get or create SMP client for SMPServer getSMPServerClient' :: SMPClientAgent p -> SMPServer -> ExceptT SMPClientError IO SMPClient getSMPServerClient' ca srv = snd <$> getSMPServerClient'' ca srv @@ -217,7 +226,7 @@ getSMPServerClient'' ca@SMPClientAgent {agentCfg, smpClients, smpSessions, worke newSMPClient :: SMPClientVar -> IO (Either SMPClientError (OwnServer, SMPClient)) newSMPClient v = do - r <- connectClient ca srv v `E.catch` (pure . Left . PCEIOError) + r <- connectClient ca srv v `E.catches` clientHandlers case r of Right smp -> do logInfo . decodeUtf8 $ "Agent connected to " <> showServer srv @@ -226,8 +235,7 @@ getSMPServerClient'' ca@SMPClientAgent {agentCfg, smpClients, smpSessions, worke atomically $ do putTMVar (sessionVar v) (Right c) TM.insert (sessionId $ thParams smp) c smpSessions - let serviceId_ = (\THClientService {serviceId} -> serviceId) <$> smpClientService smp - notify ca $ CAConnected srv serviceId_ + notify ca $ CAConnected srv $ smpClientServiceId smp pure $ Right c Left e -> do let ei = persistErrorInterval agentCfg @@ -248,15 +256,24 @@ isOwnServer SMPClientAgent {agentCfg} ProtocolServer {host} = -- | Run an SMP client for SMPClientVar connectClient :: SMPClientAgent p -> SMPServer -> SMPClientVar -> IO (Either SMPClientError SMPClient) -connectClient ca@SMPClientAgent {agentCfg, smpClients, smpSessions, msgQ, randomDrg, startedAt} srv v = - getProtocolClient randomDrg NRMBackground (1, srv, Nothing) (smpCfg agentCfg) [] (Just msgQ) startedAt clientDisconnected +connectClient ca@SMPClientAgent {agentCfg, dbService, smpClients, smpSessions, msgQ, randomDrg, startedAt} srv v = case dbService of + Just dbs -> runExceptT $ do + creds <- ExceptT $ getCredentials dbs srv + smp <- ExceptT $ getClient cfg {serviceCredentials = Just creds} + whenM (atomically $ activeClientSession ca smp srv) $ + ExceptT $ updateServiceId dbs srv $ smpClientServiceId smp + pure smp + Nothing -> getClient cfg where + cfg = smpCfg agentCfg + getClient cfg' = getProtocolClient randomDrg NRMBackground (1, srv, Nothing) cfg' [] (Just msgQ) startedAt clientDisconnected + clientDisconnected :: SMPClient -> IO () clientDisconnected smp = do removeClientAndSubs smp >>= serverDown logInfo . decodeUtf8 $ "Agent disconnected from " <> showServer srv - removeClientAndSubs :: SMPClient -> IO (Maybe (ServiceId, Int64), Maybe (Map QueueId C.APrivateAuthKey)) + removeClientAndSubs :: SMPClient -> IO (Maybe ServiceSub, Maybe (Map QueueId C.APrivateAuthKey)) removeClientAndSubs smp = do -- Looking up subscription vars outside of STM transaction to reduce re-evaluation. -- It is possible because these vars are never removed, they are only added. @@ -287,7 +304,7 @@ connectClient ca@SMPClientAgent {agentCfg, smpClients, smpSessions, msgQ, random then pure Nothing else Just subs <$ addSubs_ (pendingQueueSubs ca) srv subs - serverDown :: (Maybe (ServiceId, Int64), Maybe (Map QueueId C.APrivateAuthKey)) -> IO () + serverDown :: (Maybe ServiceSub, Maybe (Map QueueId C.APrivateAuthKey)) -> IO () serverDown (sSub, qSubs) = do mapM_ (notify ca . CAServiceDisconnected srv) sSub let qIds = L.nonEmpty . M.keys =<< qSubs @@ -307,7 +324,7 @@ reconnectClient ca@SMPClientAgent {active, agentCfg, smpSubWorkers, workerSeq} s (Just <$> getSessVar workerSeq srv smpSubWorkers ts) newSubWorker :: SessionVar (Async ()) -> IO () newSubWorker v = do - a <- async $ void (E.tryAny runSubWorker) >> atomically (cleanup v) + a <- async $ void (E.try @E.SomeException runSubWorker) >> atomically (cleanup v) atomically $ putTMVar (sessionVar v) a runSubWorker = withRetryInterval (reconnectInterval agentCfg) $ \_ loop -> do @@ -317,7 +334,7 @@ reconnectClient ca@SMPClientAgent {active, agentCfg, smpSubWorkers, workerSeq} s loop ProtocolClientConfig {networkConfig = NetworkConfig {tcpConnectTimeout}} = smpCfg agentCfg noPending (sSub, qSubs) = isNothing sSub && maybe True M.null qSubs - getPending :: Monad m => (forall a. SMPServer -> TMap SMPServer a -> m (Maybe a)) -> (forall a. TVar a -> m a) -> m (Maybe (ServiceId, Int64), Maybe (Map QueueId C.APrivateAuthKey)) + getPending :: Monad m => (forall a. SMPServer -> TMap SMPServer a -> m (Maybe a)) -> (forall a. TVar a -> m a) -> m (Maybe ServiceSub, Maybe (Map QueueId C.APrivateAuthKey)) getPending lkup rd = do sSub <- lkup srv (pendingServiceSubs ca) $>>= rd qSubs <- lkup srv (pendingQueueSubs ca) >>= mapM rd @@ -329,7 +346,7 @@ reconnectClient ca@SMPClientAgent {active, agentCfg, smpSubWorkers, workerSeq} s whenM (isEmptyTMVar $ sessionVar v) retry removeSessVar v srv smpSubWorkers -reconnectSMPClient :: forall p. SMPClientAgent p -> SMPServer -> (Maybe (ServiceId, Int64), Maybe (Map QueueId C.APrivateAuthKey)) -> ExceptT SMPClientError IO () +reconnectSMPClient :: forall p. SMPClientAgent p -> SMPServer -> (Maybe ServiceSub, Maybe (Map QueueId C.APrivateAuthKey)) -> ExceptT SMPClientError IO () reconnectSMPClient ca@SMPClientAgent {agentCfg, agentParty} srv (sSub_, qSubs_) = withSMP ca srv $ \smp -> liftIO $ case serviceParty agentParty of Just Dict -> resubscribe smp @@ -430,11 +447,11 @@ smpSubscribeQueues ca smp srv subs = do let acc@(_, _, (qOks, sQs), notPending) = foldr (groupSub pending) (False, [], ([], []), []) (L.zip subs rs) unless (null qOks) $ addActiveSubs ca srv qOks unless (null sQs) $ forM_ smpServiceId $ \serviceId -> - updateActiveServiceSub ca srv ((serviceId, fromIntegral $ length sQs), sessId) + updateActiveServiceSub ca srv (ServiceSub serviceId (fromIntegral $ length sQs) (queueIdsHash sQs), sessId) unless (null notPending) $ removePendingSubs ca srv notPending pure acc sessId = sessionId $ thParams smp - smpServiceId = (\THClientService {serviceId} -> serviceId) <$> smpClientService smp + smpServiceId = smpClientServiceId smp groupSub :: Map QueueId C.APrivateAuthKey -> ((QueueId, C.APrivateAuthKey), Either SMPClientError (Maybe ServiceId)) -> @@ -454,24 +471,24 @@ smpSubscribeQueues ca smp srv subs = do notify_ :: (SMPServer -> NonEmpty a -> SMPClientAgentEvent) -> [a] -> IO () notify_ evt qs = mapM_ (notify ca . evt srv) $ L.nonEmpty qs -subscribeServiceNtfs :: SMPClientAgent 'NotifierService -> SMPServer -> (ServiceId, Int64) -> IO () +subscribeServiceNtfs :: SMPClientAgent 'NotifierService -> SMPServer -> ServiceSub -> IO () subscribeServiceNtfs = subscribeService_ {-# INLINE subscribeServiceNtfs #-} -subscribeService_ :: (PartyI p, ServiceParty p) => SMPClientAgent p -> SMPServer -> (ServiceId, Int64) -> IO () +subscribeService_ :: (PartyI p, ServiceParty p) => SMPClientAgent p -> SMPServer -> ServiceSub -> IO () subscribeService_ ca srv serviceSub = do atomically $ setPendingServiceSub ca srv $ Just serviceSub runExceptT (getSMPServerClient' ca srv) >>= \case Right smp -> smpSubscribeService ca smp srv serviceSub Left _ -> pure () -- no call to reconnectClient - failing getSMPServerClient' does that -smpSubscribeService :: (PartyI p, ServiceParty p) => SMPClientAgent p -> SMPClient -> SMPServer -> (ServiceId, Int64) -> IO () -smpSubscribeService ca smp srv serviceSub@(serviceId, _) = case smpClientService smp of +smpSubscribeService :: (PartyI p, ServiceParty p) => SMPClientAgent p -> SMPClient -> SMPServer -> ServiceSub -> IO () +smpSubscribeService ca smp srv serviceSub@(ServiceSub serviceId n idsHash) = case smpClientService smp of Just service | serviceAvailable service -> subscribe _ -> notifyUnavailable where subscribe = do - r <- runExceptT $ subscribeService smp $ agentParty ca + r <- runExceptT $ subscribeService smp (agentParty ca) n idsHash ok <- atomically $ ifM @@ -480,14 +497,14 @@ smpSubscribeService ca smp srv serviceSub@(serviceId, _) = case smpClientService (pure False) if ok then case r of - Right n -> notify ca $ CAServiceSubscribed srv serviceSub n + Right serviceSub' -> notify ca $ CAServiceSubscribed srv serviceSub serviceSub' Left e | smpClientServiceError e -> notifyUnavailable | temporaryClientError e -> reconnectClient ca srv | otherwise -> notify ca $ CAServiceSubError srv serviceSub e else reconnectClient ca srv - processSubscription = mapM_ $ \n -> do - setActiveServiceSub ca srv $ Just ((serviceId, n), sessId) + processSubscription = mapM_ $ \serviceSub' -> do -- TODO [certs rcv] validate hash here? + setActiveServiceSub ca srv $ Just (serviceSub', sessId) setPendingServiceSub ca srv Nothing serviceAvailable THClientService {serviceRole, serviceId = serviceId'} = serviceId == serviceId' && partyServiceRole (agentParty ca) == serviceRole @@ -529,11 +546,11 @@ addSubs_ subs srv ss = Just m -> TM.union ss m _ -> TM.insertM srv (newTVar ss) subs -setActiveServiceSub :: SMPClientAgent p -> SMPServer -> Maybe ((ServiceId, Int64), SessionId) -> STM () +setActiveServiceSub :: SMPClientAgent p -> SMPServer -> Maybe (ServiceSub, SessionId) -> STM () setActiveServiceSub = setServiceSub_ activeServiceSubs {-# INLINE setActiveServiceSub #-} -setPendingServiceSub :: SMPClientAgent p -> SMPServer -> Maybe (ServiceId, Int64) -> STM () +setPendingServiceSub :: SMPClientAgent p -> SMPServer -> Maybe ServiceSub -> STM () setPendingServiceSub = setServiceSub_ pendingServiceSubs {-# INLINE setPendingServiceSub #-} @@ -548,12 +565,12 @@ setServiceSub_ subsSel ca srv sub = Just v -> writeTVar v sub Nothing -> TM.insertM srv (newTVar sub) (subsSel ca) -updateActiveServiceSub :: SMPClientAgent p -> SMPServer -> ((ServiceId, Int64), SessionId) -> STM () -updateActiveServiceSub ca srv sub@((serviceId', n'), sessId') = +updateActiveServiceSub :: SMPClientAgent p -> SMPServer -> (ServiceSub, SessionId) -> STM () +updateActiveServiceSub ca srv sub@(ServiceSub serviceId' n' idsHash', sessId') = TM.lookup srv (activeServiceSubs ca) >>= \case Just v -> modifyTVar' v $ \case - Just ((serviceId, n), sessId) | serviceId == serviceId' && sessId == sessId' -> - Just ((serviceId, n + n'), sessId) + Just (ServiceSub serviceId n idsHash, sessId) | serviceId == serviceId' && sessId == sessId' -> + Just (ServiceSub serviceId (n + n') (idsHash <> idsHash'), sessId) _ -> Just sub Nothing -> TM.insertM srv (newTVar $ Just sub) (activeServiceSubs ca) diff --git a/src/Simplex/Messaging/Crypto.hs b/src/Simplex/Messaging/Crypto.hs index 9cc78acb3..c7b539641 100644 --- a/src/Simplex/Messaging/Crypto.hs +++ b/src/Simplex/Messaging/Crypto.hs @@ -87,6 +87,8 @@ module Simplex.Messaging.Crypto signatureKeyPair, publicToX509, encodeASNObj, + decodeASNKey, + asnKeyError, -- * key encoding/decoding encodePubKey, @@ -176,6 +178,7 @@ module Simplex.Messaging.Crypto sha512Hash, sha3_256, sha3_384, + md5Hash, -- * Message padding / un-padding canPad, @@ -214,7 +217,7 @@ import Crypto.Cipher.AES (AES256) import qualified Crypto.Cipher.Types as AES import qualified Crypto.Cipher.XSalsa as XSalsa import qualified Crypto.Error as CE -import Crypto.Hash (Digest, SHA3_256, SHA3_384, SHA256 (..), SHA512 (..), hash, hashDigestSize) +import Crypto.Hash (Digest, MD5, SHA3_256, SHA3_384, SHA256 (..), SHA512 (..), hash, hashDigestSize) import qualified Crypto.KDF.HKDF as H import qualified Crypto.MAC.Poly1305 as Poly1305 import qualified Crypto.PubKey.Curve25519 as X25519 @@ -1022,6 +1025,9 @@ sha3_384 :: ByteString -> ByteString sha3_384 = BA.convert . (hash :: ByteString -> Digest SHA3_384) {-# INLINE sha3_384 #-} +md5Hash :: ByteString -> ByteString +md5Hash = BA.convert . (hash :: ByteString -> Digest MD5) + -- | AEAD-GCM encryption with associated data. -- -- Used as part of double ratchet encryption. @@ -1493,11 +1499,11 @@ encodeASNObj k = toStrict . encodeASN1 DER $ toASN1 k [] -- Decoding of binary X509 'CryptoPublicKey'. decodePubKey :: CryptoPublicKey k => ByteString -> Either String k -decodePubKey = decodeKey >=> x509ToPublic >=> pubKey +decodePubKey = decodeASNKey >=> x509ToPublic >=> pubKey -- Decoding of binary PKCS8 'PrivateKey'. decodePrivKey :: CryptoPrivateKey k => ByteString -> Either String k -decodePrivKey = decodeKey >=> x509ToPrivate >=> privKey +decodePrivKey = decodeASNKey >=> x509ToPrivate >=> privKey x509ToPublic :: (X.PubKey, [ASN1]) -> Either String APublicKey x509ToPublic = \case @@ -1505,7 +1511,7 @@ x509ToPublic = \case (X.PubKeyEd448 k, []) -> Right . APublicKey SEd448 $ PublicKeyEd448 k (X.PubKeyX25519 k, []) -> Right . APublicKey SX25519 $ PublicKeyX25519 k (X.PubKeyX448 k, []) -> Right . APublicKey SX448 $ PublicKeyX448 k - r -> keyError r + r -> asnKeyError r x509ToPublic' :: CryptoPublicKey k => X.PubKey -> Either String k x509ToPublic' k = x509ToPublic (k, []) >>= pubKey @@ -1517,16 +1523,16 @@ x509ToPrivate = \case (X.PrivKeyEd448 k, []) -> Right $ APrivateKey SEd448 $ PrivateKeyEd448 k (X.PrivKeyX25519 k, []) -> Right $ APrivateKey SX25519 $ PrivateKeyX25519 k (X.PrivKeyX448 k, []) -> Right $ APrivateKey SX448 $ PrivateKeyX448 k - r -> keyError r + r -> asnKeyError r x509ToPrivate' :: CryptoPrivateKey k => X.PrivKey -> Either String k x509ToPrivate' pk = x509ToPrivate (pk, []) >>= privKey {-# INLINE x509ToPrivate' #-} -decodeKey :: ASN1Object a => ByteString -> Either String (a, [ASN1]) -decodeKey = fromASN1 <=< first show . decodeASN1 DER . fromStrict +decodeASNKey :: ASN1Object a => ByteString -> Either String (a, [ASN1]) +decodeASNKey = fromASN1 <=< first show . decodeASN1 DER . fromStrict -keyError :: (a, [ASN1]) -> Either String b -keyError = \case +asnKeyError :: (a, [ASN1]) -> Either String b +asnKeyError = \case (_, []) -> Left "unknown key algorithm" _ -> Left "more than one key" diff --git a/src/Simplex/Messaging/Notifications/Protocol.hs b/src/Simplex/Messaging/Notifications/Protocol.hs index bf5f7fbb1..45376a780 100644 --- a/src/Simplex/Messaging/Notifications/Protocol.hs +++ b/src/Simplex/Messaging/Notifications/Protocol.hs @@ -34,7 +34,7 @@ module Simplex.Messaging.Notifications.Protocol NTInvalidReason (..), encodePNMessages, pnMessagesP, - ntfShouldSubscribe, + subscribeNtfStatuses, allowTokenVerification, allowNtfSubCommands, checkEntity, @@ -518,17 +518,9 @@ data NtfSubStatus NSErr ByteString deriving (Eq, Ord, Show) -ntfShouldSubscribe :: NtfSubStatus -> Bool -ntfShouldSubscribe = \case - NSNew -> True - NSPending -> True - NSActive -> True - NSInactive -> True - NSEnd -> False - NSDeleted -> False - NSAuth -> False - NSService -> True - NSErr _ -> False +-- if these statuses change, the queue ID hashes for services need to be updated in a new migration (see m20250830_queue_ids_hash) +subscribeNtfStatuses :: [NtfSubStatus] +subscribeNtfStatuses = [NSNew, NSPending, NSActive, NSInactive] instance Encoding NtfSubStatus where smpEncode = \case diff --git a/src/Simplex/Messaging/Notifications/Server.hs b/src/Simplex/Messaging/Notifications/Server.hs index a508b2a17..2938e1f64 100644 --- a/src/Simplex/Messaging/Notifications/Server.hs +++ b/src/Simplex/Messaging/Notifications/Server.hs @@ -67,7 +67,7 @@ import Simplex.Messaging.Notifications.Server.Store (NtfSTMStore, TokenNtfMessag import Simplex.Messaging.Notifications.Server.Store.Postgres import Simplex.Messaging.Notifications.Server.Store.Types import Simplex.Messaging.Notifications.Transport -import Simplex.Messaging.Protocol (EntityId (..), ErrorType (..), NotifierId, Party (..), ProtocolServer (host), SMPServer, ServiceId, SignedTransmission, Transmission, pattern NoEntity, pattern SMPServer, encodeTransmission, tGetServer, tPut) +import Simplex.Messaging.Protocol (EntityId (..), ErrorType (..), NotifierId, Party (..), ProtocolServer (host), SMPServer, ServiceSub (..), SignedTransmission, Transmission, pattern NoEntity, pattern SMPServer, encodeTransmission, tGetServer, tPut) import qualified Simplex.Messaging.Protocol as SMP import Simplex.Messaging.Server import Simplex.Messaging.Server.Control (CPClientRole (..)) @@ -262,9 +262,9 @@ ntfServer cfg@NtfServerConfig {transports, transportConfig = tCfg, startOptions} srvSubscribers <- getSMPWorkerMetrics a smpSubscribers srvClients <- getSMPWorkerMetrics a smpClients srvSubWorkers <- getSMPWorkerMetrics a smpSubWorkers - ntfActiveServiceSubs <- getSMPServiceSubMetrics a activeServiceSubs $ snd . fst + ntfActiveServiceSubs <- getSMPServiceSubMetrics a activeServiceSubs $ smpQueueCount . fst ntfActiveQueueSubs <- getSMPSubMetrics a activeQueueSubs - ntfPendingServiceSubs <- getSMPServiceSubMetrics a pendingServiceSubs snd + ntfPendingServiceSubs <- getSMPServiceSubMetrics a pendingServiceSubs smpQueueCount ntfPendingQueueSubs <- getSMPSubMetrics a pendingQueueSubs smpSessionCount <- M.size <$> readTVarIO smpSessions apnsPushQLength <- atomically $ lengthTBQueue pushQ @@ -457,13 +457,13 @@ resubscribe NtfSubscriber {smpAgent = ca} = do counts <- mapConcurrently (subscribeSrvSubs ca st batchSize) srvs logNote $ "Completed all SMP resubscriptions for " <> tshow (length srvs) <> " servers (" <> tshow (sum counts) <> " subscriptions)" -subscribeSrvSubs :: SMPClientAgent 'NotifierService -> NtfPostgresStore -> Int -> (SMPServer, Int64, Maybe (ServiceId, Int64)) -> IO Int +subscribeSrvSubs :: SMPClientAgent 'NotifierService -> NtfPostgresStore -> Int -> (SMPServer, Int64, Maybe ServiceSub) -> IO Int subscribeSrvSubs ca st batchSize (srv, srvId, service_) = do let srvStr = safeDecodeUtf8 (strEncode $ L.head $ host srv) logNote $ "Starting SMP resubscriptions for " <> srvStr - forM_ service_ $ \(serviceId, n) -> do - logNote $ "Subscribing service to " <> srvStr <> " with " <> tshow n <> " associated queues" - subscribeServiceNtfs ca srv (serviceId, n) + forM_ service_ $ \serviceSub -> do + logNote $ "Subscribing service to " <> srvStr <> " with " <> tshow (smpQueueCount serviceSub) <> " associated queues" + subscribeServiceNtfs ca srv serviceSub n <- subscribeLoop 0 Nothing logNote $ "Completed SMP resubscriptions for " <> srvStr <> " (" <> tshow n <> " subscriptions)" pure n @@ -529,7 +529,7 @@ ntfSubscriber NtfSubscriber {smpAgent = ca@SMPClientAgent {msgQ, agentQ}} = NtfPushServer {pushQ} <- asks pushServer stats <- asks serverStats liftIO $ forever $ do - ((_, srv@(SMPServer (h :| _) _ _), _), _thVersion, sessionId, ts) <- atomically $ readTBQueue msgQ + ((_, srv@(SMPServer (h :| _) _ _), _), THandleParams {sessionId}, ts) <- atomically $ readTBQueue msgQ forM ts $ \(ntfId, t) -> case t of STUnexpectedError e -> logError $ "SMP client unexpected error: " <> tshow e -- uncorrelated response, should not happen STResponse {} -> pure () -- it was already reported as timeout error @@ -578,12 +578,13 @@ ntfSubscriber NtfSubscriber {smpAgent = ca@SMPClientAgent {msgQ, agentQ}} = forM_ (L.nonEmpty $ mapMaybe (\(nId, err) -> (nId,) <$> queueSubErrorStatus err) $ L.toList errs) $ \subStatuses -> do updated <- batchUpdateSrvSubErrors st srv subStatuses logSubErrors srv subStatuses updated - -- TODO [certs] resubscribe queues with statuses NSErr and NSService + -- TODO [certs rcv] resubscribe queues with statuses NSErr and NSService CAServiceDisconnected srv serviceSub -> logNote $ "SMP server service disconnected " <> showService srv serviceSub - CAServiceSubscribed srv serviceSub@(_, expected) n - | expected == n -> logNote msg - | otherwise -> logWarn $ msg <> ", confirmed subs: " <> tshow n + CAServiceSubscribed srv serviceSub@(ServiceSub _ n idsHash) (ServiceSub _ n' idsHash') + | n /= n' -> logWarn $ msg <> ", confirmed subs: " <> tshow n' + | idsHash /= idsHash' -> logWarn $ msg <> ", different IDs hash" + | otherwise -> logNote msg where msg = "SMP server service subscribed " <> showService srv serviceSub CAServiceSubError srv serviceSub e -> @@ -592,13 +593,13 @@ ntfSubscriber NtfSubscriber {smpAgent = ca@SMPClientAgent {msgQ, agentQ}} = logError $ "SMP server service subscription error " <> showService srv serviceSub <> ": " <> tshow e CAServiceUnavailable srv serviceSub -> do logError $ "SMP server service unavailable: " <> showService srv serviceSub - removeServiceAssociation st srv >>= \case + removeServiceAndAssociations st srv >>= \case Right (srvId, updated) -> do logSubStatus srv "removed service association" updated updated void $ subscribeSrvSubs ca st batchSize (srv, srvId, Nothing) Left e -> logError $ "SMP server update and resubscription error " <> tshow e where - showService srv (serviceId, n) = showServer' srv <> ", service ID " <> decodeLatin1 (strEncode serviceId) <> ", " <> tshow n <> " subs" + showService srv (ServiceSub serviceId n _) = showServer' srv <> ", service ID " <> decodeLatin1 (strEncode serviceId) <> ", " <> tshow n <> " subs" logSubErrors :: SMPServer -> NonEmpty (SMP.NotifierId, NtfSubStatus) -> Int -> IO () logSubErrors srv subs updated = forM_ (L.group $ L.sort $ L.map snd subs) $ \ss -> do @@ -607,7 +608,7 @@ ntfSubscriber NtfSubscriber {smpAgent = ca@SMPClientAgent {msgQ, agentQ}} = queueSubErrorStatus :: SMPClientError -> Maybe NtfSubStatus queueSubErrorStatus = \case PCEProtocolError AUTH -> Just NSAuth - -- TODO [certs] we could allow making individual subscriptions within service session to handle SERVICE error. + -- TODO [certs rcv] we could allow making individual subscriptions within service session to handle SERVICE error. -- This would require full stack changes in SMP server, SMP client and SMP service agent. PCEProtocolError SERVICE -> Just NSService PCEProtocolError e -> updateErr "SMP error " e diff --git a/src/Simplex/Messaging/Notifications/Server/Env.hs b/src/Simplex/Messaging/Notifications/Server/Env.hs index 654428602..19e1a2044 100644 --- a/src/Simplex/Messaging/Notifications/Server/Env.hs +++ b/src/Simplex/Messaging/Notifications/Server/Env.hs @@ -4,6 +4,7 @@ {-# LANGUAGE LambdaCase #-} {-# LANGUAGE KindSignatures #-} {-# LANGUAGE NamedFieldPuns #-} +{-# LANGUAGE OverloadedLists #-} {-# LANGUAGE OverloadedStrings #-} module Simplex.Messaging.Notifications.Server.Env @@ -25,8 +26,8 @@ module Simplex.Messaging.Notifications.Server.Env ) where import Control.Concurrent (ThreadId) -import Control.Logger.Simple -import Control.Monad +import Control.Monad.Except +import Control.Monad.Trans.Except import Crypto.Random import Data.Int (Int64) import Data.List.NonEmpty (NonEmpty) @@ -37,28 +38,26 @@ import qualified Data.X509.Validation as XV import Network.Socket import qualified Network.TLS as TLS import Numeric.Natural -import Simplex.Messaging.Client (ProtocolClientConfig (..)) +import Simplex.Messaging.Client (ProtocolClientError (..), SMPClientError) import Simplex.Messaging.Client.Agent import qualified Simplex.Messaging.Crypto as C import Simplex.Messaging.Notifications.Protocol import Simplex.Messaging.Notifications.Server.Push.APNS import Simplex.Messaging.Notifications.Server.Stats -import Simplex.Messaging.Notifications.Server.Store (newNtfSTMStore) import Simplex.Messaging.Notifications.Server.Store.Postgres import Simplex.Messaging.Notifications.Server.Store.Types -import Simplex.Messaging.Notifications.Server.StoreLog (readWriteNtfSTMStore) import Simplex.Messaging.Notifications.Transport (NTFVersion, VersionRangeNTF) -import Simplex.Messaging.Protocol (BasicAuth, CorrId, Party (..), SMPServer, SParty (..), Transmission) +import Simplex.Messaging.Protocol (BasicAuth, CorrId, Party (..), SMPServer, SParty (..), ServiceId, Transmission) import Simplex.Messaging.Server.Env.STM (StartOptions (..)) import Simplex.Messaging.Server.Expiration import Simplex.Messaging.Server.QueueStore.Postgres.Config (PostgresStoreCfg (..)) -import Simplex.Messaging.Server.StoreLog (closeStoreLog) import Simplex.Messaging.Session import Simplex.Messaging.TMap (TMap) import qualified Simplex.Messaging.TMap as TM import Simplex.Messaging.Transport (ASrvTransport, SMPServiceRole (..), ServiceCredentials (..), THandleParams, TransportPeer (..)) +import Simplex.Messaging.Transport.Credentials (genCredentials, tlsCredentials) import Simplex.Messaging.Transport.Server (AddHTTP, ServerCredentials, TransportServerConfig, loadFingerprint, loadServerCredential) -import System.Exit (exitFailure) +import Simplex.Messaging.Util (liftEitherWith) import System.Mem.Weak (Weak) import UnliftIO.STM @@ -112,33 +111,35 @@ data NtfEnv = NtfEnv } newNtfServerEnv :: NtfServerConfig -> IO NtfEnv -newNtfServerEnv config@NtfServerConfig {pushQSize, smpAgentCfg, apnsConfig, dbStoreConfig, ntfCredentials, useServiceCreds, startOptions} = do - when (compactLog startOptions) $ compactDbStoreLog $ dbStoreLogPath dbStoreConfig +newNtfServerEnv config@NtfServerConfig {pushQSize, smpAgentCfg, apnsConfig, dbStoreConfig, ntfCredentials, useServiceCreds} = do random <- C.newRandom store <- newNtfDbStore dbStoreConfig tlsServerCreds <- loadServerCredential ntfCredentials - serviceCertHash@(XV.Fingerprint fp) <- loadFingerprint ntfCredentials - smpAgentCfg' <- - if useServiceCreds - then do - serviceSignKey <- case C.x509ToPrivate' $ snd tlsServerCreds of - Right pk -> pure pk - Left e -> putStrLn ("Server has no valid key: " <> show e) >> exitFailure - let service = ServiceCredentials {serviceRole = SRNotifier, serviceCreds = tlsServerCreds, serviceCertHash, serviceSignKey} - pure smpAgentCfg {smpCfg = (smpCfg smpAgentCfg) {serviceCredentials = Just service}} - else pure smpAgentCfg - subscriber <- newNtfSubscriber smpAgentCfg' random + XV.Fingerprint fp <- loadFingerprint ntfCredentials + let dbService = if useServiceCreds then Just $ mkDbService random store else Nothing + subscriber <- newNtfSubscriber smpAgentCfg dbService random pushServer <- newNtfPushServer pushQSize apnsConfig serverStats <- newNtfServerStats =<< getCurrentTime pure NtfEnv {config, subscriber, pushServer, store, random, tlsServerCreds, serverIdentity = C.KeyHash fp, serverStats} where - compactDbStoreLog = \case - Just f -> do - logNote $ "compacting store log " <> T.pack f - newNtfSTMStore >>= readWriteNtfSTMStore False f >>= closeStoreLog - Nothing -> do - logError "Error: `--compact-log` used without `enable: on` option in STORE_LOG section of INI file" - exitFailure + mkDbService g st = DBService {getCredentials, updateServiceId} + where + getCredentials :: SMPServer -> IO (Either SMPClientError ServiceCredentials) + getCredentials srv = runExceptT $ do + ExceptT (withClientDB "" st $ \db -> getNtfServiceCredentials db srv >>= mapM (mkServiceCreds db)) >>= \case + Just (C.KeyHash kh, serviceCreds) -> do + serviceSignKey <- liftEitherWith PCEIOError $ C.x509ToPrivate' $ snd serviceCreds + pure ServiceCredentials {serviceRole = SRNotifier, serviceCreds, serviceCertHash = XV.Fingerprint kh, serviceSignKey} + Nothing -> throwE PCEServiceUnavailable -- this error cannot happen, as clients never connect to unknown servers + mkServiceCreds db = \case + (_, Just tlsCreds) -> pure tlsCreds + (srvId, Nothing) -> do + cred <- genCredentials g Nothing (25, 24 * 999999) "simplex" + let tlsCreds = tlsCredentials [cred] + setNtfServiceCredentials db srvId tlsCreds + pure tlsCreds + updateServiceId :: SMPServer -> Maybe ServiceId -> IO (Either SMPClientError ()) + updateServiceId srv serviceId_ = withClientDB "" st $ \db -> updateNtfServiceId db srv serviceId_ data NtfSubscriber = NtfSubscriber { smpSubscribers :: TMap SMPServer SMPSubscriberVar, @@ -148,11 +149,11 @@ data NtfSubscriber = NtfSubscriber type SMPSubscriberVar = SessionVar SMPSubscriber -newNtfSubscriber :: SMPClientAgentConfig -> TVar ChaChaDRG -> IO NtfSubscriber -newNtfSubscriber smpAgentCfg random = do +newNtfSubscriber :: SMPClientAgentConfig -> Maybe DBService -> TVar ChaChaDRG -> IO NtfSubscriber +newNtfSubscriber smpAgentCfg dbService random = do smpSubscribers <- TM.emptyIO subscriberSeq <- newTVarIO 0 - smpAgent <- newSMPClientAgent SNotifierService smpAgentCfg random + smpAgent <- newSMPClientAgent SNotifierService smpAgentCfg dbService random pure NtfSubscriber {smpSubscribers, subscriberSeq, smpAgent} data SMPSubscriber = SMPSubscriber diff --git a/src/Simplex/Messaging/Notifications/Server/Main.hs b/src/Simplex/Messaging/Notifications/Server/Main.hs index 24d59a0d8..047f897fd 100644 --- a/src/Simplex/Messaging/Notifications/Server/Main.hs +++ b/src/Simplex/Messaging/Notifications/Server/Main.hs @@ -20,42 +20,32 @@ import Data.Functor (($>)) import Data.Ini (lookupValue, readIniFile) import Data.Int (Int64) import Data.Maybe (fromMaybe) -import Data.Set (Set) -import qualified Data.Set as S import qualified Data.Text as T import Data.Text.Encoding (encodeUtf8) import qualified Data.Text.IO as T import Network.Socket (HostName, ServiceName) import Options.Applicative -import Simplex.Messaging.Agent.Store.Postgres (checkSchemaExists) import Simplex.Messaging.Agent.Store.Postgres.Options (DBOpts (..)) import Simplex.Messaging.Agent.Store.Shared (MigrationConfirmation (..)) import Simplex.Messaging.Client (HostMode (..), NetworkConfig (..), ProtocolClientConfig (..), SMPWebPortServers (..), SocksMode (..), defaultNetworkConfig, textToHostMode) import Simplex.Messaging.Client.Agent (SMPClientAgentConfig (..), defaultSMPClientAgentConfig) import qualified Simplex.Messaging.Crypto as C -import Simplex.Messaging.Notifications.Protocol (NtfTokenId) -import Simplex.Messaging.Notifications.Server (runNtfServer, restoreServerLastNtfs) +import Simplex.Messaging.Notifications.Server (runNtfServer) import Simplex.Messaging.Notifications.Server.Env (NtfServerConfig (..), defaultInactiveClientExpiration) import Simplex.Messaging.Notifications.Server.Push.APNS (defaultAPNSPushClientConfig) -import Simplex.Messaging.Notifications.Server.Store (newNtfSTMStore) -import Simplex.Messaging.Notifications.Server.Store.Postgres (exportNtfDbStore, importNtfSTMStore, newNtfDbStore) -import Simplex.Messaging.Notifications.Server.StoreLog (readWriteNtfSTMStore) import Simplex.Messaging.Notifications.Transport (alpnSupportedNTFHandshakes, supportedServerNTFVRange) import Simplex.Messaging.Protocol (ProtoServerWithAuth (..), pattern NtfServer) import Simplex.Messaging.Server.CLI import Simplex.Messaging.Server.Env.STM (StartOptions (..)) import Simplex.Messaging.Server.Expiration -import Simplex.Messaging.Server.Main (strParse) import Simplex.Messaging.Server.Main.Init (iniDbOpts) import Simplex.Messaging.Server.QueueStore.Postgres.Config (PostgresStoreCfg (..)) -import Simplex.Messaging.Server.StoreLog (closeStoreLog) import Simplex.Messaging.Transport (ASrvTransport) import Simplex.Messaging.Transport.Client (TransportHost (..)) import Simplex.Messaging.Transport.HTTP2 (httpALPN) import Simplex.Messaging.Transport.Server (AddHTTP, ServerCredentials (..), mkTransportServerConfig) -import Simplex.Messaging.Util (eitherToMaybe, ifM, tshow) -import System.Directory (createDirectoryIfMissing, doesFileExist, renameFile) -import System.Exit (exitFailure) +import Simplex.Messaging.Util (eitherToMaybe, tshow) +import System.Directory (createDirectoryIfMissing, doesFileExist) import System.FilePath (combine) import System.IO (BufferMode (..), hSetBuffering, stderr, stdout) import Text.Read (readMaybe) @@ -76,69 +66,11 @@ ntfServerCLI cfgPath logPath = deleteDirIfExists cfgPath deleteDirIfExists logPath putStrLn "Deleted configuration and log files" - Database cmd dbOpts@DBOpts {connstr, schema} -> withIniFile $ \ini -> do - schemaExists <- checkSchemaExists connstr schema - storeLogExists <- doesFileExist storeLogFilePath - lastNtfsExists <- doesFileExist defaultLastNtfsFile - case cmd of - SCImport skipTokens - | schemaExists && (storeLogExists || lastNtfsExists) -> exitConfigureNtfStore connstr schema - | schemaExists -> do - putStrLn $ "Schema " <> B.unpack schema <> " already exists in PostrgreSQL database: " <> B.unpack connstr - exitFailure - | not storeLogExists -> do - putStrLn $ storeLogFilePath <> " file does not exist." - exitFailure - | not lastNtfsExists -> do - putStrLn $ defaultLastNtfsFile <> " file does not exist." - exitFailure - | otherwise -> do - storeLogFile <- getRequiredStoreLogFile ini - confirmOrExit - ("WARNING: store log file " <> storeLogFile <> " will be compacted and imported to PostrgreSQL database: " <> B.unpack connstr <> ", schema: " <> B.unpack schema) - "Notification server store not imported" - stmStore <- newNtfSTMStore - sl <- readWriteNtfSTMStore True storeLogFile stmStore - closeStoreLog sl - restoreServerLastNtfs stmStore defaultLastNtfsFile - let storeCfg = PostgresStoreCfg {dbOpts = dbOpts {createSchema = True}, dbStoreLogPath = Nothing, confirmMigrations = MCConsole, deletedTTL = iniDeletedTTL ini} - ps <- newNtfDbStore storeCfg - (tCnt, sCnt, nCnt, serviceCnt) <- importNtfSTMStore ps stmStore skipTokens - renameFile storeLogFile $ storeLogFile <> ".bak" - putStrLn $ "Import completed: " <> show tCnt <> " tokens, " <> show sCnt <> " subscriptions, " <> show serviceCnt <> " service associations, " <> show nCnt <> " last token notifications." - putStrLn "Configure database options in INI file." - SCExport - | schemaExists && storeLogExists -> exitConfigureNtfStore connstr schema - | not schemaExists -> do - putStrLn $ "Schema " <> B.unpack schema <> " does not exist in PostrgreSQL database: " <> B.unpack connstr - exitFailure - | storeLogExists -> do - putStrLn $ storeLogFilePath <> " file already exists." - exitFailure - | lastNtfsExists -> do - putStrLn $ defaultLastNtfsFile <> " file already exists." - exitFailure - | otherwise -> do - confirmOrExit - ("WARNING: PostrgreSQL database schema " <> B.unpack schema <> " (database: " <> B.unpack connstr <> ") will be exported to store log file " <> storeLogFilePath) - "Notification server store not imported" - let storeCfg = PostgresStoreCfg {dbOpts, dbStoreLogPath = Just storeLogFilePath, confirmMigrations = MCConsole, deletedTTL = iniDeletedTTL ini} - st <- newNtfDbStore storeCfg - (tCnt, sCnt, nCnt) <- exportNtfDbStore st defaultLastNtfsFile - putStrLn $ "Export completed: " <> show tCnt <> " tokens, " <> show sCnt <> " subscriptions, " <> show nCnt <> " last token notifications." where withIniFile a = doesFileExist iniFile >>= \case True -> readIniFile iniFile >>= either exitError a _ -> exitError $ "Error: server is not initialized (" <> iniFile <> " does not exist).\nRun `" <> executableName <> " init`." - getRequiredStoreLogFile ini = do - case enableStoreLog' ini $> storeLogFilePath of - Just storeLogFile -> do - ifM - (doesFileExist storeLogFile) - (pure storeLogFile) - (putStrLn ("Store log file " <> storeLogFile <> " not found") >> exitFailure) - Nothing -> putStrLn "Store log disabled, see `[STORE_LOG] enable`" >> exitFailure iniFile = combine cfgPath "ntf-server.ini" serverVersion = "SMP notifications server v" <> simplexmqVersionCommit defaultServerPort = "443" @@ -292,11 +224,6 @@ ntfServerCLI cfgPath logPath = startOptions } iniDeletedTTL ini = readIniDefault (86400 * defaultDeletedTTL) "STORE_LOG" "db_deleted_ttl" ini - defaultLastNtfsFile = combine logPath "ntf-server-last-notifications.log" - exitConfigureNtfStore connstr schema = do - putStrLn $ "Error: both " <> storeLogFilePath <> " file and " <> B.unpack schema <> " schema are present (database: " <> B.unpack connstr <> ")." - putStrLn "Configure notification server storage." - exitFailure printNtfServerConfig :: [(ServiceName, ASrvTransport, AddHTTP)] -> PostgresStoreCfg -> IO () printNtfServerConfig transports PostgresStoreCfg {dbOpts = DBOpts {connstr, schema}, dbStoreLogPath} = do @@ -308,9 +235,6 @@ data CliCommand | OnlineCert CertOptions | Start StartOptions | Delete - | Database StoreCmd DBOpts - -data StoreCmd = SCImport (Set NtfTokenId) | SCExport data InitOptions = InitOptions { enableStoreLog :: Bool, @@ -341,22 +265,8 @@ cliCommandP cfgPath logPath iniFile = <> command "cert" (info (OnlineCert <$> certOptionsP) (progDesc $ "Generate new online TLS server credentials (configuration: " <> iniFile <> ")")) <> command "start" (info (Start <$> startOptionsP) (progDesc $ "Start server (configuration: " <> iniFile <> ")")) <> command "delete" (info (pure Delete) (progDesc "Delete configuration and log files")) - <> command "database" (info (Database <$> databaseCmdP <*> dbOptsP defaultNtfDBOpts) (progDesc "Import/export notifications server store to/from PostgreSQL database")) ) where - databaseCmdP = - hsubparser - ( command "import" (info (SCImport <$> skipTokensP) (progDesc $ "Import store logs into a new PostgreSQL database schema")) - <> command "export" (info (pure SCExport) (progDesc $ "Export PostgreSQL database schema to store logs")) - ) - skipTokensP :: Parser (Set NtfTokenId) - skipTokensP = - option - strParse - ( long "skip-tokens" - <> help "Skip tokens during import" - <> value S.empty - ) initP :: Parser InitOptions initP = do enableStoreLog <- diff --git a/src/Simplex/Messaging/Notifications/Server/Store/Migrations.hs b/src/Simplex/Messaging/Notifications/Server/Store/Migrations.hs index 139403fc3..eae815a5b 100644 --- a/src/Simplex/Messaging/Notifications/Server/Store/Migrations.hs +++ b/src/Simplex/Messaging/Notifications/Server/Store/Migrations.hs @@ -9,13 +9,16 @@ where import Data.List (sortOn) import Data.Text (Text) +import Simplex.Messaging.Agent.Store.Postgres.Migrations.Util import Simplex.Messaging.Agent.Store.Shared import Text.RawString.QQ (r) ntfServerSchemaMigrations :: [(String, Text, Maybe Text)] ntfServerSchemaMigrations = [ ("20250417_initial", m20250417_initial, Nothing), - ("20250517_service_cert", m20250517_service_cert, Just down_m20250517_service_cert) + ("20250517_service_cert", m20250517_service_cert, Just down_m20250517_service_cert), + ("20250830_queue_ids_hash", m20250830_queue_ids_hash, Just down_m20250830_queue_ids_hash), + ("20251219_service_cert_per_server", m20251219_service_cert_per_server, Just down_m20251219_service_cert_per_server) ] -- | The list of migrations in ascending order by date @@ -104,3 +107,158 @@ ALTER TABLE smp_servers DROP COLUMN ntf_service_id; ALTER TABLE subscriptions DROP COLUMN ntf_service_assoc; |] + +m20250830_queue_ids_hash :: Text +m20250830_queue_ids_hash = + createXorHashFuncs + <> [r| +ALTER TABLE smp_servers + ADD COLUMN smp_notifier_count BIGINT NOT NULL DEFAULT 0, + ADD COLUMN smp_notifier_ids_hash BYTEA NOT NULL DEFAULT '\x00000000000000000000000000000000'; + +CREATE FUNCTION should_subscribe_status(p_status TEXT) RETURNS BOOLEAN +LANGUAGE plpgsql IMMUTABLE STRICT +AS $$ +BEGIN + RETURN p_status IN ('NEW', 'PENDING', 'ACTIVE', 'INACTIVE'); +END; +$$; + +CREATE FUNCTION update_all_aggregates() RETURNS VOID +LANGUAGE plpgsql +AS $$ +BEGIN + WITH acc AS ( + SELECT + s.smp_server_id, + count(smp_notifier_id) as notifier_count, + xor_aggregate(public.digest(s.smp_notifier_id, 'md5')) AS notifier_hash + FROM subscriptions s + WHERE s.ntf_service_assoc = true AND should_subscribe_status(s.status) + GROUP BY s.smp_server_id + ) + UPDATE smp_servers srv + SET smp_notifier_count = COALESCE(acc.notifier_count, 0), + smp_notifier_ids_hash = COALESCE(acc.notifier_hash, '\x00000000000000000000000000000000') + FROM acc + WHERE srv.smp_server_id = acc.smp_server_id; +END; +$$; + +SELECT update_all_aggregates(); + +CREATE FUNCTION update_aggregates(p_server_id BIGINT, p_change BIGINT, p_notifier_id BYTEA) RETURNS VOID +LANGUAGE plpgsql +AS $$ +BEGIN + UPDATE smp_servers + SET smp_notifier_count = smp_notifier_count + p_change, + smp_notifier_ids_hash = xor_combine(smp_notifier_ids_hash, public.digest(p_notifier_id, 'md5')) + WHERE smp_server_id = p_server_id; +END; +$$; + +CREATE FUNCTION on_subscription_insert() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF NEW.ntf_service_assoc = true AND should_subscribe_status(NEW.status) THEN + PERFORM update_aggregates(NEW.smp_server_id, 1, NEW.smp_notifier_id); + END IF; + RETURN NEW; +END; +$$; + +CREATE FUNCTION on_subscription_delete() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF OLD.ntf_service_assoc = true AND should_subscribe_status(OLD.status) THEN + PERFORM update_aggregates(OLD.smp_server_id, -1, OLD.smp_notifier_id); + END IF; + RETURN OLD; +END; +$$; + +CREATE FUNCTION on_subscription_update() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF OLD.ntf_service_assoc = true AND should_subscribe_status(OLD.status) THEN + IF NOT (NEW.ntf_service_assoc = true AND should_subscribe_status(NEW.status)) THEN + PERFORM update_aggregates(OLD.smp_server_id, -1, OLD.smp_notifier_id); + END IF; + ELSIF NEW.ntf_service_assoc = true AND should_subscribe_status(NEW.status) THEN + PERFORM update_aggregates(NEW.smp_server_id, 1, NEW.smp_notifier_id); + END IF; + RETURN NEW; +END; +$$; + +CREATE TRIGGER tr_subscriptions_insert +AFTER INSERT ON subscriptions +FOR EACH ROW EXECUTE PROCEDURE on_subscription_insert(); + +CREATE TRIGGER tr_subscriptions_delete +AFTER DELETE ON subscriptions +FOR EACH ROW EXECUTE PROCEDURE on_subscription_delete(); + +CREATE TRIGGER tr_subscriptions_update +AFTER UPDATE ON subscriptions +FOR EACH ROW EXECUTE PROCEDURE on_subscription_update(); + |] + +down_m20250830_queue_ids_hash :: Text +down_m20250830_queue_ids_hash = + [r| +DROP TRIGGER tr_subscriptions_insert ON subscriptions; +DROP TRIGGER tr_subscriptions_delete ON subscriptions; +DROP TRIGGER tr_subscriptions_update ON subscriptions; + +DROP FUNCTION on_subscription_insert; +DROP FUNCTION on_subscription_delete; +DROP FUNCTION on_subscription_update; + +DROP FUNCTION update_aggregates; +DROP FUNCTION update_all_aggregates; + +DROP FUNCTION should_subscribe_status; + +ALTER TABLE smp_servers + DROP COLUMN smp_notifier_count, + DROP COLUMN smp_notifier_ids_hash; + |] + <> dropXorHashFuncs + +m20251219_service_cert_per_server :: Text +m20251219_service_cert_per_server = + [r| +ALTER TABLE smp_servers + ADD COLUMN ntf_service_cert BYTEA, + ADD COLUMN ntf_service_cert_hash BYTEA, + ADD COLUMN ntf_service_priv_key BYTEA; + |] + <> resetNtfServices + +down_m20251219_service_cert_per_server :: Text +down_m20251219_service_cert_per_server = + [r| +ALTER TABLE smp_servers + DROP COLUMN ntf_service_cert, + DROP COLUMN ntf_service_cert_hash, + DROP COLUMN ntf_service_priv_key; + |] + <> resetNtfServices + +resetNtfServices :: Text +resetNtfServices = + [r| +ALTER TABLE subscriptions DISABLE TRIGGER tr_subscriptions_update; +UPDATE subscriptions SET ntf_service_assoc = FALSE; +ALTER TABLE subscriptions ENABLE TRIGGER tr_subscriptions_update; + +UPDATE smp_servers +SET ntf_service_id = NULL, + smp_notifier_count = 0, + smp_notifier_ids_hash = DEFAULT; + |] diff --git a/src/Simplex/Messaging/Notifications/Server/Store/Postgres.hs b/src/Simplex/Messaging/Notifications/Server/Store/Postgres.hs index c66aa5e70..aca573d21 100644 --- a/src/Simplex/Messaging/Notifications/Server/Store/Postgres.hs +++ b/src/Simplex/Messaging/Notifications/Server/Store/Postgres.hs @@ -29,6 +29,9 @@ module Simplex.Messaging.Notifications.Server.Store.Postgres deleteNtfToken, updateTknCronInterval, getUsedSMPServers, + getNtfServiceCredentials, + setNtfServiceCredentials, + updateNtfServiceId, getServerNtfSubscriptions, findNtfSubscription, getNtfSubscription, @@ -44,15 +47,13 @@ module Simplex.Messaging.Notifications.Server.Store.Postgres updateSrvSubStatus, batchUpdateSrvSubStatus, batchUpdateSrvSubErrors, - removeServiceAssociation, + removeServiceAndAssociations, addTokenLastNtf, getEntityCounts, - importNtfSTMStore, - exportNtfDbStore, withDB', + withClientDB, ) where -import Control.Concurrent.STM import qualified Control.Exception as E import Control.Logger.Simple import Control.Monad @@ -60,19 +61,13 @@ import Control.Monad.Except import Control.Monad.IO.Class import Control.Monad.Trans.Except import Data.Bitraversable (bimapM) -import qualified Data.ByteString.Base64.URL as B64 import Data.ByteString.Char8 (ByteString) -import qualified Data.ByteString.Char8 as B -import Data.Containers.ListUtils (nubOrd) import Data.Either (fromRight) import Data.Functor (($>)) import Data.Int (Int64) -import Data.List (findIndex, foldl') import Data.List.NonEmpty (NonEmpty (..)) import qualified Data.List.NonEmpty as L -import qualified Data.Map.Strict as M import Data.Maybe (fromMaybe, isJust, mapMaybe) -import qualified Data.Set as S import Data.Text (Text) import qualified Data.Text as T import Data.Text.Encoding (decodeLatin1, encodeUtf8) @@ -85,31 +80,30 @@ import Database.PostgreSQL.Simple.FromField (FromField (..)) import Database.PostgreSQL.Simple.SqlQQ (sql) import Database.PostgreSQL.Simple.ToField (ToField (..)) import Network.Socket (ServiceName) +import qualified Network.TLS as TLS import Simplex.Messaging.Agent.Store.AgentStore () import Simplex.Messaging.Agent.Store.Postgres (closeDBStore, createDBStore) import Simplex.Messaging.Agent.Store.Postgres.Common import Simplex.Messaging.Agent.Store.Postgres.DB (fromTextField_) import Simplex.Messaging.Agent.Store.Shared (MigrationConfig (..)) +import Simplex.Messaging.Client (ProtocolClientError (..), SMPClientError) import Simplex.Messaging.Encoding import Simplex.Messaging.Encoding.String import qualified Simplex.Messaging.Crypto as C import Simplex.Messaging.Notifications.Protocol -import Simplex.Messaging.Notifications.Server.Store (NtfSTMStore (..), NtfSubData (..), NtfTknData (..), TokenNtfMessageRecord (..), ntfSubServer) import Simplex.Messaging.Notifications.Server.Store.Migrations import Simplex.Messaging.Notifications.Server.Store.Types -import Simplex.Messaging.Notifications.Server.StoreLog -import Simplex.Messaging.Protocol (EntityId (..), EncNMsgMeta, ErrorType (..), NotifierId, NtfPrivateAuthKey, NtfPublicAuthKey, SMPServer, ServiceId, pattern SMPServer) -import Simplex.Messaging.Server.QueueStore.Postgres (handleDuplicate, withLog_) +import Simplex.Messaging.Protocol (EntityId (..), EncNMsgMeta, ErrorType (..), IdsHash (..), NotifierId, NtfPrivateAuthKey, NtfPublicAuthKey, ProtocolServer (..), SMPServer, ServiceId, ServiceSub (..), pattern SMPServer) +import Simplex.Messaging.Server.QueueStore.Postgres (handleDuplicate) import Simplex.Messaging.Server.QueueStore.Postgres.Config (PostgresStoreCfg (..)) -import Simplex.Messaging.Server.StoreLog (openWriteStoreLog) import Simplex.Messaging.SystemTime import Simplex.Messaging.Transport.Client (TransportHost) -import Simplex.Messaging.Util (anyM, firstRow, maybeFirstRow, toChunks, tshow) +import Simplex.Messaging.Util (firstRow, maybeFirstRow, tshow) import System.Exit (exitFailure) -import System.IO (IOMode (..), hFlush, stdout, withFile) import Text.Hex (decodeHex) #if !defined(dbPostgres) +import qualified Data.X509 as X import Simplex.Messaging.Agent.Store.Postgres.DB (blobFieldDecoder) import Simplex.Messaging.Parsers (parseAll) import Simplex.Messaging.Util (eitherToMaybe) @@ -117,7 +111,6 @@ import Simplex.Messaging.Util (eitherToMaybe) data NtfPostgresStore = NtfPostgresStore { dbStore :: DBStore, - dbStoreLog :: Maybe (StoreLog 'WriteMode), deletedTTL :: Int64 } @@ -133,25 +126,22 @@ data NtfEntityRec (e :: NtfEntity) where NtfSub :: NtfSubRec -> NtfEntityRec 'Subscription newNtfDbStore :: PostgresStoreCfg -> IO NtfPostgresStore -newNtfDbStore PostgresStoreCfg {dbOpts, dbStoreLogPath, confirmMigrations, deletedTTL} = do +newNtfDbStore PostgresStoreCfg {dbOpts, confirmMigrations, deletedTTL} = do dbStore <- either err pure =<< createDBStore dbOpts ntfServerMigrations (MigrationConfig confirmMigrations Nothing) - dbStoreLog <- mapM (openWriteStoreLog True) dbStoreLogPath - pure NtfPostgresStore {dbStore, dbStoreLog, deletedTTL} + pure NtfPostgresStore {dbStore, deletedTTL} where err e = do logError $ "STORE: newNtfStore, error opening PostgreSQL database, " <> tshow e exitFailure closeNtfDbStore :: NtfPostgresStore -> IO () -closeNtfDbStore NtfPostgresStore {dbStore, dbStoreLog} = do - closeDBStore dbStore - mapM_ closeStoreLog dbStoreLog +closeNtfDbStore NtfPostgresStore {dbStore} = closeDBStore dbStore addNtfToken :: NtfPostgresStore -> NtfTknRec -> IO (Either ErrorType ()) addNtfToken st tkn = withFastDB "addNtfToken" st $ \db -> - E.try (DB.execute db insertNtfTknQuery $ ntfTknToRow tkn) - >>= bimapM handleDuplicate (\_ -> withLog "addNtfToken" st (`logCreateToken` tkn)) + E.try (void $ DB.execute db insertNtfTknQuery $ ntfTknToRow tkn) + >>= bimapM handleDuplicate pure insertNtfTknQuery :: Query insertNtfTknQuery = @@ -162,7 +152,7 @@ insertNtfTknQuery = |] replaceNtfToken :: NtfPostgresStore -> NtfTknRec -> IO (Either ErrorType ()) -replaceNtfToken st NtfTknRec {ntfTknId, token = token@(DeviceToken pp ppToken), tknStatus, tknRegCode = code@(NtfRegCode regCode)} = +replaceNtfToken st NtfTknRec {ntfTknId, token = DeviceToken pp ppToken, tknStatus, tknRegCode = NtfRegCode regCode} = withFastDB "replaceNtfToken" st $ \db -> runExceptT $ do ExceptT $ assertUpdated <$> DB.execute @@ -173,7 +163,6 @@ replaceNtfToken st NtfTknRec {ntfTknId, token = token@(DeviceToken pp ppToken), WHERE token_id = ? |] (pp, Binary ppToken, tknStatus, Binary regCode, ntfTknId) - withLog "replaceNtfToken" st $ \sl -> logUpdateToken sl ntfTknId token code ntfTknToRow :: NtfTknRec -> NtfTknRow ntfTknToRow NtfTknRec {ntfTknId, token, tknStatus, tknVerifyKey, tknDhPrivKey, tknDhSecret, tknRegCode, tknCronInterval, tknUpdatedAt} = @@ -194,15 +183,14 @@ getNtfToken_ :: ToRow q => NtfPostgresStore -> Query -> q -> IO (Either ErrorTyp getNtfToken_ st cond params = withFastDB' "getNtfToken" st $ \db -> do tkn_ <- maybeFirstRow rowToNtfTkn $ DB.query db (ntfTknQuery <> cond) params - mapM_ (updateTokenDate st db) tkn_ + mapM_ (updateTokenDate db) tkn_ pure tkn_ -updateTokenDate :: NtfPostgresStore -> DB.Connection -> NtfTknRec -> IO () -updateTokenDate st db NtfTknRec {ntfTknId, tknUpdatedAt} = do +updateTokenDate :: DB.Connection -> NtfTknRec -> IO () +updateTokenDate db NtfTknRec {ntfTknId, tknUpdatedAt} = do ts <- getSystemDate when (maybe True (ts /=) tknUpdatedAt) $ do void $ DB.execute db "UPDATE tokens SET updated_at = ? WHERE token_id = ?" (ts, ntfTknId) - withLog "updateTokenDate" st $ \sl -> logUpdateTokenTime sl ntfTknId ts type NtfTknRow = (NtfTokenId, PushProvider, Binary ByteString, NtfTknStatus, NtfPublicAuthKey, C.PrivateKeyX25519, C.DhSecretX25519, Binary ByteString, Word16, Maybe SystemDate) @@ -240,7 +228,6 @@ deleteNtfToken st tknId = |] (Only tknId) liftIO $ void $ DB.execute db "DELETE FROM tokens WHERE token_id = ?" (Only tknId) - withLog "deleteNtfToken" st (`logDeleteToken` tknId) pure subs where toServerSubs :: SMPServerRow :. Only Text -> (SMPServer, [NotifierId]) @@ -269,11 +256,10 @@ updateTknCronInterval st tknId cronInt = withFastDB "updateTknCronInterval" st $ \db -> runExceptT $ do ExceptT $ assertUpdated <$> DB.execute db "UPDATE tokens SET cron_interval = ? WHERE token_id = ?" (cronInt, tknId) - withLog "updateTknCronInterval" st $ \sl -> logTokenCron sl tknId 0 -- Reads servers that have subscriptions that need subscribing. -- It is executed on server start, and it is supposed to crash on database error -getUsedSMPServers :: NtfPostgresStore -> IO [(SMPServer, Int64, Maybe (ServiceId, Int64))] +getUsedSMPServers :: NtfPostgresStore -> IO [(SMPServer, Int64, Maybe ServiceSub)] getUsedSMPServers st = withTransaction (dbStore st) $ \db -> map rowToSrvSubs <$> @@ -281,25 +267,84 @@ getUsedSMPServers st = db [sql| SELECT - p.smp_host, p.smp_port, p.smp_keyhash, p.smp_server_id, p.ntf_service_id, - SUM(CASE WHEN s.ntf_service_assoc THEN s.subs_count ELSE 0 END) :: BIGINT as service_subs_count - FROM smp_servers p - JOIN ( - SELECT - smp_server_id, - ntf_service_assoc, - COUNT(1) as subs_count - FROM subscriptions - WHERE status IN ? - GROUP BY smp_server_id, ntf_service_assoc - ) s ON s.smp_server_id = p.smp_server_id - GROUP BY p.smp_host, p.smp_port, p.smp_keyhash, p.smp_server_id, p.ntf_service_id + smp_host, smp_port, smp_keyhash, smp_server_id, + ntf_service_id, smp_notifier_count, smp_notifier_ids_hash + FROM smp_servers + WHERE EXISTS (SELECT 1 FROM subscriptions WHERE status IN ?) |] - (Only (In [NSNew, NSPending, NSActive, NSInactive])) + (Only (In subscribeNtfStatuses)) + where + rowToSrvSubs :: SMPServerRow :. (Int64, Maybe ServiceId, Int64, IdsHash) -> (SMPServer, Int64, Maybe ServiceSub) + rowToSrvSubs ((host, port, kh) :. (srvId, serviceId_, n, idsHash)) = + let service_ = (\serviceId -> ServiceSub serviceId n idsHash) <$> serviceId_ + in (SMPServer host port kh, srvId, service_) + +getNtfServiceCredentials :: DB.Connection -> SMPServer -> IO (Maybe (Int64, Maybe (C.KeyHash, TLS.Credential))) +getNtfServiceCredentials db srv = + maybeFirstRow toService $ + DB.query + db + [sql| + SELECT smp_server_id, ntf_service_cert_hash, ntf_service_cert, ntf_service_priv_key + FROM smp_servers + WHERE smp_host = ? AND smp_port = ? AND smp_keyhash = ? + FOR UPDATE + |] + (host srv, port srv, keyHash srv) + where + toService (Only srvId :. creds) = (srvId, toCredentials creds) + toCredentials = \case + (Just kh, Just cert, Just pk) -> Just (kh, (cert, pk)) + _ -> Nothing + +setNtfServiceCredentials :: DB.Connection -> Int64 -> (C.KeyHash, TLS.Credential) -> IO () +setNtfServiceCredentials db srvId (kh, (cert, pk)) = + void $ DB.execute + db + [sql| + UPDATE smp_servers + SET ntf_service_cert_hash = ?, ntf_service_cert = ?, ntf_service_priv_key = ? + WHERE smp_server_id = ? + |] + (kh, cert, pk, srvId) + +updateNtfServiceId :: DB.Connection -> SMPServer -> Maybe ServiceId -> IO () +updateNtfServiceId db srv newServiceId_ = do + maybeFirstRow id (getSMPServiceForUpdate_ db srv) >>= mapM_ updateService where - rowToSrvSubs :: SMPServerRow :. (Int64, Maybe ServiceId, Int64) -> (SMPServer, Int64, Maybe (ServiceId, Int64)) - rowToSrvSubs ((host, port, kh) :. (srvId, serviceId_, subsCount)) = - (SMPServer host port kh, srvId, (,subsCount) <$> serviceId_) + updateService (srvId, currServiceId_) = unless (currServiceId_ == newServiceId_) $ do + when (isJust currServiceId_) $ do + void $ removeServiceAssociation_ db srvId + logError $ "STORE: service ID for " <> enc (host srv) <> toServiceId <> ", removed sub associations" + void $ case newServiceId_ of + Just newServiceId -> + DB.execute + db + [sql| + UPDATE smp_servers + SET ntf_service_id = ?, + smp_notifier_count = 0, + smp_notifier_ids_hash = DEFAULT + WHERE smp_server_id = ? + |] + (newServiceId, srvId) + Nothing -> + DB.execute + db + [sql| + UPDATE smp_servers + SET ntf_service_id = NULL, + ntf_service_cert = NULL, + ntf_service_cert_hash = NULL, + ntf_service_priv_key = NULL, + smp_notifier_count = 0, + smp_notifier_ids_hash = DEFAULT + WHERE smp_server_id = ? + |] + (Only srvId) + toServiceId = maybe " removed" ((" changed to " <>) . enc) newServiceId_ + enc :: StrEncoding a => a -> Text + enc = decodeLatin1 . strEncode getServerNtfSubscriptions :: NtfPostgresStore -> Int64 -> Maybe NtfSubscriptionId -> Int -> IO (Either ErrorType [ServerNtfSub]) getServerNtfSubscriptions st srvId afterSubId_ count = @@ -307,9 +352,9 @@ getServerNtfSubscriptions st srvId afterSubId_ count = subs <- map toServerNtfSub <$> case afterSubId_ of Nothing -> - DB.query db (query <> orderLimit) (srvId, statusIn, count) + DB.query db (query <> orderLimit) (srvId, In subscribeNtfStatuses, count) Just afterSubId -> - DB.query db (query <> " AND subscription_id > ?" <> orderLimit) (srvId, statusIn, afterSubId, count) + DB.query db (query <> " AND subscription_id > ?" <> orderLimit) (srvId, In subscribeNtfStatuses, afterSubId, count) void $ DB.executeMany db @@ -330,7 +375,6 @@ getServerNtfSubscriptions st srvId afterSubId_ count = WHERE smp_server_id = ? AND NOT ntf_service_assoc AND status IN ? |] orderLimit = " ORDER BY subscription_id LIMIT ?" - statusIn = In [NSNew, NSPending, NSActive, NSInactive] toServerNtfSub (ntfSubId, notifierId, notifierKey) = (ntfSubId, (notifierId, notifierKey)) -- Returns token and subscription. @@ -340,7 +384,7 @@ findNtfSubscription st tknId q = withFastDB "findNtfSubscription" st $ \db -> runExceptT $ do tkn@NtfTknRec {ntfTknId, tknStatus} <- ExceptT $ getNtfToken st tknId unless (allowNtfSubCommands tknStatus) $ throwE AUTH - liftIO $ updateTokenDate st db tkn + liftIO $ updateTokenDate db tkn sub_ <- liftIO $ maybeFirstRow (rowToNtfSub q) $ DB.query @@ -373,7 +417,7 @@ getNtfSubscription st subId = WHERE s.subscription_id = ? |] (Only subId) - liftIO $ updateTokenDate st db tkn + liftIO $ updateTokenDate db tkn unless (allowNtfSubCommands tknStatus) $ throwE AUTH pure r @@ -395,36 +439,30 @@ mkNtfSubRec ntfSubId (NewNtfSub tokenId smpQueue notifierKey) = updateTknStatus :: NtfPostgresStore -> NtfTknRec -> NtfTknStatus -> IO (Either ErrorType ()) updateTknStatus st tkn status = - withFastDB' "updateTknStatus" st $ \db -> updateTknStatus_ st db tkn status + withFastDB' "updateTknStatus" st $ \db -> updateTknStatus_ db tkn status -updateTknStatus_ :: NtfPostgresStore -> DB.Connection -> NtfTknRec -> NtfTknStatus -> IO () -updateTknStatus_ st db NtfTknRec {ntfTknId} status = do - updated <- DB.execute db "UPDATE tokens SET status = ? WHERE token_id = ? AND status != ?" (status, ntfTknId, status) - when (updated > 0) $ withLog "updateTknStatus" st $ \sl -> logTokenStatus sl ntfTknId status +updateTknStatus_ :: DB.Connection -> NtfTknRec -> NtfTknStatus -> IO () +updateTknStatus_ db NtfTknRec {ntfTknId} status = + void $ DB.execute db "UPDATE tokens SET status = ? WHERE token_id = ? AND status != ?" (status, ntfTknId, status) -- unless it was already active setTknStatusConfirmed :: NtfPostgresStore -> NtfTknRec -> IO (Either ErrorType ()) setTknStatusConfirmed st NtfTknRec {ntfTknId} = - withFastDB' "updateTknStatus" st $ \db -> do - updated <- DB.execute db "UPDATE tokens SET status = ? WHERE token_id = ? AND status != ? AND status != ?" (NTConfirmed, ntfTknId, NTConfirmed, NTActive) - when (updated > 0) $ withLog "updateTknStatus" st $ \sl -> logTokenStatus sl ntfTknId NTConfirmed + withFastDB' "updateTknStatus" st $ \db -> + void $ DB.execute db "UPDATE tokens SET status = ? WHERE token_id = ? AND status != ? AND status != ?" (NTConfirmed, ntfTknId, NTConfirmed, NTActive) setTokenActive :: NtfPostgresStore -> NtfTknRec -> IO (Either ErrorType ()) setTokenActive st tkn@NtfTknRec {ntfTknId, token = DeviceToken pp ppToken} = withFastDB' "setTokenActive" st $ \db -> do - updateTknStatus_ st db tkn NTActive + updateTknStatus_ db tkn NTActive -- this removes other instances of the same token, e.g. because of repeated token registration attempts - tknIds <- - liftIO $ map fromOnly <$> - DB.query - db - [sql| - DELETE FROM tokens - WHERE push_provider = ? AND push_provider_token = ? AND token_id != ? - RETURNING token_id - |] - (pp, Binary ppToken, ntfTknId) - withLog "deleteNtfToken" st $ \sl -> mapM_ (logDeleteToken sl) tknIds + void $ DB.execute + db + [sql| + DELETE FROM tokens + WHERE push_provider = ? AND push_provider_token = ? AND token_id != ? + |] + (pp, Binary ppToken, ntfTknId) withPeriodicNtfTokens :: NtfPostgresStore -> Int64 -> (NtfTknRec -> IO ()) -> IO Int withPeriodicNtfTokens st now notify = @@ -442,7 +480,6 @@ addNtfSubscription st sub = withFastDB "addNtfSubscription" st $ \db -> runExceptT $ do srvId :: Int64 <- ExceptT $ upsertServer db $ ntfSubServer' sub n <- liftIO $ DB.execute db insertNtfSubQuery $ ntfSubToRow srvId sub - withLog "addNtfSubscription" st (`logCreateSubscription` sub) pure (srvId, n > 0) where -- It is possible to combine these two statements into one with CTEs, @@ -485,76 +522,66 @@ ntfSubToRow srvId NtfSubRec {ntfSubId, tokenId, smpQueue = SMPQueueNtf _ nId, no deleteNtfSubscription :: NtfPostgresStore -> NtfSubscriptionId -> IO (Either ErrorType ()) deleteNtfSubscription st subId = - withFastDB "deleteNtfSubscription" st $ \db -> runExceptT $ do - ExceptT $ assertUpdated <$> + withFastDB "deleteNtfSubscription" st $ \db -> + assertUpdated <$> DB.execute db "DELETE FROM subscriptions WHERE subscription_id = ?" (Only subId) - withLog "deleteNtfSubscription" st (`logDeleteSubscription` subId) updateSubStatus :: NtfPostgresStore -> Int64 -> NotifierId -> NtfSubStatus -> IO (Either ErrorType ()) updateSubStatus st srvId nId status = withFastDB' "updateSubStatus" st $ \db -> do - sub_ :: Maybe (NtfSubscriptionId, NtfAssociatedService) <- - maybeFirstRow id $ - DB.query - db - [sql| - UPDATE subscriptions SET status = ? - WHERE smp_server_id = ? AND smp_notifier_id = ? AND status != ? - RETURNING subscription_id, ntf_service_assoc - |] - (status, srvId, nId, status) - forM_ sub_ $ \(subId, serviceAssoc) -> - withLog "updateSubStatus" st $ \sl -> logSubscriptionStatus sl (subId, status, serviceAssoc) + void $ + DB.execute + db + [sql| + UPDATE subscriptions SET status = ? + WHERE smp_server_id = ? AND smp_notifier_id = ? AND status != ? + |] + (status, srvId, nId, status) updateSrvSubStatus :: NtfPostgresStore -> SMPQueueNtf -> NtfSubStatus -> IO (Either ErrorType ()) updateSrvSubStatus st q status = - withFastDB' "updateSrvSubStatus" st $ \db -> do - sub_ :: Maybe (NtfSubscriptionId, NtfAssociatedService) <- - maybeFirstRow id $ - DB.query - db - [sql| - UPDATE subscriptions s - SET status = ? - FROM smp_servers p - WHERE p.smp_server_id = s.smp_server_id - AND p.smp_host = ? AND p.smp_port = ? AND p.smp_keyhash = ? AND s.smp_notifier_id = ? - AND s.status != ? - RETURNING s.subscription_id, s.ntf_service_assoc - |] - (Only status :. smpQueueToRow q :. Only status) - forM_ sub_ $ \(subId, serviceAssoc) -> - withLog "updateSrvSubStatus" st $ \sl -> logSubscriptionStatus sl (subId, status, serviceAssoc) + withFastDB' "updateSrvSubStatus" st $ \db -> + void $ + DB.execute + db + [sql| + UPDATE subscriptions s + SET status = ? + FROM smp_servers p + WHERE p.smp_server_id = s.smp_server_id + AND p.smp_host = ? AND p.smp_port = ? AND p.smp_keyhash = ? AND s.smp_notifier_id = ? + AND s.status != ? + |] + (Only status :. smpQueueToRow q :. Only status) batchUpdateSrvSubStatus :: NtfPostgresStore -> SMPServer -> Maybe ServiceId -> NonEmpty NotifierId -> NtfSubStatus -> IO Int batchUpdateSrvSubStatus st srv newServiceId nIds status = fmap (fromRight (-1)) $ withDB "batchUpdateSrvSubStatus" st $ \db -> runExceptT $ do - (srvId :: Int64, currServiceId) <- ExceptT $ getSMPServerService db + (srvId, currServiceId) <- ExceptT $ firstRow id AUTH $ getSMPServiceForUpdate_ db srv + -- TODO [certs rcv] should this remove associations/credentials when newServiceId is Nothing or different unless (currServiceId == newServiceId) $ liftIO $ void $ DB.execute db "UPDATE smp_servers SET ntf_service_id = ? WHERE smp_server_id = ?" (newServiceId, srvId) let params = L.toList $ L.map (srvId,isJust newServiceId,status,) nIds liftIO $ fromIntegral <$> DB.executeMany db updateSubStatusQuery params - where - getSMPServerService db = - firstRow id AUTH $ - DB.query - db - [sql| - SELECT smp_server_id, ntf_service_id - FROM smp_servers - WHERE smp_host = ? AND smp_port = ? AND smp_keyhash = ? - FOR UPDATE - |] - (srvToRow srv) + +getSMPServiceForUpdate_ :: DB.Connection -> SMPServer -> IO [(Int64, Maybe ServiceId)] +getSMPServiceForUpdate_ db srv = + DB.query + db + [sql| + SELECT smp_server_id, ntf_service_id + FROM smp_servers + WHERE smp_host = ? AND smp_port = ? AND smp_keyhash = ? + FOR UPDATE + |] + (srvToRow srv) batchUpdateSrvSubErrors :: NtfPostgresStore -> SMPServer -> NonEmpty (NotifierId, NtfSubStatus) -> IO Int batchUpdateSrvSubErrors st srv subs = fmap (fromRight (-1)) $ withDB "batchUpdateSrvSubErrors" st $ \db -> runExceptT $ do srvId :: Int64 <- ExceptT $ getSMPServerId db let params = map (\(nId, status) -> (srvId, False, status, nId)) $ L.toList subs - subs' <- liftIO $ DB.returning db (updateSubStatusQuery <> " RETURNING s.subscription_id, s.status, s.ntf_service_assoc") params - withLog "batchUpdateStatus_" st $ forM_ subs' . logSubscriptionStatus - pure $ length subs' + liftIO $ fromIntegral <$> DB.executeMany db updateSubStatusQuery params where getSMPServerId db = firstRow fromOnly AUTH $ @@ -578,36 +605,51 @@ updateSubStatusQuery = AND (s.status != upd.status OR s.ntf_service_assoc != upd.ntf_service_assoc) |] -removeServiceAssociation :: NtfPostgresStore -> SMPServer -> IO (Either ErrorType (Int64, Int)) -removeServiceAssociation st srv = do - withDB "removeServiceAssociation" st $ \db -> runExceptT $ do - srvId <- ExceptT $ removeServerService db - subs <- - liftIO $ - DB.query - db - [sql| - UPDATE subscriptions s - SET status = ?, ntf_service_assoc = FALSE - WHERE smp_server_id = ? - AND (s.status != ? OR s.ntf_service_assoc != FALSE) - RETURNING s.subscription_id, s.status, s.ntf_service_assoc - |] - (NSInactive, srvId, NSInactive) - withLog "removeServiceAssociation" st $ forM_ subs . logSubscriptionStatus - pure (srvId, length subs) +removeServiceAssociation_ :: DB.Connection -> Int64 -> IO Int64 +removeServiceAssociation_ db srvId = + DB.execute + db + [sql| + UPDATE subscriptions s + SET status = ?, ntf_service_assoc = FALSE + WHERE smp_server_id = ? + AND (s.status != ? OR s.ntf_service_assoc != FALSE) + |] + (NSInactive, srvId, NSInactive) + +removeServiceAndAssociations :: NtfPostgresStore -> SMPServer -> IO (Either ErrorType (Int64, Int)) +removeServiceAndAssociations st srv = do + withDB "removeServiceAndAssociations" st $ \db -> runExceptT $ do + srvId <- ExceptT $ getServerId db + subsCount <- liftIO $ removeServiceAssociation_ db srvId + liftIO $ void $ removeServerService db srvId + pure (srvId, fromIntegral subsCount) where - removeServerService db = + getServerId db = firstRow fromOnly AUTH $ DB.query db [sql| - UPDATE smp_servers - SET ntf_service_id = NULL + SELECT smp_server_id + FROM smp_servers WHERE smp_host = ? AND smp_port = ? AND smp_keyhash = ? - RETURNING smp_server_id + FOR UPDATE |] (srvToRow srv) + removeServerService db srvId = + DB.execute + db + [sql| + UPDATE smp_servers + SET ntf_service_id = NULL, + ntf_service_cert = NULL, + ntf_service_cert_hash = NULL, + ntf_service_priv_key = NULL, + smp_notifier_count = 0, + smp_notifier_ids_hash = DEFAULT + WHERE smp_server_id = ? + |] + (Only srvId) addTokenLastNtf :: NtfPostgresStore -> PNMessageData -> IO (Either ErrorType (NtfTknRec, NonEmpty PNMessageData)) addTokenLastNtf st newNtf = @@ -689,216 +731,6 @@ getEntityCounts st = count (Only n : _) = n count [] = 0 -importNtfSTMStore :: NtfPostgresStore -> NtfSTMStore -> S.Set NtfTokenId -> IO (Int64, Int64, Int64, Int64) -importNtfSTMStore NtfPostgresStore {dbStore = s} stmStore skipTokens = do - (tIds, tCnt) <- importTokens - subLookup <- readTVarIO $ subscriptionLookup stmStore - sCnt <- importSubscriptions tIds subLookup - nCnt <- importLastNtfs tIds subLookup - serviceCnt <- importNtfServiceIds - pure (tCnt, sCnt, nCnt, serviceCnt) - where - importTokens = do - allTokens <- M.elems <$> readTVarIO (tokens stmStore) - tokens <- filterTokens allTokens - let skipped = length allTokens - length tokens - when (skipped /= 0) $ putStrLn $ "Total skipped tokens " <> show skipped - -- uncomment this line instead of the next two to import tokens one by one. - -- tCnt <- withConnection s $ \db -> foldM (importTkn db) 0 tokens - -- token interval is reset to 0 to only send notifications to devices with periodic mode, - -- and before clients are upgraded - to all active devices. - tRows <- mapM (fmap (ntfTknToRow . (\t -> t {tknCronInterval = 0} :: NtfTknRec)) . mkTknRec) tokens - tCnt <- withConnection s $ \db -> DB.executeMany db insertNtfTknQuery tRows - let tokenIds = S.fromList $ map (\NtfTknData {ntfTknId} -> ntfTknId) tokens - (tokenIds,) <$> checkCount "token" (length tokens) tCnt - where - filterTokens tokens = do - let deviceTokens = foldl' (\m t -> M.alter (Just . (t :) . fromMaybe []) (tokenKey t) m) M.empty tokens - tokenSubs <- readTVarIO (tokenSubscriptions stmStore) - filterM (keepTokenRegistration deviceTokens tokenSubs) tokens - tokenKey NtfTknData {token, tknVerifyKey} = strEncode token <> ":" <> C.toPubKey C.pubKeyBytes tknVerifyKey - keepTokenRegistration deviceTokens tokenSubs tkn@NtfTknData {ntfTknId, tknStatus} = - case M.lookup (tokenKey tkn) deviceTokens of - Just ts - | length ts < 2 -> pure True - | ntfTknId `S.member` skipTokens -> False <$ putStrLn ("Skipped token " <> enc ntfTknId <> " from --skip-tokens") - | otherwise -> - readTVarIO tknStatus >>= \case - NTConfirmed -> do - hasSubs <- maybe (pure False) (\v -> not . S.null <$> readTVarIO v) $ M.lookup ntfTknId tokenSubs - if hasSubs - then pure True - else do - anyBetterToken <- anyM $ map (\NtfTknData {tknStatus = tknStatus'} -> activeOrInvalid <$> readTVarIO tknStatus') ts - if anyBetterToken - then False <$ putStrLn ("Skipped duplicate inactive token " <> enc ntfTknId) - else case findIndex (\NtfTknData {ntfTknId = tId} -> tId == ntfTknId) ts of - Just 0 -> pure True -- keeping the first token - Just _ -> False <$ putStrLn ("Skipped duplicate inactive token " <> enc ntfTknId <> " (no active token)") - Nothing -> True <$ putStrLn "Error: no device token in the list" - _ -> pure True - Nothing -> True <$ putStrLn "Error: no device token in lookup map" - activeOrInvalid = \case - NTActive -> True - NTInvalid _ -> True - _ -> False - -- importTkn db !n tkn@NtfTknData {ntfTknId} = do - -- tknRow <- ntfTknToRow <$> mkTknRec tkn - -- (DB.execute db insertNtfTknQuery tknRow >>= pure . (n + )) `E.catch` \(e :: E.SomeException) -> - -- putStrLn ("Error inserting token " <> enc ntfTknId <> " " <> show e) $> n - importSubscriptions :: S.Set NtfTokenId -> M.Map SMPQueueNtf NtfSubscriptionId -> IO Int64 - importSubscriptions tIds subLookup = do - subs <- filterSubs . M.elems =<< readTVarIO (subscriptions stmStore) - srvIds <- importServers subs - putStrLn $ "Importing " <> show (length subs) <> " subscriptions..." - -- uncomment this line instead of the next to import subs one by one. - -- (sCnt, errTkns) <- withConnection s $ \db -> foldM (importSub db srvIds) (0, M.empty) subs - sCnt <- foldM (importSubs srvIds) 0 $ toChunks 500000 subs - checkCount "subscription" (length subs) sCnt - where - filterSubs allSubs = do - let subs = filter (\NtfSubData {tokenId} -> S.member tokenId tIds) allSubs - skipped = length allSubs - length subs - when (skipped /= 0) $ putStrLn $ "Skipped " <> show skipped <> " subscriptions of missing tokens" - let (removedSubTokens, removeSubs, dupQueues) = foldl' addSubToken (S.empty, S.empty, S.empty) subs - unless (null removeSubs) $ putStrLn $ "Skipped " <> show (S.size removeSubs) <> " duplicate subscriptions of " <> show (S.size removedSubTokens) <> " tokens for " <> show (S.size dupQueues) <> " queues" - pure $ filter (\NtfSubData {ntfSubId} -> S.notMember ntfSubId removeSubs) subs - where - addSubToken acc@(!stIds, !sIds, !qs) NtfSubData {ntfSubId, smpQueue, tokenId} = - case M.lookup smpQueue subLookup of - Just sId | sId /= ntfSubId -> - (S.insert tokenId stIds, S.insert ntfSubId sIds, S.insert smpQueue qs) - _ -> acc - importSubs srvIds !n subs = do - rows <- mapM (ntfSubRow srvIds) subs - cnt <- withConnection s $ \db -> DB.executeMany db insertNtfSubQuery $ L.toList rows - let n' = n + cnt - putStr $ "Imported " <> show n' <> " subscriptions" <> "\r" - hFlush stdout - pure n' - -- importSub db srvIds (!n, !errTkns) sub@NtfSubData {ntfSubId = sId, tokenId} = do - -- subRow <- ntfSubRow srvIds sub - -- E.try (DB.execute db insertNtfSubQuery subRow) >>= \case - -- Right i -> do - -- let n' = n + i - -- when (n' `mod` 100000 == 0) $ do - -- putStr $ "Imported " <> show n' <> " subscriptions" <> "\r" - -- hFlush stdout - -- pure (n', errTkns) - -- Left (e :: E.SomeException) -> do - -- when (n `mod` 100000 == 0) $ putStrLn "" - -- putStrLn $ "Error inserting subscription " <> enc sId <> " for token " <> enc tokenId <> " " <> show e - -- pure (n, M.alter (Just . maybe [sId] (sId :)) tokenId errTkns) - ntfSubRow srvIds sub = case M.lookup srv srvIds of - Just sId -> ntfSubToRow sId <$> mkSubRec sub - Nothing -> E.throwIO $ userError $ "no matching server ID for server " <> show srv - where - srv = ntfSubServer sub - importServers subs = do - sIds <- withConnection s $ \db -> map fromOnly <$> DB.returning db srvQuery (map srvToRow srvs) - void $ checkCount "server" (length srvs) (length sIds) - pure $ M.fromList $ zip srvs sIds - where - srvQuery = "INSERT INTO smp_servers (smp_host, smp_port, smp_keyhash) VALUES (?, ?, ?) RETURNING smp_server_id" - srvs = nubOrd $ map ntfSubServer subs - importLastNtfs :: S.Set NtfTokenId -> M.Map SMPQueueNtf NtfSubscriptionId -> IO Int64 - importLastNtfs tIds subLookup = do - ntfs <- readTVarIO (tokenLastNtfs stmStore) - ntfRows <- filterLastNtfRows ntfs - nCnt <- withConnection s $ \db -> DB.executeMany db lastNtfQuery ntfRows - checkCount "last notification" (length ntfRows) nCnt - where - lastNtfQuery = "INSERT INTO last_notifications(token_id, subscription_id, sent_at, nmsg_nonce, nmsg_data) VALUES (?,?,?,?,?)" - filterLastNtfRows ntfs = do - (skippedTkns, ntfCnt, (skippedQueues, ntfRows)) <- foldM lastNtfRows (S.empty, 0, (S.empty, [])) $ M.assocs ntfs - let skipped = ntfCnt - length ntfRows - when (skipped /= 0) $ putStrLn $ "Skipped last notifications " <> show skipped <> " for " <> show (S.size skippedTkns) <> " missing tokens and " <> show (S.size skippedQueues) <> " missing subscriptions with token present" - pure ntfRows - lastNtfRows (!stIds, !cnt, !acc) (tId, ntfVar) = do - ntfs <- L.toList <$> readTVarIO ntfVar - let cnt' = cnt + length ntfs - pure $ - if S.member tId tIds - then (stIds, cnt', foldl' ntfRow acc ntfs) - else (S.insert tId stIds, cnt', acc) - where - ntfRow (!qs, !rows) PNMessageData {smpQueue, ntfTs, nmsgNonce, encNMsgMeta} = case M.lookup smpQueue subLookup of - Just ntfSubId -> - let row = (tId, ntfSubId, systemToUTCTime ntfTs, nmsgNonce, Binary encNMsgMeta) - in (qs, row : rows) - Nothing -> (S.insert smpQueue qs, rows) - importNtfServiceIds = do - ss <- M.assocs <$> readTVarIO (ntfServices stmStore) - withConnection s $ \db -> DB.executeMany db serviceQuery $ map serviceToRow ss - where - serviceQuery = - [sql| - INSERT INTO smp_servers (smp_host, smp_port, smp_keyhash, ntf_service_id) - VALUES (?, ?, ?, ?) - ON CONFLICT (smp_host, smp_port, smp_keyhash) - DO UPDATE SET ntf_service_id = EXCLUDED.ntf_service_id - |] - serviceToRow (srv, serviceId) = srvToRow srv :. Only serviceId - checkCount name expected inserted - | fromIntegral expected == inserted = do - putStrLn $ "Imported " <> show inserted <> " " <> name <> "s." - pure inserted - | otherwise = do - putStrLn $ "Incorrect " <> name <> " count: expected " <> show expected <> ", imported " <> show inserted - putStrLn "Import aborted, fix data and repeat" - exitFailure - enc = B.unpack . B64.encode . unEntityId - -exportNtfDbStore :: NtfPostgresStore -> FilePath -> IO (Int, Int, Int) -exportNtfDbStore NtfPostgresStore {dbStoreLog = Nothing} _ = - putStrLn "Internal error: export requires store log" >> exitFailure -exportNtfDbStore NtfPostgresStore {dbStore = s, dbStoreLog = Just sl} lastNtfsFile = - (,,) <$> exportTokens <*> exportSubscriptions <*> exportLastNtfs - where - exportTokens = do - tCnt <- withConnection s $ \db -> DB.fold_ db ntfTknQuery 0 $ \ !i tkn -> - logCreateToken sl (rowToNtfTkn tkn) $> (i + 1) - putStrLn $ "Exported " <> show tCnt <> " tokens" - pure tCnt - exportSubscriptions = do - sCnt <- withConnection s $ \db -> DB.fold_ db ntfSubQuery 0 $ \ !i sub -> do - let i' = i + 1 - logCreateSubscription sl (toNtfSub sub) - when (i' `mod` 500000 == 0) $ do - putStr $ "Exported " <> show i' <> " subscriptions" <> "\r" - hFlush stdout - pure i' - putStrLn $ "Exported " <> show sCnt <> " subscriptions" - pure sCnt - where - ntfSubQuery = - [sql| - SELECT s.token_id, s.subscription_id, s.smp_notifier_key, s.status, s.ntf_service_assoc, - p.smp_host, p.smp_port, p.smp_keyhash, s.smp_notifier_id - FROM subscriptions s - JOIN smp_servers p ON p.smp_server_id = s.smp_server_id - |] - toNtfSub :: Only NtfTokenId :. NtfSubRow :. SMPQueueNtfRow -> NtfSubRec - toNtfSub (Only tokenId :. (ntfSubId, notifierKey, subStatus, ntfServiceAssoc) :. qRow) = - let smpQueue = rowToSMPQueue qRow - in NtfSubRec {ntfSubId, tokenId, smpQueue, notifierKey, subStatus, ntfServiceAssoc} - exportLastNtfs = - withFile lastNtfsFile WriteMode $ \h -> - withConnection s $ \db -> DB.fold_ db lastNtfsQuery 0 $ \ !i (Only tknId :. ntfRow) -> - B.hPutStr h (encodeLastNtf tknId $ toLastNtf ntfRow) $> (i + 1) - where - -- Note that the order here is ascending, to be compatible with how it is imported - lastNtfsQuery = - [sql| - SELECT s.token_id, p.smp_host, p.smp_port, p.smp_keyhash, s.smp_notifier_id, - n.sent_at, n.nmsg_nonce, n.nmsg_data - FROM last_notifications n - JOIN subscriptions s ON s.subscription_id = n.subscription_id - JOIN smp_servers p ON p.smp_server_id = s.smp_server_id - ORDER BY token_ntf_id ASC - |] - encodeLastNtf tknId ntf = strEncode (TNMRv1 tknId ntf) `B.snoc` '\n' - withFastDB' :: Text -> NtfPostgresStore -> (DB.Connection -> IO a) -> IO (Either ErrorType a) withFastDB' op st action = withFastDB op st $ fmap Right . action {-# INLINE withFastDB' #-} @@ -924,9 +756,12 @@ withDB_ op st priority action = where err = op <> ", withDB, " <> tshow e -withLog :: MonadIO m => Text -> NtfPostgresStore -> (StoreLog 'WriteMode -> IO ()) -> m () -withLog op NtfPostgresStore {dbStoreLog} = withLog_ op dbStoreLog -{-# INLINE withLog #-} +withClientDB :: Text -> NtfPostgresStore -> (DB.Connection -> IO a) -> IO (Either SMPClientError a) +withClientDB op st action = + E.uninterruptibleMask_ $ E.try (withTransaction (dbStore st) action) >>= bimapM logErr pure + where + logErr :: E.SomeException -> IO SMPClientError + logErr e = logError ("STORE: " <> op <> ", withDB, " <> tshow e) $> PCEIOError (E.displayException e) assertUpdated :: Int64 -> Either ErrorType () assertUpdated 0 = Left AUTH @@ -964,4 +799,9 @@ instance ToField C.KeyHash where toField = toField . Binary . strEncode instance FromField C.CbNonce where fromField = blobFieldDecoder $ parseAll smpP instance ToField C.CbNonce where toField = toField . Binary . smpEncode + +instance ToField X.PrivKey where toField = toField . Binary . C.encodeASNObj + +instance FromField X.PrivKey where + fromField = blobFieldDecoder $ C.decodeASNKey >=> \case (pk, []) -> Right pk; r -> C.asnKeyError r #endif diff --git a/src/Simplex/Messaging/Notifications/Server/Store/ntf_server_schema.sql b/src/Simplex/Messaging/Notifications/Server/Store/ntf_server_schema.sql index 3b155fa1a..801208aaa 100644 --- a/src/Simplex/Messaging/Notifications/Server/Store/ntf_server_schema.sql +++ b/src/Simplex/Messaging/Notifications/Server/Store/ntf_server_schema.sql @@ -15,6 +15,123 @@ SET row_security = off; CREATE SCHEMA ntf_server; + +CREATE FUNCTION ntf_server.on_subscription_delete() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF OLD.ntf_service_assoc = true AND should_subscribe_status(OLD.status) THEN + PERFORM update_aggregates(OLD.smp_server_id, -1, OLD.smp_notifier_id); + END IF; + RETURN OLD; +END; +$$; + + + +CREATE FUNCTION ntf_server.on_subscription_insert() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF NEW.ntf_service_assoc = true AND should_subscribe_status(NEW.status) THEN + PERFORM update_aggregates(NEW.smp_server_id, 1, NEW.smp_notifier_id); + END IF; + RETURN NEW; +END; +$$; + + + +CREATE FUNCTION ntf_server.on_subscription_update() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF OLD.ntf_service_assoc = true AND should_subscribe_status(OLD.status) THEN + IF NOT (NEW.ntf_service_assoc = true AND should_subscribe_status(NEW.status)) THEN + PERFORM update_aggregates(OLD.smp_server_id, -1, OLD.smp_notifier_id); + END IF; + ELSIF NEW.ntf_service_assoc = true AND should_subscribe_status(NEW.status) THEN + PERFORM update_aggregates(NEW.smp_server_id, 1, NEW.smp_notifier_id); + END IF; + RETURN NEW; +END; +$$; + + + +CREATE FUNCTION ntf_server.should_subscribe_status(p_status text) RETURNS boolean + LANGUAGE plpgsql IMMUTABLE STRICT + AS $$ +BEGIN + RETURN p_status IN ('NEW', 'PENDING', 'ACTIVE', 'INACTIVE'); +END; +$$; + + + +CREATE FUNCTION ntf_server.update_aggregates(p_server_id bigint, p_change bigint, p_notifier_id bytea) RETURNS void + LANGUAGE plpgsql + AS $$ +BEGIN + UPDATE smp_servers + SET smp_notifier_count = smp_notifier_count + p_change, + smp_notifier_ids_hash = xor_combine(smp_notifier_ids_hash, public.digest(p_notifier_id, 'md5')) + WHERE smp_server_id = p_server_id; +END; +$$; + + + +CREATE FUNCTION ntf_server.update_all_aggregates() RETURNS void + LANGUAGE plpgsql + AS $$ +BEGIN + WITH acc AS ( + SELECT + s.smp_server_id, + count(smp_notifier_id) as notifier_count, + xor_aggregate(public.digest(s.smp_notifier_id, 'md5')) AS notifier_hash + FROM subscriptions s + WHERE s.ntf_service_assoc = true AND should_subscribe_status(s.status) + GROUP BY s.smp_server_id + ) + UPDATE smp_servers srv + SET smp_notifier_count = COALESCE(acc.notifier_count, 0), + smp_notifier_ids_hash = COALESCE(acc.notifier_hash, '\x00000000000000000000000000000000') + FROM acc + WHERE srv.smp_server_id = acc.smp_server_id; +END; +$$; + + + +CREATE FUNCTION ntf_server.xor_combine(state bytea, value bytea) RETURNS bytea + LANGUAGE plpgsql IMMUTABLE STRICT + AS $$ +DECLARE + result BYTEA := state; + i INTEGER; + len INTEGER := octet_length(value); +BEGIN + IF octet_length(state) != len THEN + RAISE EXCEPTION 'Inputs must be equal length (% != %)', octet_length(state), len; + END IF; + FOR i IN 0..len-1 LOOP + result := set_byte(result, i, get_byte(state, i) # get_byte(value, i)); + END LOOP; + RETURN result; +END; +$$; + + + +CREATE AGGREGATE ntf_server.xor_aggregate(bytea) ( + SFUNC = ntf_server.xor_combine, + STYPE = bytea, + INITCOND = '\x00000000000000000000000000000000' +); + + SET default_table_access_method = heap; @@ -53,7 +170,12 @@ CREATE TABLE ntf_server.smp_servers ( smp_host text NOT NULL, smp_port text NOT NULL, smp_keyhash bytea NOT NULL, - ntf_service_id bytea + ntf_service_id bytea, + smp_notifier_count bigint DEFAULT 0 NOT NULL, + smp_notifier_ids_hash bytea DEFAULT '\x00000000000000000000000000000000'::bytea NOT NULL, + ntf_service_cert bytea, + ntf_service_cert_hash bytea, + ntf_service_priv_key bytea ); @@ -158,6 +280,18 @@ CREATE INDEX idx_tokens_status_cron_interval_sent_at ON ntf_server.tokens USING +CREATE TRIGGER tr_subscriptions_delete AFTER DELETE ON ntf_server.subscriptions FOR EACH ROW EXECUTE FUNCTION ntf_server.on_subscription_delete(); + + + +CREATE TRIGGER tr_subscriptions_insert AFTER INSERT ON ntf_server.subscriptions FOR EACH ROW EXECUTE FUNCTION ntf_server.on_subscription_insert(); + + + +CREATE TRIGGER tr_subscriptions_update AFTER UPDATE ON ntf_server.subscriptions FOR EACH ROW EXECUTE FUNCTION ntf_server.on_subscription_update(); + + + ALTER TABLE ONLY ntf_server.last_notifications ADD CONSTRAINT last_notifications_subscription_id_fkey FOREIGN KEY (subscription_id) REFERENCES ntf_server.subscriptions(subscription_id) ON UPDATE RESTRICT ON DELETE CASCADE; diff --git a/src/Simplex/Messaging/Notifications/Server/StoreLog.hs b/src/Simplex/Messaging/Notifications/Server/StoreLog.hs deleted file mode 100644 index 7c71ddb08..000000000 --- a/src/Simplex/Messaging/Notifications/Server/StoreLog.hs +++ /dev/null @@ -1,177 +0,0 @@ -{-# LANGUAGE DataKinds #-} -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE GADTs #-} -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE NamedFieldPuns #-} -{-# LANGUAGE OverloadedStrings #-} -{-# LANGUAGE StrictData #-} -{-# OPTIONS_GHC -fno-warn-ambiguous-fields #-} - -module Simplex.Messaging.Notifications.Server.StoreLog - ( StoreLog, - NtfStoreLogRecord (..), - readWriteNtfSTMStore, - logCreateToken, - logTokenStatus, - logUpdateToken, - logTokenCron, - logDeleteToken, - logUpdateTokenTime, - logCreateSubscription, - logSubscriptionStatus, - logDeleteSubscription, - closeStoreLog, - ) -where - -import Control.Applicative (optional, (<|>)) -import Control.Concurrent.STM -import Control.Monad -import qualified Data.Attoparsec.ByteString.Char8 as A -import qualified Data.ByteString.Base64.URL as B64 -import qualified Data.ByteString.Char8 as B -import Data.Functor (($>)) -import qualified Data.Map.Strict as M -import Data.Maybe (fromMaybe) -import Data.Word (Word16) -import Simplex.Messaging.Encoding.String -import Simplex.Messaging.Notifications.Protocol -import Simplex.Messaging.Notifications.Server.Store -import Simplex.Messaging.Notifications.Server.Store.Types -import Simplex.Messaging.Protocol (EntityId (..), SMPServer, ServiceId) -import Simplex.Messaging.Server.StoreLog -import Simplex.Messaging.SystemTime -import System.IO - -data NtfStoreLogRecord - = CreateToken NtfTknRec - | TokenStatus NtfTokenId NtfTknStatus - | UpdateToken NtfTokenId DeviceToken NtfRegCode - | TokenCron NtfTokenId Word16 - | DeleteToken NtfTokenId - | UpdateTokenTime NtfTokenId SystemDate - | CreateSubscription NtfSubRec - | SubscriptionStatus NtfSubscriptionId NtfSubStatus NtfAssociatedService - | DeleteSubscription NtfSubscriptionId - | SetNtfService SMPServer (Maybe ServiceId) - deriving (Show) - -instance StrEncoding NtfStoreLogRecord where - strEncode = \case - CreateToken tknRec -> strEncode (Str "TCREATE", tknRec) - TokenStatus tknId tknStatus -> strEncode (Str "TSTATUS", tknId, tknStatus) - UpdateToken tknId token regCode -> strEncode (Str "TUPDATE", tknId, token, regCode) - TokenCron tknId cronInt -> strEncode (Str "TCRON", tknId, cronInt) - DeleteToken tknId -> strEncode (Str "TDELETE", tknId) - UpdateTokenTime tknId ts -> strEncode (Str "TTIME", tknId, ts) - CreateSubscription subRec -> strEncode (Str "SCREATE", subRec) - SubscriptionStatus subId subStatus serviceAssoc -> strEncode (Str "SSTATUS", subId, subStatus) <> serviceStr - where - serviceStr = if serviceAssoc then " service=" <> strEncode True else "" - DeleteSubscription subId -> strEncode (Str "SDELETE", subId) - SetNtfService srv serviceId -> strEncode (Str "SERVICE", srv) <> " service=" <> maybe "off" strEncode serviceId - strP = - A.choice - [ "TCREATE " *> (CreateToken <$> strP), - "TSTATUS " *> (TokenStatus <$> strP_ <*> strP), - "TUPDATE " *> (UpdateToken <$> strP_ <*> strP_ <*> strP), - "TCRON " *> (TokenCron <$> strP_ <*> strP), - "TDELETE " *> (DeleteToken <$> strP), - "TTIME " *> (UpdateTokenTime <$> strP_ <*> strP), - "SCREATE " *> (CreateSubscription <$> strP), - "SSTATUS " *> (SubscriptionStatus <$> strP_ <*> strP <*> (fromMaybe False <$> optional (" service=" *> strP))), - "SDELETE " *> (DeleteSubscription <$> strP), - "SERVICE " *> (SetNtfService <$> strP <* " service=" <*> ("off" $> Nothing <|> strP)) - ] - -logNtfStoreRecord :: StoreLog 'WriteMode -> NtfStoreLogRecord -> IO () -logNtfStoreRecord = writeStoreLogRecord -{-# INLINE logNtfStoreRecord #-} - -logCreateToken :: StoreLog 'WriteMode -> NtfTknRec -> IO () -logCreateToken s = logNtfStoreRecord s . CreateToken - -logTokenStatus :: StoreLog 'WriteMode -> NtfTokenId -> NtfTknStatus -> IO () -logTokenStatus s tknId tknStatus = logNtfStoreRecord s $ TokenStatus tknId tknStatus - -logUpdateToken :: StoreLog 'WriteMode -> NtfTokenId -> DeviceToken -> NtfRegCode -> IO () -logUpdateToken s tknId token regCode = logNtfStoreRecord s $ UpdateToken tknId token regCode - -logTokenCron :: StoreLog 'WriteMode -> NtfTokenId -> Word16 -> IO () -logTokenCron s tknId cronInt = logNtfStoreRecord s $ TokenCron tknId cronInt - -logDeleteToken :: StoreLog 'WriteMode -> NtfTokenId -> IO () -logDeleteToken s tknId = logNtfStoreRecord s $ DeleteToken tknId - -logUpdateTokenTime :: StoreLog 'WriteMode -> NtfTokenId -> SystemDate -> IO () -logUpdateTokenTime s tknId t = logNtfStoreRecord s $ UpdateTokenTime tknId t - -logCreateSubscription :: StoreLog 'WriteMode -> NtfSubRec -> IO () -logCreateSubscription s = logNtfStoreRecord s . CreateSubscription - -logSubscriptionStatus :: StoreLog 'WriteMode -> (NtfSubscriptionId, NtfSubStatus, NtfAssociatedService) -> IO () -logSubscriptionStatus s (subId, subStatus, serviceAssoc) = logNtfStoreRecord s $ SubscriptionStatus subId subStatus serviceAssoc - -logDeleteSubscription :: StoreLog 'WriteMode -> NtfSubscriptionId -> IO () -logDeleteSubscription s subId = logNtfStoreRecord s $ DeleteSubscription subId - -logSetNtfService :: StoreLog 'WriteMode -> SMPServer -> Maybe ServiceId -> IO () -logSetNtfService s srv serviceId = logNtfStoreRecord s $ SetNtfService srv serviceId - -readWriteNtfSTMStore :: Bool -> FilePath -> NtfSTMStore -> IO (StoreLog 'WriteMode) -readWriteNtfSTMStore tty = readWriteStoreLog (readNtfStore tty) writeNtfStore - -readNtfStore :: Bool -> FilePath -> NtfSTMStore -> IO () -readNtfStore tty f st = readLogLines tty f $ \_ -> processLine - where - processLine s = either printError procNtfLogRecord (strDecode s) - where - printError e = B.putStrLn $ "Error parsing log: " <> B.pack e <> " - " <> B.take 100 s - procNtfLogRecord = \case - CreateToken r@NtfTknRec {ntfTknId} -> do - tkn <- mkTknData r - atomically $ stmAddNtfToken st ntfTknId tkn - TokenStatus tknId status -> do - tkn_ <- stmGetNtfTokenIO st tknId - forM_ tkn_ $ \tkn@NtfTknData {tknStatus} -> do - atomically $ writeTVar tknStatus status - when (status == NTActive) $ void $ atomically $ stmRemoveInactiveTokenRegistrations st tkn - UpdateToken tknId token' tknRegCode -> do - stmGetNtfTokenIO st tknId - >>= mapM_ - ( \tkn@NtfTknData {tknStatus} -> do - atomically $ stmRemoveTokenRegistration st tkn - atomically $ writeTVar tknStatus NTRegistered - atomically $ stmAddNtfToken st tknId tkn {token = token', tknRegCode} - ) - TokenCron tknId cronInt -> - stmGetNtfTokenIO st tknId - >>= mapM_ (\NtfTknData {tknCronInterval} -> atomically $ writeTVar tknCronInterval cronInt) - DeleteToken tknId -> - atomically $ void $ stmDeleteNtfToken st tknId - UpdateTokenTime tknId t -> - stmGetNtfTokenIO st tknId - >>= mapM_ (\NtfTknData {tknUpdatedAt} -> atomically $ writeTVar tknUpdatedAt $ Just t) - CreateSubscription r@NtfSubRec {tokenId, ntfSubId} -> do - sub <- mkSubData r - atomically (stmAddNtfSubscription st ntfSubId sub) >>= \case - Just () -> pure () - Nothing -> B.putStrLn $ "Warning: no token " <> enc tokenId <> ", subscription " <> enc ntfSubId - where - enc = B64.encode . unEntityId - SubscriptionStatus subId status serviceAssoc -> do - stmGetNtfSubscriptionIO st subId >>= mapM_ update - where - update NtfSubData {subStatus, ntfServiceAssoc} = atomically $ do - writeTVar subStatus status - writeTVar ntfServiceAssoc serviceAssoc - DeleteSubscription subId -> - atomically $ stmDeleteNtfSubscription st subId - SetNtfService srv serviceId -> - atomically $ stmSetNtfService st srv serviceId - -writeNtfStore :: StoreLog 'WriteMode -> NtfSTMStore -> IO () -writeNtfStore s NtfSTMStore {tokens, subscriptions, ntfServices} = do - mapM_ (logCreateToken s <=< mkTknRec) =<< readTVarIO tokens - mapM_ (logCreateSubscription s <=< mkSubRec) =<< readTVarIO subscriptions - mapM_ (\(srv, serviceId) -> logSetNtfService s srv $ Just serviceId) . M.assocs =<< readTVarIO ntfServices diff --git a/src/Simplex/Messaging/Protocol.hs b/src/Simplex/Messaging/Protocol.hs index cb4b5aa2b..fa58d8843 100644 --- a/src/Simplex/Messaging/Protocol.hs +++ b/src/Simplex/Messaging/Protocol.hs @@ -140,6 +140,15 @@ module Simplex.Messaging.Protocol RcvMessage (..), MsgId, MsgBody, + IdsHash (..), + ServiceSub (..), + ServiceSubResult (..), + ServiceSubError (..), + serviceSubResult, + queueIdsHash, + queueIdHash, + addServiceSubs, + subtractServiceSubs, MaxMessageLen, MaxRcvMessageLen, EncRcvMsgBody (..), @@ -223,6 +232,8 @@ import qualified Data.Aeson.TH as J import Data.Attoparsec.ByteString.Char8 (Parser, ()) import qualified Data.Attoparsec.ByteString.Char8 as A import Data.Bifunctor (bimap, first) +import Data.Bits (xor) +import qualified Data.ByteString as BS import qualified Data.ByteString.Base64 as B64 import Data.ByteString.Char8 (ByteString) import qualified Data.ByteString.Char8 as B @@ -232,6 +243,7 @@ import Data.Constraint (Dict (..)) import Data.Functor (($>)) import Data.Int (Int64) import Data.Kind +import Data.List (foldl') import Data.List.NonEmpty (NonEmpty (..)) import qualified Data.List.NonEmpty as L import Data.Maybe (isJust, isNothing) @@ -241,7 +253,7 @@ import qualified Data.Text as T import Data.Text.Encoding (decodeLatin1, encodeUtf8) import Data.Time.Clock.System (SystemTime (..), systemToUTCTime) import Data.Type.Equality -import Data.Word (Word16) +import Data.Word (Word8, Word16) import GHC.TypeLits (ErrorMessage (..), TypeError, type (+)) import qualified GHC.TypeLits as TE import qualified GHC.TypeLits as Type @@ -548,7 +560,8 @@ data Command (p :: Party) where NEW :: NewQueueReq -> Command Creator SUB :: Command Recipient -- | subscribe all associated queues. Service ID must be used as entity ID, and service session key must sign the command. - SUBS :: Command RecipientService + -- Parameters are expected queue count and hash of all subscribed queues, it allows to monitor "state drift" on the server + SUBS :: Int64 -> IdsHash -> Command RecipientService KEY :: SndPublicAuthKey -> Command Recipient RKEY :: NonEmpty RcvPublicAuthKey -> Command Recipient LSET :: LinkId -> QueueLinkData -> Command Recipient @@ -572,7 +585,7 @@ data Command (p :: Party) where -- SMP notification subscriber commands NSUB :: Command Notifier -- | subscribe all associated queues. Service ID must be used as entity ID, and service session key must sign the command. - NSUBS :: Command NotifierService + NSUBS :: Int64 -> IdsHash -> Command NotifierService PRXY :: SMPServer -> Maybe BasicAuth -> Command ProxiedClient -- request a relay server connection by URI -- Transmission to proxy: -- - entity ID: ID of the session with relay returned in PKEY (response to PRXY) @@ -600,7 +613,7 @@ data NewQueueReq = NewQueueReq data SubscriptionMode = SMSubscribe | SMOnlyCreate deriving (Eq, Show) --- SenderId must be computed client-side as `sha3-256(corr_id)`, `corr_id` - a random transmission ID. +-- SenderId must be computed client-side as the first 24 bytes of `sha3-384(corr_id)`, `corr_id` - a random transmission ID. -- The server must verify and reject it if it does not match (and in case of collision). -- This allows to include SenderId in FixedDataBytes in full connection request, -- and at the same time prevents the possibility of checking whether a queue with a known ID exists. @@ -698,12 +711,14 @@ data BrokerMsg where LNK :: SenderId -> QueueLinkData -> BrokerMsg -- | Service subscription success - confirms when queue was associated with the service SOK :: Maybe ServiceId -> BrokerMsg - -- | The number of queues subscribed with SUBS command - SOKS :: Int64 -> BrokerMsg + -- | The number of queues and XOR-hash of their IDs subscribed with SUBS command + SOKS :: Int64 -> IdsHash -> BrokerMsg -- MSG v1/2 has to be supported for encoding/decoding -- v1: MSG :: MsgId -> SystemTime -> MsgBody -> BrokerMsg -- v2: MsgId -> SystemTime -> MsgFlags -> MsgBody -> BrokerMsg MSG :: RcvMessage -> BrokerMsg + -- sent once delivering messages to SUBS command is complete + ALLS :: BrokerMsg NID :: NotifierId -> RcvNtfPublicDhKey -> BrokerMsg NMSG :: C.CbNonce -> EncNMsgMeta -> BrokerMsg -- Should include certificate chain @@ -711,7 +726,7 @@ data BrokerMsg where RRES :: EncFwdResponse -> BrokerMsg -- relay to proxy PRES :: EncResponse -> BrokerMsg -- proxy to client END :: BrokerMsg - ENDS :: Int64 -> BrokerMsg + ENDS :: Int64 -> IdsHash -> BrokerMsg DELD :: BrokerMsg INFO :: QueueInfo -> BrokerMsg OK :: BrokerMsg @@ -940,6 +955,7 @@ data BrokerMsgTag | SOK_ | SOKS_ | MSG_ + | ALLS_ | NID_ | NMSG_ | PKEY_ @@ -1032,6 +1048,7 @@ instance Encoding BrokerMsgTag where SOK_ -> "SOK" SOKS_ -> "SOKS" MSG_ -> "MSG" + ALLS_ -> "ALLS" NID_ -> "NID" NMSG_ -> "NMSG" PKEY_ -> "PKEY" @@ -1053,6 +1070,7 @@ instance ProtocolMsgTag BrokerMsgTag where "SOK" -> Just SOK_ "SOKS" -> Just SOKS_ "MSG" -> Just MSG_ + "ALLS" -> Just ALLS_ "NID" -> Just NID_ "NMSG" -> Just NMSG_ "PKEY" -> Just PKEY_ @@ -1455,6 +1473,66 @@ type MsgId = ByteString -- | SMP message body. type MsgBody = ByteString +data ServiceSub = ServiceSub + { smpServiceId :: ServiceId, + smpQueueCount :: Int64, + smpQueueIdsHash :: IdsHash + } + deriving (Eq, Show) + +data ServiceSubResult = ServiceSubResult (Maybe ServiceSubError) ServiceSub + deriving (Eq, Show) + +data ServiceSubError + = SSErrorServiceId {expectedServiceId :: ServiceId, subscribedServiceId :: ServiceId} + | SSErrorQueueCount {expectedQueueCount :: Int64, subscribedQueueCount :: Int64} + | SSErrorQueueIdsHash {expectedQueueIdsHash :: IdsHash, subscribedQueueIdsHash :: IdsHash} + deriving (Eq, Show) + +serviceSubResult :: ServiceSub -> ServiceSub -> ServiceSubResult +serviceSubResult s s' = ServiceSubResult subError_ s' + where + subError_ + | smpServiceId s /= smpServiceId s' = Just $ SSErrorServiceId (smpServiceId s) (smpServiceId s') + | smpQueueCount s /= smpQueueCount s' = Just $ SSErrorQueueCount (smpQueueCount s) (smpQueueCount s') + | smpQueueIdsHash s /= smpQueueIdsHash s' = Just $ SSErrorQueueIdsHash (smpQueueIdsHash s) (smpQueueIdsHash s') + | otherwise = Nothing + +newtype IdsHash = IdsHash {unIdsHash :: BS.ByteString} + deriving (Eq, Show) + deriving newtype (Encoding, FromField) + +instance ToField IdsHash where + toField (IdsHash s) = toField (Binary s) + {-# INLINE toField #-} + +instance Semigroup IdsHash where + (IdsHash s1) <> (IdsHash s2) = IdsHash $! BS.pack $ BS.zipWith xor s1 s2 + +instance Monoid IdsHash where + mempty = IdsHash $ BS.replicate 16 0 + mconcat ss = + let !s' = BS.pack $ foldl' (\ !r (IdsHash s) -> zipWith xor' r (BS.unpack s)) (replicate 16 0) ss -- to prevent packing/unpacking in <> on each step with default mappend + in IdsHash s' + +xor' :: Word8 -> Word8 -> Word8 +xor' x y = let !r = xor x y in r + +queueIdsHash :: [QueueId] -> IdsHash +queueIdsHash = mconcat . map queueIdHash + +queueIdHash :: QueueId -> IdsHash +queueIdHash = IdsHash . C.md5Hash . unEntityId +{-# INLINE queueIdHash #-} + +addServiceSubs :: (Int64, IdsHash) -> (Int64, IdsHash) -> (Int64, IdsHash) +addServiceSubs (n', idsHash') (n, idsHash) = (n + n', idsHash <> idsHash') + +subtractServiceSubs :: (Int64, IdsHash) -> (Int64, IdsHash) -> (Int64, IdsHash) +subtractServiceSubs (n', idsHash') (n, idsHash) + | n > n' = (n - n', idsHash <> idsHash') -- concat is a reversible xor: (x `xor` y) `xor` y == x + | otherwise = (0, mempty) + data ProtocolErrorType = PECmdSyntax | PECmdUnknown | PESession | PEBlock -- | Type for protocol errors. @@ -1688,7 +1766,9 @@ instance PartyI p => ProtocolEncoding SMPVersion ErrorType (Command p) where new = e (NEW_, ' ', rKey, dhKey) auth = maybe "" (e . ('A',)) auth_ SUB -> e SUB_ - SUBS -> e SUBS_ + SUBS n idsHash + | v >= rcvServiceSMPVersion -> e (SUBS_, ' ', n, idsHash) + | otherwise -> e SUBS_ KEY k -> e (KEY_, ' ', k) RKEY ks -> e (RKEY_, ' ', ks) LSET lnkId d -> e (LSET_, ' ', lnkId, d) @@ -1704,7 +1784,9 @@ instance PartyI p => ProtocolEncoding SMPVersion ErrorType (Command p) where SEND flags msg -> e (SEND_, ' ', flags, ' ', Tail msg) PING -> e PING_ NSUB -> e NSUB_ - NSUBS -> e NSUBS_ + NSUBS n idsHash + | v >= rcvServiceSMPVersion -> e (NSUBS_, ' ', n, idsHash) + | otherwise -> e NSUBS_ LKEY k -> e (LKEY_, ' ', k) LGET -> e LGET_ PRXY host auth_ -> e (PRXY_, ' ', host, auth_) @@ -1795,7 +1877,9 @@ instance ProtocolEncoding SMPVersion ErrorType Cmd where OFF_ -> pure OFF DEL_ -> pure DEL QUE_ -> pure QUE - CT SRecipientService SUBS_ -> pure $ Cmd SRecipientService SUBS + CT SRecipientService SUBS_ + | v >= rcvServiceSMPVersion -> Cmd SRecipientService <$> (SUBS <$> _smpP <*> smpP) + | otherwise -> pure $ Cmd SRecipientService $ SUBS (-1) mempty CT SSender tag -> Cmd SSender <$> case tag of SKEY_ -> SKEY <$> _smpP @@ -1812,7 +1896,9 @@ instance ProtocolEncoding SMPVersion ErrorType Cmd where PFWD_ -> PFWD <$> _smpP <*> smpP <*> (EncTransmission . unTail <$> smpP) PRXY_ -> PRXY <$> _smpP <*> smpP CT SNotifier NSUB_ -> pure $ Cmd SNotifier NSUB - CT SNotifierService NSUBS_ -> pure $ Cmd SNotifierService NSUBS + CT SNotifierService NSUBS_ + | v >= rcvServiceSMPVersion -> Cmd SNotifierService <$> (NSUBS <$> _smpP <*> smpP) + | otherwise -> pure $ Cmd SNotifierService $ NSUBS (-1) mempty fromProtocolError = fromProtocolError @SMPVersion @ErrorType @BrokerMsg {-# INLINE fromProtocolError #-} @@ -1835,16 +1921,17 @@ instance ProtocolEncoding SMPVersion ErrorType BrokerMsg where SOK serviceId_ | v >= serviceCertsSMPVersion -> e (SOK_, ' ', serviceId_) | otherwise -> e OK_ -- won't happen, the association with the service requires v >= serviceCertsSMPVersion - SOKS n -> e (SOKS_, ' ', n) + SOKS n idsHash -> serviceResp SOKS_ n idsHash MSG RcvMessage {msgId, msgBody = EncRcvMsgBody body} -> e (MSG_, ' ', msgId, Tail body) + ALLS -> e ALLS_ NID nId srvNtfDh -> e (NID_, ' ', nId, srvNtfDh) NMSG nmsgNonce encNMsgMeta -> e (NMSG_, ' ', nmsgNonce, encNMsgMeta) PKEY sid vr certKey -> e (PKEY_, ' ', sid, vr, certKey) RRES (EncFwdResponse encBlock) -> e (RRES_, ' ', Tail encBlock) PRES (EncResponse encBlock) -> e (PRES_, ' ', Tail encBlock) END -> e END_ - ENDS n -> e (ENDS_, ' ', n) + ENDS n idsHash -> serviceResp ENDS_ n idsHash DELD | v >= deletedEventSMPVersion -> e DELD_ | otherwise -> e END_ @@ -1861,6 +1948,9 @@ instance ProtocolEncoding SMPVersion ErrorType BrokerMsg where where e :: Encoding a => a -> ByteString e = smpEncode + serviceResp tag n idsHash + | v >= rcvServiceSMPVersion = e (tag, ' ', n, idsHash) + | otherwise = e (tag, ' ', n) protocolP v = \case MSG_ -> do @@ -1868,6 +1958,7 @@ instance ProtocolEncoding SMPVersion ErrorType BrokerMsg where MSG . RcvMessage msgId <$> bodyP where bodyP = EncRcvMsgBody . unTail <$> smpP + ALLS_ -> pure ALLS IDS_ | v >= newNtfCredsSMPVersion -> ids smpP smpP smpP smpP | v >= serviceCertsSMPVersion -> ids smpP smpP smpP nothing @@ -1888,19 +1979,23 @@ instance ProtocolEncoding SMPVersion ErrorType BrokerMsg where pure $ IDS QIK {rcvId, sndId, rcvPublicDhKey, queueMode, linkId, serviceId, serverNtfCreds} LNK_ -> LNK <$> _smpP <*> smpP SOK_ -> SOK <$> _smpP - SOKS_ -> SOKS <$> _smpP + SOKS_ -> serviceRespP SOKS NID_ -> NID <$> _smpP <*> smpP NMSG_ -> NMSG <$> _smpP <*> smpP PKEY_ -> PKEY <$> _smpP <*> smpP <*> smpP RRES_ -> RRES <$> (EncFwdResponse . unTail <$> _smpP) PRES_ -> PRES <$> (EncResponse . unTail <$> _smpP) END_ -> pure END - ENDS_ -> ENDS <$> _smpP + ENDS_ -> serviceRespP ENDS DELD_ -> pure DELD INFO_ -> INFO <$> _smpP OK_ -> pure OK ERR_ -> ERR <$> _smpP PONG_ -> pure PONG + where + serviceRespP resp + | v >= rcvServiceSMPVersion = resp <$> _smpP <*> smpP + | otherwise = resp <$> _smpP <*> pure mempty fromProtocolError = \case PECmdSyntax -> CMD SYNTAX @@ -1918,6 +2013,7 @@ instance ProtocolEncoding SMPVersion ErrorType BrokerMsg where PONG -> noEntityMsg PKEY {} -> noEntityMsg RRES _ -> noEntityMsg + ALLS -> noEntityMsg -- other broker responses must have queue ID _ | B.null entId -> Left $ CMD NO_ENTITY diff --git a/src/Simplex/Messaging/Server.hs b/src/Simplex/Messaging/Server.hs index ec75a07d4..3d977dc8c 100644 --- a/src/Simplex/Messaging/Server.hs +++ b/src/Simplex/Messaging/Server.hs @@ -6,6 +6,7 @@ {-# LANGUAGE GADTs #-} {-# LANGUAGE KindSignatures #-} {-# LANGUAGE LambdaCase #-} +{-# LANGUAGE MultiWayIf #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE NumericUnderscores #-} {-# LANGUAGE OverloadedLists #-} @@ -45,6 +46,7 @@ module Simplex.Messaging.Server where import Control.Concurrent.STM (throwSTM) +import qualified Control.Exception as E import Control.Logger.Simple import Control.Monad import Control.Monad.Except @@ -95,7 +97,7 @@ import Network.Socket (ServiceName, Socket, socketToHandle) import qualified Network.TLS as TLS import Numeric.Natural (Natural) import Simplex.Messaging.Agent.Lock -import Simplex.Messaging.Client (ProtocolClient (thParams), ProtocolClientError (..), SMPClient, SMPClientError, forwardSMPTransmission, smpProxyError, temporaryClientError) +import Simplex.Messaging.Client (ProtocolClient (thParams), ProtocolClientError (..), SMPClient, SMPClientError, clientHandlers, forwardSMPTransmission, smpProxyError, temporaryClientError) import Simplex.Messaging.Client.Agent (OwnServer, SMPClientAgent (..), SMPClientAgentEvent (..), closeSMPClientAgent, getSMPServerClient'', isOwnServer, lookupSMPServerClient, getConnectedSMPServerClient) import qualified Simplex.Messaging.Crypto as C import Simplex.Messaging.Encoding @@ -165,8 +167,8 @@ type AttachHTTP = Socket -> TLS.Context -> IO () -- actions used in serverThread to reduce STM transaction scope data ClientSubAction = CSAEndSub QueueId -- end single direct queue subscription - | CSAEndServiceSub -- end service subscription to one queue - | CSADecreaseSubs Int64 -- reduce service subscriptions when cancelling. Fixed number is used to correctly handle race conditions when service resubscribes + | CSAEndServiceSub QueueId -- end service subscription to one queue + | CSADecreaseSubs (Int64, IdsHash) -- reduce service subscriptions when cancelling. Fixed number is used to correctly handle race conditions when service resubscribes type PrevClientSub s = (Client s, ClientSubAction, (EntityId, BrokerMsg)) @@ -250,7 +252,7 @@ smpServer started cfg@ServerConfig {transports, transportConfig = tCfg, startOpt Server s -> (Server s -> ServerSubscribers s) -> (Client s -> TMap QueueId sub) -> - (Client s -> TVar Int64) -> + (Client s -> TVar (Int64, IdsHash)) -> Maybe (sub -> IO ()) -> M s () serverThread label srv srvSubscribers clientSubs clientServiceSubs unsub_ = do @@ -276,7 +278,7 @@ smpServer started cfg@ServerConfig {transports, transportConfig = tCfg, startOpt as'' <- if prevServiceId == serviceId_ then pure [] else endServiceSub prevServiceId qId END case serviceId_ of Just serviceId -> do - modifyTVar' totalServiceSubs (+ 1) -- server count for all services + modifyTVar' totalServiceSubs $ addServiceSubs (1, queueIdHash qId) -- server count and IDs hash for all services as <- endQueueSub qId END as' <- cancelServiceSubs serviceId =<< upsertSubscribedClient serviceId c serviceSubscribers pure $ as ++ as' ++ as'' @@ -288,9 +290,9 @@ smpServer started cfg@ServerConfig {transports, transportConfig = tCfg, startOpt as <- endQueueSub qId DELD as' <- endServiceSub serviceId qId DELD pure $ as ++ as' - CSService serviceId count -> do + CSService serviceId changedSubs -> do modifyTVar' subClients $ IS.insert clntId -- add ID to server's subscribed cients - modifyTVar' totalServiceSubs (+ count) -- server count for all services + modifyTVar' totalServiceSubs $ subtractServiceSubs changedSubs -- server count and IDs hash for all services cancelServiceSubs serviceId =<< upsertSubscribedClient serviceId c serviceSubscribers updateSubDisconnected = case clntSub of -- do not insert client if it is already disconnected, but send END/DELD to any other client subscribed to this queue or service @@ -308,15 +310,15 @@ smpServer started cfg@ServerConfig {transports, transportConfig = tCfg, startOpt endQueueSub qId msg = prevSub qId msg (CSAEndSub qId) =<< lookupDeleteSubscribedClient qId queueSubscribers endServiceSub :: Maybe ServiceId -> QueueId -> BrokerMsg -> STM [PrevClientSub s] endServiceSub Nothing _ _ = pure [] - endServiceSub (Just serviceId) qId msg = prevSub qId msg CSAEndServiceSub =<< lookupSubscribedClient serviceId serviceSubscribers + endServiceSub (Just serviceId) qId msg = prevSub qId msg (CSAEndServiceSub qId) =<< lookupSubscribedClient serviceId serviceSubscribers prevSub :: QueueId -> BrokerMsg -> ClientSubAction -> Maybe (Client s) -> STM [PrevClientSub s] prevSub qId msg action = checkAnotherClient $ \c -> pure [(c, action, (qId, msg))] cancelServiceSubs :: ServiceId -> Maybe (Client s) -> STM [PrevClientSub s] cancelServiceSubs serviceId = checkAnotherClient $ \c -> do - n <- swapTVar (clientServiceSubs c) 0 - pure [(c, CSADecreaseSubs n, (serviceId, ENDS n))] + changedSubs@(n, idsHash) <- swapTVar (clientServiceSubs c) (0, mempty) + pure [(c, CSADecreaseSubs changedSubs, (serviceId, ENDS n idsHash))] checkAnotherClient :: (Client s -> STM [PrevClientSub s]) -> Maybe (Client s) -> STM [PrevClientSub s] checkAnotherClient mkSub = \case Just c@Client {clientId, connected} | clntId /= clientId -> @@ -331,20 +333,21 @@ smpServer started cfg@ServerConfig {transports, transportConfig = tCfg, startOpt where a (Just unsub) (Just s) = unsub s a _ _ = pure () - CSAEndServiceSub -> atomically $ do + CSAEndServiceSub qId -> atomically $ do modifyTVar' (clientServiceSubs c) decrease modifyTVar' totalServiceSubs decrease where - decrease n = max 0 (n - 1) - -- TODO [certs rcv] for SMP subscriptions CSADecreaseSubs should also remove all delivery threads of the passed client - CSADecreaseSubs n' -> atomically $ modifyTVar' totalServiceSubs $ \n -> max 0 (n - n') + decrease = subtractServiceSubs (1, queueIdHash qId) + CSADecreaseSubs changedSubs -> do + atomically $ modifyTVar' totalServiceSubs $ subtractServiceSubs changedSubs + forM_ unsub_ $ \unsub -> atomically (swapTVar (clientSubs c) M.empty) >>= mapM_ unsub where endSub :: Client s -> QueueId -> STM (Maybe sub) endSub c qId = TM.lookupDelete qId (clientSubs c) >>= (removeWhenNoSubs c $>) -- remove client from server's subscribed cients removeWhenNoSubs c = do noClientSubs <- null <$> readTVar (clientSubs c) - noServiceSubs <- (0 ==) <$> readTVar (clientServiceSubs c) + noServiceSubs <- ((0 ==) . fst) <$> readTVar (clientServiceSubs c) when (noClientSubs && noServiceSubs) $ modifyTVar' subClients $ IS.delete (clientId c) deliverNtfsThread :: Server s -> M s () @@ -922,7 +925,7 @@ smpServer started cfg@ServerConfig {transports, transportConfig = tCfg, startOpt putSubscribersInfo protoName ServerSubscribers {queueSubscribers, subClients} showIds = do activeSubs <- getSubscribedClients queueSubscribers hPutStrLn h $ protoName <> " subscriptions: " <> show (M.size activeSubs) - -- TODO [certs] service subscriptions + -- TODO [certs rcv] service subscriptions clnts <- countSubClients activeSubs hPutStrLn h $ protoName <> " subscribed clients: " <> show (IS.size clnts) <> (if showIds then " " <> show (IS.toList clnts) else "") clnts' <- readTVarIO subClients @@ -1111,10 +1114,10 @@ clientDisconnected c@Client {clientId, subscriptions, ntfSubscriptions, serviceS updateSubscribers subs ServerSubscribers {queueSubscribers, subClients} = do mapM_ (\qId -> deleteSubcribedClient qId c queueSubscribers) (M.keys subs) atomically $ modifyTVar' subClients $ IS.delete clientId - updateServiceSubs :: ServiceId -> TVar Int64 -> ServerSubscribers s -> IO () + updateServiceSubs :: ServiceId -> TVar (Int64, IdsHash) -> ServerSubscribers s -> IO () updateServiceSubs serviceId subsCount ServerSubscribers {totalServiceSubs, serviceSubscribers} = do deleteSubcribedClient serviceId c serviceSubscribers - atomically . modifyTVar' totalServiceSubs . subtract =<< readTVarIO subsCount + atomically . modifyTVar' totalServiceSubs . subtractServiceSubs =<< readTVarIO subsCount cancelSub :: Sub -> IO () cancelSub s = case subThread s of @@ -1247,7 +1250,7 @@ verifyQueueTransmission service thAuth (tAuth, authorized, (corrId, entId, comma vc SCreator (NEW NewQueueReq {rcvAuthKey = k}) = verifiedWith k vc SRecipient SUB = verifyQueue $ \q -> verifiedWithKeys $ recipientKeys (snd q) vc SRecipient _ = verifyQueue $ \q -> verifiedWithKeys $ recipientKeys (snd q) - vc SRecipientService SUBS = verifyServiceCmd + vc SRecipientService SUBS {} = verifyServiceCmd vc SSender (SKEY k) = verifySecure k -- SEND will be accepted without authorization before the queue is secured with KEY, SKEY or LSKEY command vc SSender SEND {} = verifyQueue $ \q -> if maybe (isNothing tAuth) verify (senderKey $ snd q) then VRVerified q_ else VRFailed AUTH @@ -1255,7 +1258,7 @@ verifyQueueTransmission service thAuth (tAuth, authorized, (corrId, entId, comma vc SSenderLink (LKEY k) = verifySecure k vc SSenderLink LGET = verifyQueue $ \q -> if isContactQueue (snd q) then VRVerified q_ else VRFailed AUTH vc SNotifier NSUB = verifyQueue $ \q -> maybe dummyVerify (\n -> verifiedWith $ notifierKey n) (notifier $ snd q) - vc SNotifierService NSUBS = verifyServiceCmd + vc SNotifierService NSUBS {} = verifyServiceCmd vc SProxiedClient _ = VRVerified Nothing vc SProxyService (RFWD _) = VRVerified Nothing checkRole = case (service, partyClientRole p) of @@ -1356,10 +1359,9 @@ forkClient Client {endThreads, endThreadSeq} label action = do client :: forall s. MsgStoreClass s => Server s -> s -> Client s -> M s () client - -- TODO [certs rcv] rcv subscriptions Server {subscribers, ntfSubscribers} ms - clnt@Client {clientId, ntfSubscriptions, ntfServiceSubscribed, serviceSubsCount = _todo', ntfServiceSubsCount, rcvQ, sndQ, clientTHParams = thParams'@THandleParams {sessionId}, procThreads} = do + clnt@Client {clientId, rcvQ, sndQ, msgQ, clientTHParams = thParams'@THandleParams {sessionId}, procThreads} = do labelMyThread . B.unpack $ "client $" <> encode sessionId <> " commands" let THandleParams {thVersion} = thParams' clntServiceId = (\THClientService {serviceId} -> serviceId) <$> (peerClientService =<< thAuth thParams') @@ -1384,7 +1386,7 @@ client Just r -> Just <$> proxyServerResponse a r Nothing -> forkProxiedCmd $ - liftIO (runExceptT (getSMPServerClient'' a srv) `catch` (pure . Left . PCEIOError)) + liftIO (runExceptT (getSMPServerClient'' a srv) `E.catches` clientHandlers) >>= proxyServerResponse a proxyServerResponse :: SMPClientAgent 'Sender -> Either SMPClientError (OwnServer, SMPClient) -> M s BrokerMsg proxyServerResponse a smp_ = do @@ -1421,7 +1423,7 @@ client inc own pRequests if v >= sendingProxySMPVersion then forkProxiedCmd $ do - liftIO (runExceptT (forwardSMPTransmission smp corrId fwdV pubKey encBlock) `catch` (pure . Left . PCEIOError)) >>= \case + liftIO (runExceptT (forwardSMPTransmission smp corrId fwdV pubKey encBlock) `E.catches` clientHandlers) >>= \case Right r -> PRES r <$ inc own pSuccesses Left e -> ERR (smpProxyError e) <$ case e of PCEProtocolError {} -> inc own pSuccesses @@ -1465,8 +1467,8 @@ client Cmd SNotifier NSUB -> response . (corrId,entId,) <$> case q_ of Just (q, QueueRec {notifier = Just ntfCreds}) -> subscribeNotifications q ntfCreds _ -> pure $ ERR INTERNAL - Cmd SNotifierService NSUBS -> response . (corrId,entId,) <$> case clntServiceId of - Just serviceId -> subscribeServiceNotifications serviceId + Cmd SNotifierService (NSUBS n idsHash) -> response . (corrId,entId,) <$> case clntServiceId of + Just serviceId -> subscribeServiceNotifications serviceId (n, idsHash) Nothing -> pure $ ERR INTERNAL Cmd SCreator (NEW nqr@NewQueueReq {auth_}) -> response <$> ifM allowNew (createQueue nqr) (pure (corrId, entId, ERR AUTH)) @@ -1495,7 +1497,9 @@ client OFF -> response <$> maybe (pure $ err INTERNAL) suspendQueue_ q_ DEL -> response <$> maybe (pure $ err INTERNAL) delQueueAndMsgs q_ QUE -> withQueue $ \q qr -> (corrId,entId,) <$> getQueueInfo q qr - Cmd SRecipientService SUBS -> pure $ response $ err (CMD PROHIBITED) -- "TODO [certs rcv]" + Cmd SRecipientService (SUBS n idsHash)-> response . (corrId,entId,) <$> case clntServiceId of + Just serviceId -> subscribeServiceMessages serviceId (n, idsHash) + Nothing -> pure $ ERR INTERNAL -- it's "internal" because it should never get to this branch where createQueue :: NewQueueReq -> M s (Transmission BrokerMsg) createQueue NewQueueReq {rcvAuthKey, rcvDhKey, subMode, queueReqData, ntfCreds} @@ -1615,11 +1619,13 @@ client suspendQueue_ :: (StoreQueue s, QueueRec) -> M s (Transmission BrokerMsg) suspendQueue_ (q, _) = liftIO $ either err (const ok) <$> suspendQueue (queueStore ms) q - -- TODO [certs rcv] if serviceId is passed, associate with the service and respond with SOK subscribeQueueAndDeliver :: StoreQueue s -> QueueRec -> M s ResponseAndMessage - subscribeQueueAndDeliver q qr = + subscribeQueueAndDeliver q qr@QueueRec {rcvServiceId} = liftIO (TM.lookupIO entId $ subscriptions clnt) >>= \case - Nothing -> subscribeRcvQueue qr >>= deliver False + Nothing -> + sharedSubscribeQueue q SRecipientService rcvServiceId subscribers subscriptions serviceSubsCount (newSubscription NoSub) rcvServices >>= \case + Left e -> pure (err e, Nothing) + Right s -> deliver s Just s@Sub {subThread} -> do stats <- asks serverStats case subThread of @@ -1629,32 +1635,34 @@ client pure (err (CMD PROHIBITED), Nothing) _ -> do incStat $ qSubDuplicate stats - atomically (writeTVar (delivered s) Nothing) >> deliver True s + atomically (writeTVar (delivered s) Nothing) >> deliver (True, Just s) where - deliver :: Bool -> Sub -> M s ResponseAndMessage - deliver hasSub sub = do + deliver :: (Bool, Maybe Sub) -> M s ResponseAndMessage + deliver (hasSub, sub_) = do stats <- asks serverStats fmap (either ((,Nothing) . err) id) $ liftIO $ runExceptT $ do msg_ <- tryPeekMsg ms q msg' <- forM msg_ $ \msg -> liftIO $ do ts <- getSystemSeconds + sub <- maybe (atomically getSub) pure sub_ atomically $ setDelivered sub msg ts unless hasSub $ incStat $ qSub stats pure (NoCorrId, entId, MSG (encryptMsg qr msg)) pure ((corrId, entId, SOK clntServiceId), msg') - -- TODO [certs rcv] combine with subscribing ntf queues - subscribeRcvQueue :: QueueRec -> M s Sub - subscribeRcvQueue QueueRec {rcvServiceId} = atomically $ do - writeTQueue (subQ subscribers) (CSClient entId rcvServiceId Nothing, clientId) - sub <- newSubscription NoSub - TM.insert entId sub $ subscriptions clnt - pure sub + getSub :: STM Sub + getSub = + TM.lookup entId (subscriptions clnt) >>= \case + Just sub -> pure sub + Nothing -> do + sub <- newSubscription NoSub + TM.insert entId sub $ subscriptions clnt + pure sub subscribeNewQueue :: RecipientId -> QueueRec -> M s () subscribeNewQueue rId QueueRec {rcvServiceId} = do case rcvServiceId of - Just _ -> atomically $ modifyTVar' (serviceSubsCount clnt) (+ 1) + Just _ -> atomically $ modifyTVar' (serviceSubsCount clnt) $ addServiceSubs (1, queueIdHash rId) Nothing -> do sub <- atomically $ newSubscription NoSub atomically $ TM.insert rId sub $ subscriptions clnt @@ -1719,74 +1727,148 @@ client else liftIO (updateQueueTime (queueStore ms) q t) >>= either (pure . err') (action q) subscribeNotifications :: StoreQueue s -> NtfCreds -> M s BrokerMsg - subscribeNotifications q NtfCreds {ntfServiceId} = do + subscribeNotifications q NtfCreds {ntfServiceId} = + sharedSubscribeQueue q SNotifierService ntfServiceId ntfSubscribers ntfSubscriptions ntfServiceSubsCount (pure ()) ntfServices >>= \case + Left e -> pure $ ERR e + Right (hasSub, _) -> do + when (isNothing clntServiceId) $ + asks serverStats >>= incStat . (if hasSub then ntfSubDuplicate else ntfSub) + pure $ SOK clntServiceId + + sharedSubscribeQueue :: + (PartyI p, ServiceParty p) => + StoreQueue s -> + SParty p -> + Maybe ServiceId -> + ServerSubscribers s -> + (Client s -> TMap QueueId sub) -> + (Client s -> TVar (Int64, IdsHash)) -> + STM sub -> + (ServerStats -> ServiceStats) -> + M s (Either ErrorType (Bool, Maybe sub)) + sharedSubscribeQueue q party queueServiceId srvSubscribers clientSubs clientServiceSubs mkSub servicesSel = do stats <- asks serverStats - let incNtfSrvStat sel = incStat $ sel $ ntfServices stats - case clntServiceId of + let incSrvStat sel = incStat $ sel $ servicesSel stats + writeSub = writeTQueue (subQ srvSubscribers) (CSClient entId queueServiceId clntServiceId, clientId) + liftIO $ case clntServiceId of Just serviceId - | ntfServiceId == Just serviceId -> do + | queueServiceId == Just serviceId -> do -- duplicate queue-service association - can only happen in case of response error/timeout - hasSub <- atomically $ ifM hasServiceSub (pure True) (False <$ newServiceQueueSub) + hasSub <- atomically $ ifM hasServiceSub (pure True) (False <$ incServiceQueueSubs) unless hasSub $ do - incNtfSrvStat srvSubCount - incNtfSrvStat srvSubQueues - incNtfSrvStat srvAssocDuplicate - pure $ SOK $ Just serviceId - | otherwise -> + atomically writeSub + incSrvStat srvSubCount + incSrvStat srvSubQueues + incSrvStat srvAssocDuplicate + pure $ Right (hasSub, Nothing) + | otherwise -> runExceptT $ do -- new or updated queue-service association - liftIO (setQueueService (queueStore ms) q SNotifierService (Just serviceId)) >>= \case - Left e -> pure $ ERR e - Right () -> do - hasSub <- atomically $ (<$ newServiceQueueSub) =<< hasServiceSub - unless hasSub $ incNtfSrvStat srvSubCount - incNtfSrvStat srvSubQueues - incNtfSrvStat $ maybe srvAssocNew (const srvAssocUpdated) ntfServiceId - pure $ SOK $ Just serviceId + ExceptT $ setQueueService (queueStore ms) q party (Just serviceId) + hasSub <- atomically $ (<$ incServiceQueueSubs) =<< hasServiceSub + atomically writeSub + liftIO $ do + unless hasSub $ incSrvStat srvSubCount + incSrvStat srvSubQueues + incSrvStat $ maybe srvAssocNew (const srvAssocUpdated) queueServiceId + pure (hasSub, Nothing) where - hasServiceSub = (0 /=) <$> readTVar ntfServiceSubsCount - -- This function is used when queue is associated with the service. - newServiceQueueSub = do - writeTQueue (subQ ntfSubscribers) (CSClient entId ntfServiceId (Just serviceId), clientId) - modifyTVar' ntfServiceSubsCount (+ 1) -- service count - modifyTVar' (totalServiceSubs ntfSubscribers) (+ 1) -- server count for all services - Nothing -> case ntfServiceId of - Just _ -> - liftIO (setQueueService (queueStore ms) q SNotifierService Nothing) >>= \case - Left e -> pure $ ERR e - Right () -> do - -- hasSubscription should never be True in this branch, because queue was associated with service. - -- So unless storage and session states diverge, this check is redundant. - hasSub <- atomically $ hasSubscription >>= newSub - incNtfSrvStat srvAssocRemoved - sok hasSub + hasServiceSub = ((0 /=) . fst) <$> readTVar (clientServiceSubs clnt) + -- This function is used when queue association with the service is created. + incServiceQueueSubs = modifyTVar' (clientServiceSubs clnt) $ addServiceSubs (1, queueIdHash (recipientId q)) -- service count and IDs hash + Nothing -> case queueServiceId of + Just _ -> runExceptT $ do + ExceptT $ setQueueService (queueStore ms) q party Nothing + liftIO $ incSrvStat srvAssocRemoved + -- getSubscription may be Just for receiving service, where clientSubs also hold active deliveries for service subscriptions. + -- For notification service it can only be Just if storage and session states diverge. + r <- atomically $ getSubscription >>= newSub + atomically writeSub + pure r Nothing -> do - hasSub <- atomically $ ifM hasSubscription (pure True) (newSub False) - sok hasSub + r@(hasSub, _) <- atomically $ getSubscription >>= newSub + unless hasSub $ atomically writeSub + pure $ Right r where - hasSubscription = TM.member entId ntfSubscriptions - newSub hasSub = do - writeTQueue (subQ ntfSubscribers) (CSClient entId ntfServiceId Nothing, clientId) - unless (hasSub) $ TM.insert entId () ntfSubscriptions - pure hasSub - sok hasSub = do - incStat $ if hasSub then ntfSubDuplicate stats else ntfSub stats - pure $ SOK Nothing - - subscribeServiceNotifications :: ServiceId -> M s BrokerMsg - subscribeServiceNotifications serviceId = do - subscribed <- readTVarIO ntfServiceSubscribed - if subscribed - then SOKS <$> readTVarIO ntfServiceSubsCount - else - liftIO (getServiceQueueCount @(StoreQueue s) (queueStore ms) SNotifierService serviceId) >>= \case - Left e -> pure $ ERR e - Right !count' -> do - incCount <- atomically $ do - writeTVar ntfServiceSubscribed True - count <- swapTVar ntfServiceSubsCount count' - pure $ count' - count - atomically $ writeTQueue (subQ ntfSubscribers) (CSService serviceId incCount, clientId) - pure $ SOKS count' + getSubscription = TM.lookup entId $ clientSubs clnt + newSub = \case + Just sub -> pure (True, Just sub) + Nothing -> do + sub <- mkSub + TM.insert entId sub $ clientSubs clnt + pure (False, Just sub) + + subscribeServiceMessages :: ServiceId -> (Int64, IdsHash) -> M s BrokerMsg + subscribeServiceMessages serviceId expected = + sharedSubscribeService SRecipientService serviceId expected subscribers serviceSubscribed serviceSubsCount rcvServices >>= \case + Left e -> pure $ ERR e + Right (hasSub, (count, idsHash)) -> do + stats <- asks serverStats + unless hasSub $ forkClient clnt "deliverServiceMessages" $ liftIO $ deliverServiceMessages stats count + pure $ SOKS count idsHash + where + deliverServiceMessages stats expectedCnt = do + foldRcvServiceMessages ms serviceId deliverQueueMsg (0, 0, 0, [(NoCorrId, NoEntity, ALLS)]) >>= \case + Right (qCnt, msgCnt, dupCnt, evts) -> do + atomically $ writeTBQueue msgQ evts + atomicModifyIORef'_ (rcvServicesSubMsg stats) (+ msgCnt) + atomicModifyIORef'_ (rcvServicesSubDuplicate stats) (+ dupCnt) + let logMsg = "Subscribed service " <> tshow serviceId <> " (" + if qCnt == expectedCnt + then logNote $ logMsg <> tshow qCnt <> " queues)" + else logError $ logMsg <> "expected " <> tshow expectedCnt <> "," <> tshow qCnt <> " queues)" + Left e -> do + logError $ "Service subscription error for " <> tshow serviceId <> ": " <> tshow e + atomically $ writeTBQueue msgQ [(NoCorrId, NoEntity, ERR e)] + deliverQueueMsg :: (Int64, Int, Int, NonEmpty (Transmission BrokerMsg)) -> RecipientId -> Either ErrorType (Maybe (QueueRec, Message)) -> IO (Int64, Int, Int, NonEmpty (Transmission BrokerMsg)) + deliverQueueMsg (!qCnt, !msgCnt, !dupCnt, evts) rId = \case + Left e -> pure (qCnt + 1, msgCnt, dupCnt, (NoCorrId, rId, ERR e) <| evts) + Right qMsg_ -> case qMsg_ of + Nothing -> pure (qCnt + 1, msgCnt, dupCnt, evts) + Just (qr, msg) -> + atomically (getSubscription rId) >>= \case + Nothing -> pure (qCnt + 1, msgCnt, dupCnt + 1, evts) + Just sub -> do + ts <- getSystemSeconds + atomically $ setDelivered sub msg ts + atomically $ writeTBQueue msgQ [(NoCorrId, rId, MSG (encryptMsg qr msg))] + pure (qCnt + 1, msgCnt + 1, dupCnt, evts) + getSubscription rId = + TM.lookup rId (subscriptions clnt) >>= \case + -- If delivery subscription already exists, then there is no need to deliver message. + -- It may have been created when the message is sent after service subscription is created. + Just _sub -> pure Nothing + Nothing -> do + sub <- newSubscription NoSub + TM.insert rId sub $ subscriptions clnt + pure $ Just sub + + subscribeServiceNotifications :: ServiceId -> (Int64, IdsHash) -> M s BrokerMsg + subscribeServiceNotifications serviceId expected = + either ERR (uncurry SOKS . snd) <$> sharedSubscribeService SNotifierService serviceId expected ntfSubscribers ntfServiceSubscribed ntfServiceSubsCount ntfServices + + sharedSubscribeService :: (PartyI p, ServiceParty p) => SParty p -> ServiceId -> (Int64, IdsHash) -> ServerSubscribers s -> (Client s -> TVar Bool) -> (Client s -> TVar (Int64, IdsHash)) -> (ServerStats -> ServiceStats) -> M s (Either ErrorType (Bool, (Int64, IdsHash))) + sharedSubscribeService party serviceId (count, idsHash) srvSubscribers clientServiceSubscribed clientServiceSubs servicesSel = do + subscribed <- readTVarIO $ clientServiceSubscribed clnt + stats <- asks serverStats + liftIO $ runExceptT $ + (subscribed,) + <$> if subscribed + then readTVarIO $ clientServiceSubs clnt + else do + subs'@(count', idsHash') <- ExceptT $ getServiceQueueCountHash @(StoreQueue s) (queueStore ms) party serviceId + subsChange <- atomically $ do + writeTVar (clientServiceSubscribed clnt) True + currSubs <- swapTVar (clientServiceSubs clnt) subs' + pure $ subtractServiceSubs currSubs subs' + let incSrvStat sel n = liftIO $ atomicModifyIORef'_ (sel $ servicesSel stats) (+ n) + diff = fromIntegral $ count' - count + if -- `count == -1` only for subscriptions by old NTF servers + | count == -1 && (diff == 0 && idsHash == idsHash') -> incSrvStat srvSubOk 1 + | diff > 0 -> incSrvStat srvSubMore 1 >> incSrvStat srvSubMoreTotal diff + | diff < 0 -> incSrvStat srvSubFewer 1 >> incSrvStat srvSubFewerTotal (- diff) + | otherwise -> incSrvStat srvSubDiff 1 + atomically $ writeTQueue (subQ srvSubscribers) (CSService serviceId subsChange, clientId) + pure (count', idsHash') acknowledgeMsg :: MsgId -> StoreQueue s -> QueueRec -> M s (Transmission BrokerMsg) acknowledgeMsg msgId q qr = @@ -1904,10 +1986,13 @@ client tryDeliverMessage msg = -- the subscribed client var is read outside of STM to avoid transaction cost -- in case no client is subscribed. - getSubscribedClient rId (queueSubscribers subscribers) + getSubscribed $>>= deliverToSub >>= mapM_ forkDeliver where + getSubscribed = case rcvServiceId qr of + Just serviceId -> getSubscribedClient serviceId $ serviceSubscribers subscribers + Nothing -> getSubscribedClient rId $ queueSubscribers subscribers rId = recipientId q deliverToSub rcv = do ts <- getSystemSeconds @@ -1918,6 +2003,7 @@ client -- the new client will receive message in response to SUB. readTVar rcv $>>= \rc@Client {subscriptions = subs, sndQ = sndQ'} -> TM.lookup rId subs + >>= maybe (newServiceDeliverySub subs) (pure . Just) $>>= \s@Sub {subThread, delivered} -> case subThread of ProhibitSub -> pure Nothing ServerSub st -> readTVar st >>= \case @@ -1930,6 +2016,12 @@ client (writeTVar st SubPending $> Just (rc, s, st)) (deliver sndQ' s ts $> Nothing) _ -> pure Nothing + newServiceDeliverySub subs + | isJust (rcvServiceId qr) = do + sub <- newSubscription NoSub + TM.insert rId sub subs + pure $ Just sub + | otherwise = pure Nothing deliver sndQ' s ts = do let encMsg = encryptMsg qr msg writeTBQueue sndQ' ([(NoCorrId, rId, MSG encMsg)], []) @@ -2051,6 +2143,7 @@ client -- we delete subscription here, so the client with no subscriptions can be disconnected. sub <- atomically $ TM.lookupDelete entId $ subscriptions clnt liftIO $ mapM_ cancelSub sub + when (isJust rcvServiceId) $ atomically $ modifyTVar' (serviceSubsCount clnt) $ subtractServiceSubs (1, queueIdHash (recipientId q)) atomically $ writeTQueue (subQ subscribers) (CSDeleted entId rcvServiceId, clientId) forM_ (notifier qr) $ \NtfCreds {notifierId = nId, ntfServiceId} -> do -- queue is deleted by a different client from the one subscribed to notifications, diff --git a/src/Simplex/Messaging/Server/Env/STM.hs b/src/Simplex/Messaging/Server/Env/STM.hs index 24cd6dfcc..574111c15 100644 --- a/src/Simplex/Messaging/Server/Env/STM.hs +++ b/src/Simplex/Messaging/Server/Env/STM.hs @@ -363,7 +363,7 @@ data ServerSubscribers s = ServerSubscribers { subQ :: TQueue (ClientSub, ClientId), queueSubscribers :: SubscribedClients s, serviceSubscribers :: SubscribedClients s, -- service clients with long-term certificates that have subscriptions - totalServiceSubs :: TVar Int64, + totalServiceSubs :: TVar (Int64, IdsHash), subClients :: TVar IntSet, -- clients with individual or service subscriptions pendingEvents :: TVar (IntMap (NonEmpty (EntityId, BrokerMsg))) } @@ -426,7 +426,7 @@ sameClient c cv = maybe False (sameClientId c) <$> readTVar cv data ClientSub = CSClient QueueId (Maybe ServiceId) (Maybe ServiceId) -- includes previous and new associated service IDs | CSDeleted QueueId (Maybe ServiceId) -- includes previously associated service IDs - | CSService ServiceId Int64 -- only send END to idividual client subs on message delivery, not of SSUB/NSSUB + | CSService ServiceId (Int64, IdsHash) -- only send END to idividual client subs on message delivery, not of SSUB/NSSUB newtype ProxyAgent = ProxyAgent { smpAgent :: SMPClientAgent 'Sender @@ -440,8 +440,8 @@ data Client s = Client ntfSubscriptions :: TMap NotifierId (), serviceSubscribed :: TVar Bool, -- set independently of serviceSubsCount, to track whether service subscription command was received ntfServiceSubscribed :: TVar Bool, - serviceSubsCount :: TVar Int64, -- only one service can be subscribed, based on its certificate, this is subscription count - ntfServiceSubsCount :: TVar Int64, -- only one service can be subscribed, based on its certificate, this is subscription count + serviceSubsCount :: TVar (Int64, IdsHash), -- only one service can be subscribed, based on its certificate, this is subscription count + ntfServiceSubsCount :: TVar (Int64, IdsHash), -- only one service can be subscribed, based on its certificate, this is subscription count rcvQ :: TBQueue (NonEmpty (VerifiedTransmission s)), sndQ :: TBQueue (NonEmpty (Transmission BrokerMsg), [Transmission BrokerMsg]), msgQ :: TBQueue (NonEmpty (Transmission BrokerMsg)), @@ -502,7 +502,7 @@ newServerSubscribers = do subQ <- newTQueueIO queueSubscribers <- SubscribedClients <$> TM.emptyIO serviceSubscribers <- SubscribedClients <$> TM.emptyIO - totalServiceSubs <- newTVarIO 0 + totalServiceSubs <- newTVarIO (0, mempty) subClients <- newTVarIO IS.empty pendingEvents <- newTVarIO IM.empty pure ServerSubscribers {subQ, queueSubscribers, serviceSubscribers, totalServiceSubs, subClients, pendingEvents} @@ -513,8 +513,8 @@ newClient clientId qSize clientTHParams createdAt = do ntfSubscriptions <- TM.emptyIO serviceSubscribed <- newTVarIO False ntfServiceSubscribed <- newTVarIO False - serviceSubsCount <- newTVarIO 0 - ntfServiceSubsCount <- newTVarIO 0 + serviceSubsCount <- newTVarIO (0, mempty) + ntfServiceSubsCount <- newTVarIO (0, mempty) rcvQ <- newTBQueueIO qSize sndQ <- newTBQueueIO qSize msgQ <- newTBQueueIO qSize @@ -706,7 +706,7 @@ mkJournalStoreConfig queueStoreCfg storePath msgQueueQuota maxJournalMsgCount ma newSMPProxyAgent :: SMPClientAgentConfig -> TVar ChaChaDRG -> IO ProxyAgent newSMPProxyAgent smpAgentCfg random = do - smpAgent <- newSMPClientAgent SSender smpAgentCfg random + smpAgent <- newSMPClientAgent SSender smpAgentCfg Nothing random pure ProxyAgent {smpAgent} readWriteQueueStore :: forall q. StoreQueueClass q => Bool -> (RecipientId -> QueueRec -> IO q) -> FilePath -> STMQueueStore q -> IO (StoreLog 'WriteMode) diff --git a/src/Simplex/Messaging/Server/Main.hs b/src/Simplex/Messaging/Server/Main.hs index 92f0b0821..f7461f392 100644 --- a/src/Simplex/Messaging/Server/Main.hs +++ b/src/Simplex/Messaging/Server/Main.hs @@ -40,7 +40,7 @@ module Simplex.Messaging.Server.Main ) where import Control.Concurrent.STM -import Control.Exception (SomeException, finally, try) +import Control.Exception (finally) import Control.Logger.Simple import Control.Monad import qualified Data.Attoparsec.ByteString.Char8 as A @@ -50,10 +50,8 @@ import Data.Char (isAlpha, isAscii, toUpper) import Data.Either (fromRight) import Data.Functor (($>)) import Data.Ini (Ini, lookupValue, readIniFile) -import Data.Int (Int64) import Data.List (find, isPrefixOf) import qualified Data.List.NonEmpty as L -import qualified Data.Map.Strict as M import Data.Maybe (fromMaybe, isJust, isNothing) import Data.Text (Text) import qualified Data.Text as T @@ -84,14 +82,17 @@ import Simplex.Messaging.Transport (supportedProxyClientSMPRelayVRange, alpnSupp import Simplex.Messaging.Transport.Client (TransportHost (..), defaultSocksProxy) import Simplex.Messaging.Transport.HTTP2 (httpALPN) import Simplex.Messaging.Transport.Server (ServerCredentials (..), mkTransportServerConfig) -import Simplex.Messaging.Util (eitherToMaybe, ifM, unlessM) +import Simplex.Messaging.Util (eitherToMaybe, ifM) import System.Directory (createDirectoryIfMissing, doesDirectoryExist, doesFileExist) import System.Exit (exitFailure) import System.FilePath (combine) -import System.IO (BufferMode (..), IOMode (..), hSetBuffering, stderr, stdout, withFile) +import System.IO (BufferMode (..), hSetBuffering, stderr, stdout) import Text.Read (readMaybe) #if defined(dbServerPostgres) +import Control.Exception (SomeException, try) +import Data.Int (Int64) +import qualified Data.Map.Strict as M import Data.Semigroup (Sum (..)) import Simplex.Messaging.Agent.Store.Postgres (checkSchemaExists) import Simplex.Messaging.Server.MsgStore.Journal (JournalQueue) @@ -102,7 +103,9 @@ import Simplex.Messaging.Server.QueueStore.Postgres (batchInsertQueues, batchIns import Simplex.Messaging.Server.QueueStore.STM (STMQueueStore (..)) import Simplex.Messaging.Server.QueueStore.Types import Simplex.Messaging.Server.StoreLog (closeStoreLog, logNewService, logCreateQueue, openWriteStoreLog) +import Simplex.Messaging.Util (unlessM) import System.Directory (renameFile) +import System.IO (IOMode (..), withFile) #endif smpServerCLI :: FilePath -> FilePath -> IO () @@ -579,7 +582,7 @@ smpServerCLI_ generateSite serveStaticFiles attachStaticFiles cfgPath logPath = mkTransportServerConfig (fromMaybe False $ iniOnOff "TRANSPORT" "log_tls_errors" ini) (Just $ alpnSupportedSMPHandshakes <> httpALPN) - (fromMaybe True $ iniOnOff "TRANSPORT" "accept_service_credentials" ini), -- TODO [certs] remove this option + True, controlPort = eitherToMaybe $ T.unpack <$> lookupValue "TRANSPORT" "control_port" ini, smpAgentCfg = defaultSMPClientAgentConfig diff --git a/src/Simplex/Messaging/Server/MsgStore/Journal.hs b/src/Simplex/Messaging/Server/MsgStore/Journal.hs index 5038c8826..c65660c93 100644 --- a/src/Simplex/Messaging/Server/MsgStore/Journal.hs +++ b/src/Simplex/Messaging/Server/MsgStore/Journal.hs @@ -355,8 +355,8 @@ instance QueueStoreClass (JournalQueue s) (QStore s) where {-# INLINE setQueueService #-} getQueueNtfServices = withQS (getQueueNtfServices @(JournalQueue s)) {-# INLINE getQueueNtfServices #-} - getServiceQueueCount = withQS (getServiceQueueCount @(JournalQueue s)) - {-# INLINE getServiceQueueCount #-} + getServiceQueueCountHash = withQS (getServiceQueueCountHash @(JournalQueue s)) + {-# INLINE getServiceQueueCountHash #-} makeQueue_ :: JournalMsgStore s -> RecipientId -> QueueRec -> Lock -> IO (JournalQueue s) makeQueue_ JournalMsgStore {sharedLock} rId qr queueLock = do @@ -444,6 +444,26 @@ instance MsgStoreClass (JournalMsgStore s) where getLoadedQueue :: JournalQueue s -> IO (JournalQueue s) getLoadedQueue q = fromMaybe q <$> TM.lookupIO (recipientId q) (loadedQueues $ queueStore_ ms) + foldRcvServiceMessages :: JournalMsgStore s -> ServiceId -> (a -> RecipientId -> Either ErrorType (Maybe (QueueRec, Message)) -> IO a) -> a -> IO (Either ErrorType a) + foldRcvServiceMessages ms serviceId f acc = case queueStore_ ms of + MQStore st -> fmap Right $ foldRcvServiceQueues st serviceId f' acc + where + f' a (q, qr) = runExceptT (tryPeekMsg ms q) >>= f a (recipientId q) . ((qr,) <$$>) +#if defined(dbServerPostgres) + PQStore st -> foldRcvServiceQueueRecs st serviceId f' acc + where + JournalMsgStore {queueLocks, sharedLock} = ms + f' a (rId, qr) = do + q <- mkQueue ms False rId qr + qMsg_ <- + withSharedWaitLock rId queueLocks sharedLock $ runExceptT $ tryStore' "foldRcvServiceMessages" rId $ + (qr,) . snd <$$> (getLoadedQueue q >>= unStoreIO . getPeekMsgQueue ms) + f a rId qMsg_ + -- Use cached queue if available. + -- Also see the comment in loadQueue in PostgresQueueStore + getLoadedQueue q = fromMaybe q <$> TM.lookupIO (recipientId q) (loadedQueues $ queueStore_ ms) +#endif + logQueueStates :: JournalMsgStore s -> IO () logQueueStates ms = withActiveMsgQueues ms $ unStoreIO . logQueueState diff --git a/src/Simplex/Messaging/Server/MsgStore/Postgres.hs b/src/Simplex/Messaging/Server/MsgStore/Postgres.hs index a0eb1d1ca..edf7f481c 100644 --- a/src/Simplex/Messaging/Server/MsgStore/Postgres.hs +++ b/src/Simplex/Messaging/Server/MsgStore/Postgres.hs @@ -119,6 +119,34 @@ instance MsgStoreClass PostgresMsgStore where toMessageStats (expiredMsgsCount, storedMsgsCount, storedQueues) = MessageStats {expiredMsgsCount, storedMsgsCount, storedQueues} + foldRcvServiceMessages :: PostgresMsgStore -> ServiceId -> (a -> RecipientId -> Either ErrorType (Maybe (QueueRec, Message)) -> IO a) -> a -> IO (Either ErrorType a) + foldRcvServiceMessages ms serviceId f acc = + runExceptT $ withDB' "foldRcvServiceMessages" (queueStore_ ms) $ \db -> + DB.fold + db + [sql| + SELECT q.recipient_id, q.recipient_keys, q.rcv_dh_secret, + q.sender_id, q.sender_key, q.queue_mode, + q.notifier_id, q.notifier_key, q.rcv_ntf_dh_secret, q.ntf_service_id, + q.status, q.updated_at, q.link_id, q.rcv_service_id, + m.msg_id, m.msg_ts, m.msg_quota, m.msg_ntf_flag, m.msg_body + FROM msg_queues q + LEFT JOIN ( + SELECT recipient_id, msg_id, msg_ts, msg_quota, msg_ntf_flag, msg_body, + ROW_NUMBER() OVER (PARTITION BY recipient_id ORDER BY message_id ASC) AS row_num + FROM messages + ) m ON q.recipient_id = m.recipient_id AND m.row_num = 1 + WHERE q.rcv_service_id = ? AND q.deleted_at IS NULL; + |] + (Only serviceId) + acc + f' + where + f' a (qRow :. mRow) = + let (rId, qr) = rowToQueueRec qRow + msg_ = toMaybeMessage mRow + in f a rId $ Right ((qr,) <$> msg_) + logQueueStates _ = error "logQueueStates not used" logQueueState _ = error "logQueueState not used" @@ -247,6 +275,11 @@ uninterruptibleMask_ :: ExceptT ErrorType IO a -> ExceptT ErrorType IO a uninterruptibleMask_ = ExceptT . E.uninterruptibleMask_ . runExceptT {-# INLINE uninterruptibleMask_ #-} +toMaybeMessage :: (Maybe (Binary MsgId), Maybe Int64, Maybe Bool, Maybe Bool, Maybe (Binary MsgBody)) -> Maybe Message +toMaybeMessage = \case + (Just msgId, Just ts, Just msgQuota, Just ntf, Just body) -> Just $ toMessage (msgId, ts, msgQuota, ntf, body) + _ -> Nothing + toMessage :: (Binary MsgId, Int64, Bool, Bool, Binary MsgBody) -> Message toMessage (Binary msgId, ts, msgQuota, ntf, Binary body) | msgQuota = MessageQuota {msgId, msgTs} diff --git a/src/Simplex/Messaging/Server/MsgStore/STM.hs b/src/Simplex/Messaging/Server/MsgStore/STM.hs index 73e1bf398..f118e007c 100644 --- a/src/Simplex/Messaging/Server/MsgStore/STM.hs +++ b/src/Simplex/Messaging/Server/MsgStore/STM.hs @@ -87,6 +87,11 @@ instance MsgStoreClass STMMsgStore where expireOldMessages _tty ms now ttl = withLoadedQueues (queueStore_ ms) $ atomically . expireQueueMsgs ms now (now - ttl) + foldRcvServiceMessages :: STMMsgStore -> ServiceId -> (a -> RecipientId -> Either ErrorType (Maybe (QueueRec, Message)) -> IO a) -> a -> IO (Either ErrorType a) + foldRcvServiceMessages ms serviceId f = fmap Right . foldRcvServiceQueues (queueStore_ ms) serviceId f' + where + f' a (q, qr) = runExceptT (tryPeekMsg ms q) >>= f a (recipientId q) . ((qr,) <$$>) + logQueueStates _ = pure () {-# INLINE logQueueStates #-} logQueueState _ = pure () diff --git a/src/Simplex/Messaging/Server/MsgStore/Types.hs b/src/Simplex/Messaging/Server/MsgStore/Types.hs index ffecf584c..acb661a40 100644 --- a/src/Simplex/Messaging/Server/MsgStore/Types.hs +++ b/src/Simplex/Messaging/Server/MsgStore/Types.hs @@ -63,6 +63,7 @@ class (Monad (StoreMonad s), QueueStoreClass (StoreQueue s) (QueueStore s)) => M unsafeWithAllMsgQueues :: Monoid a => Bool -> s -> (StoreQueue s -> IO a) -> IO a -- tty, store, now, ttl expireOldMessages :: Bool -> s -> Int64 -> Int64 -> IO MessageStats + foldRcvServiceMessages :: s -> ServiceId -> (a -> RecipientId -> Either ErrorType (Maybe (QueueRec, Message)) -> IO a) -> a -> IO (Either ErrorType a) logQueueStates :: s -> IO () logQueueState :: StoreQueue s -> StoreMonad s () queueStore :: s -> QueueStore s diff --git a/src/Simplex/Messaging/Server/Prometheus.hs b/src/Simplex/Messaging/Server/Prometheus.hs index f61b4d946..33ccbd0be 100644 --- a/src/Simplex/Messaging/Server/Prometheus.hs +++ b/src/Simplex/Messaging/Server/Prometheus.hs @@ -123,6 +123,8 @@ prometheusMetrics sm rtm ts = _pMsgFwdsRecv, _rcvServices, _ntfServices, + _rcvServicesSubMsg, + _rcvServicesSubDuplicate, _qCount, _msgCount, _ntfCount @@ -388,6 +390,14 @@ prometheusMetrics sm rtm ts = \# HELP simplex_smp_ntf_services_queues_count The count of queues associated with notification services.\n\ \# TYPE simplex_smp_ntf_services_queues_count gauge\n\ \simplex_smp_ntf_services_queues_count " <> mshow (ntfServiceQueuesCount entityCounts) <> "\n# ntfServiceQueuesCount\n\ + \\n\ + \# HELP simplex_smp_rcv_services_sub_msg The count of subscribed service queues with messages.\n\ + \# TYPE simplex_smp_rcv_services_sub_msg counter\n\ + \simplex_smp_rcv_services_sub_msg " <> mshow _rcvServicesSubMsg <> "\n# rcvServicesSubMsg\n\ + \\n\ + \# HELP simplex_smp_rcv_services_sub_duplicate The count of duplicate subscribed service queues.\n\ + \# TYPE simplex_smp_rcv_services_sub_duplicate counter\n\ + \simplex_smp_rcv_services_sub_duplicate " <> mshow _rcvServicesSubDuplicate <> "\n# rcvServicesSubDuplicate\n\ \\n" <> showServices _rcvServices "rcv" "receiving" <> showServices _ntfServices "ntf" "notification" @@ -423,6 +433,30 @@ prometheusMetrics sm rtm ts = \# HELP simplex_smp_" <> pfx <> "_services_sub_end Ended subscriptions with " <> name <> " services.\n\ \# TYPE simplex_smp_" <> pfx <> "_services_sub_end gauge\n\ \simplex_smp_" <> pfx <> "_services_sub_end " <> mshow (_srvSubEnd ss) <> "\n# " <> pfx <> ".srvSubEnd\n\ + \\n\ + \# HELP simplex_smp_" <> pfx <> "_services_sub_ok Service subscriptions for " <> name <> " services.\n\ + \# TYPE simplex_smp_" <> pfx <> "_services_sub_ok gauge\n\ + \simplex_smp_" <> pfx <> "_services_sub_ok " <> mshow (_srvSubOk ss) <> "\n# " <> pfx <> ".srvSubOk\n\ + \\n\ + \# HELP simplex_smp_" <> pfx <> "_services_sub_more Service subscriptions for " <> name <> " services with more queues than in the client.\n\ + \# TYPE simplex_smp_" <> pfx <> "_services_sub_more gauge\n\ + \simplex_smp_" <> pfx <> "_services_sub_more " <> mshow (_srvSubMore ss) <> "\n# " <> pfx <> ".srvSubMore\n\ + \\n\ + \# HELP simplex_smp_" <> pfx <> "_services_sub_fewer Service subscriptions for " <> name <> " services with fewer queues than in the client.\n\ + \# TYPE simplex_smp_" <> pfx <> "_services_sub_fewer gauge\n\ + \simplex_smp_" <> pfx <> "_services_sub_fewer " <> mshow (_srvSubFewer ss) <> "\n# " <> pfx <> ".srvSubFewer\n\ + \\n\ + \# HELP simplex_smp_" <> pfx <> "_services_sub_diff Service subscriptions for " <> name <> " services with different hash than in the client.\n\ + \# TYPE simplex_smp_" <> pfx <> "_services_sub_diff gauge\n\ + \simplex_smp_" <> pfx <> "_services_sub_diff " <> mshow (_srvSubDiff ss) <> "\n# " <> pfx <> ".srvSubDiff\n\ + \\n\ + \# HELP simplex_smp_" <> pfx <> "_services_sub_more_total Service subscriptions for " <> name <> " services with more queues than in the client total.\n\ + \# TYPE simplex_smp_" <> pfx <> "_services_sub_more_total gauge\n\ + \simplex_smp_" <> pfx <> "_services_sub_more_total " <> mshow (_srvSubMoreTotal ss) <> "\n# " <> pfx <> ".srvSubMoreTotal\n\ + \\n\ + \# HELP simplex_smp_" <> pfx <> "_services_sub_fewer_total Service subscriptions for " <> name <> " services with fewer queues than in the client total.\n\ + \# TYPE simplex_smp_" <> pfx <> "_services_sub_fewer_total gauge\n\ + \simplex_smp_" <> pfx <> "_services_sub_fewer_total " <> mshow (_srvSubFewerTotal ss) <> "\n# " <> pfx <> ".srvSubFewerTotal\n\ \\n" info = "# Info\n\ diff --git a/src/Simplex/Messaging/Server/QueueStore.hs b/src/Simplex/Messaging/Server/QueueStore.hs index 44aad047b..3904871bf 100644 --- a/src/Simplex/Messaging/Server/QueueStore.hs +++ b/src/Simplex/Messaging/Server/QueueStore.hs @@ -71,6 +71,7 @@ data ServiceRec = ServiceRec serviceCert :: X.CertificateChain, serviceCertHash :: XV.Fingerprint, -- SHA512 hash of long-term service client certificate. See comment for ClientHandshake. serviceCreatedAt :: SystemDate + -- entitiesHash :: IdsHash -- a xor-hash of all associated entities } deriving (Show) diff --git a/src/Simplex/Messaging/Server/QueueStore/Postgres.hs b/src/Simplex/Messaging/Server/QueueStore/Postgres.hs index e86bec07b..a8c8c040a 100644 --- a/src/Simplex/Messaging/Server/QueueStore/Postgres.hs +++ b/src/Simplex/Messaging/Server/QueueStore/Postgres.hs @@ -24,9 +24,11 @@ module Simplex.Messaging.Server.QueueStore.Postgres batchInsertServices, batchInsertQueues, foldServiceRecs, + foldRcvServiceQueueRecs, foldQueueRecs, foldRecentQueueRecs, handleDuplicate, + rowToQueueRec, withLog_, withDB, withDB', @@ -522,15 +524,11 @@ instance StoreQueueClass q => QueueStoreClass q (PostgresQueueStore q) where let (sNtfs, restNtfs) = partition (\(nId, _) -> S.member nId snIds) ntfs' in ((serviceId, sNtfs) : ssNtfs, restNtfs) - getServiceQueueCount :: (PartyI p, ServiceParty p) => PostgresQueueStore q -> SParty p -> ServiceId -> IO (Either ErrorType Int64) - getServiceQueueCount st party serviceId = - E.uninterruptibleMask_ $ runExceptT $ withDB' "getServiceQueueCount" st $ \db -> - maybeFirstRow' 0 fromOnly $ - DB.query db query (Only serviceId) - where - query = case party of - SRecipientService -> "SELECT count(1) FROM msg_queues WHERE rcv_service_id = ? AND deleted_at IS NULL" - SNotifierService -> "SELECT count(1) FROM msg_queues WHERE ntf_service_id = ? AND deleted_at IS NULL" + getServiceQueueCountHash :: (PartyI p, ServiceParty p) => PostgresQueueStore q -> SParty p -> ServiceId -> IO (Either ErrorType (Int64, IdsHash)) + getServiceQueueCountHash st party serviceId = + E.uninterruptibleMask_ $ runExceptT $ withDB' "getServiceQueueCountHash" st $ \db -> + maybeFirstRow' (0, mempty) id $ + DB.query db ("SELECT queue_count, queue_ids_hash FROM services WHERE service_id = ? AND service_role = ?") (serviceId, partyServiceRole party) batchInsertServices :: [STMService] -> PostgresQueueStore q -> IO Int64 batchInsertServices services' toStore = @@ -577,12 +575,17 @@ insertServiceQuery = VALUES (?,?,?,?,?) |] -foldServiceRecs :: forall a q. Monoid a => PostgresQueueStore q -> (ServiceRec -> IO a) -> IO a +foldServiceRecs :: Monoid a => PostgresQueueStore q -> (ServiceRec -> IO a) -> IO a foldServiceRecs st f = withTransaction (dbStore st) $ \db -> DB.fold_ db "SELECT service_id, service_role, service_cert, service_cert_hash, created_at FROM services" mempty $ \ !acc -> fmap (acc <>) . f . rowToServiceRec +foldRcvServiceQueueRecs :: PostgresQueueStore q -> ServiceId -> (a -> (RecipientId, QueueRec) -> IO a) -> a -> IO (Either ErrorType a) +foldRcvServiceQueueRecs st serviceId f acc = + runExceptT $ withDB' "foldRcvServiceQueueRecs" st $ \db -> + DB.fold db (queueRecQuery <> " WHERE rcv_service_id = ? AND deleted_at IS NULL") (Only serviceId) acc $ \a -> f a . rowToQueueRec + foldQueueRecs :: Monoid a => Bool -> Bool -> PostgresQueueStore q -> ((RecipientId, QueueRec) -> IO a) -> IO a foldQueueRecs withData = foldQueueRecs_ foldRecs where @@ -769,10 +772,6 @@ instance ToField SMPServiceRole where toField = toField . decodeLatin1 . smpEnco instance FromField SMPServiceRole where fromField = fromTextField_ $ eitherToMaybe . smpDecode . encodeUtf8 -instance ToField X.CertificateChain where toField = toField . Binary . smpEncode . C.encodeCertChain - -instance FromField X.CertificateChain where fromField = blobFieldDecoder (parseAll C.certChainP) - #if !defined(dbPostgres) instance ToField EntityId where toField (EntityId s) = toField $ Binary s @@ -790,6 +789,10 @@ instance ToField C.APublicAuthKey where toField = toField . Binary . C.encodePub instance FromField C.APublicAuthKey where fromField = blobFieldDecoder C.decodePubKey +instance ToField IdsHash where toField (IdsHash s) = toField (Binary s) + +deriving newtype instance FromField IdsHash + instance ToField EncDataBytes where toField (EncDataBytes s) = toField (Binary s) deriving newtype instance FromField EncDataBytes @@ -797,4 +800,8 @@ deriving newtype instance FromField EncDataBytes deriving newtype instance ToField (RoundedSystemTime t) deriving newtype instance FromField (RoundedSystemTime t) + +instance ToField X.CertificateChain where toField = toField . Binary . smpEncode . C.encodeCertChain + +instance FromField X.CertificateChain where fromField = blobFieldDecoder (parseAll C.certChainP) #endif diff --git a/src/Simplex/Messaging/Server/QueueStore/Postgres/Migrations.hs b/src/Simplex/Messaging/Server/QueueStore/Postgres/Migrations.hs index 3c4da6458..fdcdeba0a 100644 --- a/src/Simplex/Messaging/Server/QueueStore/Postgres/Migrations.hs +++ b/src/Simplex/Messaging/Server/QueueStore/Postgres/Migrations.hs @@ -10,6 +10,7 @@ where import Data.List (sortOn) import Data.Text (Text) import Simplex.Messaging.Agent.Store.Shared +import Simplex.Messaging.Agent.Store.Postgres.Migrations.Util import Text.RawString.QQ (r) serverSchemaMigrations :: [(String, Text, Maybe Text)] @@ -18,7 +19,8 @@ serverSchemaMigrations = ("20250319_updated_index", m20250319_updated_index, Just down_m20250319_updated_index), ("20250320_short_links", m20250320_short_links, Just down_m20250320_short_links), ("20250514_service_certs", m20250514_service_certs, Just down_m20250514_service_certs), - ("20250903_store_messages", m20250903_store_messages, Just down_m20250903_store_messages) + ("20250903_store_messages", m20250903_store_messages, Just down_m20250903_store_messages), + ("20250915_queue_ids_hash", m20250915_queue_ids_hash, Just down_m20250915_queue_ids_hash) ] -- | The list of migrations in ascending order by date @@ -450,3 +452,139 @@ ALTER TABLE msg_queues DROP TABLE messages; |] + +m20250915_queue_ids_hash :: Text +m20250915_queue_ids_hash = + createXorHashFuncs + <> [r| +ALTER TABLE services + ADD COLUMN queue_count BIGINT NOT NULL DEFAULT 0, + ADD COLUMN queue_ids_hash BYTEA NOT NULL DEFAULT '\x00000000000000000000000000000000'; + +CREATE FUNCTION update_all_aggregates() RETURNS VOID +LANGUAGE plpgsql +AS $$ +BEGIN + WITH acc AS ( + SELECT + s.service_id, + count(1) as q_count, + xor_aggregate(public.digest(CASE WHEN s.service_role = 'M' THEN q.recipient_id ELSE COALESCE(q.notifier_id, '\x00000000000000000000000000000000') END, 'md5')) AS q_ids_hash + FROM services s + JOIN msg_queues q ON (s.service_id = q.rcv_service_id AND s.service_role = 'M') OR (s.service_id = q.ntf_service_id AND s.service_role = 'N') + WHERE q.deleted_at IS NULL + GROUP BY s.service_id + ) + UPDATE services s + SET queue_count = COALESCE(acc.q_count, 0), + queue_ids_hash = COALESCE(acc.q_ids_hash, '\x00000000000000000000000000000000') + FROM acc + WHERE s.service_id = acc.service_id; +END; +$$; + +SELECT update_all_aggregates(); + +CREATE FUNCTION update_aggregates(p_service_id BYTEA, p_role TEXT, p_queue_id BYTEA, p_change BIGINT) RETURNS VOID +LANGUAGE plpgsql +AS $$ +BEGIN + UPDATE services + SET queue_count = queue_count + p_change, + queue_ids_hash = xor_combine(queue_ids_hash, public.digest(p_queue_id, 'md5')) + WHERE service_id = p_service_id AND service_role = p_role; +END; +$$; + +CREATE FUNCTION on_queue_insert() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF NEW.rcv_service_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.rcv_service_id, 'M', NEW.recipient_id, 1); + END IF; + IF NEW.ntf_service_id IS NOT NULL AND NEW.notifier_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.ntf_service_id, 'N', NEW.notifier_id, 1); + END IF; + RETURN NEW; +END; +$$; + +CREATE FUNCTION on_queue_delete() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF OLD.deleted_at IS NULL THEN + IF OLD.rcv_service_id IS NOT NULL THEN + PERFORM update_aggregates(OLD.rcv_service_id, 'M', OLD.recipient_id, -1); + END IF; + IF OLD.ntf_service_id IS NOT NULL AND OLD.notifier_id IS NOT NULL THEN + PERFORM update_aggregates(OLD.ntf_service_id, 'N', OLD.notifier_id, -1); + END IF; + END IF; + RETURN OLD; +END; +$$; + +CREATE FUNCTION on_queue_update() RETURNS TRIGGER +LANGUAGE plpgsql +AS $$ +BEGIN + IF OLD.deleted_at IS NULL AND OLD.rcv_service_id IS NOT NULL THEN + IF NOT (NEW.deleted_at IS NULL AND NEW.rcv_service_id IS NOT NULL) THEN + PERFORM update_aggregates(OLD.rcv_service_id, 'M', OLD.recipient_id, -1); + ELSIF OLD.rcv_service_id IS DISTINCT FROM NEW.rcv_service_id THEN + PERFORM update_aggregates(OLD.rcv_service_id, 'M', OLD.recipient_id, -1); + PERFORM update_aggregates(NEW.rcv_service_id, 'M', NEW.recipient_id, 1); + END IF; + ELSIF NEW.deleted_at IS NULL AND NEW.rcv_service_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.rcv_service_id, 'M', NEW.recipient_id, 1); + END IF; + + IF OLD.deleted_at IS NULL AND OLD.ntf_service_id IS NOT NULL AND OLD.notifier_id IS NOT NULL THEN + IF NOT (NEW.deleted_at IS NULL AND NEW.ntf_service_id IS NOT NULL AND NEW.notifier_id IS NOT NULL) THEN + PERFORM update_aggregates(OLD.ntf_service_id, 'N', OLD.notifier_id, -1); + ELSIF OLD.ntf_service_id IS DISTINCT FROM NEW.ntf_service_id OR OLD.notifier_id IS DISTINCT FROM NEW.notifier_id THEN + PERFORM update_aggregates(OLD.ntf_service_id, 'N', OLD.notifier_id, -1); + PERFORM update_aggregates(NEW.ntf_service_id, 'N', NEW.notifier_id, 1); + END IF; + ELSIF NEW.deleted_at IS NULL AND NEW.ntf_service_id IS NOT NULL AND NEW.notifier_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.ntf_service_id, 'N', NEW.notifier_id, 1); + END IF; + RETURN NEW; +END; +$$; + +CREATE TRIGGER tr_queue_insert +AFTER INSERT ON msg_queues +FOR EACH ROW EXECUTE PROCEDURE on_queue_insert(); + +CREATE TRIGGER tr_queue_delete +AFTER DELETE ON msg_queues +FOR EACH ROW EXECUTE PROCEDURE on_queue_delete(); + +CREATE TRIGGER tr_queue_update +AFTER UPDATE ON msg_queues +FOR EACH ROW EXECUTE PROCEDURE on_queue_update(); + |] + +down_m20250915_queue_ids_hash :: Text +down_m20250915_queue_ids_hash = + [r| +DROP TRIGGER tr_queue_insert ON msg_queues; +DROP TRIGGER tr_queue_delete ON msg_queues; +DROP TRIGGER tr_queue_update ON msg_queues; + +DROP FUNCTION on_queue_insert; +DROP FUNCTION on_queue_delete; +DROP FUNCTION on_queue_update; + +DROP FUNCTION update_aggregates; + +DROP FUNCTION update_all_aggregates; + +ALTER TABLE services + DROP COLUMN queue_count, + DROP COLUMN queue_ids_hash; + |] + <> dropXorHashFuncs diff --git a/src/Simplex/Messaging/Server/QueueStore/Postgres/server_schema.sql b/src/Simplex/Messaging/Server/QueueStore/Postgres/server_schema.sql index 433d45473..f0da5272d 100644 --- a/src/Simplex/Messaging/Server/QueueStore/Postgres/server_schema.sql +++ b/src/Simplex/Messaging/Server/QueueStore/Postgres/server_schema.sql @@ -104,6 +104,71 @@ $$; +CREATE FUNCTION smp_server.on_queue_delete() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF OLD.deleted_at IS NULL THEN + IF OLD.rcv_service_id IS NOT NULL THEN + PERFORM update_aggregates(OLD.rcv_service_id, 'M', OLD.recipient_id, -1); + END IF; + IF OLD.ntf_service_id IS NOT NULL AND OLD.notifier_id IS NOT NULL THEN + PERFORM update_aggregates(OLD.ntf_service_id, 'N', OLD.notifier_id, -1); + END IF; + END IF; + RETURN OLD; +END; +$$; + + + +CREATE FUNCTION smp_server.on_queue_insert() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF NEW.rcv_service_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.rcv_service_id, 'M', NEW.recipient_id, 1); + END IF; + IF NEW.ntf_service_id IS NOT NULL AND NEW.notifier_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.ntf_service_id, 'N', NEW.notifier_id, 1); + END IF; + RETURN NEW; +END; +$$; + + + +CREATE FUNCTION smp_server.on_queue_update() RETURNS trigger + LANGUAGE plpgsql + AS $$ +BEGIN + IF OLD.deleted_at IS NULL AND OLD.rcv_service_id IS NOT NULL THEN + IF NOT (NEW.deleted_at IS NULL AND NEW.rcv_service_id IS NOT NULL) THEN + PERFORM update_aggregates(OLD.rcv_service_id, 'M', OLD.recipient_id, -1); + ELSIF OLD.rcv_service_id IS DISTINCT FROM NEW.rcv_service_id THEN + PERFORM update_aggregates(OLD.rcv_service_id, 'M', OLD.recipient_id, -1); + PERFORM update_aggregates(NEW.rcv_service_id, 'M', NEW.recipient_id, 1); + END IF; + ELSIF NEW.deleted_at IS NULL AND NEW.rcv_service_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.rcv_service_id, 'M', NEW.recipient_id, 1); + END IF; + + IF OLD.deleted_at IS NULL AND OLD.ntf_service_id IS NOT NULL AND OLD.notifier_id IS NOT NULL THEN + IF NOT (NEW.deleted_at IS NULL AND NEW.ntf_service_id IS NOT NULL AND NEW.notifier_id IS NOT NULL) THEN + PERFORM update_aggregates(OLD.ntf_service_id, 'N', OLD.notifier_id, -1); + ELSIF OLD.ntf_service_id IS DISTINCT FROM NEW.ntf_service_id OR OLD.notifier_id IS DISTINCT FROM NEW.notifier_id THEN + PERFORM update_aggregates(OLD.ntf_service_id, 'N', OLD.notifier_id, -1); + PERFORM update_aggregates(NEW.ntf_service_id, 'N', NEW.notifier_id, 1); + END IF; + ELSIF NEW.deleted_at IS NULL AND NEW.ntf_service_id IS NOT NULL AND NEW.notifier_id IS NOT NULL THEN + PERFORM update_aggregates(NEW.ntf_service_id, 'N', NEW.notifier_id, 1); + END IF; + RETURN NEW; +END; +$$; + + + CREATE FUNCTION smp_server.try_del_msg(p_recipient_id bytea, p_msg_id bytea) RETURNS TABLE(r_msg_id bytea, r_msg_ts bigint, r_msg_quota boolean, r_msg_ntf_flag boolean, r_msg_body bytea) LANGUAGE plpgsql AS $$ @@ -225,6 +290,43 @@ $$; +CREATE FUNCTION smp_server.update_aggregates(p_service_id bytea, p_role text, p_queue_id bytea, p_change bigint) RETURNS void + LANGUAGE plpgsql + AS $$ +BEGIN + UPDATE services + SET queue_count = queue_count + p_change, + queue_ids_hash = xor_combine(queue_ids_hash, public.digest(p_queue_id, 'md5')) + WHERE service_id = p_service_id AND service_role = p_role; +END; +$$; + + + +CREATE FUNCTION smp_server.update_all_aggregates() RETURNS void + LANGUAGE plpgsql + AS $$ +BEGIN + WITH acc AS ( + SELECT + s.service_id, + count(1) as q_count, + xor_aggregate(public.digest(CASE WHEN s.service_role = 'M' THEN q.recipient_id ELSE COALESCE(q.notifier_id, '\x00000000000000000000000000000000') END, 'md5')) AS q_ids_hash + FROM services s + JOIN msg_queues q ON (s.service_id = q.rcv_service_id AND s.service_role = 'M') OR (s.service_id = q.ntf_service_id AND s.service_role = 'N') + WHERE q.deleted_at IS NULL + GROUP BY s.service_id + ) + UPDATE services s + SET queue_count = COALESCE(acc.q_count, 0), + queue_ids_hash = COALESCE(acc.q_ids_hash, '\x00000000000000000000000000000000') + FROM acc + WHERE s.service_id = acc.service_id; +END; +$$; + + + CREATE FUNCTION smp_server.write_message(p_recipient_id bytea, p_msg_id bytea, p_msg_ts bigint, p_msg_quota boolean, p_msg_ntf_flag boolean, p_msg_body bytea, p_quota integer) RETURNS TABLE(quota_written boolean, was_empty boolean) LANGUAGE plpgsql AS $$ @@ -256,6 +358,34 @@ END; $$; + +CREATE FUNCTION smp_server.xor_combine(state bytea, value bytea) RETURNS bytea + LANGUAGE plpgsql IMMUTABLE STRICT + AS $$ +DECLARE + result BYTEA := state; + i INTEGER; + len INTEGER := octet_length(value); +BEGIN + IF octet_length(state) != len THEN + RAISE EXCEPTION 'Inputs must be equal length (% != %)', octet_length(state), len; + END IF; + FOR i IN 0..len-1 LOOP + result := set_byte(result, i, get_byte(state, i) # get_byte(value, i)); + END LOOP; + RETURN result; +END; +$$; + + + +CREATE AGGREGATE smp_server.xor_aggregate(bytea) ( + SFUNC = smp_server.xor_combine, + STYPE = bytea, + INITCOND = '\x00000000000000000000000000000000' +); + + SET default_table_access_method = heap; @@ -320,7 +450,9 @@ CREATE TABLE smp_server.services ( service_role text NOT NULL, service_cert bytea NOT NULL, service_cert_hash bytea NOT NULL, - created_at bigint NOT NULL + created_at bigint NOT NULL, + queue_count bigint DEFAULT 0 NOT NULL, + queue_ids_hash bytea DEFAULT '\x00000000000000000000000000000000'::bytea NOT NULL ); @@ -390,6 +522,18 @@ CREATE INDEX idx_services_service_role ON smp_server.services USING btree (servi +CREATE TRIGGER tr_queue_delete AFTER DELETE ON smp_server.msg_queues FOR EACH ROW EXECUTE FUNCTION smp_server.on_queue_delete(); + + + +CREATE TRIGGER tr_queue_insert AFTER INSERT ON smp_server.msg_queues FOR EACH ROW EXECUTE FUNCTION smp_server.on_queue_insert(); + + + +CREATE TRIGGER tr_queue_update AFTER UPDATE ON smp_server.msg_queues FOR EACH ROW EXECUTE FUNCTION smp_server.on_queue_update(); + + + ALTER TABLE ONLY smp_server.messages ADD CONSTRAINT messages_recipient_id_fkey FOREIGN KEY (recipient_id) REFERENCES smp_server.msg_queues(recipient_id) ON UPDATE RESTRICT ON DELETE CASCADE; diff --git a/src/Simplex/Messaging/Server/QueueStore/STM.hs b/src/Simplex/Messaging/Server/QueueStore/STM.hs index ad98698db..3a236076c 100644 --- a/src/Simplex/Messaging/Server/QueueStore/STM.hs +++ b/src/Simplex/Messaging/Server/QueueStore/STM.hs @@ -17,6 +17,7 @@ module Simplex.Messaging.Server.QueueStore.STM ( STMQueueStore (..), STMService (..), + foldRcvServiceQueues, setStoreLog, withLog', readQueueRecIO, @@ -27,6 +28,7 @@ where import qualified Control.Exception as E import Control.Logger.Simple import Control.Monad +import Data.Bifunctor (first) import Data.Bitraversable (bimapM) import Data.Functor (($>)) import Data.Int (Int64) @@ -45,7 +47,7 @@ import Simplex.Messaging.SystemTime import Simplex.Messaging.TMap (TMap) import qualified Simplex.Messaging.TMap as TM import Simplex.Messaging.Transport (SMPServiceRole (..)) -import Simplex.Messaging.Util (anyM, ifM, tshow, ($>>), ($>>=), (<$$)) +import Simplex.Messaging.Util (anyM, ifM, tshow, ($>>), ($>>=), (<$$), (<$$>)) import System.IO import UnliftIO.STM @@ -61,8 +63,8 @@ data STMQueueStore q = STMQueueStore data STMService = STMService { serviceRec :: ServiceRec, - serviceRcvQueues :: TVar (Set RecipientId), - serviceNtfQueues :: TVar (Set NotifierId) + serviceRcvQueues :: TVar (Set RecipientId, IdsHash), + serviceNtfQueues :: TVar (Set NotifierId, IdsHash) } setStoreLog :: STMQueueStore q -> StoreLog 'WriteMode -> IO () @@ -112,7 +114,7 @@ instance StoreQueueClass q => QueueStoreClass q (STMQueueStore q) where } where serviceCount role = M.foldl' (\ !n s -> if serviceRole (serviceRec s) == role then n + 1 else n) 0 - serviceQueuesCount serviceSel = foldM (\n s -> (n +) . S.size <$> readTVarIO (serviceSel s)) 0 + serviceQueuesCount serviceSel = foldM (\n s -> (n +) . S.size . fst <$> readTVarIO (serviceSel s)) 0 addQueue_ :: STMQueueStore q -> (RecipientId -> QueueRec -> IO q) -> RecipientId -> QueueRec -> IO (Either ErrorType q) addQueue_ st mkQ rId qr@QueueRec {senderId = sId, notifier, queueData, rcvServiceId} = do @@ -303,8 +305,8 @@ instance StoreQueueClass q => QueueStoreClass q (STMQueueStore q) where TM.insert fp newSrvId serviceCerts pure $ Right (newSrvId, True) newSTMService = do - serviceRcvQueues <- newTVar S.empty - serviceNtfQueues <- newTVar S.empty + serviceRcvQueues <- newTVar (S.empty, mempty) + serviceNtfQueues <- newTVar (S.empty, mempty) pure STMService {serviceRec = sr, serviceRcvQueues, serviceNtfQueues} setQueueService :: (PartyI p, ServiceParty p) => STMQueueStore q -> q -> SParty p -> Maybe ServiceId -> IO (Either ErrorType ()) @@ -330,7 +332,7 @@ instance StoreQueueClass q => QueueStoreClass q (STMQueueStore q) where let !q' = Just q {notifier = Just nc {ntfServiceId = serviceId}} updateServiceQueues serviceNtfQueues nId prevNtfSrvId writeTVar qr q' $> Right () - updateServiceQueues :: (STMService -> TVar (Set QueueId)) -> QueueId -> Maybe ServiceId -> STM () + updateServiceQueues :: (STMService -> TVar (Set QueueId, IdsHash)) -> QueueId -> Maybe ServiceId -> STM () updateServiceQueues serviceSel qId prevSrvId = do mapM_ (removeServiceQueue st serviceSel qId) prevSrvId mapM_ (addServiceQueue st serviceSel qId) serviceId @@ -345,20 +347,30 @@ instance StoreQueueClass q => QueueStoreClass q (STMQueueStore q) where pure $ Right (ssNtfs', deleteNtfs) where addService (ssNtfs, ntfs') (serviceId, s) = do - snIds <- readTVarIO $ serviceNtfQueues s + (snIds, _) <- readTVarIO $ serviceNtfQueues s let (sNtfs, restNtfs) = partition (\(nId, _) -> S.member nId snIds) ntfs' pure ((Just serviceId, sNtfs) : ssNtfs, restNtfs) - getServiceQueueCount :: (PartyI p, ServiceParty p) => STMQueueStore q -> SParty p -> ServiceId -> IO (Either ErrorType Int64) - getServiceQueueCount st party serviceId = + getServiceQueueCountHash :: (PartyI p, ServiceParty p) => STMQueueStore q -> SParty p -> ServiceId -> IO (Either ErrorType (Int64, IdsHash)) + getServiceQueueCountHash st party serviceId = TM.lookupIO serviceId (services st) >>= - maybe (pure $ Left AUTH) (fmap (Right . fromIntegral . S.size) . readTVarIO . serviceSel) + maybe (pure $ Left AUTH) (fmap (Right . first (fromIntegral . S.size)) . readTVarIO . serviceSel) where - serviceSel :: STMService -> TVar (Set QueueId) + serviceSel :: STMService -> TVar (Set QueueId, IdsHash) serviceSel = case party of SRecipientService -> serviceRcvQueues SNotifierService -> serviceNtfQueues +foldRcvServiceQueues :: StoreQueueClass q => STMQueueStore q -> ServiceId -> (a -> (q, QueueRec) -> IO a) -> a -> IO a +foldRcvServiceQueues st serviceId f acc = + TM.lookupIO serviceId (services st) >>= \case + Nothing -> pure acc + Just s -> + readTVarIO (serviceRcvQueues s) + >>= foldM (\a -> get >=> maybe (pure a) (f a)) acc . fst + where + get rId = TM.lookupIO rId (queues st) $>>= \q -> (q,) <$$> readTVarIO (queueRec q) + withQueueRec :: TVar (Maybe QueueRec) -> (QueueRec -> STM a) -> IO (Either ErrorType a) withQueueRec qr a = atomically $ readQueueRec qr >>= mapM a @@ -368,16 +380,23 @@ setStatus qr status = Just q -> (Right (), Just q {status}) Nothing -> (Left AUTH, Nothing) -addServiceQueue :: STMQueueStore q -> (STMService -> TVar (Set QueueId)) -> QueueId -> ServiceId -> STM () -addServiceQueue st serviceSel qId serviceId = - TM.lookup serviceId (services st) >>= mapM_ (\s -> modifyTVar' (serviceSel s) (S.insert qId)) +addServiceQueue :: STMQueueStore q -> (STMService -> TVar (Set QueueId, IdsHash)) -> QueueId -> ServiceId -> STM () +addServiceQueue = setServiceQueues_ S.insert {-# INLINE addServiceQueue #-} -removeServiceQueue :: STMQueueStore q -> (STMService -> TVar (Set QueueId)) -> QueueId -> ServiceId -> STM () -removeServiceQueue st serviceSel qId serviceId = - TM.lookup serviceId (services st) >>= mapM_ (\s -> modifyTVar' (serviceSel s) (S.delete qId)) +removeServiceQueue :: STMQueueStore q -> (STMService -> TVar (Set QueueId, IdsHash)) -> QueueId -> ServiceId -> STM () +removeServiceQueue = setServiceQueues_ S.delete {-# INLINE removeServiceQueue #-} +setServiceQueues_ :: (QueueId -> Set QueueId -> Set QueueId) -> STMQueueStore q -> (STMService -> TVar (Set QueueId, IdsHash)) -> QueueId -> ServiceId -> STM () +setServiceQueues_ updateSet st serviceSel qId serviceId = + TM.lookup serviceId (services st) >>= mapM_ (\v -> modifyTVar' (serviceSel v) update) + where + update (s, idsHash) = + let !s' = updateSet qId s + !idsHash' = queueIdHash qId <> idsHash + in (s', idsHash') + removeNotifier :: STMQueueStore q -> NtfCreds -> STM () removeNotifier st NtfCreds {notifierId = nId, ntfServiceId} = do TM.delete nId $ notifiers st diff --git a/src/Simplex/Messaging/Server/QueueStore/Types.hs b/src/Simplex/Messaging/Server/QueueStore/Types.hs index df7947a62..7d1d439bd 100644 --- a/src/Simplex/Messaging/Server/QueueStore/Types.hs +++ b/src/Simplex/Messaging/Server/QueueStore/Types.hs @@ -52,7 +52,7 @@ class StoreQueueClass q => QueueStoreClass q s where getCreateService :: s -> ServiceRec -> IO (Either ErrorType ServiceId) setQueueService :: (PartyI p, ServiceParty p) => s -> q -> SParty p -> Maybe ServiceId -> IO (Either ErrorType ()) getQueueNtfServices :: s -> [(NotifierId, a)] -> IO (Either ErrorType ([(Maybe ServiceId, [(NotifierId, a)])], [(NotifierId, a)])) - getServiceQueueCount :: (PartyI p, ServiceParty p) => s -> SParty p -> ServiceId -> IO (Either ErrorType Int64) + getServiceQueueCountHash :: (PartyI p, ServiceParty p) => s -> SParty p -> ServiceId -> IO (Either ErrorType (Int64, IdsHash)) data EntityCounts = EntityCounts { queueCount :: Int, diff --git a/src/Simplex/Messaging/Server/Stats.hs b/src/Simplex/Messaging/Server/Stats.hs index 60b2a372f..e8291759e 100644 --- a/src/Simplex/Messaging/Server/Stats.hs +++ b/src/Simplex/Messaging/Server/Stats.hs @@ -119,6 +119,8 @@ data ServerStats = ServerStats pMsgFwdsRecv :: IORef Int, rcvServices :: ServiceStats, ntfServices :: ServiceStats, + rcvServicesSubMsg :: IORef Int, + rcvServicesSubDuplicate :: IORef Int, qCount :: IORef Int, msgCount :: IORef Int, ntfCount :: IORef Int @@ -178,6 +180,8 @@ data ServerStatsData = ServerStatsData _pMsgFwdsRecv :: Int, _ntfServices :: ServiceStatsData, _rcvServices :: ServiceStatsData, + _rcvServicesSubMsg :: Int, + _rcvServicesSubDuplicate :: Int, _qCount :: Int, _msgCount :: Int, _ntfCount :: Int @@ -239,6 +243,8 @@ newServerStats ts = do pMsgFwdsRecv <- newIORef 0 rcvServices <- newServiceStats ntfServices <- newServiceStats + rcvServicesSubMsg <- newIORef 0 + rcvServicesSubDuplicate <- newIORef 0 qCount <- newIORef 0 msgCount <- newIORef 0 ntfCount <- newIORef 0 @@ -297,6 +303,8 @@ newServerStats ts = do pMsgFwdsRecv, rcvServices, ntfServices, + rcvServicesSubMsg, + rcvServicesSubDuplicate, qCount, msgCount, ntfCount @@ -357,6 +365,8 @@ getServerStatsData s = do _pMsgFwdsRecv <- readIORef $ pMsgFwdsRecv s _rcvServices <- getServiceStatsData $ rcvServices s _ntfServices <- getServiceStatsData $ ntfServices s + _rcvServicesSubMsg <- readIORef $ rcvServicesSubMsg s + _rcvServicesSubDuplicate <- readIORef $ rcvServicesSubDuplicate s _qCount <- readIORef $ qCount s _msgCount <- readIORef $ msgCount s _ntfCount <- readIORef $ ntfCount s @@ -415,6 +425,8 @@ getServerStatsData s = do _pMsgFwdsRecv, _rcvServices, _ntfServices, + _rcvServicesSubMsg, + _rcvServicesSubDuplicate, _qCount, _msgCount, _ntfCount @@ -476,6 +488,8 @@ setServerStats s d = do writeIORef (pMsgFwdsRecv s) $! _pMsgFwdsRecv d setServiceStats (rcvServices s) $! _rcvServices d setServiceStats (ntfServices s) $! _ntfServices d + writeIORef (rcvServicesSubMsg s) $! _rcvServicesSubMsg d + writeIORef (rcvServicesSubDuplicate s) $! _rcvServicesSubDuplicate d writeIORef (qCount s) $! _qCount d writeIORef (msgCount s) $! _msgCount d writeIORef (ntfCount s) $! _ntfCount d @@ -669,6 +683,8 @@ instance StrEncoding ServerStatsData where _pMsgFwdsRecv, _rcvServices, _ntfServices, + _rcvServicesSubMsg = 0, + _rcvServicesSubDuplicate = 0, _qCount, _msgCount = 0, _ntfCount = 0 @@ -854,7 +870,15 @@ data ServiceStats = ServiceStats srvSubCount :: IORef Int, srvSubDuplicate :: IORef Int, srvSubQueues :: IORef Int, - srvSubEnd :: IORef Int + srvSubEnd :: IORef Int, + -- counts of subscriptions + srvSubOk :: IORef Int, -- server has the same queues as expected + srvSubMore :: IORef Int, -- server has more queues than expected + srvSubFewer :: IORef Int, -- server has fewer queues than expected + srvSubDiff :: IORef Int, -- server has the same count, but different queues than expected (based on xor-hash) + -- adds actual deviations + srvSubMoreTotal :: IORef Int, -- server has more queues than expected, adds diff + srvSubFewerTotal :: IORef Int } data ServiceStatsData = ServiceStatsData @@ -865,7 +889,13 @@ data ServiceStatsData = ServiceStatsData _srvSubCount :: Int, _srvSubDuplicate :: Int, _srvSubQueues :: Int, - _srvSubEnd :: Int + _srvSubEnd :: Int, + _srvSubOk :: Int, + _srvSubMore :: Int, + _srvSubFewer :: Int, + _srvSubDiff :: Int, + _srvSubMoreTotal :: Int, + _srvSubFewerTotal :: Int } deriving (Show) @@ -879,7 +909,13 @@ newServiceStatsData = _srvSubCount = 0, _srvSubDuplicate = 0, _srvSubQueues = 0, - _srvSubEnd = 0 + _srvSubEnd = 0, + _srvSubOk = 0, + _srvSubMore = 0, + _srvSubFewer = 0, + _srvSubDiff = 0, + _srvSubMoreTotal = 0, + _srvSubFewerTotal = 0 } newServiceStats :: IO ServiceStats @@ -892,6 +928,12 @@ newServiceStats = do srvSubDuplicate <- newIORef 0 srvSubQueues <- newIORef 0 srvSubEnd <- newIORef 0 + srvSubOk <- newIORef 0 + srvSubMore <- newIORef 0 + srvSubFewer <- newIORef 0 + srvSubDiff <- newIORef 0 + srvSubMoreTotal <- newIORef 0 + srvSubFewerTotal <- newIORef 0 pure ServiceStats { srvAssocNew, @@ -901,7 +943,13 @@ newServiceStats = do srvSubCount, srvSubDuplicate, srvSubQueues, - srvSubEnd + srvSubEnd, + srvSubOk, + srvSubMore, + srvSubFewer, + srvSubDiff, + srvSubMoreTotal, + srvSubFewerTotal } getServiceStatsData :: ServiceStats -> IO ServiceStatsData @@ -914,6 +962,12 @@ getServiceStatsData s = do _srvSubDuplicate <- readIORef $ srvSubDuplicate s _srvSubQueues <- readIORef $ srvSubQueues s _srvSubEnd <- readIORef $ srvSubEnd s + _srvSubOk <- readIORef $ srvSubOk s + _srvSubMore <- readIORef $ srvSubMore s + _srvSubFewer <- readIORef $ srvSubFewer s + _srvSubDiff <- readIORef $ srvSubDiff s + _srvSubMoreTotal <- readIORef $ srvSubMoreTotal s + _srvSubFewerTotal <- readIORef $ srvSubFewerTotal s pure ServiceStatsData { _srvAssocNew, @@ -923,7 +977,13 @@ getServiceStatsData s = do _srvSubCount, _srvSubDuplicate, _srvSubQueues, - _srvSubEnd + _srvSubEnd, + _srvSubOk, + _srvSubMore, + _srvSubFewer, + _srvSubDiff, + _srvSubMoreTotal, + _srvSubFewerTotal } getResetServiceStatsData :: ServiceStats -> IO ServiceStatsData @@ -936,6 +996,12 @@ getResetServiceStatsData s = do _srvSubDuplicate <- atomicSwapIORef (srvSubDuplicate s) 0 _srvSubQueues <- atomicSwapIORef (srvSubQueues s) 0 _srvSubEnd <- atomicSwapIORef (srvSubEnd s) 0 + _srvSubOk <- atomicSwapIORef (srvSubOk s) 0 + _srvSubMore <- atomicSwapIORef (srvSubMore s) 0 + _srvSubFewer <- atomicSwapIORef (srvSubFewer s) 0 + _srvSubDiff <- atomicSwapIORef (srvSubDiff s) 0 + _srvSubMoreTotal <- atomicSwapIORef (srvSubMoreTotal s) 0 + _srvSubFewerTotal <- atomicSwapIORef (srvSubFewerTotal s) 0 pure ServiceStatsData { _srvAssocNew, @@ -945,7 +1011,13 @@ getResetServiceStatsData s = do _srvSubCount, _srvSubDuplicate, _srvSubQueues, - _srvSubEnd + _srvSubEnd, + _srvSubOk, + _srvSubMore, + _srvSubFewer, + _srvSubDiff, + _srvSubMoreTotal, + _srvSubFewerTotal } -- this function is not thread safe, it is used on server start only @@ -959,6 +1031,12 @@ setServiceStats s d = do writeIORef (srvSubDuplicate s) $! _srvSubDuplicate d writeIORef (srvSubQueues s) $! _srvSubQueues d writeIORef (srvSubEnd s) $! _srvSubEnd d + writeIORef (srvSubOk s) $! _srvSubOk d + writeIORef (srvSubMore s) $! _srvSubMore d + writeIORef (srvSubFewer s) $! _srvSubFewer d + writeIORef (srvSubDiff s) $! _srvSubDiff d + writeIORef (srvSubMoreTotal s) $! _srvSubMoreTotal d + writeIORef (srvSubFewerTotal s) $! _srvSubFewerTotal d instance StrEncoding ServiceStatsData where strEncode ServiceStatsData {_srvAssocNew, _srvAssocDuplicate, _srvAssocUpdated, _srvAssocRemoved, _srvSubCount, _srvSubDuplicate, _srvSubQueues, _srvSubEnd} = @@ -996,7 +1074,13 @@ instance StrEncoding ServiceStatsData where _srvSubCount, _srvSubDuplicate, _srvSubQueues, - _srvSubEnd + _srvSubEnd, + _srvSubOk = 0, + _srvSubMore = 0, + _srvSubFewer = 0, + _srvSubDiff = 0, + _srvSubMoreTotal = 0, + _srvSubFewerTotal = 0 } data TimeBuckets = TimeBuckets diff --git a/src/Simplex/Messaging/Server/StoreLog/ReadWrite.hs b/src/Simplex/Messaging/Server/StoreLog/ReadWrite.hs index 81adcc04d..6bf327f70 100644 --- a/src/Simplex/Messaging/Server/StoreLog/ReadWrite.hs +++ b/src/Simplex/Messaging/Server/StoreLog/ReadWrite.hs @@ -64,7 +64,7 @@ readQueueStore tty mkQ f st = readLogLines tty f $ \_ -> processLine Left e -> logError $ errPfx <> tshow e where errPfx = "STORE: getCreateService, stored service " <> decodeLatin1 (strEncode serviceId) <> ", " - QueueService rId (ASP party) serviceId -> withQueue rId "QueueService" $ \q -> setQueueService st q party serviceId + QueueService qId (ASP party) serviceId -> withQueue qId "QueueService" $ \q -> setQueueService st q party serviceId printError :: String -> IO () printError e = B.putStrLn $ "Error parsing log: " <> B.pack e <> " - " <> s withQueue :: forall a. RecipientId -> T.Text -> (q -> IO (Either ErrorType a)) -> IO () diff --git a/src/Simplex/Messaging/Transport.hs b/src/Simplex/Messaging/Transport.hs index e2e912875..f1eb1a8bd 100644 --- a/src/Simplex/Messaging/Transport.hs +++ b/src/Simplex/Messaging/Transport.hs @@ -56,6 +56,7 @@ module Simplex.Messaging.Transport serviceCertsSMPVersion, newNtfCredsSMPVersion, clientNoticesSMPVersion, + rcvServiceSMPVersion, simplexMQVersion, smpBlockSize, TransportConfig (..), @@ -170,6 +171,7 @@ smpBlockSize = 16384 -- 16 - service certificates (5/31/2025) -- 17 - create notification credentials with NEW (7/12/2025) -- 18 - support client notices (10/10/2025) +-- 19 - service subscriptions to messages (10/20/2025) data SMPVersion @@ -218,6 +220,9 @@ newNtfCredsSMPVersion = VersionSMP 17 clientNoticesSMPVersion :: VersionSMP clientNoticesSMPVersion = VersionSMP 18 +rcvServiceSMPVersion :: VersionSMP +rcvServiceSMPVersion = VersionSMP 19 + minClientSMPRelayVersion :: VersionSMP minClientSMPRelayVersion = VersionSMP 6 @@ -225,13 +230,13 @@ minServerSMPRelayVersion :: VersionSMP minServerSMPRelayVersion = VersionSMP 6 currentClientSMPRelayVersion :: VersionSMP -currentClientSMPRelayVersion = VersionSMP 18 +currentClientSMPRelayVersion = VersionSMP 19 legacyServerSMPRelayVersion :: VersionSMP legacyServerSMPRelayVersion = VersionSMP 6 currentServerSMPRelayVersion :: VersionSMP -currentServerSMPRelayVersion = VersionSMP 18 +currentServerSMPRelayVersion = VersionSMP 19 -- Max SMP protocol version to be used in e2e encrypted -- connection between client and server, as defined by SMP proxy. @@ -239,7 +244,7 @@ currentServerSMPRelayVersion = VersionSMP 18 -- to prevent client version fingerprinting by the -- destination relays when clients upgrade at different times. proxiedSMPRelayVersion :: VersionSMP -proxiedSMPRelayVersion = VersionSMP 17 +proxiedSMPRelayVersion = VersionSMP 18 -- minimal supported protocol version is 6 -- TODO remove code that supports sending commands without batching @@ -555,7 +560,6 @@ data SMPClientHandshake = SMPClientHandshake keyHash :: C.KeyHash, -- | pub key to agree shared secret for entity ID encryption, shared secret for command authorization is agreed using per-queue keys. authPubKey :: Maybe C.PublicKeyX25519, - -- TODO [certs] remove proxyServer, as serviceInfo includes it as clientRole -- | Whether connecting client is a proxy server (send from SMP v12). -- This property, if True, disables additional transport encrytion inside TLS. -- (Proxy server connection already has additional encryption, so this layer is not needed there). @@ -823,7 +827,7 @@ smpClientHandshake c ks_ keyHash@(C.KeyHash kh) vRange proxyServer serviceKeys_ serviceKeys = case serviceKeys_ of Just sks | v >= serviceCertsSMPVersion && certificateSent c -> Just sks _ -> Nothing - clientService = mkClientService <$> serviceKeys + clientService = mkClientService v =<< serviceKeys hs = SMPClientHandshake {smpVersion = v, keyHash, authPubKey = fst <$> ks_, proxyServer, clientService} sendHandshake th hs service <- mapM getClientService serviceKeys @@ -831,10 +835,12 @@ smpClientHandshake c ks_ keyHash@(C.KeyHash kh) vRange proxyServer serviceKeys_ Nothing -> throwE TEVersion where th@THandle {params = THandleParams {sessionId}} = smpTHandle c - mkClientService :: (ServiceCredentials, C.KeyPairEd25519) -> SMPClientHandshakeService - mkClientService (ServiceCredentials {serviceRole, serviceCreds, serviceSignKey}, (k, _)) = - let sk = C.signX509 serviceSignKey $ C.publicToX509 k - in SMPClientHandshakeService {serviceRole, serviceCertKey = CertChainPubKey (fst serviceCreds) sk} + mkClientService :: VersionSMP -> (ServiceCredentials, C.KeyPairEd25519) -> Maybe SMPClientHandshakeService + mkClientService v (ServiceCredentials {serviceRole, serviceCreds, serviceSignKey}, (k, _)) + | serviceRole == SRMessaging && v < rcvServiceSMPVersion = Nothing + | otherwise = + let sk = C.signX509 serviceSignKey $ C.publicToX509 k + in Just SMPClientHandshakeService {serviceRole, serviceCertKey = CertChainPubKey (fst serviceCreds) sk} getClientService :: (ServiceCredentials, C.KeyPairEd25519) -> ExceptT TransportError IO THClientService getClientService (ServiceCredentials {serviceRole, serviceCertHash}, (_, pk)) = getHandshake th >>= \case diff --git a/src/Simplex/Messaging/Transport/HTTP2/Client.hs b/src/Simplex/Messaging/Transport/HTTP2/Client.hs index 4c38472d8..ca0714225 100644 --- a/src/Simplex/Messaging/Transport/HTTP2/Client.hs +++ b/src/Simplex/Messaging/Transport/HTTP2/Client.hs @@ -24,7 +24,7 @@ module Simplex.Messaging.Transport.HTTP2.Client ) where import Control.Concurrent.Async -import Control.Exception (IOException, try) +import Control.Exception (Handler (..), IOException, SomeAsyncException, SomeException) import qualified Control.Exception as E import Control.Monad import Data.Functor (($>)) @@ -103,9 +103,16 @@ defaultHTTP2ClientConfig = suportedTLSParams = http2TLSParams } -data HTTP2ClientError = HCResponseTimeout | HCNetworkError NetworkError | HCIOError IOException +data HTTP2ClientError = HCResponseTimeout | HCNetworkError NetworkError | HCIOError String deriving (Show) +httpClientHandlers :: [Handler (Either HTTP2ClientError a)] +httpClientHandlers = + [ Handler $ \(e :: IOException) -> pure $ Left $ HCIOError $ E.displayException e, + Handler $ \(e :: SomeAsyncException) -> E.throwIO e, + Handler $ \(e :: SomeException) -> pure $ Left $ HCNetworkError $ toNetworkError e + ] + getHTTP2Client :: HostName -> ServiceName -> Maybe XS.CertificateStore -> HTTP2ClientConfig -> IO () -> IO (Either HTTP2ClientError HTTP2Client) getHTTP2Client host port = getVerifiedHTTP2Client Nothing (THDomainName host) port Nothing @@ -124,7 +131,7 @@ attachHTTP2Client config host port disconnected bufferSize tls = getVerifiedHTTP getVerifiedHTTP2ClientWith :: forall p. TransportPeerI p => HTTP2ClientConfig -> TransportHost -> ServiceName -> IO () -> ((TLS p -> H.Client HTTP2Response) -> IO HTTP2Response) -> IO (Either HTTP2ClientError HTTP2Client) getVerifiedHTTP2ClientWith config host port disconnected setup = (mkHTTPS2Client >>= runClient) - `E.catch` \(e :: IOException) -> pure . Left $ HCIOError e + `E.catches` httpClientHandlers where mkHTTPS2Client :: IO HClient mkHTTPS2Client = do @@ -190,9 +197,9 @@ sendRequest HTTP2Client {client_ = HClient {config, reqQ}} req reqTimeout_ = do sendRequestDirect :: HTTP2Client -> Request -> Maybe Int -> IO (Either HTTP2ClientError HTTP2Response) sendRequestDirect HTTP2Client {client_ = HClient {config, disconnected}, sendReq} req reqTimeout_ = do let reqTimeout = http2RequestTimeout config reqTimeout_ - reqTimeout `timeout` try (sendReq req process) >>= \case + reqTimeout `timeout` ((Right <$> sendReq req process) `E.catches` httpClientHandlers) >>= \case Just (Right r) -> pure $ Right r - Just (Left e) -> disconnected $> Left (HCIOError e) + Just (Left e) -> disconnected $> Left e Nothing -> pure $ Left HCResponseTimeout where process r = do diff --git a/src/Simplex/Messaging/Util.hs b/src/Simplex/Messaging/Util.hs index 3eab9c923..6c1937144 100644 --- a/src/Simplex/Messaging/Util.hs +++ b/src/Simplex/Messaging/Util.hs @@ -1,3 +1,4 @@ +{-# LANGUAGE BangPatterns #-} {-# LANGUAGE MonadComprehensions #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ScopedTypeVariables #-} @@ -24,8 +25,7 @@ module Simplex.Messaging.Util bindRight, forME, mapAccumLM, - mapAccumLM_List, - mapAccumLM_NonEmpty, + packZipWith, tryWriteTBQueue, catchAll, catchAll_, @@ -78,6 +78,7 @@ import qualified Data.Aeson as J import Data.Bifunctor (first, second) import Data.ByteString.Char8 (ByteString) import qualified Data.ByteString.Char8 as B +import Data.ByteString.Internal (toForeignPtr, unsafeCreate) import qualified Data.ByteString.Lazy.Char8 as LB import Data.IORef import Data.Int (Int64) @@ -92,6 +93,9 @@ import qualified Data.Text as T import Data.Text.Encoding (decodeUtf8With, encodeUtf8) import Data.Time (NominalDiffTime) import Data.Tuple (swap) +import Data.Word (Word8) +import Foreign.ForeignPtr (withForeignPtr) +import Foreign.Storable (peekByteOff, pokeByteOff) import GHC.Conc (labelThread, myThreadId, threadDelay) import UnliftIO hiding (atomicModifyIORef') import qualified UnliftIO.Exception as UE @@ -228,6 +232,27 @@ mapAccumLM_NonEmpty :: mapAccumLM_NonEmpty f s (x :| xs) = [(s2, x' :| xs') | (s1, x') <- f s x, (s2, xs') <- mapAccumLM_List f s1 xs] +-- | Optimized from bytestring package for GHC 8.10.7 compatibility +packZipWith :: (Word8 -> Word8 -> Word8) -> ByteString -> ByteString -> ByteString +packZipWith f s1 s2 = + unsafeCreate len $ \r -> + withForeignPtr fp1 $ \p1 -> + withForeignPtr fp2 $ \p2 -> zipWith_ p1 p2 r + where + zipWith_ p1 p2 r = go 0 + where + go :: Int -> IO () + go !n + | n >= len = pure () + | otherwise = do + x <- peekByteOff p1 (off1 + n) + y <- peekByteOff p2 (off2 + n) + pokeByteOff r n (f x y) + go (n + 1) + (fp1, off1, l1) = toForeignPtr s1 + (fp2, off2, l2) = toForeignPtr s2 + len = min l1 l2 + tryWriteTBQueue :: TBQueue a -> a -> STM Bool tryWriteTBQueue q a = do full <- isFullTBQueue q diff --git a/tests/AgentTests/FunctionalAPITests.hs b/tests/AgentTests/FunctionalAPITests.hs index b87ebcfad..b824b61c3 100644 --- a/tests/AgentTests/FunctionalAPITests.hs +++ b/tests/AgentTests/FunctionalAPITests.hs @@ -67,7 +67,7 @@ import Data.ByteString.Char8 (ByteString) import qualified Data.ByteString.Char8 as B import Data.Either (isRight) import Data.Int (Int64) -import Data.List (find, isSuffixOf, nub) +import Data.List (find, isPrefixOf, isSuffixOf, nub) import Data.List.NonEmpty (NonEmpty) import qualified Data.Map as M import Data.Maybe (isJust, isNothing) @@ -86,7 +86,7 @@ import Simplex.Messaging.Agent hiding (acceptContact, createConnection, deleteCo import qualified Simplex.Messaging.Agent as A import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..), ServerQueueInfo (..), UserNetworkInfo (..), UserNetworkType (..), waitForUserNetwork) import Simplex.Messaging.Agent.Env.SQLite (AgentConfig (..), Env (..), InitialAgentServers (..), createAgentStore) -import Simplex.Messaging.Agent.Protocol hiding (CON, CONF, INFO, REQ, SENT, INV, JOINED) +import Simplex.Messaging.Agent.Protocol hiding (CON, CONF, INFO, REQ, SENT) import qualified Simplex.Messaging.Agent.Protocol as A import Simplex.Messaging.Agent.Store (Connection' (..), SomeConn' (..), StoredRcvQueue (..)) import Simplex.Messaging.Agent.Store.AgentStore (getConn) @@ -114,7 +114,7 @@ import Simplex.Messaging.Util (bshow, diffToMicroseconds) import Simplex.Messaging.Version (VersionRange (..)) import qualified Simplex.Messaging.Version as V import Simplex.Messaging.Version.Internal (Version (..)) -import System.Directory (copyFile, renameFile) +import System.Directory (copyFile, removeFile, renameFile) import Test.Hspec hiding (fit, it) import UnliftIO import Util @@ -122,15 +122,18 @@ import XFTPClient (testXFTPServer) #if defined(dbPostgres) import Fixtures +import Simplex.Messaging.Agent.Store (RcvQueue, RcvQueueSub (..), ServiceAssoc) +import Simplex.Messaging.Agent.Store.AgentStore (deleteClientService, getSubscriptionService, getUserServerRcvQueueSubs, removeRcvServiceAssocs, setRcvServiceAssocs) #endif #if defined(dbServerPostgres) import qualified Database.PostgreSQL.Simple as PSQL -import Simplex.Messaging.Agent.Store (Connection' (..), StoredRcvQueue (..), SomeConn' (..)) -import Simplex.Messaging.Agent.Store.AgentStore (getConn) +import qualified Simplex.Messaging.Agent.Store.Postgres as Postgres +import qualified Simplex.Messaging.Agent.Store.Postgres.Common as Postgres import Simplex.Messaging.Server.MsgStore.Journal (JournalQueue) import Simplex.Messaging.Server.MsgStore.Postgres (PostgresQueue) import Simplex.Messaging.Server.MsgStore.Types (QSType (..)) import Simplex.Messaging.Server.QueueStore.Postgres +import Simplex.Messaging.Server.QueueStore.Postgres.Migrations import Simplex.Messaging.Server.QueueStore.Types (QueueStoreClass (..)) #endif @@ -223,12 +226,6 @@ pattern Rcvd agentMsgId <- RCVD MsgMeta {integrity = MsgOk} [MsgReceipt {agentMs pattern Rcvd' :: AgentMsgId -> AgentMsgId -> AEvent 'AEConn pattern Rcvd' aMsgId rcvdMsgId <- RCVD MsgMeta {integrity = MsgOk, recipient = (aMsgId, _)} [MsgReceipt {agentMsgId = rcvdMsgId, msgRcptStatus = MROk}] -pattern INV :: AConnectionRequestUri -> AEvent 'AEConn -pattern INV cReq = A.INV cReq Nothing - -pattern JOINED :: SndQueueSecured -> AEvent 'AEConn -pattern JOINED sndSecure = A.JOINED sndSecure Nothing - smpCfgVPrev :: ProtocolClientConfig SMPVersion smpCfgVPrev = (smpCfg agentCfg) {serverVRange = prevRange $ serverVRange $ smpCfg agentCfg} @@ -286,16 +283,16 @@ inAnyOrder g rs = withFrozenCallStack $ do createConnection :: ConnectionModeI c => AgentClient -> UserId -> Bool -> SConnectionMode c -> Maybe CRClientData -> SubscriptionMode -> AE (ConnId, ConnectionRequestUri c) createConnection c userId enableNtfs cMode clientData subMode = do - (connId, (CCLink cReq _, Nothing)) <- A.createConnection c NRMInteractive userId enableNtfs True cMode Nothing clientData IKPQOn subMode + (connId, CCLink cReq _) <- A.createConnection c NRMInteractive userId enableNtfs True cMode Nothing clientData IKPQOn subMode pure (connId, cReq) joinConnection :: AgentClient -> UserId -> Bool -> ConnectionRequestUri c -> ConnInfo -> SubscriptionMode -> AE (ConnId, SndQueueSecured) joinConnection c userId enableNtfs cReq connInfo subMode = do connId <- A.prepareConnectionToJoin c userId enableNtfs cReq PQSupportOn - (sndSecure, Nothing) <- A.joinConnection c NRMInteractive userId connId enableNtfs cReq connInfo PQSupportOn subMode + sndSecure <- A.joinConnection c NRMInteractive userId connId enableNtfs cReq connInfo PQSupportOn subMode pure (connId, sndSecure) -acceptContact :: AgentClient -> UserId -> ConnId -> Bool -> ConfirmationId -> ConnInfo -> PQSupport -> SubscriptionMode -> AE (SndQueueSecured, Maybe ClientServiceId) +acceptContact :: AgentClient -> UserId -> ConnId -> Bool -> ConfirmationId -> ConnInfo -> PQSupport -> SubscriptionMode -> AE SndQueueSecured acceptContact c = A.acceptContact c NRMInteractive subscribeConnection :: AgentClient -> ConnId -> AE () @@ -491,6 +488,10 @@ functionalAPITests ps = do testUsersNoServer ps it "should connect two users and switch session mode" $ withSmpServer ps testTwoUsers + describe "Client service certificates" $ do + it "should connect, subscribe and reconnect as a service" $ testClientServiceConnection ps + it "should re-subscribe when service ID changed" $ testClientServiceIDChange ps + it "migrate connections to and from service" $ testMigrateConnectionsToService ps describe "Connection switch" $ do describe "should switch delivery to the new queue" $ testServerMatrix2 ps testSwitchConnection @@ -717,9 +718,9 @@ runAgentClientTest pqSupport sqSecured viaProxy alice bob baseId = runAgentClientTestPQ :: HasCallStack => SndQueueSecured -> Bool -> (AgentClient, InitialKeys) -> (AgentClient, PQSupport) -> AgentMsgId -> IO () runAgentClientTestPQ sqSecured viaProxy (alice, aPQ) (bob, bPQ) baseId = runRight_ $ do - (bobId, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing aPQ SMSubscribe + (bobId, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing aPQ SMSubscribe aliceId <- A.prepareConnectionToJoin bob 1 True qInfo bPQ - (sqSecured', Nothing) <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" bPQ SMSubscribe + sqSecured' <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" bPQ SMSubscribe liftIO $ sqSecured' `shouldBe` sqSecured ("", _, A.CONF confId pqSup' _ "bob's connInfo") <- get alice liftIO $ pqSup' `shouldBe` CR.connPQEncryption aPQ @@ -790,7 +791,7 @@ runAgentClientStressTestOneWay n pqSupport sqSecured viaProxy alice bob baseId = msgId = subtract baseId . fst runAgentClientStressTestConc :: HasCallStack => Int64 -> PQSupport -> SndQueueSecured -> Bool -> AgentClient -> AgentClient -> AgentMsgId -> IO () -runAgentClientStressTestConc n pqSupport sqSecured viaProxy alice bob baseId = runRight_ $ do +runAgentClientStressTestConc n pqSupport sqSecured viaProxy alice bob _baseId = runRight_ $ do (aliceId, bobId) <- makeConnection_ pqSupport sqSecured alice bob amId <- newTVarIO 0 bmId <- newTVarIO 0 @@ -807,7 +808,6 @@ runAgentClientStressTestConc n pqSupport sqSecured viaProxy alice bob baseId = r liftIO $ noMessagesIngoreQCONT alice "nothing else should be delivered to alice" liftIO $ noMessagesIngoreQCONT bob "nothing else should be delivered to bob" where - msgId = subtract baseId . fst pqEnc = PQEncryption $ supportPQ pqSupport proxySrv = if viaProxy then Just testSMPServer else Nothing message i = "message " <> bshow i @@ -820,11 +820,11 @@ runAgentClientStressTestConc n pqSupport sqSecured viaProxy alice bob baseId = r timeout 100000 (get a) >>= mapM_ (\case ("", _, QCONT) -> drain; r -> expectationFailure $ "unexpected: " <> show r) loop (0, 0, 0, 0) = pure () - loop acc@(!s, !m, !r, !o) = + loop acc@(s, !m, !r, !o) = timeout 3000000 (get a) >>= \case Nothing -> error $ "timeout " <> show acc Just evt -> case evt of - ("", c, A.SENT mId srv) -> do + ("", c, A.SENT _mId srv) -> do liftIO $ c == bId && srv == proxySrv `shouldBe` True unless (s > 0) $ error "unexpected SENT" loop (s - 1, m, r, o) @@ -838,7 +838,7 @@ runAgentClientStressTestConc n pqSupport sqSecured viaProxy alice bob baseId = r ackMessageAsync a "123" bId mId (Just "") unless (m > 0) $ error "unexpected MSG" loop (s, m - 1, r, o) - ("", c, Rcvd' mId rcvdMsgId) -> do + ("", c, Rcvd' mId _rcvdMsgId) -> do liftIO $ (mId >) <$> atomically (swapTVar mIdVar mId) `shouldReturn` True liftIO $ c == bId `shouldBe` True ackMessageAsync a "123" bId mId Nothing @@ -947,14 +947,14 @@ runAgentClientContactTest pqSupport sqSecured viaProxy alice bob baseId = runAgentClientContactTestPQ :: HasCallStack => SndQueueSecured -> Bool -> PQSupport -> (AgentClient, InitialKeys) -> (AgentClient, PQSupport) -> AgentMsgId -> IO () runAgentClientContactTestPQ sqSecured viaProxy reqPQSupport (alice, aPQ) (bob, bPQ) baseId = runRight_ $ do - (_, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive 1 True True SCMContact Nothing Nothing aPQ SMSubscribe + (_, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive 1 True True SCMContact Nothing Nothing aPQ SMSubscribe aliceId <- A.prepareConnectionToJoin bob 1 True qInfo bPQ - (sqSecuredJoin, Nothing) <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" bPQ SMSubscribe + sqSecuredJoin <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" bPQ SMSubscribe liftIO $ sqSecuredJoin `shouldBe` False -- joining via contact address connection ("", _, A.REQ invId pqSup' _ "bob's connInfo") <- get alice liftIO $ pqSup' `shouldBe` reqPQSupport bobId <- A.prepareConnectionToAccept alice 1 True invId (CR.connPQEncryption aPQ) - (sqSecured', Nothing) <- acceptContact alice 1 bobId True invId "alice's connInfo" (CR.connPQEncryption aPQ) SMSubscribe + sqSecured' <- acceptContact alice 1 bobId True invId "alice's connInfo" (CR.connPQEncryption aPQ) SMSubscribe liftIO $ sqSecured' `shouldBe` sqSecured ("", _, A.CONF confId pqSup'' _ "alice's connInfo") <- get bob liftIO $ pqSup'' `shouldBe` bPQ @@ -991,7 +991,7 @@ runAgentClientContactTestPQ sqSecured viaProxy reqPQSupport (alice, aPQ) (bob, b runAgentClientContactTestPQ3 :: HasCallStack => Bool -> (AgentClient, InitialKeys) -> (AgentClient, PQSupport) -> (AgentClient, PQSupport) -> AgentMsgId -> IO () runAgentClientContactTestPQ3 viaProxy (alice, aPQ) (bob, bPQ) (tom, tPQ) baseId = runRight_ $ do - (_, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive 1 True True SCMContact Nothing Nothing aPQ SMSubscribe + (_, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive 1 True True SCMContact Nothing Nothing aPQ SMSubscribe (bAliceId, bobId, abPQEnc) <- connectViaContact bob bPQ qInfo sentMessages abPQEnc alice bobId bob bAliceId (tAliceId, tomId, atPQEnc) <- connectViaContact tom tPQ qInfo @@ -1000,12 +1000,12 @@ runAgentClientContactTestPQ3 viaProxy (alice, aPQ) (bob, bPQ) (tom, tPQ) baseId msgId = subtract baseId . fst connectViaContact b pq qInfo = do aId <- A.prepareConnectionToJoin b 1 True qInfo pq - (sqSecuredJoin, Nothing) <- A.joinConnection b NRMInteractive 1 aId True qInfo "bob's connInfo" pq SMSubscribe + sqSecuredJoin <- A.joinConnection b NRMInteractive 1 aId True qInfo "bob's connInfo" pq SMSubscribe liftIO $ sqSecuredJoin `shouldBe` False -- joining via contact address connection ("", _, A.REQ invId pqSup' _ "bob's connInfo") <- get alice liftIO $ pqSup' `shouldBe` PQSupportOn bId <- A.prepareConnectionToAccept alice 1 True invId (CR.connPQEncryption aPQ) - (sqSecuredAccept, Nothing) <- acceptContact alice 1 bId True invId "alice's connInfo" (CR.connPQEncryption aPQ) SMSubscribe + sqSecuredAccept <- acceptContact alice 1 bId True invId "alice's connInfo" (CR.connPQEncryption aPQ) SMSubscribe liftIO $ sqSecuredAccept `shouldBe` False -- agent cfg is v8 ("", _, A.CONF confId pqSup'' _ "alice's connInfo") <- get b liftIO $ pqSup'' `shouldBe` pq @@ -1044,9 +1044,9 @@ noMessages_ ingoreQCONT c err = tryGet `shouldReturn` () testRejectContactRequest :: HasCallStack => IO () testRejectContactRequest = withAgentClients2 $ \alice bob -> runRight_ $ do - (_addrConnId, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive 1 True True SCMContact Nothing Nothing IKPQOn SMSubscribe + (_addrConnId, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive 1 True True SCMContact Nothing Nothing IKPQOn SMSubscribe aliceId <- A.prepareConnectionToJoin bob 1 True qInfo PQSupportOn - (sqSecured, Nothing) <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe + sqSecured <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe liftIO $ sqSecured `shouldBe` False -- joining via contact address connection ("", _, A.REQ invId PQSupportOn _ "bob's connInfo") <- get alice rejectContact alice invId @@ -1056,10 +1056,10 @@ testUpdateConnectionUserId :: HasCallStack => IO () testUpdateConnectionUserId = withAgentClients2 $ \alice bob -> runRight_ $ do (connId, qInfo) <- createConnection alice 1 True SCMInvitation Nothing SMSubscribe - newUserId <- createUser alice [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] + newUserId <- createUser alice False [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] _ <- changeConnectionUser alice 1 connId newUserId aliceId <- A.prepareConnectionToJoin bob 1 True qInfo PQSupportOn - (sqSecured', Nothing) <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe + sqSecured' <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe liftIO $ sqSecured' `shouldBe` True ("", _, A.CONF confId pqSup' _ "bob's connInfo") <- get alice liftIO $ pqSup' `shouldBe` PQSupportOn @@ -1243,7 +1243,7 @@ testInvitationErrors ps restart = do threadDelay 200000 let loopConfirm n = runExceptT (A.joinConnection b' NRMInteractive 1 aId True cReq "bob's connInfo" PQSupportOn SMSubscribe) >>= \case - Right (True, Nothing) -> pure n + Right True -> pure n Right r -> error $ "unexpected result " <> show r Left _ -> putStrLn "retrying confirm" >> threadDelay 200000 >> loopConfirm (n + 1) n <- loopConfirm 1 @@ -1305,7 +1305,7 @@ testContactErrors ps restart = do let loopSend = do -- sends the invitation to testPort runExceptT (A.joinConnection b'' NRMInteractive 1 aId True cReq "bob's connInfo" PQSupportOn SMSubscribe) >>= \case - Right (False, Nothing) -> pure () + Right False -> pure () Right r -> error $ "unexpected result " <> show r Left _ -> putStrLn "retrying send" >> threadDelay 200000 >> loopSend loopSend @@ -1334,7 +1334,7 @@ testContactErrors ps restart = do ("", "", UP _ [_]) <- nGet b'' let loopConfirm n = runExceptT (acceptContact a' 1 bId True invId "alice's connInfo" PQSupportOn SMSubscribe) >>= \case - Right (True, Nothing) -> pure n + Right True -> pure n Right r -> error $ "unexpected result " <> show r Left _ -> putStrLn "retrying accept confirm" >> threadDelay 200000 >> loopConfirm (n + 1) n <- loopConfirm 1 @@ -1371,7 +1371,7 @@ testInvitationShortLink viaProxy a b = withAgent 3 agentCfg initAgentServers testDB3 $ \c -> do let userData = UserLinkData "some user data" newLinkData = UserInvLinkData userData - (bId, (CCLink connReq (Just shortLink), Nothing)) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMInvitation (Just newLinkData) Nothing CR.IKUsePQ SMSubscribe + (bId, CCLink connReq (Just shortLink)) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMInvitation (Just newLinkData) Nothing CR.IKUsePQ SMSubscribe (FixedLinkData {linkConnReq = connReq'}, connData') <- runRight $ getConnShortLink b 1 shortLink strDecode (strEncode shortLink) `shouldBe` Right shortLink connReq' `shouldBe` connReq @@ -1393,7 +1393,7 @@ testInvitationShortLink viaProxy a b = testJoinConn_ :: Bool -> Bool -> AgentClient -> ConnId -> AgentClient -> ConnectionRequestUri c -> ExceptT AgentErrorType IO () testJoinConn_ viaProxy sndSecure a bId b connReq = do aId <- A.prepareConnectionToJoin b 1 True connReq PQSupportOn - (sndSecure', Nothing) <- A.joinConnection b NRMInteractive 1 aId True connReq "bob's connInfo" PQSupportOn SMSubscribe + sndSecure' <- A.joinConnection b NRMInteractive 1 aId True connReq "bob's connInfo" PQSupportOn SMSubscribe liftIO $ sndSecure' `shouldBe` sndSecure ("", _, CONF confId _ "bob's connInfo") <- get a allowConnection a bId confId "alice's connInfo" @@ -1407,14 +1407,14 @@ testInvitationShortLinkPrev viaProxy sndSecure a b = runRight_ $ do let userData = UserLinkData "some user data" newLinkData = UserInvLinkData userData -- can't create short link with previous version - (bId, (CCLink connReq Nothing, Nothing)) <- A.createConnection a NRMInteractive 1 True True SCMInvitation (Just newLinkData) Nothing CR.IKPQOn SMSubscribe + (bId, CCLink connReq Nothing) <- A.createConnection a NRMInteractive 1 True True SCMInvitation (Just newLinkData) Nothing CR.IKPQOn SMSubscribe testJoinConn_ viaProxy sndSecure a bId b connReq testInvitationShortLinkAsync :: HasCallStack => Bool -> AgentClient -> AgentClient -> IO () testInvitationShortLinkAsync viaProxy a b = do let userData = UserLinkData "some user data" newLinkData = UserInvLinkData userData - (bId, (CCLink connReq (Just shortLink), Nothing)) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMInvitation (Just newLinkData) Nothing CR.IKUsePQ SMSubscribe + (bId, CCLink connReq (Just shortLink)) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMInvitation (Just newLinkData) Nothing CR.IKUsePQ SMSubscribe (FixedLinkData {linkConnReq = connReq'}, connData') <- runRight $ getConnShortLink b 1 shortLink strDecode (strEncode shortLink) `shouldBe` Right shortLink connReq' `shouldBe` connReq @@ -1441,7 +1441,7 @@ testContactShortLink viaProxy a b = let userData = UserLinkData "some user data" userCtData = UserContactData {direct = True, owners = [], relays = [], userData} newLinkData = UserContactLinkData userCtData - (contactId, (CCLink connReq0 (Just shortLink), Nothing)) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMContact (Just newLinkData) Nothing CR.IKPQOn SMSubscribe + (contactId, CCLink connReq0 (Just shortLink)) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMContact (Just newLinkData) Nothing CR.IKPQOn SMSubscribe Right connReq <- pure $ smpDecode (smpEncode connReq0) (FixedLinkData {linkConnReq = connReq'}, ContactLinkData _ userCtData') <- runRight $ getConnShortLink b 1 shortLink strDecode (strEncode shortLink) `shouldBe` Right shortLink @@ -1460,7 +1460,7 @@ testContactShortLink viaProxy a b = liftIO $ sndSecure `shouldBe` False ("", _, REQ invId _ "bob's connInfo") <- get a bId <- A.prepareConnectionToAccept a 1 True invId PQSupportOn - (sndSecure', Nothing) <- acceptContact a 1 bId True invId "alice's connInfo" PQSupportOn SMSubscribe + sndSecure' <- acceptContact a 1 bId True invId "alice's connInfo" PQSupportOn SMSubscribe liftIO $ sndSecure' `shouldBe` True ("", _, CONF confId _ "alice's connInfo") <- get b allowConnection b aId confId "bob's connInfo" @@ -1488,7 +1488,7 @@ testContactShortLink viaProxy a b = testAddContactShortLink :: HasCallStack => Bool -> AgentClient -> AgentClient -> IO () testAddContactShortLink viaProxy a b = withAgent 3 agentCfg initAgentServers testDB3 $ \c -> do - (contactId, (CCLink connReq0 Nothing, Nothing)) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMContact Nothing Nothing CR.IKPQOn SMSubscribe + (contactId, CCLink connReq0 Nothing) <- runRight $ A.createConnection a NRMInteractive 1 True True SCMContact Nothing Nothing CR.IKPQOn SMSubscribe Right connReq <- pure $ smpDecode (smpEncode connReq0) -- let userData = UserLinkData "some user data" userCtData = UserContactData {direct = True, owners = [], relays = [], userData} @@ -1511,7 +1511,7 @@ testAddContactShortLink viaProxy a b = liftIO $ sndSecure `shouldBe` False ("", _, REQ invId _ "bob's connInfo") <- get a bId <- A.prepareConnectionToAccept a 1 True invId PQSupportOn - (sndSecure', Nothing) <- acceptContact a 1 bId True invId "alice's connInfo" PQSupportOn SMSubscribe + sndSecure' <- acceptContact a 1 bId True invId "alice's connInfo" PQSupportOn SMSubscribe liftIO $ sndSecure' `shouldBe` True ("", _, CONF confId _ "alice's connInfo") <- get b allowConnection b aId confId "bob's connInfo" @@ -1533,7 +1533,7 @@ testInvitationShortLinkRestart :: HasCallStack => (ASrvTransport, AStoreType) -> testInvitationShortLinkRestart ps = withAgentClients2 $ \a b -> do let userData = UserLinkData "some user data" newLinkData = UserInvLinkData userData - (bId, (CCLink connReq (Just shortLink), Nothing)) <- withSmpServer ps $ + (bId, CCLink connReq (Just shortLink)) <- withSmpServer ps $ runRight $ A.createConnection a NRMInteractive 1 True True SCMInvitation (Just newLinkData) Nothing CR.IKUsePQ SMOnlyCreate withSmpServer ps $ do runRight_ $ subscribeConnection a bId @@ -1547,7 +1547,7 @@ testContactShortLinkRestart ps = withAgentClients2 $ \a b -> do let userData = UserLinkData "some user data" userCtData = UserContactData {direct = True, owners = [], relays = [], userData} newLinkData = UserContactLinkData userCtData - (contactId, (CCLink connReq0 (Just shortLink), Nothing)) <- withSmpServer ps $ + (contactId, CCLink connReq0 (Just shortLink)) <- withSmpServer ps $ runRight $ A.createConnection a NRMInteractive 1 True True SCMContact (Just newLinkData) Nothing CR.IKPQOn SMOnlyCreate Right connReq <- pure $ smpDecode (smpEncode connReq0) let updatedData = UserLinkData "updated user data" @@ -1571,7 +1571,7 @@ testAddContactShortLinkRestart ps = withAgentClients2 $ \a b -> do let userData = UserLinkData "some user data" userCtData = UserContactData {direct = True, owners = [], relays = [], userData} newLinkData = UserContactLinkData userCtData - ((contactId, (CCLink connReq0 Nothing, Nothing)), shortLink) <- withSmpServer ps $ runRight $ do + ((contactId, CCLink connReq0 Nothing), shortLink) <- withSmpServer ps $ runRight $ do r@(contactId, _) <- A.createConnection a NRMInteractive 1 True True SCMContact Nothing Nothing CR.IKPQOn SMOnlyCreate (r,) <$> setConnShortLink a contactId SCMContact newLinkData Nothing Right connReq <- pure $ smpDecode (smpEncode connReq0) @@ -1593,7 +1593,7 @@ testAddContactShortLinkRestart ps = withAgentClients2 $ \a b -> do testOldContactQueueShortLink :: HasCallStack => (ASrvTransport, AStoreType) -> IO () testOldContactQueueShortLink ps@(_, msType) = withAgentClients2 $ \a b -> do - (contactId, (CCLink connReq Nothing, Nothing)) <- withSmpServer ps $ runRight $ + (contactId, CCLink connReq Nothing) <- withSmpServer ps $ runRight $ A.createConnection a NRMInteractive 1 True True SCMContact Nothing Nothing CR.IKPQOn SMOnlyCreate -- make it an "old" queue let updateStoreLog f = replaceSubstringInFile f " queue_mode=C" "" @@ -1668,7 +1668,7 @@ testPrepareCreateConnectionLink ps = withSmpServer ps $ withAgentClients2 $ \a b liftIO $ sndSecure `shouldBe` False ("", _, REQ invId _ "bob's connInfo") <- get a aId <- A.prepareConnectionToAccept a 1 True invId PQSupportOn - (sndSecure', Nothing) <- acceptContact a 1 aId True invId "alice's connInfo" PQSupportOn SMSubscribe + sndSecure' <- acceptContact a 1 aId True invId "alice's connInfo" PQSupportOn SMSubscribe liftIO $ sndSecure' `shouldBe` True ("", _, CONF confId _ "alice's connInfo") <- get b allowConnection b bId confId "bob's connInfo" @@ -2368,9 +2368,9 @@ makeConnectionForUsers = makeConnectionForUsers_ PQSupportOn True makeConnectionForUsers_ :: HasCallStack => PQSupport -> SndQueueSecured -> AgentClient -> UserId -> AgentClient -> UserId -> ExceptT AgentErrorType IO (ConnId, ConnId) makeConnectionForUsers_ pqSupport sqSecured alice aliceUserId bob bobUserId = do - (bobId, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive aliceUserId True True SCMInvitation Nothing Nothing (IKLinkPQ pqSupport) SMSubscribe + (bobId, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive aliceUserId True True SCMInvitation Nothing Nothing (IKLinkPQ pqSupport) SMSubscribe aliceId <- A.prepareConnectionToJoin bob bobUserId True qInfo pqSupport - (sqSecured', Nothing) <- A.joinConnection bob NRMInteractive bobUserId aliceId True qInfo "bob's connInfo" pqSupport SMSubscribe + sqSecured' <- A.joinConnection bob NRMInteractive bobUserId aliceId True qInfo "bob's connInfo" pqSupport SMSubscribe liftIO $ sqSecured' `shouldBe` sqSecured ("", _, A.CONF confId pqSup' _ "bob's connInfo") <- get alice liftIO $ pqSup' `shouldBe` pqSupport @@ -2701,7 +2701,7 @@ testSetConnShortLinkAsync ps = withAgentClients2 $ \alice bob -> let userData = UserLinkData "test user data" userCtData = UserContactData {direct = True, owners = [], relays = [], userData} newLinkData = UserContactLinkData userCtData - (cId, (CCLink qInfo (Just shortLink), _)) <- A.createConnection alice NRMInteractive 1 True True SCMContact (Just newLinkData) Nothing IKPQOn SMSubscribe + (cId, CCLink qInfo (Just shortLink)) <- A.createConnection alice NRMInteractive 1 True True SCMContact (Just newLinkData) Nothing IKPQOn SMSubscribe -- verify initial link data (_, ContactLinkData _ userCtData') <- getConnShortLink bob 1 shortLink liftIO $ userCtData' `shouldBe` userCtData @@ -2720,7 +2720,7 @@ testSetConnShortLinkAsync ps = withAgentClients2 $ \alice bob -> (aliceId, _) <- joinConnection bob 1 True qInfo "bob's connInfo" SMSubscribe ("", _, REQ invId _ "bob's connInfo") <- get alice bobId <- A.prepareConnectionToAccept alice 1 True invId PQSupportOn - (_, Nothing) <- acceptContact alice 1 bobId True invId "alice's connInfo" PQSupportOn SMSubscribe + _ <- acceptContact alice 1 bobId True invId "alice's connInfo" PQSupportOn SMSubscribe ("", _, CONF confId _ "alice's connInfo") <- get bob allowConnection bob aliceId confId "bob's connInfo" get alice ##> ("", bobId, INFO "bob's connInfo") @@ -2733,7 +2733,7 @@ testGetConnShortLinkAsync ps = withAgentClients2 $ \alice bob -> let userData = UserLinkData "test user data" userCtData = UserContactData {direct = True, owners = [], relays = [], userData} newLinkData = UserContactLinkData userCtData - (_, (CCLink qInfo (Just shortLink), _)) <- A.createConnection alice NRMInteractive 1 True True SCMContact (Just newLinkData) Nothing IKPQOn SMSubscribe + (_, CCLink qInfo (Just shortLink)) <- A.createConnection alice NRMInteractive 1 True True SCMContact (Just newLinkData) Nothing IKPQOn SMSubscribe -- get link data async - creates new connection for bob newId <- getConnShortLinkAsync bob 1 "1" shortLink ("1", newId', LDATA FixedLinkData {linkConnReq = qInfo'} (ContactLinkData _ userCtData')) <- get bob @@ -2748,7 +2748,7 @@ testGetConnShortLinkAsync ps = withAgentClients2 $ \alice bob -> -- complete connection ("", _, REQ invId _ "bob's connInfo") <- get alice bobId <- A.prepareConnectionToAccept alice 1 True invId PQSupportOn - (_, Nothing) <- acceptContact alice 1 bobId True invId "alice's connInfo" PQSupportOn SMSubscribe + _ <- acceptContact alice 1 bobId True invId "alice's connInfo" PQSupportOn SMSubscribe ("", _, CONF confId _ "alice's connInfo") <- get bob allowConnection bob aliceId confId "bob's connInfo" get alice ##> ("", bobId, INFO "bob's connInfo") @@ -3129,7 +3129,7 @@ testUsers = withAgentClients2 $ \a b -> runRight_ $ do (aId, bId) <- makeConnection a b exchangeGreetings a bId b aId - auId <- createUser a [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] + auId <- createUser a False [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] (aId', bId') <- makeConnectionForUsers a auId b 1 exchangeGreetings a bId' b aId' deleteUser a auId True @@ -3144,7 +3144,7 @@ testDeleteUserQuietly = withAgentClients2 $ \a b -> runRight_ $ do (aId, bId) <- makeConnection a b exchangeGreetings a bId b aId - auId <- createUser a [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] + auId <- createUser a False [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] (aId', bId') <- makeConnectionForUsers a auId b 1 exchangeGreetings a bId' b aId' deleteUser a auId False @@ -3156,7 +3156,7 @@ testUsersNoServer ps = withAgentClientsCfg2 aCfg agentCfg $ \a b -> do (aId, bId, auId, _aId', bId') <- withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do (aId, bId) <- makeConnection a b exchangeGreetings a bId b aId - auId <- createUser a [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] + auId <- createUser a False [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] (aId', bId') <- makeConnectionForUsers a auId b 1 exchangeGreetings a bId' b aId' pure (aId, bId, auId, aId', bId') @@ -3738,7 +3738,7 @@ testTwoUsers = withAgentClients2 $ \a b -> do exchangeGreetings a bId1' b aId1' a `hasClients` 1 b `hasClients` 1 - liftIO $ setNetworkConfig a nc {sessionMode = TSMEntity} + setNetworkConfig a nc {sessionMode = TSMEntity} liftIO $ threadDelay 250000 ("", "", DOWN _ _) <- nGet a ("", "", UP _ _) <- nGet a @@ -3748,7 +3748,7 @@ testTwoUsers = withAgentClients2 $ \a b -> do exchangeGreetingsMsgId 4 a bId1 b aId1 exchangeGreetingsMsgId 4 a bId1' b aId1' liftIO $ threadDelay 250000 - liftIO $ setNetworkConfig a nc {sessionMode = TSMUser} + setNetworkConfig a nc {sessionMode = TSMUser} liftIO $ threadDelay 250000 ("", "", DOWN _ _) <- nGet a ("", "", DOWN _ _) <- nGet a @@ -3756,14 +3756,14 @@ testTwoUsers = withAgentClients2 $ \a b -> do ("", "", UP _ _) <- nGet a a `hasClients` 1 - aUserId2 <- createUser a [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] + aUserId2 <- createUser a False [noAuthSrvCfg testSMPServer] [noAuthSrvCfg testXFTPServer] (aId2, bId2) <- makeConnectionForUsers a aUserId2 b 1 exchangeGreetings a bId2 b aId2 (aId2', bId2') <- makeConnectionForUsers a aUserId2 b 1 exchangeGreetings a bId2' b aId2' a `hasClients` 2 b `hasClients` 1 - liftIO $ setNetworkConfig a nc {sessionMode = TSMEntity} + setNetworkConfig a nc {sessionMode = TSMEntity} liftIO $ threadDelay 250000 ("", "", DOWN _ _) <- nGet a ("", "", DOWN _ _) <- nGet a @@ -3777,7 +3777,7 @@ testTwoUsers = withAgentClients2 $ \a b -> do exchangeGreetingsMsgId 4 a bId2 b aId2 exchangeGreetingsMsgId 4 a bId2' b aId2' liftIO $ threadDelay 250000 - liftIO $ setNetworkConfig a nc {sessionMode = TSMUser} + setNetworkConfig a nc {sessionMode = TSMUser} liftIO $ threadDelay 250000 ("", "", DOWN _ _) <- nGet a ("", "", DOWN _ _) <- nGet a @@ -3797,10 +3797,265 @@ testTwoUsers = withAgentClients2 $ \a b -> do hasClients :: HasCallStack => AgentClient -> Int -> ExceptT AgentErrorType IO () hasClients c n = liftIO $ M.size <$> readTVarIO (smpClients c) `shouldReturn` n +testClientServiceConnection :: HasCallStack => (ASrvTransport, AStoreType) -> IO () +testClientServiceConnection ps = do + ((sId, uId), qIdHash) <- withSmpServerStoreLogOn ps testPort $ \_ -> do + conns@(sId, uId) <- withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> runRight $ do + conns@(sId, uId) <- makeConnection service user + exchangeGreetings service uId user sId + pure conns + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> runRight $ do + liftIO $ threadDelay 250000 + [(_, Right (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 1 qIdHash)))] <- M.toList <$> subscribeClientServices service 1 + ("", "", SERVICE_ALL _) <- nGet service + subscribeConnection user sId + exchangeGreetingsMsgId 4 service uId user sId + pure (conns, qIdHash) + (uId', sId') <- withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> do + withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + liftIO $ threadDelay 250000 + subscribeAllConnections service False Nothing + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 1 qIdHash')))) -> qIdHash' == qIdHash; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + subscribeConnection user sId + exchangeGreetingsMsgId 6 service uId user sId + ("", "", DOWN _ [_]) <- nGet user + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 1 qIdHash')) <- nGet service + qIdHash' `shouldBe` qIdHash + withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + ("", "", UP _ [_]) <- nGet user + -- Nothing in ServiceSubResult confirms that both counts and IDs hash match + -- SERVICE_ALL may be deliverd before SERVICE_UP event in case there are no messages to deliver + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 1 qIdHash'')))) -> qIdHash'' == qIdHash; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + exchangeGreetingsMsgId 8 service uId user sId + conns'@(uId', sId') <- makeConnection user service -- opposite direction + exchangeGreetings user sId' service uId' + pure conns' + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> do + withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + liftIO $ threadDelay 250000 + subscribeAllConnections service False Nothing + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 2 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + -- TODO [certs rcv] test message delivery during subscription + subscribeAllConnections user False Nothing + ("", "", UP _ [_, _]) <- nGet user + exchangeGreetingsMsgId 4 user sId' service uId' + exchangeGreetingsMsgId 10 service uId user sId + +testClientServiceIDChange :: HasCallStack => (ASrvTransport, AStoreType) -> IO () +testClientServiceIDChange ps@(_, ASType qs _) = do + (sId, uId) <- withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> do + conns <- withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + conns@(sId, uId) <- makeConnection service user + exchangeGreetings service uId user sId + pure conns + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 1 _)) <- nGet service + ("", "", DOWN _ [_]) <- nGet user + withSmpServerStoreLogOn ps testPort $ \_ -> do + getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 1 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + ("", "", UP _ [_]) <- nGet user + pure () + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 1 _)) <- nGet service + ("", "", DOWN _ [_]) <- nGet user + pure conns + _ :: () <- case qs of + SQSPostgres -> do +#if defined(dbServerPostgres) + st <- either (error . show) pure =<< Postgres.createDBStore testStoreDBOpts serverMigrations (MigrationConfig MCError Nothing) + void $ Postgres.withTransaction st (`PSQL.execute_` "DELETE FROM services") +#else + pure () +#endif + SQSMemory -> do + s <- readFile testStoreLogFile + removeFile testStoreLogFile + writeFile testStoreLogFile $ unlines $ filter (not . ("NEW_SERVICE" `isPrefixOf`)) $ lines s + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> do + withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + liftIO $ threadDelay 250000 + subscribeAllConnections service False Nothing + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult (Just (SMP.SSErrorQueueCount 1 0)) (SMP.ServiceSub _ 0 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False, + \case ("", "", AEvt SAENone (UP _ [_])) -> True; _ -> False + ] + subscribeAllConnections user False Nothing + ("", "", UP _ [_]) <- nGet user + exchangeGreetingsMsgId 4 service uId user sId + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 1 _)) <- nGet service + ("", "", DOWN _ [_]) <- nGet user + pure () + -- disable service in the client + withAgentClientsServers2 (agentCfg, initAgentServers) (agentCfg, initAgentServers) $ \notService user -> do + withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + subscribeAllConnections notService False Nothing + ("", "", UP _ [_]) <- nGet notService + subscribeAllConnections user False Nothing + ("", "", UP _ [_]) <- nGet user + exchangeGreetingsMsgId 6 notService uId user sId + +testMigrateConnectionsToService :: HasCallStack => (ASrvTransport, AStoreType) -> IO () +testMigrateConnectionsToService ps = do + (((sId1, uId1), (uId2, sId2)), ((sId3, uId3), (uId4, sId4)), ((sId5, uId5), (uId6, sId6))) <- + withSmpServerStoreLogOn ps testPort $ \_ -> do + -- starting without service + cs12@((sId1, uId1), (uId2, sId2)) <- + withAgentClientsServers2 (agentCfg, initAgentServers) (agentCfg, initAgentServers) $ \notService user -> + runRight $ (,) <$> makeConnection notService user <*> makeConnection user notService + -- migrating to service + cs34@((sId3, uId3), (uId4, sId4)) <- + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> runRight $ do + subscribeAllConnections service False Nothing + service `up` 2 + subscribeAllConnections user False Nothing + user `up` 2 + exchangeGreetingsMsgId 2 service uId1 user sId1 + exchangeGreetingsMsgId 2 service uId2 user sId2 + (,) <$> makeConnection service user <*> makeConnection user service + -- starting as service + cs56 <- + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> runRight $ do + subscribeAllConnections service False Nothing + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 4 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + subscribeAllConnections user False Nothing + user `up` 4 + exchangeGreetingsMsgId 4 service uId1 user sId1 + exchangeGreetingsMsgId 4 service uId2 user sId2 + exchangeGreetingsMsgId 2 service uId3 user sId3 + exchangeGreetingsMsgId 2 service uId4 user sId4 + (,) <$> makeConnection service user <*> makeConnection user service + pure (cs12, cs34, cs56) + -- server reconnecting resubscribes service + let testSendMessages6 s u n = do + exchangeGreetingsMsgId (n + 4) s uId1 u sId1 + exchangeGreetingsMsgId (n + 4) s uId2 u sId2 + exchangeGreetingsMsgId (n + 2) s uId3 u sId3 + exchangeGreetingsMsgId (n + 2) s uId4 u sId4 + exchangeGreetingsMsgId n s uId5 u sId5 + exchangeGreetingsMsgId n s uId6 u sId6 + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> do + withSmpServerStoreLogOn ps testPort $ \_ -> runRight_ $ do + subscribeAllConnections service False Nothing + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 6 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + subscribeAllConnections user False Nothing + user `up` 6 + testSendMessages6 service user 2 + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 6 _)) <- nGet service + user `down` 6 + withSmpServerStoreLogOn ps testPort $ \_ -> runRight_ $ do + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 6 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + user `up` 6 + testSendMessages6 service user 4 + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 6 _)) <- nGet service + user `down` 6 + -- disabling service and adding connections + ((sId7, uId7), (uId8, sId8)) <- + withAgentClientsServers2 (agentCfg, initAgentServers) (agentCfg, initAgentServers) $ \notService user -> do + cs78@((sId7, uId7), (uId8, sId8)) <- + withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + subscribeAllConnections notService False Nothing + notService `up` 6 + subscribeAllConnections user False Nothing + user `up` 6 + testSendMessages6 notService user 6 + (,) <$> makeConnection notService user <*> makeConnection user notService + notService `down` 8 + user `down` 8 + withSmpServerStoreLogOn ps testPort $ \_ -> runRight $ do + notService `up` 8 + user `up` 8 + testSendMessages6 notService user 8 + exchangeGreetingsMsgId 2 notService uId7 user sId7 + exchangeGreetingsMsgId 2 notService uId8 user sId8 + notService `down` 8 + user `down` 8 + pure cs78 + let testSendMessages8 s u n = do + testSendMessages6 s u (n + 8) + exchangeGreetingsMsgId (n + 2) s uId7 u sId7 + exchangeGreetingsMsgId (n + 2) s uId8 u sId8 + -- re-enabling service and adding connections + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> do + withSmpServerStoreLogOn ps testPort $ \_ -> runRight_ $ do + subscribeAllConnections service False Nothing + service `up` 8 + subscribeAllConnections user False Nothing + user `up` 8 + testSendMessages8 service user 2 + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 8 _)) <- nGet service + user `down` 8 + -- re-connect to server + withSmpServerStoreLogOn ps testPort $ \_ -> runRight_ $ do + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 8 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + user `up` 8 + testSendMessages8 service user 4 + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ _ _)) <- nGet service -- should be 8 here + user `down` 8 + -- restart agents + withAgentClientsServers2 (agentCfg, initAgentServersClientService) (agentCfg, initAgentServers) $ \service user -> do + withSmpServerStoreLogOn ps testPort $ \_ -> runRight_ $ do + subscribeAllConnections service False Nothing + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 8 _)))) -> True; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + subscribeAllConnections user False Nothing + user `up` 8 + testSendMessages8 service user 6 + ("", "", SERVICE_DOWN _ (SMP.ServiceSub _ 8 _)) <- nGet service + user `down` 8 + runRight_ $ do + void $ sendMessage user sId7 SMP.noMsgFlags "hello 1" + void $ sendMessage user sId8 SMP.noMsgFlags "hello 2" + -- re-connect to server + withSmpServerStoreLogOn ps testPort $ \_ -> runRight_ $ do + liftIO $ getInAnyOrder service + [ \case ("", "", AEvt SAENone (SERVICE_UP _ (SMP.ServiceSubResult Nothing (SMP.ServiceSub _ 8 _)))) -> True; _ -> False, + \case ("", c, AEvt SAEConn (Msg "hello 1")) -> c == uId7; _ -> False, + \case ("", c, AEvt SAEConn (Msg "hello 2")) -> c == uId8; _ -> False, + \case ("", "", AEvt SAENone (SERVICE_ALL _)) -> True; _ -> False + ] + liftIO $ getInAnyOrder user + [ \case ("", "", AEvt SAENone (UP _ [_, _, _, _, _, _, _, _])) -> True; _ -> False, + \case ("", c, AEvt SAEConn (SENT 10)) -> c == sId7; _ -> False, + \case ("", c, AEvt SAEConn (SENT 10)) -> c == sId8; _ -> False + ] + testSendMessages6 service user 16 + where + up c n = do + ("", "", UP _ conns) <- nGet c + liftIO $ length conns `shouldBe` n + down c n = do + ("", "", DOWN _ conns) <- nGet c + liftIO $ length conns `shouldBe` n + getSMPAgentClient' :: Int -> AgentConfig -> InitialAgentServers -> String -> IO AgentClient getSMPAgentClient' clientId cfg' initServers dbPath = do Right st <- liftIO $ createStore dbPath - c <- getSMPAgentClient_ clientId cfg' initServers st False + Right c <- runExceptT $ getSMPAgentClient_ clientId cfg' initServers st False when (dbNew st) $ insertUser st pure c diff --git a/tests/AgentTests/SQLiteTests.hs b/tests/AgentTests/SQLiteTests.hs index b68731ff7..c52f5c5bd 100644 --- a/tests/AgentTests/SQLiteTests.hs +++ b/tests/AgentTests/SQLiteTests.hs @@ -227,7 +227,7 @@ rcvQueue1 = sndId = EntityId "2345", queueMode = Just QMMessaging, shortLink = Nothing, - clientService = Nothing, + rcvServiceAssoc = False, status = New, enableNtfs = True, clientNoticeId = Nothing, @@ -441,7 +441,7 @@ testUpgradeSndConnToDuplex = sndId = EntityId "4567", queueMode = Just QMMessaging, shortLink = Nothing, - clientService = Nothing, + rcvServiceAssoc = False, status = New, enableNtfs = True, clientNoticeId = Nothing, diff --git a/tests/AgentTests/ServerChoice.hs b/tests/AgentTests/ServerChoice.hs index a27678cb6..8412c6761 100644 --- a/tests/AgentTests/ServerChoice.hs +++ b/tests/AgentTests/ServerChoice.hs @@ -64,6 +64,7 @@ initServers = ntf = [testNtfServer], xftp = userServers [testXFTPServer], netCfg = defaultNetworkConfig, + useServices = M.empty, presetDomains = [], presetServers = [] } diff --git a/tests/CoreTests/BatchingTests.hs b/tests/CoreTests/BatchingTests.hs index d013c0db4..8a285721b 100644 --- a/tests/CoreTests/BatchingTests.hs +++ b/tests/CoreTests/BatchingTests.hs @@ -334,7 +334,7 @@ randomSUBv6 = randomSUB_ C.SEd25519 minServerSMPRelayVersion randomSUB :: ByteString -> IO (Either TransportError (Maybe TAuthorizations, ByteString)) randomSUB = randomSUB_ C.SEd25519 currentClientSMPRelayVersion --- TODO [certs] test with the additional certificate signature +-- TODO [certs rcv] test with the additional certificate signature randomSUB_ :: (C.AlgorithmI a, C.AuthAlgorithm a) => C.SAlgorithm a -> VersionSMP -> ByteString -> IO (Either TransportError (Maybe TAuthorizations, ByteString)) randomSUB_ a v sessId = do g <- C.newRandom diff --git a/tests/CoreTests/TSessionSubs.hs b/tests/CoreTests/TSessionSubs.hs index e3f819332..96975e9ef 100644 --- a/tests/CoreTests/TSessionSubs.hs +++ b/tests/CoreTests/TSessionSubs.hs @@ -58,9 +58,9 @@ testSessionSubs = do atomically (SS.hasPendingSubs tSess2 ss) `shouldReturn` True atomically (SS.batchAddPendingSubs tSess1 [q1, q2] ss') atomically (SS.batchAddPendingSubs tSess2 [q3] ss') - atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` M.fromList [("r1", q1), ("r2", q2)] + atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` (M.fromList [("r1", q1), ("r2", q2)], Nothing) atomically (SS.getActiveSubs tSess1 ss) `shouldReturn` M.fromList [] - atomically (SS.getPendingSubs tSess2 ss) `shouldReturn` M.fromList [("r3", q3)] + atomically (SS.getPendingSubs tSess2 ss) `shouldReturn` (M.fromList [("r3", q3)], Nothing) st <- dumpSessionSubs ss dumpSessionSubs ss' `shouldReturn` st countSubs ss `shouldReturn` (0, 3) @@ -69,41 +69,41 @@ testSessionSubs = do atomically (SS.hasPendingSub tSess1 (rcvId q4) ss) `shouldReturn` False atomically (SS.hasActiveSub tSess1 (rcvId q4) ss) `shouldReturn` False -- setting active queue without setting session ID would keep it as pending - atomically $ SS.addActiveSub tSess1 "123" q1 ss + atomically $ SS.addActiveSub' tSess1 "123" Nothing q1 False ss atomically (SS.hasPendingSub tSess1 (rcvId q1) ss) `shouldReturn` True atomically (SS.hasActiveSub tSess1 (rcvId q1) ss) `shouldReturn` False dumpSessionSubs ss `shouldReturn` st countSubs ss `shouldReturn` (0, 3) -- setting active queues atomically $ SS.setSessionId tSess1 "123" ss - atomically $ SS.addActiveSub tSess1 "123" q1 ss + atomically $ SS.addActiveSub' tSess1 "123" Nothing q1 False ss atomically (SS.hasPendingSub tSess1 (rcvId q1) ss) `shouldReturn` False atomically (SS.hasActiveSub tSess1 (rcvId q1) ss) `shouldReturn` True atomically (SS.getActiveSubs tSess1 ss) `shouldReturn` M.fromList [("r1", q1)] - atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` M.fromList [("r2", q2)] + atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` (M.fromList [("r2", q2)], Nothing) countSubs ss `shouldReturn` (1, 2) atomically $ SS.setSessionId tSess2 "456" ss - atomically $ SS.addActiveSub tSess2 "456" q4 ss + atomically $ SS.addActiveSub' tSess2 "456" Nothing q4 False ss atomically (SS.hasPendingSub tSess2 (rcvId q4) ss) `shouldReturn` False atomically (SS.hasActiveSub tSess2 (rcvId q4) ss) `shouldReturn` True atomically (SS.hasActiveSub tSess1 (rcvId q4) ss) `shouldReturn` False -- wrong transport session atomically (SS.getActiveSubs tSess2 ss) `shouldReturn` M.fromList [("r4", q4)] - atomically (SS.getPendingSubs tSess2 ss) `shouldReturn` M.fromList [("r3", q3)] + atomically (SS.getPendingSubs tSess2 ss) `shouldReturn` (M.fromList [("r3", q3)], Nothing) countSubs ss `shouldReturn` (2, 2) -- setting pending queues st' <- dumpSessionSubs ss - atomically (SS.setSubsPending TSMUser tSess1 "abc" ss) `shouldReturn` M.empty -- wrong session + atomically (SS.setSubsPending TSMUser tSess1 "abc" ss) `shouldReturn` (M.empty, Nothing) -- wrong session dumpSessionSubs ss `shouldReturn` st' - atomically (SS.setSubsPending TSMUser tSess1 "123" ss) `shouldReturn` M.fromList [("r1", q1)] + atomically (SS.setSubsPending TSMUser tSess1 "123" ss) `shouldReturn` (M.fromList [("r1", q1)], Nothing) atomically (SS.getActiveSubs tSess1 ss) `shouldReturn` M.fromList [] - atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` M.fromList [("r1", q1), ("r2", q2)] + atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` (M.fromList [("r1", q1), ("r2", q2)], Nothing) countSubs ss `shouldReturn` (1, 3) -- delete subs atomically $ SS.deletePendingSub tSess1 (rcvId q1) ss - atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` M.fromList [("r2", q2)] + atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` (M.fromList [("r2", q2)], Nothing) countSubs ss `shouldReturn` (1, 2) atomically $ SS.deleteSub tSess1 (rcvId q2) ss - atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` M.fromList [] + atomically (SS.getPendingSubs tSess1 ss) `shouldReturn` (M.fromList [], Nothing) countSubs ss `shouldReturn` (1, 1) atomically (SS.getActiveSubs tSess2 ss) `shouldReturn` M.fromList [("r4", q4)] atomically $ SS.deleteSub tSess2 (rcvId q4) ss diff --git a/tests/Fixtures.hs b/tests/Fixtures.hs index 2360a7ba6..f2f314fed 100644 --- a/tests/Fixtures.hs +++ b/tests/Fixtures.hs @@ -3,7 +3,9 @@ module Fixtures where import Data.ByteString (ByteString) +import qualified Data.ByteString.Char8 as B import Database.PostgreSQL.Simple (ConnectInfo (..), defaultConnectInfo) +import Simplex.Messaging.Agent.Store.Postgres.Options testDBConnstr :: ByteString testDBConnstr = "postgresql://test_agent_user@/test_agent_db" @@ -14,3 +16,6 @@ testDBConnectInfo = connectUser = "test_agent_user", connectDatabase = "test_agent_db" } + +testDBOpts :: String -> DBOpts +testDBOpts schema' = DBOpts testDBConnstr (B.pack schema') 1 True diff --git a/tests/SMPAgentClient.hs b/tests/SMPAgentClient.hs index 02bee9ae7..41aab2039 100644 --- a/tests/SMPAgentClient.hs +++ b/tests/SMPAgentClient.hs @@ -65,6 +65,7 @@ initAgentServers = ntf = [testNtfServer], xftp = userServers [testXFTPServer], netCfg = defaultNetworkConfig {tcpTimeout = NetworkTimeout 500000 500000, tcpConnectTimeout = NetworkTimeout 500000 500000}, + useServices = M.empty, presetDomains = [], presetServers = [] } @@ -82,6 +83,9 @@ initAgentServersProxy_ smpProxyMode smpProxyFallback = initAgentServersProxy2 :: InitialAgentServers initAgentServersProxy2 = initAgentServersProxy {smp = userServers [testSMPServer2]} +initAgentServersClientService :: InitialAgentServers +initAgentServersClientService = initAgentServers {useServices = M.fromList [(1, True)]} + agentCfg :: AgentConfig agentCfg = defaultAgentConfig diff --git a/tests/SMPClient.hs b/tests/SMPClient.hs index c51079d5e..d043fd3c8 100644 --- a/tests/SMPClient.hs +++ b/tests/SMPClient.hs @@ -15,10 +15,14 @@ module SMPClient where +import Control.Monad import Control.Monad.Except (runExceptT) import Data.ByteString.Char8 (ByteString) import Data.List.NonEmpty (NonEmpty) +import qualified Data.X509 as X +import qualified Data.X509.Validation as XV import Network.Socket +import qualified Network.TLS as TLS import Simplex.Messaging.Agent.Store.Postgres.Options (DBOpts (..)) import Simplex.Messaging.Agent.Store.Shared (MigrationConfirmation (..)) import Simplex.Messaging.Client (ProtocolClientConfig (..), chooseTransportHost, defaultNetworkConfig) @@ -33,6 +37,7 @@ import Simplex.Messaging.Server.QueueStore.Postgres.Config (PostgresStoreCfg (.. import Simplex.Messaging.Transport import Simplex.Messaging.Transport.Client import Simplex.Messaging.Transport.Server +import Simplex.Messaging.Transport.Shared (ChainCertificates (..), chainIdCaCerts) import Simplex.Messaging.Util (ifM) import Simplex.Messaging.Version import Simplex.Messaging.Version.Internal @@ -151,13 +156,26 @@ testSMPClient = testSMPClientVR supportedClientSMPRelayVRange testSMPClientVR :: Transport c => VersionRangeSMP -> (THandleSMP c 'TClient -> IO a) -> IO a testSMPClientVR vr client = do Right useHost <- pure $ chooseTransportHost defaultNetworkConfig testHost - testSMPClient_ useHost testPort vr client + testSMPClient_ useHost testPort vr Nothing client -testSMPClient_ :: Transport c => TransportHost -> ServiceName -> VersionRangeSMP -> (THandleSMP c 'TClient -> IO a) -> IO a -testSMPClient_ host port vr client = do - let tcConfig = defaultTransportClientConfig {clientALPN} :: TransportClientConfig +testSMPServiceClient :: Transport c => (TLS.Credential, C.KeyPairEd25519) -> (THandleSMP c 'TClient -> IO a) -> IO a +testSMPServiceClient serviceCreds client = do + Right useHost <- pure $ chooseTransportHost defaultNetworkConfig testHost + testSMPClient_ useHost testPort supportedClientSMPRelayVRange (Just serviceCreds) client + +testSMPClient_ :: Transport c => TransportHost -> ServiceName -> VersionRangeSMP -> Maybe (TLS.Credential, C.KeyPairEd25519) -> (THandleSMP c 'TClient -> IO a) -> IO a +testSMPClient_ host port vr serviceCreds_ client = do + serviceAndKeys_ <- forM serviceCreds_ $ \(serviceCreds@(cc, pk), keys) -> do + Right serviceSignKey <- pure $ C.x509ToPrivate' pk + let idCert' = case chainIdCaCerts cc of + CCSelf cert -> cert + CCValid {idCert} -> idCert + _ -> error "bad certificate" + serviceCertHash = XV.getFingerprint idCert' X.HashSHA256 + pure (ServiceCredentials {serviceRole = SRMessaging, serviceCreds, serviceCertHash, serviceSignKey}, keys) + let tcConfig = defaultTransportClientConfig {clientALPN, clientCredentials = fst <$> serviceCreds_} :: TransportClientConfig runTransportClient tcConfig Nothing host port (Just testKeyHash) $ \h -> - runExceptT (smpClientHandshake h Nothing testKeyHash vr False Nothing) >>= \case + runExceptT (smpClientHandshake h Nothing testKeyHash vr False serviceAndKeys_) >>= \case Right th -> client th Left e -> error $ show e where @@ -165,6 +183,12 @@ testSMPClient_ host port vr client = do | authCmdsSMPVersion `isCompatible` vr = Just alpnSupportedSMPHandshakes | otherwise = Nothing +runSMPClient :: Transport c => TProxy c 'TServer -> (THandleSMP c 'TClient -> IO a) -> IO a +runSMPClient _ test' = testSMPClient test' + +runSMPServiceClient :: Transport c => TProxy c 'TServer -> (TLS.Credential, C.KeyPairEd25519) -> (THandleSMP c 'TClient -> IO a) -> IO a +runSMPServiceClient _ serviceCreds test' = testSMPServiceClient serviceCreds test' + testNtfServiceClient :: Transport c => TProxy c 'TServer -> C.KeyPairEd25519 -> (THandleSMP c 'TClient -> IO a) -> IO a testNtfServiceClient _ keys client = do tlsNtfServerCreds <- loadServerCredential ntfTestServerCredentials diff --git a/tests/SMPProxyTests.hs b/tests/SMPProxyTests.hs index b756ce7c9..0d8ccdf89 100644 --- a/tests/SMPProxyTests.hs +++ b/tests/SMPProxyTests.hs @@ -188,7 +188,7 @@ deliverMessagesViaProxy proxyServ relayServ alg unsecuredMsgs securedMsgs = do runExceptT' (proxySMPMessage pc NRMInteractive sess Nothing sndId noMsgFlags msg) `shouldReturn` Right () runExceptT' (proxySMPMessage pc NRMInteractive sess {prSessionId = "bad session"} Nothing sndId noMsgFlags msg) `shouldReturn` Left (ProxyProtocolError $ SMP.PROXY SMP.NO_SESSION) -- receive 1 - (_tSess, _v, _sid, [(_entId, STEvent (Right (SMP.MSG RcvMessage {msgId, msgBody = EncRcvMsgBody encBody})))]) <- atomically $ readTBQueue msgQ + (_tSess, _, [(_entId, STEvent (Right (SMP.MSG RcvMessage {msgId, msgBody = EncRcvMsgBody encBody})))]) <- atomically $ readTBQueue msgQ dec msgId encBody `shouldBe` Right msg runExceptT' $ ackSMPMessage rc rPriv rcvId msgId -- secure queue @@ -200,7 +200,7 @@ deliverMessagesViaProxy proxyServ relayServ alg unsecuredMsgs securedMsgs = do runExceptT' (proxySMPMessage pc NRMInteractive sess (Just sPriv) sndId noMsgFlags msg') `shouldReturn` Right () ) ( forM_ securedMsgs $ \msg' -> do - (_tSess, _v, _sid, [(_entId, STEvent (Right (SMP.MSG RcvMessage {msgId = msgId', msgBody = EncRcvMsgBody encBody'})))]) <- atomically $ readTBQueue msgQ + (_tSess, _, [(_entId, STEvent (Right (SMP.MSG RcvMessage {msgId = msgId', msgBody = EncRcvMsgBody encBody'})))]) <- atomically $ readTBQueue msgQ dec msgId' encBody' `shouldBe` Right msg' runExceptT' $ ackSMPMessage rc rPriv rcvId msgId' ) @@ -224,9 +224,9 @@ agentDeliverMessageViaProxy :: (C.AlgorithmI a, C.AuthAlgorithm a) => (NonEmpty agentDeliverMessageViaProxy aTestCfg@(aSrvs, _, aViaProxy) bTestCfg@(bSrvs, _, bViaProxy) alg msg1 msg2 baseId = withAgent 1 aCfg (servers aTestCfg) testDB $ \alice -> withAgent 2 aCfg (servers bTestCfg) testDB2 $ \bob -> runRight_ $ do - (bobId, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe + (bobId, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe aliceId <- A.prepareConnectionToJoin bob 1 True qInfo PQSupportOn - (sqSecured, Nothing) <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe + sqSecured <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe liftIO $ sqSecured `shouldBe` True ("", _, A.CONF confId pqSup' _ "bob's connInfo") <- get alice liftIO $ pqSup' `shouldBe` PQSupportOn @@ -280,9 +280,9 @@ agentDeliverMessagesViaProxyConc agentServers msgs = -- agent connections have to be set up in advance -- otherwise the CONF messages would get mixed with MSG prePair alice bob = do - (bobId, (CCLink qInfo Nothing, Nothing)) <- runExceptT' $ A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe + (bobId, CCLink qInfo Nothing) <- runExceptT' $ A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe aliceId <- runExceptT' $ A.prepareConnectionToJoin bob 1 True qInfo PQSupportOn - (sqSecured, Nothing) <- runExceptT' $ A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe + sqSecured <- runExceptT' $ A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe liftIO $ sqSecured `shouldBe` True confId <- get alice >>= \case @@ -331,7 +331,7 @@ agentViaProxyVersionError = withAgent 1 agentCfg (servers [SMPServer testHost testPort testKeyHash]) testDB $ \alice -> do Left (A.BROKER _ (TRANSPORT TEVersion)) <- withAgent 2 agentCfg (servers [SMPServer testHost2 testPort2 testKeyHash]) testDB2 $ \bob -> runExceptT $ do - (_bobId, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe + (_bobId, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe aliceId <- A.prepareConnectionToJoin bob 1 True qInfo PQSupportOn A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe pure () @@ -351,9 +351,9 @@ agentViaProxyRetryOffline = do let pqEnc = CR.PQEncOn withServer $ \_ -> do (aliceId, bobId) <- withServer2 $ \_ -> runRight $ do - (bobId, (CCLink qInfo Nothing, Nothing)) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe + (bobId, CCLink qInfo Nothing) <- A.createConnection alice NRMInteractive 1 True True SCMInvitation Nothing Nothing CR.IKPQOn SMSubscribe aliceId <- A.prepareConnectionToJoin bob 1 True qInfo PQSupportOn - (sqSecured, Nothing) <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe + sqSecured <- A.joinConnection bob NRMInteractive 1 aliceId True qInfo "bob's connInfo" PQSupportOn SMSubscribe liftIO $ sqSecured `shouldBe` True ("", _, A.CONF confId pqSup' _ "bob's connInfo") <- get alice liftIO $ pqSup' `shouldBe` PQSupportOn @@ -434,14 +434,14 @@ agentViaProxyRetryNoSession = do testNoProxy :: AStoreType -> IO () testNoProxy msType = do withSmpServerConfigOn (transport @TLS) (cfgMS msType) testPort2 $ \_ -> do - testSMPClient_ "127.0.0.1" testPort2 proxyVRangeV8 $ \(th :: THandleSMP TLS 'TClient) -> do + testSMPClient_ "127.0.0.1" testPort2 proxyVRangeV8 Nothing $ \(th :: THandleSMP TLS 'TClient) -> do (_, _, reply) <- sendRecv th (Nothing, "0", NoEntity, SMP.PRXY testSMPServer Nothing) reply `shouldBe` Right (SMP.ERR $ SMP.PROXY SMP.BASIC_AUTH) testProxyAuth :: AStoreType -> IO () testProxyAuth msType = do withSmpServerConfigOn (transport @TLS) proxyCfgAuth testPort $ \_ -> do - testSMPClient_ "127.0.0.1" testPort proxyVRangeV8 $ \(th :: THandleSMP TLS 'TClient) -> do + testSMPClient_ "127.0.0.1" testPort proxyVRangeV8 Nothing $ \(th :: THandleSMP TLS 'TClient) -> do (_, _, reply) <- sendRecv th (Nothing, "0", NoEntity, SMP.PRXY testSMPServer2 $ Just "wrong") reply `shouldBe` Right (SMP.ERR $ SMP.PROXY SMP.BASIC_AUTH) where diff --git a/tests/ServerTests.hs b/tests/ServerTests.hs index b2c2d997c..27a72d2ac 100644 --- a/tests/ServerTests.hs +++ b/tests/ServerTests.hs @@ -29,9 +29,11 @@ import Data.Bifunctor (first) import qualified Data.ByteString.Base64 as B64 import Data.ByteString.Char8 (ByteString) import qualified Data.ByteString.Char8 as B +import Data.Foldable (foldrM) import Data.Hashable (hash) import qualified Data.IntSet as IS import Data.List.NonEmpty (NonEmpty) +import Data.Maybe (catMaybes) import Data.String (IsString (..)) import Data.Type.Equality import qualified Data.X509.Validation as XV @@ -50,6 +52,7 @@ import Simplex.Messaging.Server.MsgStore.Types (MsgStoreClass (..), QSType (..), import Simplex.Messaging.Server.Stats (PeriodStatsData (..), ServerStatsData (..)) import Simplex.Messaging.Server.StoreLog (StoreLogRecord (..), closeStoreLog) import Simplex.Messaging.Transport +import Simplex.Messaging.Transport.Credentials import Simplex.Messaging.Util (whenM) import Simplex.Messaging.Version (mkVersionRange) import System.Directory (doesDirectoryExist, doesFileExist, removeDirectoryRecursive, removeFile) @@ -84,6 +87,9 @@ serverTests = do describe "GET & SUB commands" testGetSubCommands describe "Exceeding queue quota" testExceedQueueQuota describe "Concurrent sending and delivery" testConcurrentSendDelivery + describe "Service message subscriptions" $ do + testServiceDeliverSubscribe + testServiceUpgradeAndDowngrade describe "Store log" testWithStoreLog describe "Restore messages" testRestoreMessages describe "Restore messages (old / v2)" testRestoreExpireMessages @@ -111,6 +117,9 @@ pattern New rPub dhPub = NEW (NewQueueReq rPub dhPub Nothing SMSubscribe (Just ( pattern Ids :: RecipientId -> SenderId -> RcvPublicDhKey -> BrokerMsg pattern Ids rId sId srvDh <- IDS (QIK rId sId srvDh _sndSecure _linkId Nothing Nothing) +pattern Ids_ :: RecipientId -> SenderId -> RcvPublicDhKey -> ServiceId -> BrokerMsg +pattern Ids_ rId sId srvDh serviceId <- IDS (QIK rId sId srvDh _sndSecure _linkId (Just serviceId) Nothing) + pattern Msg :: MsgId -> MsgBody -> BrokerMsg pattern Msg msgId body <- MSG RcvMessage {msgId, msgBody = EncRcvMsgBody body} @@ -135,11 +144,21 @@ serviceSignSendRecv h pk serviceKey t = do [r] <- signSendRecv_ h pk (Just serviceKey) t pure r +serviceSignSendRecv2 :: forall c p. (Transport c, PartyI p) => THandleSMP c 'TClient -> C.APrivateAuthKey -> C.PrivateKeyEd25519 -> (ByteString, EntityId, Command p) -> IO (Transmission (Either ErrorType BrokerMsg), Transmission (Either ErrorType BrokerMsg)) +serviceSignSendRecv2 h pk serviceKey t = do + [r1, r2] <- signSendRecv_ h pk (Just serviceKey) t + pure (r1, r2) + signSendRecv_ :: forall c p. (Transport c, PartyI p) => THandleSMP c 'TClient -> C.APrivateAuthKey -> Maybe C.PrivateKeyEd25519 -> (ByteString, EntityId, Command p) -> IO (NonEmpty (Transmission (Either ErrorType BrokerMsg))) -signSendRecv_ h@THandle {params} (C.APrivateAuthKey a pk) serviceKey_ (corrId, qId, cmd) = do +signSendRecv_ h pk serviceKey_ t = do + signSend_ h pk serviceKey_ t + tGetClient h + +signSend_ :: forall c p. (Transport c, PartyI p) => THandleSMP c 'TClient -> C.APrivateAuthKey -> Maybe C.PrivateKeyEd25519 -> (ByteString, EntityId, Command p) -> IO () +signSend_ h@THandle {params} (C.APrivateAuthKey a pk) serviceKey_ (corrId, qId, cmd) = do let TransmissionForAuth {tForAuth, tToSend} = encodeTransmissionForAuth params (CorrId corrId, qId, cmd) Right () <- tPut1 h (authorize tForAuth, tToSend) - liftIO $ tGetClient h + pure () where authorize t = (,(`C.sign'` t) <$> serviceKey_) <$> case a of C.SEd25519 -> Just . TASignature . C.ASignature C.SEd25519 $ C.sign' pk t' @@ -660,6 +679,198 @@ testConcurrentSendDelivery = Resp "4" _ OK <- signSendRecv rh rKey ("4", rId, ACK mId2) pure () +testServiceDeliverSubscribe :: SpecWith (ASrvTransport, AStoreType) +testServiceDeliverSubscribe = + it "should create queue as service and subscribe with SUBS after reconnect" $ \(at@(ATransport t), msType) -> do + g <- C.newRandom + creds <- genCredentials g Nothing (0, 2400) "localhost" + let (_fp, tlsCred) = tlsCredentials [creds] + serviceKeys@(_, servicePK) <- atomically $ C.generateKeyPair g + let aServicePK = C.APrivateAuthKey C.SEd25519 servicePK + withSmpServerConfigOn at (cfgMS msType) testPort $ \_ -> runSMPClient t $ \h -> do + (rPub, rKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + (dhPub, dhPriv :: C.PrivateKeyX25519) <- atomically $ C.generateKeyPair g + (sPub, sKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + + (rId, sId, dec, serviceId) <- runSMPServiceClient t (tlsCred, serviceKeys) $ \sh -> do + Resp "1" NoEntity (ERR SERVICE) <- signSendRecv sh rKey ("1", NoEntity, New rPub dhPub) + Resp "2" NoEntity (Ids_ rId sId srvDh serviceId) <- serviceSignSendRecv sh rKey servicePK ("2", NoEntity, New rPub dhPub) + let dec = decryptMsgV3 $ C.dh' srvDh dhPriv + Resp "3" sId' OK <- signSendRecv h sKey ("3", sId, SKEY sPub) + sId' `shouldBe` sId + Resp "4" _ OK <- signSendRecv h sKey ("4", sId, _SEND "hello") + Resp "5" _ OK <- signSendRecv h sKey ("5", sId, _SEND "hello 2") + Resp "" rId' (Msg mId1 msg1) <- tGet1 sh + rId' `shouldBe` rId + dec mId1 msg1 `shouldBe` Right "hello" + -- ACK doesn't need service signature + Resp "6" _ (Msg mId2 msg2) <- signSendRecv sh rKey ("6", rId, ACK mId1) + dec mId2 msg2 `shouldBe` Right "hello 2" + Resp "7" _ (ERR NO_MSG) <- signSendRecv sh rKey ("7", rId, ACK mId1) + Resp "8" _ OK <- signSendRecv sh rKey ("8", rId, ACK mId2) + Resp "9" _ OK <- signSendRecv h sKey ("9", sId, _SEND "hello 3") + pure (rId, sId, dec, serviceId) + + runSMPServiceClient t (tlsCred, serviceKeys) $ \sh -> do + let idsHash = queueIdsHash [rId] + Resp "10" NoEntity (ERR (CMD NO_AUTH)) <- signSendRecv sh aServicePK ("10", NoEntity, SUBS 1 idsHash) + signSend_ sh aServicePK Nothing ("11", serviceId, SUBS 1 idsHash) + [mId3] <- + fmap catMaybes $ + receiveInAnyOrder -- race between SOKS and MSG, clients can handle it + sh + [ \case + Resp "11" serviceId' (SOKS n idsHash') -> do + n `shouldBe` 1 + idsHash' `shouldBe` idsHash + serviceId' `shouldBe` serviceId + pure $ Just Nothing + _ -> pure Nothing, + \case + Resp "" rId'' (Msg mId3 msg3) -> do + rId'' `shouldBe` rId + dec mId3 msg3 `shouldBe` Right "hello 3" + pure $ Just $ Just mId3 + _ -> pure Nothing + ] + Resp "" NoEntity ALLS <- tGet1 sh + Resp "12" _ OK <- signSendRecv sh rKey ("12", rId, ACK mId3) + Resp "14" _ OK <- signSendRecv h sKey ("14", sId, _SEND "hello 4") + Resp "" _ (Msg mId4 msg4) <- tGet1 sh + dec mId4 msg4 `shouldBe` Right "hello 4" + Resp "15" _ OK <- signSendRecv sh rKey ("15", rId, ACK mId4) + pure () + +testServiceUpgradeAndDowngrade :: SpecWith (ASrvTransport, AStoreType) +testServiceUpgradeAndDowngrade = + it "should create queue as client and switch to service and back" $ \(at@(ATransport t), msType) -> do + g <- C.newRandom + creds <- genCredentials g Nothing (0, 2400) "localhost" + let (_fp, tlsCred) = tlsCredentials [creds] + serviceKeys@(_, servicePK) <- atomically $ C.generateKeyPair g + let aServicePK = C.APrivateAuthKey C.SEd25519 servicePK + withSmpServerConfigOn at (cfgMS msType) testPort $ \_ -> runSMPClient t $ \h -> do + (rPub, rKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + (dhPub, dhPriv :: C.PrivateKeyX25519) <- atomically $ C.generateKeyPair g + (sPub, sKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + (rPub2, rKey2) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + (dhPub2, dhPriv2 :: C.PrivateKeyX25519) <- atomically $ C.generateKeyPair g + (sPub2, sKey2) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + (rPub3, rKey3) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + (dhPub3, dhPriv3 :: C.PrivateKeyX25519) <- atomically $ C.generateKeyPair g + (sPub3, sKey3) <- atomically $ C.generateAuthKeyPair C.SEd25519 g + + (rId, sId, dec) <- runSMPClient t $ \sh -> do + Resp "1" NoEntity (Ids rId sId srvDh) <- signSendRecv sh rKey ("1", NoEntity, New rPub dhPub) + let dec = decryptMsgV3 $ C.dh' srvDh dhPriv + Resp "2" sId' OK <- signSendRecv h sKey ("2", sId, SKEY sPub) + sId' `shouldBe` sId + Resp "3" _ OK <- signSendRecv h sKey ("3", sId, _SEND "hello") + Resp "" rId' (Msg mId1 msg1) <- tGet1 sh + rId' `shouldBe` rId + dec mId1 msg1 `shouldBe` Right "hello" + Resp "4" _ OK <- signSendRecv sh rKey ("4", rId, ACK mId1) + Resp "5" _ OK <- signSendRecv h sKey ("5", sId, _SEND "hello 2") + pure (rId, sId, dec) + + -- split to prevent message delivery + (rId2, sId2, dec2) <- runSMPClient t $ \sh -> do + Resp "6" NoEntity (Ids rId2 sId2 srvDh2) <- signSendRecv sh rKey2 ("6", NoEntity, New rPub2 dhPub2) + let dec2 = decryptMsgV3 $ C.dh' srvDh2 dhPriv2 + Resp "7" sId2' OK <- signSendRecv h sKey2 ("7", sId2, SKEY sPub2) + sId2' `shouldBe` sId2 + pure (rId2, sId2, dec2) + + (rId3, _sId3, _dec3) <- runSMPClient t $ \sh -> do + Resp "6" NoEntity (Ids rId3 sId3 srvDh3) <- signSendRecv sh rKey3 ("6", NoEntity, New rPub3 dhPub3) + let dec3 = decryptMsgV3 $ C.dh' srvDh3 dhPriv3 + Resp "7" sId3' OK <- signSendRecv h sKey3 ("7", sId3, SKEY sPub3) + sId3' `shouldBe` sId3 + pure (rId3, sId3, dec3) + + serviceId <- runSMPServiceClient t (tlsCred, serviceKeys) $ \sh -> do + Resp "8" _ (ERR SERVICE) <- signSendRecv sh rKey ("8", rId, SUB) + (Resp "9" rId' (SOK (Just serviceId)), Resp "" rId'' (Msg mId2 msg2)) <- serviceSignSendRecv2 sh rKey servicePK ("9", rId, SUB) + rId' `shouldBe` rId + rId'' `shouldBe` rId + dec mId2 msg2 `shouldBe` Right "hello 2" + (Resp "10" rId2' (SOK (Just serviceId'))) <- serviceSignSendRecv sh rKey2 servicePK ("10", rId2, SUB) + rId2' `shouldBe` rId2 + serviceId' `shouldBe` serviceId + Resp "10.1" _ OK <- signSendRecv sh rKey ("10.1", rId, ACK mId2) + (Resp "10.2" rId3' (SOK (Just serviceId''))) <- serviceSignSendRecv sh rKey3 servicePK ("10.2", rId3, SUB) + rId3' `shouldBe` rId3 + serviceId'' `shouldBe` serviceId + pure serviceId + + Resp "11" _ OK <- signSendRecv h sKey ("11", sId, _SEND "hello 3.1") + Resp "12" _ OK <- signSendRecv h sKey2 ("12", sId2, _SEND "hello 3.2") + + runSMPServiceClient t (tlsCred, serviceKeys) $ \sh -> do + let idsHash = queueIdsHash [rId, rId2, rId3] + signSend_ sh aServicePK Nothing ("14", serviceId, SUBS 3 idsHash) + [(rKey3_1, rId3_1, mId3_1), (rKey3_2, rId3_2, mId3_2)] <- + fmap catMaybes $ + receiveInAnyOrder -- race between SOKS and MSG, clients can handle it + sh + [ \case + Resp "14" serviceId' (SOKS n idsHash') -> do + n `shouldBe` 3 + idsHash' `shouldBe` idsHash + serviceId' `shouldBe` serviceId + pure $ Just Nothing + _ -> pure Nothing, + \case + Resp "" rId'' (Msg mId3 msg3) | rId'' == rId -> do + dec mId3 msg3 `shouldBe` Right "hello 3.1" + pure $ Just $ Just (rKey, rId, mId3) + _ -> pure Nothing, + \case + Resp "" rId'' (Msg mId3 msg3) | rId'' == rId2 -> do + dec2 mId3 msg3 `shouldBe` Right "hello 3.2" + pure $ Just $ Just (rKey2, rId2, mId3) + _ -> pure Nothing + ] + Resp "" NoEntity ALLS <- tGet1 sh + Resp "15" _ OK <- signSendRecv sh rKey3_1 ("15", rId3_1, ACK mId3_1) + Resp "16" _ OK <- signSendRecv sh rKey3_2 ("16", rId3_2, ACK mId3_2) + pure () + + Resp "17" _ OK <- signSendRecv h sKey ("17", sId, _SEND "hello 4") + + runSMPClient t $ \sh -> do + Resp "18" _ (ERR SERVICE) <- signSendRecv sh aServicePK ("18", serviceId, SUBS 3 mempty) + (Resp "19" rId' (SOK Nothing), Resp "" rId'' (Msg mId4 msg4)) <- signSendRecv2 sh rKey ("19", rId, SUB) + rId' `shouldBe` rId + rId'' `shouldBe` rId + dec mId4 msg4 `shouldBe` Right "hello 4" + Resp "20" _ OK <- signSendRecv sh rKey ("20", rId, ACK mId4) + Resp "21" _ OK <- signSendRecv h sKey ("21", sId, _SEND "hello 5") + Resp "" _ (Msg mId5 msg5) <- tGet1 sh + dec mId5 msg5 `shouldBe` Right "hello 5" + Resp "22" _ OK <- signSendRecv sh rKey ("22", rId, ACK mId5) + + Resp "23" rId2' (SOK Nothing) <- signSendRecv sh rKey2 ("23", rId2, SUB) + rId2' `shouldBe` rId2 + Resp "24" _ OK <- signSendRecv h sKey ("24", sId, _SEND "hello 6") + Resp "" _ (Msg mId6 msg6) <- tGet1 sh + dec mId6 msg6 `shouldBe` Right "hello 6" + Resp "25" _ OK <- signSendRecv sh rKey ("25", rId, ACK mId6) + pure () + +receiveInAnyOrder :: (HasCallStack, Transport c) => THandleSMP c 'TClient -> [(CorrId, EntityId, Either ErrorType BrokerMsg) -> IO (Maybe b)] -> IO [b] +receiveInAnyOrder h = fmap reverse . go [] + where + go rs [] = pure rs + go rs ps = withFrozenCallStack $ do + r <- 5000000 `timeout` tGet1 h >>= maybe (error "inAnyOrder timeout") pure + (r_, ps') <- foldrM (choose r) (Nothing, []) ps + case r_ of + Just r' -> go (r' : rs) ps' + Nothing -> error $ "unexpected event: " <> show r + choose r p (Nothing, ps') = (maybe (Nothing, p : ps') ((,ps') . Just)) <$> p r + choose _ p (Just r, ps') = pure (Just r, p : ps') + testWithStoreLog :: SpecWith (ASrvTransport, AStoreType) testWithStoreLog = it "should store simplex queues to log and restore them after server restart" $ \(at@(ATransport t), msType) -> do @@ -1123,7 +1334,7 @@ testMessageServiceNotifications = Resp "4" _ (SOK (Just serviceId')) <- serviceSignSendRecv nh2 nKey servicePK ("4", nId, NSUB) serviceId' `shouldBe` serviceId -- service subscription is terminated - Resp "" serviceId2 (ENDS 1) <- tGet1 nh1 + Resp "" serviceId2 (ENDS 1 _) <- tGet1 nh1 serviceId2 `shouldBe` serviceId deliverMessage rh rId rKey sh sId sKey nh2 "hello again" dec 1000 `timeout` tGetClient @SMPVersion @ErrorType @BrokerMsg nh1 >>= \case @@ -1159,9 +1370,11 @@ testMessageServiceNotifications = deliverMessage rh rId rKey sh sId sKey nh2 "connection 1" dec deliverMessage rh rId'' rKey'' sh sId'' sKey'' nh2 "connection 2" dec'' -- -- another client makes service subscription - Resp "12" serviceId5 (SOKS 2) <- signSendRecv nh1 (C.APrivateAuthKey C.SEd25519 servicePK) ("12", serviceId, NSUBS) + let idsHash = queueIdsHash [nId', nId''] + Resp "12" serviceId5 (SOKS 2 idsHash') <- signSendRecv nh1 (C.APrivateAuthKey C.SEd25519 servicePK) ("12", serviceId, NSUBS 2 idsHash) + idsHash' `shouldBe` idsHash serviceId5 `shouldBe` serviceId - Resp "" serviceId6 (ENDS 2) <- tGet1 nh2 + Resp "" serviceId6 (ENDS 2 _) <- tGet1 nh2 serviceId6 `shouldBe` serviceId deliverMessage rh rId rKey sh sId sKey nh1 "connection 1 one more" dec deliverMessage rh rId'' rKey'' sh sId'' sKey'' nh1 "connection 2 one more" dec'' @@ -1182,18 +1395,19 @@ testServiceNotificationsTwoRestarts = (nPub, nKey) <- atomically $ C.generateAuthKeyPair C.SEd25519 g serviceKeys@(_, servicePK) <- atomically $ C.generateKeyPair g (rcvNtfPubDhKey, _) <- atomically $ C.generateKeyPair g - (rId, rKey, sId, dec, serviceId) <- withSmpServerStoreLogOn ps testPort $ runTest2 t $ \sh rh -> do + (rId, rKey, sId, dec, nId, serviceId) <- withSmpServerStoreLogOn ps testPort $ runTest2 t $ \sh rh -> do (sId, rId, rKey, dhShared) <- createAndSecureQueue rh sPub let dec = decryptMsgV3 dhShared Resp "0" _ (NID nId _) <- signSendRecv rh rKey ("0", rId, NKEY nPub rcvNtfPubDhKey) testNtfServiceClient t serviceKeys $ \nh -> do Resp "1" _ (SOK (Just serviceId)) <- serviceSignSendRecv nh nKey servicePK ("1", nId, NSUB) deliverMessage rh rId rKey sh sId sKey nh "hello" dec - pure (rId, rKey, sId, dec, serviceId) + pure (rId, rKey, sId, dec, nId, serviceId) + let idsHash = queueIdsHash [nId] threadDelay 250000 withSmpServerStoreLogOn ps testPort $ runTest2 t $ \sh rh -> testNtfServiceClient t serviceKeys $ \nh -> do - Resp "2.1" serviceId' (SOKS n) <- signSendRecv nh (C.APrivateAuthKey C.SEd25519 servicePK) ("2.1", serviceId, NSUBS) + Resp "2.1" serviceId' (SOKS n _) <- signSendRecv nh (C.APrivateAuthKey C.SEd25519 servicePK) ("2.1", serviceId, NSUBS 1 idsHash) n `shouldBe` 1 Resp "2.2" _ (SOK Nothing) <- signSendRecv rh rKey ("2.2", rId, SUB) serviceId' `shouldBe` serviceId @@ -1201,7 +1415,7 @@ testServiceNotificationsTwoRestarts = threadDelay 250000 withSmpServerStoreLogOn ps testPort $ runTest2 t $ \sh rh -> testNtfServiceClient t serviceKeys $ \nh -> do - Resp "3.1" _ (SOKS n) <- signSendRecv nh (C.APrivateAuthKey C.SEd25519 servicePK) ("3.1", serviceId, NSUBS) + Resp "3.1" _ (SOKS n _) <- signSendRecv nh (C.APrivateAuthKey C.SEd25519 servicePK) ("3.1", serviceId, NSUBS 1 idsHash) n `shouldBe` 1 Resp "3.2" _ (SOK Nothing) <- signSendRecv rh rKey ("3.2", rId, SUB) deliverMessage rh rId rKey sh sId sKey nh "hello 3" dec diff --git a/tests/Test.hs b/tests/Test.hs index c3fe92953..d397c57db 100644 --- a/tests/Test.hs +++ b/tests/Test.hs @@ -40,6 +40,8 @@ import XFTPWebTests (xftpWebTests) #if defined(dbPostgres) import Fixtures +import SMPAgentClient (testDB) +import Simplex.Messaging.Agent.Store.Postgres.Migrations.App #else import AgentTests.SchemaDump (schemaDumpTest) #endif @@ -47,13 +49,13 @@ import AgentTests.SchemaDump (schemaDumpTest) #if defined(dbServerPostgres) import NtfServerTests (ntfServerTests) import NtfClient (ntfTestServerDBConnectInfo, ntfTestStoreDBOpts) -import PostgresSchemaDump (postgresSchemaDumpTest) import SMPClient (testServerDBConnectInfo, testStoreDBOpts) import Simplex.Messaging.Notifications.Server.Store.Migrations (ntfServerMigrations) import Simplex.Messaging.Server.QueueStore.Postgres.Migrations (serverMigrations) #endif #if defined(dbPostgres) || defined(dbServerPostgres) +import PostgresSchemaDump (postgresSchemaDumpTest) import SMPClient (postgressBracket) #endif @@ -73,10 +75,6 @@ main = do . before_ (createDirectoryIfMissing False "tests/tmp") . after_ (eventuallyRemove "tests/tmp" 3) $ do --- TODO [postgres] schema dump for postgres -#if !defined(dbPostgres) - describe "Agent SQLite schema dump" schemaDumpTest -#endif describe "Core tests" $ do describe "Batching tests" batchingTests describe "Encoding tests" encodingTests @@ -155,6 +153,17 @@ main = do describe "XRCP" remoteControlTests describe "Web" webTests describe "Server CLIs" cliTests +#if defined(dbPostgres) + around_ (postgressBracket testDBConnectInfo) $ + describe "Agent PostgreSQL schema dump" $ + postgresSchemaDumpTest + appMigrations + ["20250322_short_links"] -- snd_secure and last_broker_ts columns swap order on down migration + (testDBOpts testDB) + "src/Simplex/Messaging/Agent/Store/Postgres/Migrations/agent_postgres_schema.sql" +#else + describe "Agent SQLite schema dump" schemaDumpTest +#endif eventuallyRemove :: FilePath -> Int -> IO () eventuallyRemove path retries = case retries of