diff --git a/docs/Edge Market Data Connection.es.md b/docs/Edge Market Data Connection.es.md new file mode 100644 index 0000000..dbde7df --- /dev/null +++ b/docs/Edge Market Data Connection.es.md @@ -0,0 +1,465 @@ +--- +description: Ejecute doublezero-edge-connect para reenviar shreds de Solana a un puerto UDP local y consumir datos de mercado normalizados de Edge a través de un WebSocket local. +--- + +# Conexión Edge + +!!! warning "Al conectarme a DoubleZero acepto los [Términos de Uso de DoubleZero](https://doublezero.xyz/terms-protocol). Los datos son únicamente para sus propósitos internos y no pueden ser retransmitidos (ver Sección 2(e))." + +`doublezero-edge-connect` es un puente que se une al **multicast binario de DoubleZero Edge** y lo re-sirve localmente como dos feeds: + +1. **Reenvío de shreds de Solana** — shreds deduplicados (opcionalmente con verificación de firma) distribuidos a uno o más destinos UDP locales, listos para su validador o RPC. +2. **Datos de mercado normalizados** — feeds de venues de Edge decodificados, con precisión corregida, y re-servidos como un único WebSocket JSON en `ws://host:8081`. + +Ambos se ejecutan desde el mismo contenedor y la misma instalación de una sola línea. Habilite los feeds que su autorización onchain le otorgue. + +``` + ┌─ UDP datagrams ──▶ validator / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binary) (dedup · decode · normalize) └─ WebSocket (JSON) ──▶ trading engine + ws://host:8081 +``` + +--- + +## Requisitos + +- Host **Linux/amd64** con una dirección IPv4 pública autorizada onchain para el entorno objetivo. +- **Docker** (el instalador de una línea lo instala si no está presente). +- **Conectividad GRE** — permita el protocolo IP 47 en su proveedor de nube; en AWS deshabilite la verificación de origen/destino del ENI. +- Un **secreto de acceso DoubleZero**: un token base64 con prefijo `DZ_` o una ruta a un archivo de keypair, obtenido del proceso de [incorporación a DoubleZero](setup.md). + +--- + +## Paso 1: Instalar y Ejecutar + +Un solo comando prepara el host e inicia el contenedor puente. Se une a la red DoubleZero e inicia cada feed que su autorización otorgue — reenvío de shreds y/o el WebSocket de datos de mercado en `:8081`: + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet (privada)" + + ```bash + # Requiere un token GHCR con read:packages + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +Lo que hace el script: + +1. Verifica que el host sea Linux/amd64, se asegura de que Docker esté presente (ofrece instalarlo). +2. Prepara el kernel del host para el túnel GRE: carga `tun`/`ip_gre`, aumenta `net.core.rmem_max`, advierte sobre reglas de firewall y del proveedor de nube. +3. Carga su secreto de acceso (se solicita una vez si `DZ_SECRET` no está configurado). +4. Ejecuta el contenedor puente (`--network host`, `NET_ADMIN`/`NET_RAW`, `/dev/net/tun`) y ejecuta `doublezero connect multicast`. + +!!! tip "Instalación no interactiva" + Configure `DZ_SECRET=DZ_…` antes del pipe para ejecutar completamente desatendido — sin ningún prompt. + +--- + +## Paso 2: Configurar + +Toda la configuración se realiza mediante **variables de entorno establecidas antes del pipe**. No hay archivo de configuración. + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### Variables del instalador + +| Variable | Valor predeterminado | Propósito | +|----------|---------|---------| +| `DZ_SECRET` | *(solicitado)* | Token base64 con prefijo `DZ_` **o** ruta a un archivo de keypair. Un token se inyecta en el contenedor y nunca se escribe en disco; un archivo se monta en modo solo lectura. | +| `DZ_ENV` | por script | `mainnet-beta` \| `testnet` \| `devnet`. | +| `DZ_IMAGE` | por script | Sobrescribir la imagen del contenedor. | +| `DZ_NAME` | `doublezero-edge-connect` | Nombre del contenedor. | +| `DZ_FEEDS` | *(todos)* | Venues separados por comas para acotar la ingesta de datos de mercado (ej. `VenueA,VenueB`). No afecta el reenvío de shreds de Solana. | +| `DZ_ASSUME_YES` | `0` | Omitir prompts de confirmación (ej. el prompt de instalación de Docker). | +| `DZ_GHCR_TOKEN` | — | **Solo devnet** — un token GHCR con `read:packages` (la imagen de devnet es privada). | +| `DZ_GHCR_USER` | `malbeclabs` | **Solo devnet** — nombre de usuario GHCR para el login. | + +### Variables del puente + +El instalador reenvía **cualquier variable no vacía** del puente directamente al contenedor. Las más comunes: + +| Variable | Valor predeterminado | Propósito | +|----------|---------|---------| +| `DZ_IFACE` | `doublezero1` | Interfaz de red en la que escuchar. | +| `DZ_RECV_BUF` | — | Sobrescritura del buffer de recepción UDP (bytes). | +| `METRICS_BIND` | *(vacío / desactivado)* | Habilitar el endpoint Prometheus `/metrics` (ej. `127.0.0.1:9090`). | +| `RUST_LOG` | `info` | Nivel de log (`debug`, `warn`, etc.). | +| `DZ_SHRED_FORWARD` | — | Destino(s) UDP locales para shreds reenviados — ver [Reenvío de Shreds de Solana](#reenvio-de-shreds-de-solana). | +| `WS_BIND` | `0.0.0.0:8081` | Dirección de enlace del WebSocket de datos de mercado — ver [WebSocket de Datos de Mercado](#websocket-de-datos-de-mercado). | +| `WS_MAX_CLIENTS` | `64` | Máximo de clientes WebSocket concurrentes. | +| `WS_INPUT_COINS` | *(vacío / desactivado)* | Habilitar el respaldo público de WebSocket para los símbolos listados (ej. `BTC,ETH`). | + +**Ejemplos:** + +```bash +# Reenviar shreds a un validador/RPC local: +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# No interactivo, testnet: +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# Acotar datos de mercado a venues específicos, logging verbose, puerto WS no predeterminado: +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Habilitar métricas y un respaldo público de WS: +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + Dado que el instalador solo reenvía valores **no vacíos**, no puede pasar una sobrescritura vacía (ej. `WS_BIND=""` para deshabilitar el sink WebSocket) a través del one-liner. Use un `docker run` escrito manualmente para eso — ver [Autoalojamiento](#avanzado-autoalojamiento). + +--- + +## Reenvío de Shreds de Solana + +El puente se une a los grupos multicast `edge-solana-*` de shreds y distribuye cada datagrama a uno o más destinos UDP locales — alimentando su validador o RPC directamente desde la red Edge. Se activa automáticamente al descubrir cuando esos grupos están presentes en su autorización. + +```bash +# Predeterminado (solo dedup, reenviar al puerto local 20000): +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# Con verificación de firma: +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| Variable | Valor predeterminado | Propósito | +|----------|---------|---------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | Destino(s) para shreds reenviados (repetible). | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup` (una copia por shred), `sigverify` (+ verificación ed25519), `none` (todos los datagramas). | +| `DZ_SHRED_RPC_URL` | — | Endpoint RPC de Solana; requerido por el modo `sigverify`. | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | Tamaño de la ventana de deduplicación. | + +Ver [Reenvío de shreds](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md) para el pipeline completo y advertencias. + +--- + +## WebSocket de Datos de Mercado + +Abra un WebSocket a `ws://:8081` y lea tramas JSON. Recibirá todos los venues para los que esté autorizado. Un mensaje opcional `subscribe` acota el flujo a venues y símbolos específicos. + +Cualquier motor que hable WebSocket + JSON puede consumirlo con un adaptador ligero (~50–100 líneas). El multicast binario, la división de dos puertos por venue y el handshake de manifiesto/precisión permanecen dentro del puente; el único contrato contra el que un consumidor programa es el JSON del WebSocket. + +### Ciclo de vida de la conexión + +En cada nueva conexión el puente: + +1. **Reproduce las definiciones de instrumentos actuales** — un mensaje `instrument` por cada símbolo conocido — para que el consumidor tenga la precisión antes de la primera cotización. +2. **Reproduce la última instantánea de profundidad** por símbolo (si el feed Market-by-Order está activo). +3. **Transmite** mensajes `quote` / `trade` / `midpoint` / `depth` a medida que llegan, distribuidos a todos los consumidores conectados. + +``` +connect → instrument (×N) → depth (×M, últimos libros) → quote → trade → depth → … +``` + +### Tipos de mensaje + +Cada mensaje es un objeto JSON etiquetado con un campo `type`: + +| `type` | Significado | +|--------|---------| +| `instrument` | Definición de instrumento/precisión. | +| `quote` | Actualización del tope del libro (estado completo). | +| `trade` | Impresión de operación (última venta). | +| `midpoint` | Precio medio derivado. | +| `depth` | Instantánea completa de profundidad del libro de órdenes. | +| `status` | Transición de salud del feed a nivel de venue. | + +Los consumidores **deben ignorar valores `type` desconocidos y campos desconocidos** (compatibilidad hacia adelante). + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +Se envía al conectar y cada vez que las definiciones cambian. `price_exponent` y `qty_exponent` indican el tick size y el paso de tamaño del venue como potencias de diez. + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +Cada `quote` es **estado completo** — un mensaje perdido se auto-recupera con la siguiente cotización, sin necesidad de resincronización. Las cuatro marcas de tiempo descomponen la latencia de extremo a extremo: + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (recepción del consumidor) + libro del venue llegada por cable post-decodificación entrega al WS +``` + +`0` es el valor centinela para "no disponible" — trátelo como ausente, no como 1970. + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side` es `"buy"`, `"sell"` o `"unknown"`. Las operaciones son eventos puntuales y no se reproducen al reconectar. + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +Los `bids` están ordenados del precio más alto primero; los `asks` están ordenados del precio más bajo primero. Cada `depth` es una **instantánea completa** — reemplace, no combine. + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +Se emite en el edge cuando el multicast de cotizaciones de un venue queda en silencio (`state:"down"`) o se recupera (`state:"ok"`). Úselo para atenuar un venue en su interfaz. La entrega de cotizaciones no depende del estado — el feed se auto-recupera con la siguiente cotización. + +### Suscripciones + +Por defecto recibe todo. Envíe un mensaje de control para acotar el flujo: + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Omitir un campo coincide con cualquier valor (`{"symbol":"SOL"}` = SOL en todos los venues). `venue` se compara sin distinción de mayúsculas/minúsculas. + +**Confirmación del servidor:** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Los errores devuelven `{"channel":"error","error":""}`. + +### Heartbeat y comprobación de vida + +- El servidor envía un **Ping de WebSocket** cada 20 segundos; los clientes compatibles responden automáticamente con Pong. +- Los clientes silenciosos durante 60 segundos son cerrados y eliminados. +- Keepalive a nivel de aplicación: `{"method":"ping"}` → `{"channel":"pong"}`. + +### Esqueleto de consumidor + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # tipos desconocidos: ignorar silenciosamente (compatibilidad hacia adelante) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### Fuentes de entrada y el respaldo WebSocket + +El feed multicast de Edge está siempre activo. Un **respaldo público por WebSocket** opcional puede llenar vacíos cuando el feed Edge se detiene: + +```bash +# Habilitar el respaldo para BTC y ETH: +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Las dos fuentes compiten por cada tick `(venue, symbol, source_ts)` dentro de un árbitro compartido. En estado estable la fuente Edge gana (sub-ms vs. decenas de ms por internet); cuando Edge tiene vacíos, la copia pública los completa. La salida WebSocket es idéntica independientemente de qué fuente entregó una actualización determinada. + +--- + +## Gestionar el Contenedor + +```bash +# Transmitir logs +sudo docker logs -f doublezero-edge-connect + +# Verificar estado del túnel +sudo docker exec -it doublezero-edge-connect doublezero status + +# Verificar latencias del dispositivo +sudo docker exec -it doublezero-edge-connect doublezero latency + +# Detener y eliminar +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "Sin TLS" + El puente está diseñado para una red confiable/local. Termine TLS en un proxy inverso si expone el endpoint WebSocket externamente. + +--- + +## Monitoreo (Métricas de Prometheus) + +El endpoint de métricas está **desactivado por defecto**. Habilítelo con `METRICS_BIND`: + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Luego haga scraping: + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +Métricas clave: + +| Métrica | Qué rastrea | +|--------|---------------| +| `dz_feed_up{venue}` | `1` mientras el multicast de ese venue está activo, `0` mientras está en silencio. | +| `dz_datagrams_received_total{venue}` | Volumen de ingesta por venue. | +| `dz_emit_total{venue,kind}` | Mensajes difundidos después de la deduplicación, por tipo. | +| `dz_quotes_dropped_total{venue}` | Cotizaciones obsoletas/duplicadas suprimidas. | +| `dz_ws_clients` | Clientes WebSocket conectados actualmente. | +| `dz_ws_messages_sent_total{kind}` | Mensajes reenviados a los clientes. | +| `dz_ws_client_lagged_total` | Veces que un cliente lento fue descartado para proteger el feed. | + +Una sonda de vida `GET /healthz` también se sirve en la misma dirección de enlace. + +--- + +## Avanzado: Autoalojamiento + +El contenedor está disponible en GHCR: + +| Entorno | Imagen | Etiqueta | +|-------------|-------|-----| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet (privada) | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +Ejecútelo manualmente (necesario para opciones que el instalador no puede reenviar, como `WS_BIND=""`): + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**Compilar desde el código fuente:** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +Se recomienda un buffer de recepción del kernel más grande para feeds con ráfagas: + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## Límites y Contrapresión + +| Límite | Valor predeterminado | Comportamiento cuando se excede | +|-------|---------|------------------------| +| Clientes concurrentes (`WS_MAX_CLIENTS`) | 64 | La nueva conexión es rechazada. | +| Suscripciones por cliente (`WS_MAX_SUBS`) | 256 | El `subscribe` es rechazado con un error. | +| Mensajes de control entrantes / cliente / min (`WS_MAX_INBOUND_PER_MIN`) | 600 | El cliente es desconectado. | +| Buffer de difusión (`WS_BROADCAST_CAPACITY`) | 4096 | Un cliente lento **descarta los mensajes más antiguos** (nunca detiene el feed). | + +Dado que cada `quote` y `depth` es estado completo, un consumidor que descarta mensajes bajo contrapresión se auto-recupera con el siguiente mensaje — no se requiere handshake de resincronización. + +--- + +## Solución de Problemas + +### No llegan shreds al puerto local + +- Confirme que su acceso está autorizado para los grupos de shreds `edge-solana-*` onchain. +- Verifique que el túnel esté activo: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Revise los logs en busca de errores de unión: `sudo docker logs -f doublezero-edge-connect` +- Confirme que `DZ_SHRED_FORWARD` apunta a un destino UDP local alcanzable. + +### No hay mensajes de un venue + +- Verifique que el túnel esté activo: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Revise los logs en busca de errores de unión: `sudo docker logs -f doublezero-edge-connect` +- Confirme que su acceso está autorizado para ese venue onchain. +- Acote la ingesta a ese venue con `DZ_FEEDS=` para aislar el problema. + +### El WebSocket conecta pero no llegan cotizaciones + +- Los mensajes `instrument` siempre llegan primero; las cotizaciones siguen una vez que se completa el handshake de datos de referencia. Espere 10–20 segundos después de conectar antes de concluir que faltan datos. +- Verifique `dz_feed_up{venue}` en las métricas — `0` significa que el multicast está en silencio en su host. +- Verifique que las reglas del firewall permitan UDP multicast en la interfaz `doublezero1`. + +### Alto `dz_ws_client_lagged_total` + +Su consumidor está leyendo más lento de lo que el puente está publicando. Aumente el buffer de difusión con `WS_BROADCAST_CAPACITY`, reduzca el tiempo de procesamiento por mensaje, o agregue un hilo de lectura dedicado. + +### El contenedor sale inmediatamente + +- El puente requiere `--network host` y el dispositivo `/dev/net/tun`; un `docker run` sin esos flags fallará. +- Use el one-liner del instalador o el comando `docker run` exacto mostrado en [Autoalojamiento](#avanzado-autoalojamiento). + +### El túnel GRE no se establece + +Consulte [Solución de problemas](troubleshooting.md) y asegúrese de que el protocolo IP 47 esté permitido en su proveedor de nube. En AWS, deshabilite la verificación de origen/destino del ENI para el host. \ No newline at end of file diff --git a/docs/Edge Market Data Connection.fr.md b/docs/Edge Market Data Connection.fr.md new file mode 100644 index 0000000..ea530d4 --- /dev/null +++ b/docs/Edge Market Data Connection.fr.md @@ -0,0 +1,465 @@ +--- +description: Exécutez doublezero-edge-connect pour retransmettre les shreds Solana vers un port UDP local et consommer les données de marché normalisées Edge via un WebSocket local. +--- + +# Connexion Edge + +!!! warning "En me connectant à DoubleZero, j'accepte les [Conditions d'utilisation de DoubleZero](https://doublezero.xyz/terms-protocol). Les données sont destinées à votre usage interne uniquement et ne peuvent pas être retransmises (voir Section 2(e))." + +`doublezero-edge-connect` est un pont qui rejoint le **multicast binaire DoubleZero Edge** et le redistribue localement sous forme de deux flux : + +1. **Retransmission de shreds Solana** — shreds dédupliqués (avec vérification optionnelle de signature) distribués vers une ou plusieurs destinations UDP locales, prêts pour votre validateur ou RPC. +2. **Données de marché normalisées** — flux des venues Edge décodés, avec correction de précision, et redistribués sous forme d'un WebSocket JSON unique sur `ws://host:8081`. + +Les deux fonctionnent depuis le même conteneur et la même installation en une seule ligne. Activez les flux que votre autorisation onchain vous accorde. + +``` + ┌─ UDP datagrams ──▶ validateur / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binaire) (dedup · décodage · normalisation) └─ WebSocket (JSON) ──▶ moteur de trading + ws://host:8081 +``` + +--- + +## Prérequis + +- Hôte **Linux/amd64** avec une adresse IPv4 publique autorisée onchain pour l'environnement cible. +- **Docker** (l'installation en une ligne l'installe s'il est absent). +- **Connectivité GRE** — autorisez le protocole IP 47 chez votre fournisseur cloud ; sur AWS, désactivez la vérification source/dest de l'ENI. +- Un **secret d'accès DoubleZero** : un jeton base64 préfixé `DZ_` ou un chemin vers un fichier keypair, obtenu lors du processus d'[onboarding DoubleZero](setup.md). + +--- + +## Étape 1 : Installation et exécution + +Une seule commande prépare l'hôte et démarre le conteneur pont. Il rejoint le réseau DoubleZero et démarre chaque flux que votre autorisation accorde — retransmission de shreds et/ou WebSocket de données de marché sur `:8081` : + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet (privé)" + + ```bash + # Nécessite un jeton GHCR avec read:packages + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +Ce que fait le script : + +1. Vérifie que l'hôte est Linux/amd64, s'assure que Docker est présent (propose de l'installer). +2. Prépare le noyau de l'hôte pour le tunnel GRE : charge `tun`/`ip_gre`, augmente `net.core.rmem_max`, avertit concernant les règles de pare-feu et du fournisseur cloud. +3. Charge votre secret d'accès (demandé une fois si `DZ_SECRET` n'est pas défini). +4. Lance le conteneur pont (`--network host`, `NET_ADMIN`/`NET_RAW`, `/dev/net/tun`) et exécute `doublezero connect multicast`. + +!!! tip "Installation non interactive" + Définissez `DZ_SECRET=DZ_…` avant le pipe pour une exécution entièrement automatique — aucune invite. + +--- + +## Étape 2 : Configuration + +Toute la configuration se fait via des **variables d'environnement définies avant le pipe**. Il n'y a pas de fichier de configuration. + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### Variables de l'installateur + +| Variable | Défaut | Objectif | +|----------|--------|----------| +| `DZ_SECRET` | *(demandé)* | Jeton base64 préfixé `DZ_` **ou** chemin vers un fichier keypair. Un jeton est injecté dans le conteneur et n'est jamais écrit sur le disque ; un fichier est monté en bind en lecture seule. | +| `DZ_ENV` | selon le script | `mainnet-beta` \| `testnet` \| `devnet`. | +| `DZ_IMAGE` | selon le script | Remplacer l'image du conteneur. | +| `DZ_NAME` | `doublezero-edge-connect` | Nom du conteneur. | +| `DZ_FEEDS` | *(tous)* | Venues séparées par des virgules pour restreindre l'ingestion de données de marché (ex. `VenueA,VenueB`). N'affecte pas la retransmission de shreds Solana. | +| `DZ_ASSUME_YES` | `0` | Ignorer les invites de confirmation (ex. l'invite d'installation de Docker). | +| `DZ_GHCR_TOKEN` | — | **Devnet uniquement** — un jeton GHCR avec `read:packages` (l'image devnet est privée). | +| `DZ_GHCR_USER` | `malbeclabs` | **Devnet uniquement** — nom d'utilisateur GHCR pour la connexion. | + +### Variables du pont + +L'installateur transmet directement **toute variable de pont non vide** au conteneur. Les plus courantes : + +| Variable | Défaut | Objectif | +|----------|--------|----------| +| `DZ_IFACE` | `doublezero1` | Interface réseau d'écoute. | +| `DZ_RECV_BUF` | — | Remplacement du tampon de réception UDP (octets). | +| `METRICS_BIND` | *(vide / désactivé)* | Activer le endpoint Prometheus `/metrics` (ex. `127.0.0.1:9090`). | +| `RUST_LOG` | `info` | Niveau de log (`debug`, `warn`, etc.). | +| `DZ_SHRED_FORWARD` | — | Destination(s) UDP locale(s) pour les shreds retransmis — voir [Retransmission de shreds Solana](#retransmission-de-shreds-solana). | +| `WS_BIND` | `0.0.0.0:8081` | Adresse de liaison du WebSocket de données de marché — voir [WebSocket de données de marché](#websocket-de-donnees-de-marche). | +| `WS_MAX_CLIENTS` | `64` | Nombre maximum de clients WebSocket simultanés. | +| `WS_INPUT_COINS` | *(vide / désactivé)* | Activer le WebSocket public de secours pour les symboles listés (ex. `BTC,ETH`). | + +**Exemples :** + +```bash +# Retransmettre les shreds vers un validateur/RPC local : +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Non interactif, testnet : +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# Restreindre les données de marché à des venues spécifiques, logs verbeux, port WS non standard : +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Activer les métriques et un WS public de secours : +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + Parce que l'installateur ne transmet que les valeurs **non vides**, vous ne pouvez pas passer un remplacement vide (ex. `WS_BIND=""` pour désactiver le sink WebSocket) via la commande en une ligne. Utilisez un `docker run` écrit manuellement pour cela — voir [Auto-hébergement](#avance-auto-hebergement). + +--- + +## Retransmission de shreds Solana + +Le pont rejoint les groupes multicast de shreds `edge-solana-*` et distribue chaque datagramme vers une ou plusieurs destinations UDP locales — alimentant votre validateur ou RPC directement depuis le réseau Edge. Il s'active automatiquement à la découverte lorsque ces groupes sont présents dans votre autorisation. + +```bash +# Par défaut (dédup uniquement, retransmission vers le port local 20000) : +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# Avec vérification de signature : +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| Variable | Défaut | Objectif | +|----------|--------|----------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | Destination(s) pour les shreds retransmis (répétable). | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup` (une copie par shred), `sigverify` (+ vérification ed25519), `none` (tous les datagrammes). | +| `DZ_SHRED_RPC_URL` | — | Endpoint RPC Solana ; requis par le mode `sigverify`. | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | Taille de la fenêtre de déduplication. | + +Voir [Retransmission de shreds](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md) pour le pipeline complet et les mises en garde. + +--- + +## WebSocket de données de marché + +Ouvrez un WebSocket vers `ws://:8081` et lisez les trames JSON. Vous recevez toutes les venues pour lesquelles vous êtes autorisé. Un message `subscribe` optionnel permet de restreindre le flux à des venues et symboles spécifiques. + +Tout moteur compatible WebSocket + JSON peut le consommer avec un adaptateur léger (~50–100 lignes). Le multicast binaire, la séparation deux-ports par venue, et le handshake manifeste/précision restent tous à l'intérieur du pont ; le seul contrat contre lequel un consommateur doit coder est le WebSocket JSON. + +### Cycle de vie de la connexion + +À chaque nouvelle connexion, le pont : + +1. **Rejoue les définitions d'instruments actuelles** — un message `instrument` par symbole connu — afin que le consommateur dispose de la précision avant la première cotation. +2. **Rejoue le dernier snapshot de profondeur** par symbole (si le flux Market-by-Order est actif). +3. **Diffuse** les messages `quote` / `trade` / `midpoint` / `depth` au fur et à mesure de leur arrivée, distribués à tous les consommateurs connectés. + +``` +connexion → instrument (×N) → depth (×M, derniers carnets) → quote → trade → depth → … +``` + +### Types de messages + +Chaque message est un objet JSON identifié par un champ `type` : + +| `type` | Signification | +|--------|---------------| +| `instrument` | Définition d'instrument/précision. | +| `quote` | Mise à jour du meilleur achat/vente (état complet). | +| `trade` | Impression de transaction (dernière vente). | +| `midpoint` | Prix milieu dérivé. | +| `depth` | Snapshot complet de la profondeur du carnet d'ordres. | +| `status` | Transition de santé du flux au niveau de la venue. | + +Les consommateurs **doivent ignorer les valeurs `type` inconnues et les champs inconnus** (compatibilité ascendante). + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +Envoyé à la connexion et à chaque modification des définitions. `price_exponent` et `qty_exponent` donnent le pas de cotation et le pas de taille de la venue sous forme de puissances de dix. + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +Chaque `quote` est un **état complet** — un message perdu se corrige automatiquement à la prochaine cotation, aucune resynchronisation nécessaire. Les quatre horodatages décomposent la latence de bout en bout : + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (réception consommateur) + carnet venue arrivée fil post-décodage transfert WS +``` + +`0` est la valeur sentinelle pour « non disponible » — traitez-la comme manquante, pas comme 1970. + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side` est `"buy"`, `"sell"`, ou `"unknown"`. Les transactions sont des événements ponctuels et ne sont pas rejouées à la reconnexion. + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +Les `bids` sont triés du prix le plus élevé au plus bas ; les `asks` sont triés du prix le plus bas au plus élevé. Chaque `depth` est un **snapshot complet** — remplacez, ne fusionnez pas. + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +Émis en périphérie lorsque le multicast de cotations d'une venue devient silencieux (`state:"down"`) ou récupère (`state:"ok"`). Utilisez-le pour griser une venue dans votre interface. La livraison des cotations n'est pas conditionnée par le statut — le flux se corrige automatiquement à la prochaine cotation. + +### Abonnements + +Par défaut, vous recevez tout. Envoyez un message de contrôle pour restreindre le flux : + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Omettre un champ correspond à toute valeur (`{"symbol":"SOL"}` = SOL sur chaque venue). `venue` est comparé sans tenir compte de la casse. + +**Accusé de réception du serveur :** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Les erreurs retournent `{"channel":"error","error":""}`. + +### Heartbeat et vivacité + +- Le serveur envoie un **Ping WebSocket** toutes les 20 secondes ; les clients conformes répondent automatiquement par un Pong. +- Les clients silencieux pendant 60 secondes sont fermés et supprimés. +- Keepalive au niveau applicatif : `{"method":"ping"}` → `{"channel":"pong"}`. + +### Squelette de consommateur + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # types inconnus : ignorer silencieusement (compatibilité ascendante) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### Sources d'entrée et WebSocket de secours + +Le flux multicast Edge est toujours actif. Un **WebSocket public de secours** optionnel peut combler les lacunes lorsque le flux Edge cale : + +```bash +# Activer le secours pour BTC et ETH : +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Les deux sources sont en concurrence par tick `(venue, symbol, source_ts)` au sein d'un arbitre partagé. En régime permanent, la source Edge gagne (sub-ms contre des dizaines de ms via internet) ; lorsque l'Edge a des lacunes, la copie publique prend le relais. La sortie WebSocket est identique quel que soit la source ayant livré une mise à jour donnée. + +--- + +## Gestion du conteneur + +```bash +# Diffuser les logs +sudo docker logs -f doublezero-edge-connect + +# Vérifier le statut du tunnel +sudo docker exec -it doublezero-edge-connect doublezero status + +# Vérifier les latences des appareils +sudo docker exec -it doublezero-edge-connect doublezero latency + +# Arrêter et supprimer +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "Pas de TLS" + Le pont cible un réseau de confiance/local. Terminez le TLS au niveau d'un reverse proxy si vous exposez le endpoint WebSocket à l'extérieur. + +--- + +## Surveillance (Métriques Prometheus) + +Le endpoint de métriques est **désactivé par défaut**. Activez-le avec `METRICS_BIND` : + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Puis collectez : + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +Métriques clés : + +| Métrique | Ce qu'elle mesure | +|----------|-------------------| +| `dz_feed_up{venue}` | `1` tant que le multicast de cette venue est actif, `0` lorsqu'il est silencieux. | +| `dz_datagrams_received_total{venue}` | Volume d'ingestion par venue. | +| `dz_emit_total{venue,kind}` | Messages diffusés après déduplication, par type. | +| `dz_quotes_dropped_total{venue}` | Cotations obsolètes/dupliquées supprimées. | +| `dz_ws_clients` | Clients WebSocket actuellement connectés. | +| `dz_ws_messages_sent_total{kind}` | Messages transférés aux clients. | +| `dz_ws_client_lagged_total` | Nombre de fois qu'un client lent a été éjecté pour protéger le flux. | + +Une sonde de vivacité `GET /healthz` est également servie sur la même adresse de liaison. + +--- + +## Avancé : Auto-hébergement + +Le conteneur est disponible sur GHCR : + +| Environnement | Image | Tag | +|---------------|-------|-----| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet (privé) | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +Lancez-le manuellement (nécessaire pour les options que l'installateur ne peut pas transmettre, comme `WS_BIND=""`) : + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**Compilation depuis les sources :** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +Un tampon de réception noyau plus grand est recommandé pour les flux en rafales : + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## Limites et contre-pression + +| Limite | Défaut | Comportement en cas de dépassement | +|--------|--------|------------------------------------| +| Clients simultanés (`WS_MAX_CLIENTS`) | 64 | La nouvelle connexion est refusée. | +| Abonnements par client (`WS_MAX_SUBS`) | 256 | Le `subscribe` est refusé avec une erreur. | +| Messages de contrôle entrants / client / min (`WS_MAX_INBOUND_PER_MIN`) | 600 | Le client est déconnecté. | +| Tampon de diffusion (`WS_BROADCAST_CAPACITY`) | 4096 | Un client lent **perd les messages les plus anciens** (ne bloque jamais le flux). | + +Parce que chaque `quote` et `depth` est un état complet, un consommateur qui perd des messages sous contre-pression se corrige automatiquement au prochain message — aucun handshake de resynchronisation requis. + +--- + +## Dépannage + +### Aucun shred n'arrive au port local + +- Confirmez que votre accès est autorisé pour les groupes de shreds `edge-solana-*` onchain. +- Vérifiez que le tunnel est actif : `sudo docker exec -it doublezero-edge-connect doublezero status` +- Vérifiez les erreurs de jointure dans les logs : `sudo docker logs -f doublezero-edge-connect` +- Confirmez que `DZ_SHRED_FORWARD` pointe vers une destination UDP locale accessible. + +### Aucun message d'une venue + +- Vérifiez que le tunnel est actif : `sudo docker exec -it doublezero-edge-connect doublezero status` +- Vérifiez les erreurs de jointure dans les logs : `sudo docker logs -f doublezero-edge-connect` +- Confirmez que votre accès est autorisé pour cette venue onchain. +- Restreignez l'ingestion à cette venue avec `DZ_FEEDS=` pour isoler le problème. + +### Le WebSocket se connecte mais aucune cotation n'arrive + +- Les messages `instrument` arrivent toujours en premier ; les cotations suivent une fois le handshake de données de référence terminé. Attendez 10–20 secondes après la connexion avant de conclure que les données sont manquantes. +- Vérifiez `dz_feed_up{venue}` dans les métriques — `0` signifie que le multicast est silencieux sur votre hôte. +- Vérifiez que les règles de pare-feu autorisent le multicast UDP sur l'interface `doublezero1`. + +### `dz_ws_client_lagged_total` élevé + +Votre consommateur lit plus lentement que le pont ne publie. Augmentez le tampon de diffusion avec `WS_BROADCAST_CAPACITY`, réduisez le temps de traitement par message, ou ajoutez un thread de lecture dédié. + +### Le conteneur se ferme immédiatement + +- Le pont nécessite `--network host` et le périphérique `/dev/net/tun` ; un simple `docker run` sans ces flags échouera. +- Utilisez la commande d'installation en une ligne ou la commande `docker run` exacte indiquée dans [Auto-hébergement](#avance-auto-hebergement). + +### Le tunnel GRE ne s'établit pas + +Consultez [Dépannage](troubleshooting.md) et assurez-vous que le protocole IP 47 est autorisé chez votre fournisseur cloud. Sur AWS, désactivez la vérification source/dest de l'ENI pour l'hôte. \ No newline at end of file diff --git a/docs/Edge Market Data Connection.it.md b/docs/Edge Market Data Connection.it.md new file mode 100644 index 0000000..e26c6ce --- /dev/null +++ b/docs/Edge Market Data Connection.it.md @@ -0,0 +1,465 @@ +--- +description: Esegui doublezero-edge-connect per ri-inoltrare gli shred di Solana verso una porta UDP locale e consumare dati di mercato Edge normalizzati tramite un WebSocket locale. +--- + +# Connessione Edge + +!!! warning "Connettendomi a DoubleZero accetto i [Termini di utilizzo di DoubleZero](https://doublezero.xyz/terms-protocol). I dati sono esclusivamente per uso interno e non possono essere ritrasmessi (vedi Sezione 2(e))." + +`doublezero-edge-connect` è un bridge che si connette al **multicast binario di DoubleZero Edge** e lo ri-serve localmente come due feed: + +1. **Inoltro degli shred Solana** — shred deduplicati (opzionalmente con verifica della firma) distribuiti a una o più destinazioni UDP locali, pronti per il tuo validator o RPC. +2. **Dati di mercato normalizzati** — feed dei venue Edge decodificati, corretti in precisione e ri-serviti come singolo WebSocket JSON su `ws://host:8081`. + +Entrambi vengono eseguiti dallo stesso container e dalla stessa installazione con un singolo comando. Abilita i feed consentiti dalla tua autorizzazione onchain. + +``` + ┌─ UDP datagrams ──▶ validator / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binary) (dedup · decode · normalize) └─ WebSocket (JSON) ──▶ trading engine + ws://host:8081 +``` + +--- + +## Requisiti + +- Host **Linux/amd64** con un indirizzo IPv4 pubblico autorizzato onchain per l'ambiente target. +- **Docker** (il comando one-liner lo installa se mancante). +- **Connettività GRE** — consenti il protocollo IP 47 presso il tuo cloud provider; su AWS disabilita il controllo source/dest dell'ENI. +- Un **secret di accesso DoubleZero**: un token base64 con prefisso `DZ_` oppure un percorso a un file keypair, ottenuto dal processo di [onboarding DoubleZero](setup.md). + +--- + +## Passo 1: Installazione ed Esecuzione + +Un singolo comando prepara l'host e avvia il container bridge. Si connette alla rete DoubleZero e avvia ogni feed consentito dalla tua autorizzazione — inoltro shred e/o il WebSocket per i dati di mercato sulla porta `:8081`: + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet (privata)" + + ```bash + # Richiede un token GHCR con read:packages + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +Cosa fa lo script: + +1. Verifica che l'host sia Linux/amd64, assicura che Docker sia presente (propone l'installazione se assente). +2. Prepara il kernel dell'host per il tunnel GRE: carica `tun`/`ip_gre`, aumenta `net.core.rmem_max`, avvisa riguardo alle regole del firewall e del cloud provider. +3. Carica il tuo secret di accesso (richiesto una sola volta se `DZ_SECRET` non è impostato). +4. Esegue il container bridge (`--network host`, `NET_ADMIN`/`NET_RAW`, `/dev/net/tun`) e lancia `doublezero connect multicast`. + +!!! tip "Installazione non interattiva" + Imposta `DZ_SECRET=DZ_…` prima del pipe per eseguire in modo completamente automatico — nessun prompt. + +--- + +## Passo 2: Configurazione + +Tutta la configurazione avviene tramite **variabili d'ambiente impostate prima del pipe**. Non esiste un file di configurazione. + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### Variabili dell'installer + +| Variabile | Default | Scopo | +|-----------|---------|-------| +| `DZ_SECRET` | *(richiesto interattivamente)* | Token base64 con prefisso `DZ_` **oppure** percorso a un file keypair. Un token viene iniettato nel container e non viene mai scritto su disco; un file viene montato in sola lettura tramite bind mount. | +| `DZ_ENV` | per script | `mainnet-beta` \| `testnet` \| `devnet`. | +| `DZ_IMAGE` | per script | Sovrascrive l'immagine del container. | +| `DZ_NAME` | `doublezero-edge-connect` | Nome del container. | +| `DZ_FEEDS` | *(tutti)* | Venue separati da virgola per restringere l'ingestione dei dati di mercato (es. `VenueA,VenueB`). Non influisce sull'inoltro degli shred Solana. | +| `DZ_ASSUME_YES` | `0` | Salta i prompt di conferma (es. il prompt di installazione Docker). | +| `DZ_GHCR_TOKEN` | — | **Solo Devnet** — un token GHCR con `read:packages` (l'immagine devnet è privata). | +| `DZ_GHCR_USER` | `malbeclabs` | **Solo Devnet** — username GHCR per il login. | + +### Variabili del bridge + +L'installer inoltra **qualsiasi** variabile bridge non vuota direttamente al container. Le più comuni: + +| Variabile | Default | Scopo | +|-----------|---------|-------| +| `DZ_IFACE` | `doublezero1` | Interfaccia di rete su cui mettersi in ascolto. | +| `DZ_RECV_BUF` | — | Override del buffer di ricezione UDP (in byte). | +| `METRICS_BIND` | *(vuoto / disattivato)* | Abilita l'endpoint Prometheus `/metrics` (es. `127.0.0.1:9090`). | +| `RUST_LOG` | `info` | Livello di log (`debug`, `warn`, ecc.). | +| `DZ_SHRED_FORWARD` | — | Destinazione/i UDP locale/i per gli shred inoltrati — vedi [Inoltro Shred Solana](#inoltro-shred-solana). | +| `WS_BIND` | `0.0.0.0:8081` | Indirizzo di bind del WebSocket per i dati di mercato — vedi [WebSocket Dati di Mercato](#websocket-dati-di-mercato). | +| `WS_MAX_CLIENTS` | `64` | Numero massimo di client WebSocket simultanei. | +| `WS_INPUT_COINS` | *(vuoto / disattivato)* | Abilita il backstop WebSocket pubblico per i simboli elencati (es. `BTC,ETH`). | + +**Esempi:** + +```bash +# Inoltra gli shred a un validator/RPC locale: +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Non interattivo, testnet: +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# Restringi i dati di mercato a venue specifici, logging verboso, porta WS non predefinita: +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Abilita le metriche e un backstop WS pubblico: +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + Poiché l'installer inoltra solo valori **non vuoti**, non è possibile passare un override vuoto (es. `WS_BIND=""` per disabilitare il sink WebSocket) tramite il one-liner. Usa un `docker run` scritto manualmente per questo — vedi [Self-hosting](#avanzato-self-hosting). + +--- + +## Inoltro Shred Solana + +Il bridge si unisce ai gruppi multicast `edge-solana-*` per gli shred e inoltra ogni datagramma a una o più destinazioni UDP locali — alimentando direttamente il tuo validator o RPC dalla rete Edge. Si attiva automaticamente al discovery quando quei gruppi sono presenti nella tua autorizzazione. + +```bash +# Default (solo dedup, inoltro alla porta locale 20000): +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# Con verifica della firma: +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| Variabile | Default | Scopo | +|-----------|---------|-------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | Destinazione/i per gli shred inoltrati (ripetibile). | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup` (una copia per shred), `sigverify` (+ verifica ed25519), `none` (tutti i datagrammi). | +| `DZ_SHRED_RPC_URL` | — | Endpoint Solana RPC; richiesto dalla modalità `sigverify`. | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | Dimensione della finestra di deduplicazione. | + +Vedi [Inoltro shred](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md) per la pipeline completa e le avvertenze. + +--- + +## WebSocket Dati di Mercato + +Apri un WebSocket verso `ws://:8081` e leggi frame JSON. Ricevi tutti i venue per cui sei autorizzato. Un messaggio opzionale `subscribe` restringe lo stream a venue e simboli specifici. + +Qualsiasi engine che parli WebSocket + JSON può consumarlo con un adapter leggero (~50–100 righe). Il multicast binario, la suddivisione a due porte per venue e l'handshake manifest/precisione restano tutti all'interno del bridge; l'unico contratto su cui il consumer deve implementare è il WebSocket JSON. + +### Ciclo di vita della connessione + +Ad ogni nuova connessione il bridge: + +1. **Riproduce le definizioni degli strumenti correnti** — un messaggio `instrument` per ogni simbolo noto — così il consumer ha le informazioni di precisione prima della prima quotazione. +2. **Riproduce l'ultimo snapshot di profondità** per simbolo (se il feed Market-by-Order è attivo). +3. **Invia in streaming** messaggi `quote` / `trade` / `midpoint` / `depth` man mano che arrivano, distribuiti a tutti i consumer connessi. + +``` +connect → instrument (×N) → depth (×M, ultimi book) → quote → trade → depth → … +``` + +### Tipi di messaggio + +Ogni messaggio è un oggetto JSON identificato da un campo `type`: + +| `type` | Significato | +|--------|-------------| +| `instrument` | Definizione dello strumento/precisione. | +| `quote` | Aggiornamento top-of-book (stato completo). | +| `trade` | Stampa di trade (ultimo scambio). | +| `midpoint` | Prezzo medio derivato. | +| `depth` | Snapshot completo della profondità dell'order book. | +| `status` | Transizione dello stato di salute del feed a livello di venue. | + +I consumer **devono ignorare valori `type` sconosciuti e campi sconosciuti** (compatibilità in avanti). + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +Inviato alla connessione e ogni volta che le definizioni cambiano. `price_exponent` e `qty_exponent` indicano il tick size e lo step di dimensione del venue come potenze di dieci. + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +Ogni `quote` è **stato completo** — un messaggio perso si auto-ripristina con la quotazione successiva, nessuna risincronizzazione necessaria. I quattro timestamp decompongono la latenza end-to-end: + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (ricezione consumer) + book del venue arrivo sul wire post-decode hand-off WS +``` + +`0` è il valore sentinella per "non disponibile" — trattarlo come mancante, non come 1970. + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side` è `"buy"`, `"sell"` oppure `"unknown"`. I trade sono eventi puntuali e non vengono riprodotti alla riconnessione. + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +I `bids` sono ordinati dal prezzo più alto al più basso; gli `asks` dal prezzo più basso al più alto. Ogni `depth` è uno **snapshot completo** — sostituire, non unire. + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +Emesso all'edge quando il multicast delle quotazioni di un venue diventa silenzioso (`state:"down"`) o si riprende (`state:"ok"`). Usalo per disattivare visualmente un venue nella tua UI. La consegna delle quotazioni non è vincolata allo stato — il feed si auto-ripristina con la quotazione successiva. + +### Sottoscrizioni + +Per impostazione predefinita ricevi tutto. Invia un messaggio di controllo per restringere lo stream: + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Omettere un campo corrisponde a qualsiasi valore (`{"symbol":"SOL"}` = SOL su ogni venue). `venue` viene confrontato senza distinzione tra maiuscole e minuscole. + +**Risposta di conferma del server:** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Gli errori restituiscono `{"channel":"error","error":""}`. + +### Heartbeat e liveness + +- Il server invia un **WebSocket Ping** ogni 20 secondi; i client conformi rispondono automaticamente con Pong. +- I client silenziosi per 60 secondi vengono chiusi e rimossi. +- Keepalive a livello applicativo: `{"method":"ping"}` → `{"channel":"pong"}`. + +### Scheletro del consumer + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # tipi sconosciuti: ignorare silenziosamente (compatibilità in avanti) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### Sorgenti di input e backstop WebSocket + +Il feed multicast Edge è sempre attivo. Un **backstop WebSocket pubblico** opzionale può colmare le lacune quando il feed Edge si blocca: + +```bash +# Abilita il backstop per BTC ed ETH: +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Le due sorgenti competono per tick `(venue, symbol, source_ts)` all'interno di un arbitro condiviso. In condizioni normali la sorgente Edge vince (sub-ms vs. decine di ms su internet); quando l'Edge presenta lacune, la copia pubblica interviene. L'output WebSocket è identico indipendentemente dalla sorgente che ha consegnato un determinato aggiornamento. + +--- + +## Gestione del Container + +```bash +# Streaming dei log +sudo docker logs -f doublezero-edge-connect + +# Verifica stato del tunnel +sudo docker exec -it doublezero-edge-connect doublezero status + +# Verifica latenze del dispositivo +sudo docker exec -it doublezero-edge-connect doublezero latency + +# Stop e rimozione +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "Nessun TLS" + Il bridge è progettato per una rete trusted/locale. Termina TLS con un reverse proxy se esponi l'endpoint WebSocket esternamente. + +--- + +## Monitoraggio (Metriche Prometheus) + +L'endpoint delle metriche è **disattivato per impostazione predefinita**. Abilitalo con `METRICS_BIND`: + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Poi esegui lo scraping: + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +Metriche principali: + +| Metrica | Cosa monitora | +|---------|---------------| +| `dz_feed_up{venue}` | `1` mentre il multicast del venue è attivo, `0` mentre è silenzioso. | +| `dz_datagrams_received_total{venue}` | Volume di ingestione per venue. | +| `dz_emit_total{venue,kind}` | Messaggi trasmessi dopo la deduplicazione, per tipo. | +| `dz_quotes_dropped_total{venue}` | Quotazioni obsolete/duplicate soppresse. | +| `dz_ws_clients` | Client WebSocket attualmente connessi. | +| `dz_ws_messages_sent_total{kind}` | Messaggi inoltrati ai client. | +| `dz_ws_client_lagged_total` | Volte in cui un client lento è stato disconnesso per proteggere il feed. | + +Una sonda di liveness `GET /healthz` è anch'essa servita sullo stesso indirizzo di bind. + +--- + +## Avanzato: Self-hosting + +Il container è disponibile su GHCR: + +| Ambiente | Immagine | Tag | +|----------|----------|-----| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet (privata) | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +Eseguilo manualmente (necessario per opzioni che l'installer non può inoltrare, come `WS_BIND=""`): + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**Compilazione dal sorgente:** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +Un buffer di ricezione del kernel più grande è consigliato per feed a raffica: + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## Limiti e Backpressure + +| Limite | Default | Comportamento al superamento | +|--------|---------|------------------------------| +| Client simultanei (`WS_MAX_CLIENTS`) | 64 | La nuova connessione viene rifiutata. | +| Sottoscrizioni per client (`WS_MAX_SUBS`) | 256 | La `subscribe` viene rifiutata con un errore. | +| Messaggi di controllo in ingresso / client / min (`WS_MAX_INBOUND_PER_MIN`) | 600 | Il client viene disconnesso. | +| Buffer di broadcast (`WS_BROADCAST_CAPACITY`) | 4096 | Un client lento **perde i messaggi più vecchi** (non blocca mai il feed). | + +Poiché ogni `quote` e `depth` è stato completo, un consumer che perde messaggi sotto backpressure si auto-ripristina con il messaggio successivo — nessun handshake di risincronizzazione necessario. + +--- + +## Risoluzione dei Problemi + +### Nessuno shred in arrivo sulla porta locale + +- Conferma che il tuo accesso sia autorizzato per i gruppi shred `edge-solana-*` onchain. +- Verifica che il tunnel sia attivo: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Controlla i log per errori di join: `sudo docker logs -f doublezero-edge-connect` +- Conferma che `DZ_SHRED_FORWARD` punti a una destinazione UDP locale raggiungibile. + +### Nessun messaggio da un venue + +- Verifica che il tunnel sia attivo: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Controlla i log per errori di join: `sudo docker logs -f doublezero-edge-connect` +- Conferma che il tuo accesso sia autorizzato per quel venue onchain. +- Restringi l'ingestione a quel venue con `DZ_FEEDS=` per isolare il problema. + +### Il WebSocket si connette ma non arrivano quotazioni + +- I messaggi `instrument` arrivano sempre per primi; le quotazioni seguono una volta completato l'handshake dei dati di riferimento. Attendi 10–20 secondi dopo la connessione prima di concludere che i dati mancano. +- Controlla `dz_feed_up{venue}` nelle metriche — `0` significa che il multicast è silenzioso sul tuo host. +- Verifica che le regole del firewall consentano UDP multicast sull'interfaccia `doublezero1`. + +### `dz_ws_client_lagged_total` elevato + +Il tuo consumer sta leggendo più lentamente di quanto il bridge stia pubblicando. Aumenta il buffer di broadcast con `WS_BROADCAST_CAPACITY`, riduci il tempo di elaborazione per messaggio, oppure aggiungi un thread di lettura dedicato. + +### Il container termina immediatamente + +- Il bridge richiede `--network host` e il device `/dev/net/tun`; un semplice `docker run` senza questi flag fallirà. +- Usa il one-liner dell'installer o l'esatto comando `docker run` mostrato in [Self-hosting](#avanzato-self-hosting). + +### Il tunnel GRE non si stabilisce + +Fai riferimento a [Risoluzione dei problemi](troubleshooting.md) e assicurati che il protocollo IP 47 sia consentito presso il tuo cloud provider. Su AWS, disabilita il controllo source/dest dell'ENI per l'host. \ No newline at end of file diff --git a/docs/Edge Market Data Connection.ja.md b/docs/Edge Market Data Connection.ja.md new file mode 100644 index 0000000..ed65f7e --- /dev/null +++ b/docs/Edge Market Data Connection.ja.md @@ -0,0 +1,463 @@ +--- +description: doublezero-edge-connect を実行して Solana shred をローカル UDP ポートに再転送し、正規化された Edge マーケットデータをローカル WebSocket 経由で受信します。 +--- + +# Edge 接続 + +!!! warning "DoubleZero に接続することで、[DoubleZero 利用規約](https://doublezero.xyz/terms-protocol)に同意したものとみなされます。データは内部目的でのみ使用可能であり、再送信は禁止されています（セクション 2(e) を参照）。" + +`doublezero-edge-connect` は **DoubleZero Edge バイナリマルチキャスト** に参加し、以下の 2 つのフィードとしてローカルに再配信するブリッジです： + +1. **Solana shred 転送** — 重複排除済み（オプションで署名検証済み）の shred を 1 つ以上のローカル UDP 宛先にファンアウトし、バリデーターまたは RPC に直接配信します。 +2. **正規化マーケットデータ** — Edge 取引所フィードをデコード・精度補正し、`ws://host:8081` 上の単一 JSON WebSocket として再配信します。 + +どちらも同じコンテナと同じワンライナーインストールで動作します。オンチェーン認可で許可されたフィードを有効にしてください。 + +``` + ┌─ UDP datagrams ──▶ validator / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binary) (dedup · decode · normalize) └─ WebSocket (JSON) ──▶ trading engine + ws://host:8081 +``` + +--- + +## 要件 + +- ターゲット環境向けにオンチェーンで認可されたパブリック IPv4 アドレスを持つ **Linux/amd64** ホスト。 +- **Docker**（ワンライナーが未インストールの場合は自動インストールします）。 +- **GRE 接続** — クラウドプロバイダーで IP プロトコル 47 を許可してください。AWS では ENI のソース/宛先チェックを無効にしてください。 +- **DoubleZero アクセスシークレット**: `DZ_` プレフィックス付き base64 トークン、またはキーペアファイルのパス。[DoubleZero オンボーディング](setup.md)プロセスから取得します。 + +--- + +## ステップ 1: インストールと実行 + +1 つのコマンドでホストを準備し、ブリッジコンテナを起動します。DoubleZero ネットワークに参加し、認可で許可されたすべてのフィード（shred 転送および/または `:8081` のマーケットデータ WebSocket）を開始します： + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet (プライベート)" + + ```bash + # read:packages 権限を持つ GHCR トークンが必要です + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +スクリプトが実行する内容： + +1. ホストが Linux/amd64 であることを確認し、Docker の存在を確認します（未インストールの場合はインストールを提案）。 +2. GRE トンネル用にホストカーネルを準備します：`tun`/`ip_gre` をロードし、`net.core.rmem_max` を引き上げ、ファイアウォールとクラウドプロバイダーのルールについて警告します。 +3. アクセスシークレットを読み込みます（`DZ_SECRET` が未設定の場合は一度だけプロンプト表示）。 +4. ブリッジコンテナ（`--network host`、`NET_ADMIN`/`NET_RAW`、`/dev/net/tun`）を実行し、`doublezero connect multicast` を実行します。 + +!!! tip "非対話式インストール" + パイプの前に `DZ_SECRET=DZ_…` を設定すると、プロンプトなしで完全に無人実行できます。 + +--- + +## ステップ 2: 設定 + +すべての設定は**パイプの前に設定する環境変数**で行います。設定ファイルはありません。 + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### インストーラー変数 + +| 変数 | デフォルト | 用途 | +|----------|---------|---------| +| `DZ_SECRET` | *（プロンプト表示）* | `DZ_` プレフィックス付き base64 トークン**または**キーペアファイルのパス。トークンはコンテナに注入されディスクには書き込まれません。ファイルは読み取り専用でバインドマウントされます。 | +| `DZ_ENV` | スクリプトごと | `mainnet-beta` \| `testnet` \| `devnet`。 | +| `DZ_IMAGE` | スクリプトごと | コンテナイメージのオーバーライド。 | +| `DZ_NAME` | `doublezero-edge-connect` | コンテナ名。 | +| `DZ_FEEDS` | *（すべて）* | マーケットデータ取り込みを絞り込むカンマ区切りの取引所名（例: `VenueA,VenueB`）。Solana shred 転送には影響しません。 | +| `DZ_ASSUME_YES` | `0` | 確認プロンプトをスキップします（例: Docker インストールプロンプト）。 | +| `DZ_GHCR_TOKEN` | — | **Devnet 限定** — `read:packages` 権限を持つ GHCR トークン（devnet イメージはプライベートです）。 | +| `DZ_GHCR_USER` | `malbeclabs` | **Devnet 限定** — ログイン用の GHCR ユーザー名。 | + +### ブリッジ変数 + +インストーラーは**空でない**ブリッジ変数をすべてコンテナにそのまま転送します。主要なものは以下の通りです： + +| 変数 | デフォルト | 用途 | +|----------|---------|---------| +| `DZ_IFACE` | `doublezero1` | リッスンするネットワークインターフェース。 | +| `DZ_RECV_BUF` | — | UDP 受信バッファのオーバーライド（バイト）。 | +| `METRICS_BIND` | *（空 / 無効）* | Prometheus `/metrics` エンドポイントを有効にします（例: `127.0.0.1:9090`）。 | +| `RUST_LOG` | `info` | ログレベル（`debug`、`warn` など）。 | +| `DZ_SHRED_FORWARD` | — | 転送 shred のローカル UDP 宛先 — [Solana Shred 転送](#solana-shred-転送)を参照。 | +| `WS_BIND` | `0.0.0.0:8081` | マーケットデータ WebSocket のバインドアドレス — [マーケットデータ WebSocket](#マーケットデータ-websocket) を参照。 | +| `WS_MAX_CLIENTS` | `64` | WebSocket の最大同時接続クライアント数。 | +| `WS_INPUT_COINS` | *（空 / 無効）* | 指定シンボルのパブリック WebSocket バックストップを有効にします（例: `BTC,ETH`）。 | + +**例：** + +```bash +# ローカルのバリデーター/RPC に shred を転送： +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# 非対話式、testnet： +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# 特定の取引所に絞り込み、詳細ログ、デフォルト以外の WS ポート： +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# メトリクスとパブリック WS バックストップを有効化： +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + インストーラーは**空でない**値のみを転送するため、ワンライナーで空のオーバーライド（例: WebSocket シンクを無効にする `WS_BIND=""`）を渡すことはできません。その場合は手動の `docker run` を使用してください — [セルフホスティング](#上級-セルフホスティング)を参照。 + +--- + +## Solana Shred 転送 + +ブリッジは `edge-solana-*` shred マルチキャストグループに参加し、各データグラムを 1 つ以上のローカル UDP 宛先にファンアウトします。Edge ネットワークからバリデーターまたは RPC に直接フィードします。認可にこれらのグループが含まれている場合、検出時に自動的にアクティブになります。 + +```bash +# デフォルト（重複排除のみ、ローカルポート 20000 に転送）： +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# 署名検証付き： +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| 変数 | デフォルト | 用途 | +|----------|---------|---------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | 転送 shred の宛先（繰り返し指定可能）。 | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup`（shred ごとに 1 コピー）、`sigverify`（+ ed25519 検証）、`none`（全データグラム）。 | +| `DZ_SHRED_RPC_URL` | — | Solana RPC エンドポイント。`sigverify` モードで必須。 | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | 重複排除ウィンドウのサイズ。 | + +完全なパイプラインと注意事項については [Shred forwarding](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md) を参照してください。 + +--- + +## マーケットデータ WebSocket + +`ws://:8081` に WebSocket 接続を開き、JSON フレームを読み取ります。認可された取引所のデータをすべて受信します。オプションの `subscribe` メッセージでストリームを特定の取引所やシンボルに絞り込めます。 + +WebSocket + JSON に対応する任意のエンジンが、薄い（〜50–100 行の）アダプターで消費できます。バイナリマルチキャスト、取引所ごとの 2 ポート分割、マニフェスト/精度ハンドシェイクはすべてブリッジ内に留まります。コンシューマーがコーディングする唯一の契約は WebSocket JSON です。 + +### 接続ライフサイクル + +新しい接続ごとにブリッジは以下を行います： + +1. **現在のインストゥルメント定義をリプレイ** — 既知のシンボルごとに 1 つの `instrument` メッセージ — コンシューマーが最初の気配値の前に精度情報を得られるようにします。 +2. **シンボルごとの最新板情報スナップショットをリプレイ**（Market-by-Order フィードがアクティブな場合）。 +3. `quote` / `trade` / `midpoint` / `depth` メッセージが到着次第、接続済みのすべてのコンシューマーに**ストリーミング**します。 + +``` +connect → instrument (×N) → depth (×M, latest books) → quote → trade → depth → … +``` + +### メッセージタイプ + +すべてのメッセージは `type` フィールドでタグ付けされた JSON オブジェクトです： + +| `type` | 意味 | +|--------|---------| +| `instrument` | インストゥルメント/精度定義。 | +| `quote` | 最良気配値の更新（フルステート）。 | +| `trade` | 約定プリント（直近の取引）。 | +| `midpoint` | 算出されたミッド価格。 | +| `depth` | フルオーダーブック板情報スナップショット。 | +| `status` | 取引所レベルのフィードヘルス遷移。 | + +コンシューマーは**未知の `type` 値および未知のフィールドを無視しなければなりません**（前方互換性）。 + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +接続時および定義変更時に送信されます。`price_exponent` と `qty_exponent` は取引所のティックサイズとサイズステップを 10 のべき乗で表します。 + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +すべての `quote` は**フルステート**です — メッセージが欠落しても次の quote で自動復旧し、再同期は不要です。4 つのタイムスタンプでエンドツーエンドのレイテンシを分解できます： + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (consumer recv) + venue book wire arrival post-decode WS hand-off +``` + +`0` は「利用不可」のセンチネル値です — 1970 年ではなく欠損として扱ってください。 + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side` は `"buy"`、`"sell"`、または `"unknown"` です。約定はポイントインタイムのイベントであり、再接続時にはリプレイされません。 + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`bids` は価格の高い順にソート、`asks` は価格の低い順にソートされています。各 `depth` は**フルスナップショット**です — マージではなく置換してください。 + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +取引所の quote マルチキャストが沈黙（`state:"down"`）または回復（`state:"ok"`）した際に Edge 上で発行されます。UI で取引所をグレーアウト表示するのに使用してください。Quote の配信は status に依存しません — フィードは次の quote で自動復旧します。 + +### サブスクリプション + +デフォルトではすべてを受信します。ストリームを絞り込むにはコントロールメッセージを送信します： + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +フィールドを省略すると任意の値にマッチします（`{"symbol":"SOL"}` = すべての取引所の SOL）。`venue` は大文字小文字を区別しません。 + +**サーバー応答：** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +エラーは `{"channel":"error","error":""}` を返します。 + +### ハートビートと生存確認 + +- サーバーは 20 秒ごとに **WebSocket Ping** を送信します。準拠クライアントは自動で Pong を返します。 +- 60 秒間無応答のクライアントは切断・解放されます。 +- アプリケーションレベルのキープアライブ：`{"method":"ping"}` → `{"channel":"pong"}`。 + +### コンシューマーのスケルトン + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # unknown types: silently ignore (forward compatibility) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### 入力ソースと WebSocket バックストップ + +Edge マルチキャストフィードは常時オンです。オプションの**パブリック WebSocket バックストップ**は、Edge フィードが停止した際にギャップを補います： + +```bash +# BTC と ETH のバックストップを有効化： +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +2 つのソースは共有アービターの中で `(venue, symbol, source_ts)` ティックごとに競争します。定常状態では Edge ソースが勝利します（インターネット経由の数十 ms に対してサブ ms）。Edge にギャップが生じた場合、パブリックコピーが補完します。どのソースが更新を配信したかに関わらず、WebSocket 出力は同一です。 + +--- + +## コンテナの管理 + +```bash +# ログをストリーミング +sudo docker logs -f doublezero-edge-connect + +# トンネル状態を確認 +sudo docker exec -it doublezero-edge-connect doublezero status + +# デバイスレイテンシを確認 +sudo docker exec -it doublezero-edge-connect doublezero latency + +# 停止と削除 +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "TLS なし" + ブリッジは信頼済み/ローカルネットワークを対象としています。WebSocket エンドポイントを外部に公開する場合は、リバースプロキシで TLS を終端してください。 + +--- + +## モニタリング（Prometheus メトリクス） + +メトリクスエンドポイントは**デフォルトで無効**です。`METRICS_BIND` で有効にします： + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +スクレイプ： + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +主要メトリクス： + +| メトリクス | 追跡内容 | +|--------|---------------| +| `dz_feed_up{venue}` | 取引所のマルチキャストがライブの間 `1`、沈黙中は `0`。 | +| `dz_datagrams_received_total{venue}` | 取引所ごとの取り込み量。 | +| `dz_emit_total{venue,kind}` | 重複排除後にブロードキャストされたメッセージ（タイプ別）。 | +| `dz_quotes_dropped_total{venue}` | 抑制された古い/重複 quote。 | +| `dz_ws_clients` | 現在接続中の WebSocket クライアント数。 | +| `dz_ws_messages_sent_total{kind}` | クライアントに転送されたメッセージ。 | +| `dz_ws_client_lagged_total` | フィードを保護するために遅延クライアントが切断された回数。 | + +`GET /healthz` 生存確認プローブも同じバインドアドレスで提供されます。 + +--- + +## 上級: セルフホスティング + +コンテナは GHCR で利用可能です： + +| 環境 | イメージ | タグ | +|-------------|-------|-----| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet (プライベート) | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +手動で実行します（インストーラーが転送できないオプション、例えば `WS_BIND=""` が必要な場合）： + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**ソースからビルド：** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +バースト性の高いフィードには、より大きなカーネル受信バッファを推奨します： + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## 制限とバックプレッシャー + +| 制限 | デフォルト | 超過時の動作 | +|-------|---------|------------------------| +| 同時接続クライアント数 (`WS_MAX_CLIENTS`) | 64 | 新しい接続が拒否されます。 | +| クライアントごとのサブスクリプション数 (`WS_MAX_SUBS`) | 256 | `subscribe` がエラーで拒否されます。 | +| クライアントごとの受信制御メッセージ数/分 (`WS_MAX_INBOUND_PER_MIN`) | 600 | クライアントが切断されます。 | +| ブロードキャストバッファ (`WS_BROADCAST_CAPACITY`) | 4096 | 遅延クライアントは**最も古いメッセージをドロップ**します（フィードを停止させることはありません）。 | + +すべての `quote` と `depth` はフルステートであるため、バックプレッシャーでメッセージをドロップしたコンシューマーは次のメッセージで自動復旧します — 再同期ハンドシェイクは不要です。 + +--- + +## トラブルシューティング + +### ローカルポートに shred が到着しない + +- オンチェーンで `edge-solana-*` shred グループへのアクセスが認可されていることを確認してください。 +- トンネルが稼働中か確認してください：`sudo docker exec -it doublezero-edge-connect doublezero status` +- 参加エラーのログを確認してください：`sudo docker logs -f doublezero-edge-connect` +- `DZ_SHRED_FORWARD` が到達可能なローカル UDP 宛先を指していることを確認してください。 + +### 取引所からメッセージが来ない + +- トンネルが稼働中か確認してください：`sudo docker exec -it doublezero-edge-connect doublezero status` +- 参加エラーのログを確認してください：`sudo docker logs -f doublezero-edge-connect` +- オンチェーンでその取引所へのアクセスが認可されていることを確認してください。 +- 問題を切り分けるため、`DZ_FEEDS=` でその取引所に取り込みを絞り込んでください。 + +### WebSocket は接続するが quote が到着しない + +- `instrument` メッセージが常に最初に到着します。リファレンスデータのハンドシェイクが完了すると quote が続きます。データが欠落していると判断する前に、接続後 10〜20 秒待ってください。 +- メトリクスの `dz_feed_up{venue}` を確認してください — `0` はホスト上でマルチキャストが沈黙していることを意味します。 +- ファイアウォールルールが `doublezero1` インターフェース上のマルチキャスト UDP を許可していることを確認してください。 + +### `dz_ws_client_lagged_total` が高い + +コンシューマーの読み取り速度がブリッジの配信速度より遅くなっています。`WS_BROADCAST_CAPACITY` でブロードキャストバッファを増やすか、メッセージごとの処理時間を短縮するか、専用のリーダースレッドを追加してください。 + +### コンテナが即座に終了する + +- ブリッジには `--network host` と `/dev/net/tun` デバイスが必要です。これらのフラグなしの通常の `docker run` は失敗します。 +- インストーラーのワンライナーまたは[セルフホスティング](#上級-セルフホスティング)に記載されている正確な `docker run` コマンドを使用してください。 + +### GRE トンネルが確立されない \ No newline at end of file diff --git a/docs/Edge Market Data Connection.ko.md b/docs/Edge Market Data Connection.ko.md new file mode 100644 index 0000000..9c69325 --- /dev/null +++ b/docs/Edge Market Data Connection.ko.md @@ -0,0 +1,436 @@ +--- +description: doublezero-edge-connect를 실행하여 Solana 시레드(shred)를 로컬 UDP 포트로 재전달하고 정규화된 Edge 시장 데이터를 로컬 WebSocket을 통해 소비합니다. +--- + +# Edge 연결 + +!!! warning "DoubleZero에 연결함으로써 [DoubleZero 이용약관](https://doublezero.xyz/terms-protocol)에 동의합니다. 데이터는 내부 목적으로만 사용할 수 있으며 재전송할 수 없습니다(제2조(e) 참조)." + +`doublezero-edge-connect`는 **DoubleZero Edge 바이너리 멀티캐스트**에 참여하여 이를 로컬에서 두 가지 피드로 제공하는 브리지입니다: + +1. **Solana 시레드 포워딩** — 중복 제거된(선택적으로 서명 검증된) 시레드를 하나 이상의 로컬 UDP 대상으로 팬아웃하여 밸리데이터 또는 RPC에서 바로 사용할 수 있습니다. +2. **정규화된 시장 데이터** — Edge 거래소 피드를 디코딩하고 정밀도를 보정한 후 `ws://host:8081`에서 단일 JSON WebSocket으로 제공합니다. + +두 피드 모두 동일한 컨테이너와 동일한 원라인 설치로 실행됩니다. 온체인 인가가 부여한 피드를 활성화하세요. + +``` + ┌─ UDP datagrams ──▶ validator / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binary) (dedup · decode · normalize) └─ WebSocket (JSON) ──▶ trading engine + ws://host:8081 +``` + +--- + +## 요구 사항 + +- 대상 환경에 대해 온체인으로 인가된 공인 IPv4 주소를 가진 **Linux/amd64** 호스트. +- **Docker** (원라이너가 없는 경우 설치합니다). +- **GRE 연결** — 클라우드 제공자에서 IP 프로토콜 47을 허용하세요. AWS에서는 ENI 소스/목적지 확인을 비활성화하세요. +- **DoubleZero 액세스 시크릿**: `DZ_` 접두사가 붙은 base64 토큰 또는 키페어 파일 경로로, [DoubleZero 온보딩](setup.md) 과정에서 발급받습니다. + +--- + +## 1단계: 설치 및 실행 + +하나의 명령으로 호스트를 준비하고 브리지 컨테이너를 시작합니다. DoubleZero 네트워크에 참여하고 인가가 부여한 모든 피드(시레드 포워딩 및/또는 `:8081`의 시장 데이터 WebSocket)를 시작합니다: + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet (비공개)" + + ```bash + # read:packages 권한이 있는 GHCR 토큰이 필요합니다 + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +스크립트가 수행하는 작업: + +1. 호스트가 Linux/amd64인지 확인하고, Docker가 있는지 확인합니다(없으면 설치를 제안합니다). +2. GRE 터널을 위해 호스트 커널을 준비합니다: `tun`/`ip_gre` 로드, `net.core.rmem_max` 증가, 방화벽 및 클라우드 제공자 규칙에 대한 경고. +3. 액세스 시크릿을 로드합니다(`DZ_SECRET`이 설정되지 않은 경우 한 번 프롬프트됩니다). +4. 브리지 컨테이너를 실행하고(`--network host`, `NET_ADMIN`/`NET_RAW`, `/dev/net/tun`) `doublezero connect multicast`를 실행합니다. + +!!! tip "비대화형 설치" + 파이프 앞에 `DZ_SECRET=DZ_…`를 설정하면 프롬프트 없이 완전히 무인으로 실행됩니다. + +--- + +## 2단계: 구성 + +모든 구성은 **파이프 앞에 설정하는 환경 변수**를 통해 이루어집니다. 구성 파일은 없습니다. + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### 설치 프로그램 변수 + +| 변수 | 기본값 | 용도 | +|----------|---------|---------| +| `DZ_SECRET` | *(프롬프트)* | `DZ_` 접두사가 붙은 base64 토큰 **또는** 키페어 파일 경로. 토큰은 컨테이너에 주입되며 디스크에 기록되지 않습니다. 파일은 읽기 전용으로 바인드 마운트됩니다. | +| `DZ_ENV` | 스크립트별 | `mainnet-beta` \| `testnet` \| `devnet`. | +| `DZ_IMAGE` | 스크립트별 | 컨테이너 이미지를 재정의합니다. | +| `DZ_NAME` | `doublezero-edge-connect` | 컨테이너 이름. | +| `DZ_FEEDS` | *(전체)* | 시장 데이터 수집을 좁히는 쉼표로 구분된 거래소 목록 (예: `VenueA,VenueB`). Solana 시레드 포워딩에는 영향을 미치지 않습니다. | +| `DZ_ASSUME_YES` | `0` | 확인 프롬프트를 건너뜁니다 (예: Docker 설치 프롬프트). | +| `DZ_GHCR_TOKEN` | — | **Devnet 전용** — `read:packages` 권한이 있는 GHCR 토큰(devnet 이미지는 비공개). | +| `DZ_GHCR_USER` | `malbeclabs` | **Devnet 전용** — 로그인을 위한 GHCR 사용자명. | + +### 브리지 변수 + +설치 프로그램은 **비어 있지 않은** 모든 브리지 변수를 컨테이너로 직접 전달합니다. 주요 변수: + +| 변수 | 기본값 | 용도 | +|----------|---------|---------| +| `DZ_IFACE` | `doublezero1` | 리스닝할 네트워크 인터페이스. | +| `DZ_RECV_BUF` | — | UDP 수신 버퍼 재정의 (바이트). | +| `METRICS_BIND` | *(비어 있음 / 비활성)* | Prometheus `/metrics` 엔드포인트 활성화 (예: `127.0.0.1:9090`). | +| `RUST_LOG` | `info` | 로그 레벨 (`debug`, `warn` 등). | +| `DZ_SHRED_FORWARD` | — | 포워딩된 시레드의 로컬 UDP 대상 — [Solana 시레드 포워딩](#solana-시레드-포워딩) 참조. | +| `WS_BIND` | `0.0.0.0:8081` | 시장 데이터 WebSocket 바인드 주소 — [시장 데이터 WebSocket](#시장-데이터-websocket) 참조. | +| `WS_MAX_CLIENTS` | `64` | 최대 동시 WebSocket 클라이언트 수. | +| `WS_INPUT_COINS` | *(비어 있음 / 비활성)* | 나열된 심볼에 대한 공개 WebSocket 백스톱 활성화 (예: `BTC,ETH`). | + +**예시:** + +```bash +# 로컬 밸리데이터/RPC로 시레드 포워딩: +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# 비대화형, testnet: +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# 특정 거래소로 시장 데이터 좁히기, 상세 로깅, 비기본 WS 포트: +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# 메트릭 및 공개 WS 백스톱 활성화: +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + 설치 프로그램은 **비어 있지 않은** 값만 전달하므로, 원라이너를 통해 빈 재정의(예: WebSocket 싱크를 비활성화하기 위한 `WS_BIND=""`)를 전달할 수 없습니다. 이 경우 수동으로 `docker run`을 작성하세요 — [자체 호스팅](#고급-자체-호스팅) 참조. + +--- + +## Solana 시레드 포워딩 + +브리지는 `edge-solana-*` 시레드 멀티캐스트 그룹에 참여하고 각 데이터그램을 하나 이상의 로컬 UDP 대상으로 팬아웃합니다 — Edge 네트워크에서 직접 밸리데이터 또는 RPC에 공급합니다. 인가에 해당 그룹이 포함되어 있으면 검색 시 자동으로 활성화됩니다. + +```bash +# 기본값 (중복 제거만, 로컬 포트 20000으로 포워딩): +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# 서명 검증 포함: +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| 변수 | 기본값 | 용도 | +|----------|---------|---------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | 포워딩된 시레드의 대상 (반복 가능). | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup` (시레드당 한 복사본), `sigverify` (+ ed25519 검증), `none` (모든 데이터그램). | +| `DZ_SHRED_RPC_URL` | — | Solana RPC 엔드포인트; `sigverify` 모드에 필요합니다. | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | 중복 제거 윈도우 크기. | + +전체 파이프라인과 주의사항은 [시레드 포워딩](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md)을 참조하세요. + +--- + +## 시장 데이터 WebSocket + +`ws://:8081`에 WebSocket을 연결하고 JSON 프레임을 읽으세요. 인가된 모든 거래소의 데이터를 수신합니다. 선택적 `subscribe` 메시지를 통해 스트림을 특정 거래소와 심볼로 좁힐 수 있습니다. + +WebSocket + JSON을 지원하는 모든 엔진은 간단한 (~50–100줄) 어댑터로 이를 소비할 수 있습니다. 바이너리 멀티캐스트, 거래소별 2포트 분할, 매니페스트/정밀도 핸드셰이크는 모두 브리지 내부에서 처리됩니다. 소비자가 코딩해야 할 유일한 계약은 WebSocket JSON입니다. + +### 연결 라이프사이클 + +새 연결마다 브리지는: + +1. **현재 상품 정의를 재전송합니다** — 알려진 심볼당 하나의 `instrument` 메시지 — 소비자가 첫 번째 호가 전에 정밀도를 파악할 수 있도록 합니다. +2. **심볼당 최신 호가창 스냅샷을 재전송합니다** (Market-by-Order 피드가 활성인 경우). +3. `quote` / `trade` / `midpoint` / `depth` 메시지를 도착하는 대로 **스트리밍**하며 연결된 모든 소비자에게 팬아웃합니다. + +``` +connect → instrument (×N) → depth (×M, latest books) → quote → trade → depth → … +``` + +### 메시지 유형 + +모든 메시지는 `type` 필드로 태그된 JSON 객체입니다: + +| `type` | 의미 | +|--------|---------| +| `instrument` | 상품/정밀도 정의. | +| `quote` | 최우선 호가 업데이트 (전체 상태). | +| `trade` | 체결 내역 (최근 거래). | +| `midpoint` | 파생 중간 가격. | +| `depth` | 전체 호가창 깊이 스냅샷. | +| `status` | 거래소 수준 피드 상태 전환. | + +소비자는 **알 수 없는 `type` 값과 알 수 없는 필드를 무시해야 합니다** (전방 호환성). + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +연결 시와 정의가 변경될 때마다 전송됩니다. `price_exponent`와 `qty_exponent`는 거래소의 틱 사이즈와 수량 단위를 10의 거듭제곱으로 나타냅니다. + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +모든 `quote`는 **전체 상태**입니다 — 메시지가 손실되어도 다음 호가에서 자동 복구되며, 재동기화가 필요 없습니다. 네 개의 타임스탬프는 종단 간 지연 시간을 분해합니다: + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (consumer recv) + venue book wire arrival post-decode WS hand-off +``` + +`0`은 "사용할 수 없음"을 나타내는 센티넬 값입니다 — 1970이 아닌 누락된 값으로 처리하세요. + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side`는 `"buy"`, `"sell"` 또는 `"unknown"`입니다. 체결은 시점 이벤트이며 재연결 시 재전송되지 않습니다. + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`bids`는 최고가 우선으로 정렬되고 `asks`는 최저가 우선으로 정렬됩니다. 각 `depth`는 **전체 스냅샷**입니다 — 병합하지 말고 교체하세요. + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +거래소의 호가 멀티캐스트가 침묵할 때(`state:"down"`) 또는 복구될 때(`state:"ok"`) Edge에서 발생합니다. UI에서 거래소를 비활성 상태로 표시하는 데 사용하세요. 호가 전달은 상태에 의해 차단되지 않습니다 — 피드는 다음 호가에서 자동 복구됩니다. + +### 구독 + +기본적으로 모든 것을 수신합니다. 스트림을 좁히려면 제어 메시지를 보내세요: + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +필드를 생략하면 모든 값과 일치합니다(`{"symbol":"SOL"}` = 모든 거래소의 SOL). `venue`는 대소문자를 구분하지 않고 매칭됩니다. + +**서버 확인 응답:** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +오류는 `{"channel":"error","error":""}`으로 반환됩니다. + +### 하트비트 및 활성 확인 + +- 서버는 20초마다 **WebSocket Ping**을 보냅니다. 호환 클라이언트는 자동으로 Pong을 응답합니다. +- 60초 동안 비활성인 클라이언트는 닫히고 정리됩니다. +- 애플리케이션 수준 keepalive: `{"method":"ping"}` → `{"channel":"pong"}`. + +### 소비자 스켈레톤 + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # unknown types: silently ignore (forward compatibility) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### 입력 소스와 WebSocket 백스톱 + +Edge 멀티캐스트 피드는 항상 활성 상태입니다. 선택적 **공개 WebSocket 백스톱**은 Edge 피드가 중단될 때 격차를 메울 수 있습니다: + +```bash +# BTC와 ETH에 대한 백스톱 활성화: +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +두 소스는 공유 아비터 내에서 `(venue, symbol, source_ts)` 틱 단위로 경쟁합니다. 정상 상태에서는 Edge 소스가 승리합니다(인터넷 경유 수십 ms 대비 1ms 미만). Edge에 격차가 발생하면 공개 복사본이 채웁니다. WebSocket 출력은 어떤 소스가 특정 업데이트를 전달했는지와 관계없이 동일합니다. + +--- + +## 컨테이너 관리 + +```bash +# 로그 스트리밍 +sudo docker logs -f doublezero-edge-connect + +# 터널 상태 확인 +sudo docker exec -it doublezero-edge-connect doublezero status + +# 디바이스 지연 시간 확인 +sudo docker exec -it doublezero-edge-connect doublezero latency + +# 중지 및 제거 +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "TLS 없음" + 브리지는 신뢰할 수 있는/로컬 네트워크를 대상으로 합니다. WebSocket 엔드포인트를 외부에 노출하는 경우 리버스 프록시에서 TLS를 종단하세요. + +--- + +## 모니터링 (Prometheus 메트릭) + +메트릭 엔드포인트는 **기본적으로 비활성**입니다. `METRICS_BIND`로 활성화하세요: + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +그런 다음 스크래핑하세요: + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +주요 메트릭: + +| 메트릭 | 추적 대상 | +|--------|---------------| +| `dz_feed_up{venue}` | 해당 거래소의 멀티캐스트가 활성이면 `1`, 침묵이면 `0`. | +| `dz_datagrams_received_total{venue}` | 거래소별 수집 볼륨. | +| `dz_emit_total{venue,kind}` | 중복 제거 후 유형별 브로드캐스트된 메시지. | +| `dz_quotes_dropped_total{venue}` | 억제된 오래된/중복 호가. | +| `dz_ws_clients` | 현재 연결된 WebSocket 클라이언트 수. | +| `dz_ws_messages_sent_total{kind}` | 클라이언트에 포워딩된 메시지. | +| `dz_ws_client_lagged_total` | 피드 보호를 위해 느린 클라이언트가 제거된 횟수. | + +동일한 바인드 주소에 `GET /healthz` 활성 프로브도 제공됩니다. + +--- + +## 고급: 자체 호스팅 + +컨테이너는 GHCR에서 사용할 수 있습니다: + +| 환경 | 이미지 | 태그 | +|-------------|-------|-----| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet (비공개) | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +수동으로 실행합니다(설치 프로그램이 전달할 수 없는 옵션, 예: `WS_BIND=""`에 필요): + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**소스에서 빌드:** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +버스트 피드에는 더 큰 커널 수신 버퍼가 권장됩니다: + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## 제한 및 백프레셔 + +| 제한 | 기본값 | 초과 시 동작 | +|-------|---------|------------------------| +| 동시 클라이언트 수 (`WS_MAX_CLIENTS`) | 64 | 새 연결이 거부됩니다. | +| 클라이언트당 구독 수 (`WS_MAX_SUBS`) | 256 | `subscribe`가 오류와 함께 거부됩니다. | +| 클라이언트당 분당 인바운드 제어 메시지 수 (`WS_MAX_INBOUND_PER_MIN`) | 600 | 클라이언트 연결이 끊깁니다. | +| 브로드캐스트 버퍼 (`WS_BROADCAST_CAPACITY`) | 4096 | 느린 클라이언트는 **가장 오래된 메시지를 드롭합니다** (피드를 차단하지 않습니다). | + +모든 `quote`와 `depth`는 전체 상태이므로, 백프레셔로 인해 메시지를 드롭한 소비자도 다음 메시지에서 자동 복구됩니다 — 재동기화 핸드셰이크가 필요 없습니다. + +--- + +## 문제 해결 + +### 로컬 포트에 시레드가 도착하지 않음 + +- 온체인에서 `edge-solana-*` 시레드 그룹에 대한 접근이 인가 \ No newline at end of file diff --git a/docs/Edge Market Data Connection.md b/docs/Edge Market Data Connection.md new file mode 100644 index 0000000..1d40aa7 --- /dev/null +++ b/docs/Edge Market Data Connection.md @@ -0,0 +1,465 @@ +--- +description: Run doublezero-edge-connect to re-forward Solana shreds to a local UDP port and consume normalized Edge market data over a local WebSocket. +--- + +# Edge Connection + +!!! warning "By connecting to DoubleZero I agree to the [DoubleZero Terms of Use](https://doublezero.xyz/terms-protocol). The data is for your internal purposes only and may not be retransmitted (see Section 2(e))." + +`doublezero-edge-connect` is a bridge that joins the **DoubleZero Edge binary multicast** and re-serves it locally as two feeds: + +1. **Solana shred forwarding** — deduplicated (optionally signature-verified) shreds fanned out to one or more local UDP destinations, ready for your validator or RPC. +2. **Normalized market data** — Edge venue feeds decoded, precision-corrected, and re-served as a single JSON WebSocket on `ws://host:8081`. + +Both run from the same container and the same one-line install. Enable whichever feeds your onchain authorization grants. + +``` + ┌─ UDP datagrams ──▶ validator / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binary) (dedup · decode · normalize) └─ WebSocket (JSON) ──▶ trading engine + ws://host:8081 +``` + +--- + +## Requirements + +- **Linux/amd64** host with a public IPv4 address authorized onchain for the target environment. +- **Docker** (the one-liner installs it if missing). +- **GRE connectivity** — allow IP protocol 47 at your cloud provider; on AWS disable the ENI source/dest check. +- A **DoubleZero access secret**: a `DZ_`-prefixed base64 token or a path to a keypair file, obtained from the [DoubleZero onboarding](setup.md) process. + +--- + +## Step 1: Install and Run + +One command prepares the host and starts the bridge container. It joins the DoubleZero network and starts every feed your authorization grants — shred forwarding and/or the market-data WebSocket on `:8081`: + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet (private)" + + ```bash + # Requires a GHCR token with read:packages + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +What the script does: + +1. Checks that the host is Linux/amd64, ensures Docker is present (offers to install it). +2. Prepares the host kernel for the GRE tunnel: loads `tun`/`ip_gre`, raises `net.core.rmem_max`, warns about firewall and cloud-provider rules. +3. Loads your access secret (prompted once if `DZ_SECRET` is not set). +4. Runs the bridge container (`--network host`, `NET_ADMIN`/`NET_RAW`, `/dev/net/tun`) and executes `doublezero connect multicast`. + +!!! tip "Non-interactive install" + Set `DZ_SECRET=DZ_…` before the pipe to run completely unattended — no prompts at all. + +--- + +## Step 2: Configure + +All configuration is via **environment variables set before the pipe**. There is no config file. + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### Installer variables + +| Variable | Default | Purpose | +|----------|---------|---------| +| `DZ_SECRET` | *(prompted)* | `DZ_`-prefixed base64 token **or** path to a keypair file. A token is injected into the container and never written to disk; a file is bind-mounted read-only. | +| `DZ_ENV` | per script | `mainnet-beta` \| `testnet` \| `devnet`. | +| `DZ_IMAGE` | per script | Override the container image. | +| `DZ_NAME` | `doublezero-edge-connect` | Container name. | +| `DZ_FEEDS` | *(all)* | Comma-separated venues to narrow market-data ingestion (e.g. `VenueA,VenueB`). Does not affect Solana shred forwarding. | +| `DZ_ASSUME_YES` | `0` | Skip confirmation prompts (e.g. the Docker install prompt). | +| `DZ_GHCR_TOKEN` | — | **Devnet only** — a GHCR token with `read:packages` (devnet image is private). | +| `DZ_GHCR_USER` | `malbeclabs` | **Devnet only** — GHCR username for the login. | + +### Bridge variables + +The installer forwards **any non-empty** bridge variable straight through to the container. Common ones: + +| Variable | Default | Purpose | +|----------|---------|---------| +| `DZ_IFACE` | `doublezero1` | Network interface to listen on. | +| `DZ_RECV_BUF` | — | UDP receive buffer override (bytes). | +| `METRICS_BIND` | *(empty / off)* | Enable the Prometheus `/metrics` endpoint (e.g. `127.0.0.1:9090`). | +| `RUST_LOG` | `info` | Log level (`debug`, `warn`, etc.). | +| `DZ_SHRED_FORWARD` | — | Local UDP destination(s) for forwarded shreds — see [Solana Shred Forwarding](#solana-shred-forwarding). | +| `WS_BIND` | `0.0.0.0:8081` | Market-data WebSocket bind address — see [Market Data WebSocket](#market-data-websocket). | +| `WS_MAX_CLIENTS` | `64` | Maximum concurrent WebSocket clients. | +| `WS_INPUT_COINS` | *(empty / off)* | Enable the public WebSocket backstop for listed symbols (e.g. `BTC,ETH`). | + +**Examples:** + +```bash +# Forward shreds to a local validator/RPC: +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Non-interactive, testnet: +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# Narrow market data to specific venues, verbose logging, non-default WS port: +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Enable metrics and a public WS backstop: +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + Because the installer only forwards **non-empty** values, you cannot pass an empty override (e.g. `WS_BIND=""` to disable the WebSocket sink) through the one-liner. Use a hand-written `docker run` for that — see [Self-hosting](#advanced-self-hosting). + +--- + +## Solana Shred Forwarding + +The bridge joins the `edge-solana-*` shred multicast groups and fans each datagram to one or more local UDP destinations — feeding your validator or RPC directly off the Edge network. It activates automatically on discovery when those groups are present in your authorization. + +```bash +# Default (dedup-only, forward to local port 20000): +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# With signature verification: +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| Variable | Default | Purpose | +|----------|---------|---------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | Destination(s) for forwarded shreds (repeatable). | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup` (one copy per shred), `sigverify` (+ ed25519 verification), `none` (all datagrams). | +| `DZ_SHRED_RPC_URL` | — | Solana RPC endpoint; required by `sigverify` mode. | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | Size of the dedup window. | + +See [Shred forwarding](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md) for the full pipeline and caveats. + +--- + +## Market Data WebSocket + +Open a WebSocket to `ws://:8081` and read JSON frames. You receive all venues you are authorized for. An optional `subscribe` message narrows the stream to specific venues and symbols. + +Any engine that speaks WebSocket + JSON can consume it with a thin (~50–100 line) adapter. The binary multicast, the two-port per-venue split, and the manifest/precision handshake all stay inside the bridge; the only contract a consumer codes against is the WebSocket JSON. + +### Connection lifecycle + +On each new connection the bridge: + +1. **Replays current instrument definitions** — one `instrument` message per known symbol — so the consumer has precision before the first quote. +2. **Replays the latest depth snapshot** per symbol (if the Market-by-Order feed is active). +3. **Streams** `quote` / `trade` / `midpoint` / `depth` messages as they arrive, fanned out to all connected consumers. + +``` +connect → instrument (×N) → depth (×M, latest books) → quote → trade → depth → … +``` + +### Message types + +Every message is a JSON object tagged by a `type` field: + +| `type` | Meaning | +|--------|---------| +| `instrument` | Instrument/precision definition. | +| `quote` | Top-of-book update (full state). | +| `trade` | Trade print (last sale). | +| `midpoint` | Derived mid price. | +| `depth` | Full order-book depth snapshot. | +| `status` | Venue-level feed-health transition. | + +Consumers **must ignore unknown `type` values and unknown fields** (forward compatibility). + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +Sent on connect and whenever definitions change. `price_exponent` and `qty_exponent` give the venue's tick size and size step as powers of ten. + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +Every `quote` is **full state** — a dropped message self-heals on the next quote, no resync needed. The four timestamps decompose end-to-end latency: + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (consumer recv) + venue book wire arrival post-decode WS hand-off +``` + +`0` is the sentinel for "not available" — treat it as missing, not as 1970. + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side` is `"buy"`, `"sell"`, or `"unknown"`. Trades are point-in-time events and are not replayed on reconnect. + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`bids` are sorted highest price first; `asks` are sorted lowest price first. Each `depth` is a **full snapshot** — replace, do not merge. + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +Emitted on the edge when a venue's quote multicast goes silent (`state:"down"`) or recovers (`state:"ok"`). Use it to gray out a venue in your UI. Quote delivery is not gated on status — the feed self-heals on the next quote. + +### Subscriptions + +By default you receive everything. Send a control message to narrow the stream: + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Omitting a field matches any value (`{"symbol":"SOL"}` = SOL on every venue). `venue` is matched case-insensitively. + +**Server acknowledgement:** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Errors return `{"channel":"error","error":""}`. + +### Heartbeat and liveness + +- The server sends a **WebSocket Ping** every 20 seconds; compliant clients auto-reply Pong. +- Clients silent for 60 seconds are closed and reaped. +- App-level keepalive: `{"method":"ping"}` → `{"channel":"pong"}`. + +### Consumer skeleton + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # unknown types: silently ignore (forward compatibility) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### Input sources and the WebSocket backstop + +The Edge multicast feed is always-on. An optional **public WebSocket backstop** can fill gaps when the Edge feed stalls: + +```bash +# Enable the backstop for BTC and ETH: +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +The two sources race per `(venue, symbol, source_ts)` tick inside a shared arbiter. In steady state the Edge source wins (sub-ms vs. tens of ms over the internet); when the Edge gaps, the public copy fills in. The WebSocket output is identical regardless of which source delivered a given update. + +--- + +## Manage the Container + +```bash +# Stream logs +sudo docker logs -f doublezero-edge-connect + +# Check tunnel status +sudo docker exec -it doublezero-edge-connect doublezero status + +# Check device latencies +sudo docker exec -it doublezero-edge-connect doublezero latency + +# Stop and remove +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "No TLS" + The bridge targets a trusted/local network. Terminate TLS at a reverse proxy if you expose the WebSocket endpoint externally. + +--- + +## Monitoring (Prometheus Metrics) + +The metrics endpoint is **off by default**. Enable it with `METRICS_BIND`: + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Then scrape: + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +Key metrics: + +| Metric | What it tracks | +|--------|---------------| +| `dz_feed_up{venue}` | `1` while that venue's multicast is live, `0` while silent. | +| `dz_datagrams_received_total{venue}` | Ingest volume per venue. | +| `dz_emit_total{venue,kind}` | Messages broadcast after dedup, by type. | +| `dz_quotes_dropped_total{venue}` | Stale/duplicate quotes suppressed. | +| `dz_ws_clients` | Currently connected WebSocket clients. | +| `dz_ws_messages_sent_total{kind}` | Messages forwarded to clients. | +| `dz_ws_client_lagged_total` | Times a slow client was shed to protect the feed. | + +A `GET /healthz` liveness probe is also served on the same bind address. + +--- + +## Advanced: Self-hosting + +The container is available on GHCR: + +| Environment | Image | Tag | +|-------------|-------|-----| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet (private) | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +Run it by hand (required for options the installer can't forward, like `WS_BIND=""`): + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**Build from source:** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +A larger kernel receive buffer is recommended for bursty feeds: + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## Limits and Backpressure + +| Limit | Default | Behavior when exceeded | +|-------|---------|------------------------| +| Concurrent clients (`WS_MAX_CLIENTS`) | 64 | New connection is rejected. | +| Subscriptions per client (`WS_MAX_SUBS`) | 256 | `subscribe` is refused with an error. | +| Inbound control msgs / client / min (`WS_MAX_INBOUND_PER_MIN`) | 600 | Client is disconnected. | +| Broadcast buffer (`WS_BROADCAST_CAPACITY`) | 4096 | A slow client **drops the oldest messages** (never stalls the feed). | + +Because every `quote` and `depth` is full state, a consumer that drops messages under backpressure self-heals on the next message — no resync handshake required. + +--- + +## Troubleshooting + +### No shreds arriving at the local port + +- Confirm your access is authorized for the `edge-solana-*` shred groups onchain. +- Verify the tunnel is up: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Check logs for join errors: `sudo docker logs -f doublezero-edge-connect` +- Confirm `DZ_SHRED_FORWARD` points at a reachable local UDP destination. + +### No messages from a venue + +- Verify the tunnel is up: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Check logs for join errors: `sudo docker logs -f doublezero-edge-connect` +- Confirm your access is authorized for that venue onchain. +- Narrow ingestion to that venue with `DZ_FEEDS=` to isolate the issue. + +### WebSocket connects but no quotes arrive + +- The `instrument` messages always arrive first; quotes follow once the reference-data handshake completes. Wait 10–20 seconds after connect before concluding data is missing. +- Check `dz_feed_up{venue}` in metrics — `0` means the multicast is silent on your host. +- Verify firewall rules allow multicast UDP on the `doublezero1` interface. + +### High `dz_ws_client_lagged_total` + +Your consumer is reading slower than the bridge is publishing. Increase the broadcast buffer with `WS_BROADCAST_CAPACITY`, reduce per-message processing time, or add a dedicated reader thread. + +### Container exits immediately + +- The bridge requires `--network host` and the `/dev/net/tun` device; a plain `docker run` without those flags will fail. +- Use the installer one-liner or the exact `docker run` command shown in [Self-hosting](#advanced-self-hosting). + +### GRE tunnel not establishing + +Refer to [Troubleshooting](troubleshooting.md) and ensure IP protocol 47 is permitted at your cloud provider. On AWS, disable the ENI source/dest check for the host. diff --git a/docs/Edge Market Data Connection.pt.md b/docs/Edge Market Data Connection.pt.md new file mode 100644 index 0000000..75eb469 --- /dev/null +++ b/docs/Edge Market Data Connection.pt.md @@ -0,0 +1,465 @@ +--- +description: Execute o doublezero-edge-connect para reencaminhar shreds Solana para uma porta UDP local e consumir dados de mercado normalizados do Edge através de um WebSocket local. +--- + +# Conexão Edge + +!!! warning "Ao conectar-me ao DoubleZero, concordo com os [Termos de Uso do DoubleZero](https://doublezero.xyz/terms-protocol). Os dados são apenas para uso interno e não podem ser retransmitidos (consulte a Secção 2(e))." + +`doublezero-edge-connect` é uma ponte que se junta ao **multicast binário do DoubleZero Edge** e o re-serve localmente como dois feeds: + +1. **Encaminhamento de shreds Solana** — shreds deduplicados (opcionalmente com verificação de assinatura) distribuídos para um ou mais destinos UDP locais, prontos para o seu validador ou RPC. +2. **Dados de mercado normalizados** — feeds de venues Edge decodificados, com precisão corrigida, e re-servidos como um único WebSocket JSON em `ws://host:8081`. + +Ambos executam a partir do mesmo contêiner e da mesma instalação de uma linha. Ative os feeds que a sua autorização onchain conceder. + +``` + ┌─ UDP datagrams ──▶ validator / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binary) (dedup · decode · normalize) └─ WebSocket (JSON) ──▶ trading engine + ws://host:8081 +``` + +--- + +## Requisitos + +- Host **Linux/amd64** com um endereço IPv4 público autorizado onchain para o ambiente alvo. +- **Docker** (o comando de uma linha instala-o se estiver ausente). +- **Conectividade GRE** — permita o protocolo IP 47 no seu provedor cloud; na AWS desative a verificação source/dest da ENI. +- Um **segredo de acesso DoubleZero**: um token base64 com prefixo `DZ_` ou um caminho para um ficheiro de keypair, obtido no processo de [onboarding DoubleZero](setup.md). + +--- + +## Passo 1: Instalar e Executar + +Um único comando prepara o host e inicia o contêiner da ponte. Ele junta-se à rede DoubleZero e inicia todos os feeds que a sua autorização conceder — encaminhamento de shreds e/ou o WebSocket de dados de mercado na porta `:8081`: + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet (privada)" + + ```bash + # Requer um token GHCR com read:packages + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +O que o script faz: + +1. Verifica que o host é Linux/amd64, garante que o Docker está presente (oferece instalá-lo). +2. Prepara o kernel do host para o túnel GRE: carrega `tun`/`ip_gre`, aumenta `net.core.rmem_max`, avisa sobre regras de firewall e do provedor cloud. +3. Carrega o seu segredo de acesso (solicitado uma vez se `DZ_SECRET` não estiver definido). +4. Executa o contêiner da ponte (`--network host`, `NET_ADMIN`/`NET_RAW`, `/dev/net/tun`) e executa `doublezero connect multicast`. + +!!! tip "Instalação não interativa" + Defina `DZ_SECRET=DZ_…` antes do pipe para executar completamente sem supervisão — sem prompts. + +--- + +## Passo 2: Configurar + +Toda a configuração é feita via **variáveis de ambiente definidas antes do pipe**. Não existe ficheiro de configuração. + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### Variáveis do instalador + +| Variável | Padrão | Finalidade | +|----------|--------|------------| +| `DZ_SECRET` | *(solicitado)* | Token base64 com prefixo `DZ_` **ou** caminho para um ficheiro de keypair. Um token é injetado no contêiner e nunca escrito em disco; um ficheiro é montado em modo somente leitura. | +| `DZ_ENV` | por script | `mainnet-beta` \| `testnet` \| `devnet`. | +| `DZ_IMAGE` | por script | Substituir a imagem do contêiner. | +| `DZ_NAME` | `doublezero-edge-connect` | Nome do contêiner. | +| `DZ_FEEDS` | *(todos)* | Venues separadas por vírgula para restringir a ingestão de dados de mercado (ex.: `VenueA,VenueB`). Não afeta o encaminhamento de shreds Solana. | +| `DZ_ASSUME_YES` | `0` | Ignorar prompts de confirmação (ex.: o prompt de instalação do Docker). | +| `DZ_GHCR_TOKEN` | — | **Apenas Devnet** — um token GHCR com `read:packages` (a imagem devnet é privada). | +| `DZ_GHCR_USER` | `malbeclabs` | **Apenas Devnet** — nome de utilizador GHCR para o login. | + +### Variáveis da ponte + +O instalador encaminha **qualquer variável de ponte não vazia** diretamente para o contêiner. As mais comuns: + +| Variável | Padrão | Finalidade | +|----------|--------|------------| +| `DZ_IFACE` | `doublezero1` | Interface de rede para escutar. | +| `DZ_RECV_BUF` | — | Substituição do buffer de receção UDP (bytes). | +| `METRICS_BIND` | *(vazio / desativado)* | Ativar o endpoint Prometheus `/metrics` (ex.: `127.0.0.1:9090`). | +| `RUST_LOG` | `info` | Nível de log (`debug`, `warn`, etc.). | +| `DZ_SHRED_FORWARD` | — | Destino(s) UDP local(ais) para shreds encaminhados — veja [Encaminhamento de Shreds Solana](#encaminhamento-de-shreds-solana). | +| `WS_BIND` | `0.0.0.0:8081` | Endereço de bind do WebSocket de dados de mercado — veja [WebSocket de Dados de Mercado](#websocket-de-dados-de-mercado). | +| `WS_MAX_CLIENTS` | `64` | Máximo de clientes WebSocket simultâneos. | +| `WS_INPUT_COINS` | *(vazio / desativado)* | Ativar o backstop público via WebSocket para os símbolos listados (ex.: `BTC,ETH`). | + +**Exemplos:** + +```bash +# Encaminhar shreds para um validador/RPC local: +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Não interativo, testnet: +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# Restringir dados de mercado a venues específicas, logging verboso, porta WS não padrão: +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# Ativar métricas e um backstop WS público: +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + Como o instalador só encaminha valores **não vazios**, não é possível passar uma substituição vazia (ex.: `WS_BIND=""` para desativar o sink WebSocket) através do comando de uma linha. Use um `docker run` escrito manualmente para isso — veja [Self-hosting](#avancado-self-hosting). + +--- + +## Encaminhamento de Shreds Solana + +A ponte junta-se aos grupos multicast `edge-solana-*` de shreds e distribui cada datagrama para um ou mais destinos UDP locais — alimentando o seu validador ou RPC diretamente a partir da rede Edge. Ativa-se automaticamente na descoberta quando esses grupos estão presentes na sua autorização. + +```bash +# Padrão (apenas dedup, encaminhar para porta local 20000): +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# Com verificação de assinatura: +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| Variável | Padrão | Finalidade | +|----------|--------|------------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | Destino(s) para shreds encaminhados (repetível). | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup` (uma cópia por shred), `sigverify` (+ verificação ed25519), `none` (todos os datagramas). | +| `DZ_SHRED_RPC_URL` | — | Endpoint RPC Solana; necessário pelo modo `sigverify`. | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | Tamanho da janela de deduplicação. | + +Veja [Encaminhamento de shreds](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md) para o pipeline completo e ressalvas. + +--- + +## WebSocket de Dados de Mercado + +Abra um WebSocket para `ws://:8081` e leia frames JSON. Recebe todas as venues para as quais está autorizado. Uma mensagem opcional `subscribe` restringe o fluxo a venues e símbolos específicos. + +Qualquer engine que fale WebSocket + JSON pode consumi-lo com um adaptador simples (~50–100 linhas). O multicast binário, a divisão em duas portas por venue, e o handshake de manifesto/precisão ficam todos dentro da ponte; o único contrato contra o qual um consumidor programa é o WebSocket JSON. + +### Ciclo de vida da conexão + +Em cada nova conexão, a ponte: + +1. **Repete as definições de instrumentos atuais** — uma mensagem `instrument` por símbolo conhecido — para que o consumidor tenha a precisão antes da primeira cotação. +2. **Repete o último snapshot de profundidade** por símbolo (se o feed Market-by-Order estiver ativo). +3. **Transmite** mensagens `quote` / `trade` / `midpoint` / `depth` à medida que chegam, distribuídas a todos os consumidores conectados. + +``` +connect → instrument (×N) → depth (×M, latest books) → quote → trade → depth → … +``` + +### Tipos de mensagem + +Cada mensagem é um objeto JSON identificado por um campo `type`: + +| `type` | Significado | +|--------|-------------| +| `instrument` | Definição de instrumento/precisão. | +| `quote` | Atualização do topo do livro (estado completo). | +| `trade` | Impressão de negócio (última venda). | +| `midpoint` | Preço médio derivado. | +| `depth` | Snapshot completo de profundidade do livro de ordens. | +| `status` | Transição de saúde do feed a nível de venue. | + +Os consumidores **devem ignorar valores `type` desconhecidos e campos desconhecidos** (compatibilidade futura). + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +Enviado na conexão e sempre que as definições mudam. `price_exponent` e `qty_exponent` indicam o tick size e o step de tamanho da venue como potências de dez. + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +Cada `quote` é **estado completo** — uma mensagem perdida auto-recupera na próxima cotação, sem necessidade de ressincronização. Os quatro timestamps decompõem a latência de ponta a ponta: + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (consumer recv) + venue book wire arrival post-decode WS hand-off +``` + +`0` é o sentinela para "não disponível" — trate-o como ausente, não como 1970. + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side` é `"buy"`, `"sell"` ou `"unknown"`. Negócios são eventos pontuais no tempo e não são repetidos na reconexão. + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`bids` são ordenados do preço mais alto para o mais baixo; `asks` são ordenados do preço mais baixo para o mais alto. Cada `depth` é um **snapshot completo** — substitua, não faça merge. + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +Emitido no edge quando o multicast de cotações de uma venue fica silencioso (`state:"down"`) ou recupera (`state:"ok"`). Use-o para desativar visualmente uma venue na sua UI. A entrega de cotações não depende do status — o feed auto-recupera na próxima cotação. + +### Subscrições + +Por padrão, recebe tudo. Envie uma mensagem de controlo para restringir o fluxo: + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Omitir um campo corresponde a qualquer valor (`{"symbol":"SOL"}` = SOL em todas as venues). `venue` é comparado sem distinção de maiúsculas/minúsculas. + +**Confirmação do servidor:** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +Erros retornam `{"channel":"error","error":""}`. + +### Heartbeat e liveness + +- O servidor envia um **WebSocket Ping** a cada 20 segundos; clientes conformes respondem automaticamente com Pong. +- Clientes silenciosos por 60 segundos são fechados e eliminados. +- Keepalive a nível de aplicação: `{"method":"ping"}` → `{"channel":"pong"}`. + +### Esqueleto do consumidor + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # unknown types: silently ignore (forward compatibility) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### Fontes de entrada e o backstop WebSocket + +O feed multicast Edge está sempre ativo. Um **backstop público via WebSocket** opcional pode preencher lacunas quando o feed Edge estagna: + +```bash +# Ativar o backstop para BTC e ETH: +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +As duas fontes competem por tick `(venue, symbol, source_ts)` dentro de um árbitro partilhado. Em estado estável, a fonte Edge vence (sub-ms vs. dezenas de ms pela internet); quando o Edge falha, a cópia pública preenche. A saída WebSocket é idêntica independentemente de qual fonte entregou uma determinada atualização. + +--- + +## Gerir o Contêiner + +```bash +# Transmitir logs +sudo docker logs -f doublezero-edge-connect + +# Verificar o estado do túnel +sudo docker exec -it doublezero-edge-connect doublezero status + +# Verificar latências do dispositivo +sudo docker exec -it doublezero-edge-connect doublezero latency + +# Parar e remover +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "Sem TLS" + A ponte destina-se a uma rede confiável/local. Termine o TLS num reverse proxy se expuser o endpoint WebSocket externamente. + +--- + +## Monitorização (Métricas Prometheus) + +O endpoint de métricas está **desativado por padrão**. Ative-o com `METRICS_BIND`: + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +Depois faça scrape: + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +Métricas principais: + +| Métrica | O que monitoriza | +|---------|------------------| +| `dz_feed_up{venue}` | `1` enquanto o multicast dessa venue está ativo, `0` enquanto silencioso. | +| `dz_datagrams_received_total{venue}` | Volume de ingestão por venue. | +| `dz_emit_total{venue,kind}` | Mensagens transmitidas após dedup, por tipo. | +| `dz_quotes_dropped_total{venue}` | Cotações obsoletas/duplicadas suprimidas. | +| `dz_ws_clients` | Clientes WebSocket atualmente conectados. | +| `dz_ws_messages_sent_total{kind}` | Mensagens encaminhadas para clientes. | +| `dz_ws_client_lagged_total` | Vezes que um cliente lento foi descartado para proteger o feed. | + +Uma sonda de liveness `GET /healthz` também é servida no mesmo endereço de bind. + +--- + +## Avançado: Self-hosting + +O contêiner está disponível no GHCR: + +| Ambiente | Imagem | Tag | +|----------|--------|-----| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet (privada) | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +Execute manualmente (necessário para opções que o instalador não consegue encaminhar, como `WS_BIND=""`): + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**Compilar a partir do código-fonte:** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +Um buffer de receção do kernel maior é recomendado para feeds com rajadas: + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## Limites e Contrapressão + +| Limite | Padrão | Comportamento quando excedido | +|--------|--------|-------------------------------| +| Clientes simultâneos (`WS_MAX_CLIENTS`) | 64 | Nova conexão é rejeitada. | +| Subscrições por cliente (`WS_MAX_SUBS`) | 256 | `subscribe` é recusado com um erro. | +| Msgs de controlo de entrada / cliente / min (`WS_MAX_INBOUND_PER_MIN`) | 600 | Cliente é desconectado. | +| Buffer de broadcast (`WS_BROADCAST_CAPACITY`) | 4096 | Um cliente lento **descarta as mensagens mais antigas** (nunca bloqueia o feed). | + +Como cada `quote` e `depth` é estado completo, um consumidor que perca mensagens sob contrapressão auto-recupera na próxima mensagem — sem necessidade de handshake de ressincronização. + +--- + +## Resolução de Problemas + +### Nenhum shred a chegar à porta local + +- Confirme que o seu acesso está autorizado para os grupos de shreds `edge-solana-*` onchain. +- Verifique se o túnel está ativo: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Verifique os logs para erros de join: `sudo docker logs -f doublezero-edge-connect` +- Confirme que `DZ_SHRED_FORWARD` aponta para um destino UDP local acessível. + +### Sem mensagens de uma venue + +- Verifique se o túnel está ativo: `sudo docker exec -it doublezero-edge-connect doublezero status` +- Verifique os logs para erros de join: `sudo docker logs -f doublezero-edge-connect` +- Confirme que o seu acesso está autorizado para essa venue onchain. +- Restrinja a ingestão a essa venue com `DZ_FEEDS=` para isolar o problema. + +### WebSocket conecta mas nenhuma cotação chega + +- As mensagens `instrument` chegam sempre primeiro; as cotações seguem-se assim que o handshake de dados de referência é concluído. Aguarde 10–20 segundos após a conexão antes de concluir que os dados estão em falta. +- Verifique `dz_feed_up{venue}` nas métricas — `0` significa que o multicast está silencioso no seu host. +- Verifique se as regras de firewall permitem UDP multicast na interface `doublezero1`. + +### `dz_ws_client_lagged_total` elevado + +O seu consumidor está a ler mais lentamente do que a ponte está a publicar. Aumente o buffer de broadcast com `WS_BROADCAST_CAPACITY`, reduza o tempo de processamento por mensagem, ou adicione uma thread de leitura dedicada. + +### Contêiner termina imediatamente + +- A ponte requer `--network host` e o dispositivo `/dev/net/tun`; um `docker run` simples sem essas flags falhará. +- Use o comando de uma linha do instalador ou o comando exato `docker run` mostrado em [Self-hosting](#avancado-self-hosting). + +### Túnel GRE não estabelece + +Consulte [Resolução de Problemas](troubleshooting.md) e garanta que o protocolo IP 47 é permitido no seu provedor cloud. Na AWS, desative a verificação source/dest da ENI para o host. \ No newline at end of file diff --git a/docs/Edge Market Data Connection.zh.md b/docs/Edge Market Data Connection.zh.md new file mode 100644 index 0000000..087847a --- /dev/null +++ b/docs/Edge Market Data Connection.zh.md @@ -0,0 +1,465 @@ +--- +description: 运行 doublezero-edge-connect 将 Solana shred 重新转发到本地 UDP 端口，并通过本地 WebSocket 消费标准化的 Edge 市场数据。 +--- + +# Edge 连接 + +!!! warning "连接到 DoubleZero 即表示我同意 [DoubleZero 使用条款](https://doublezero.xyz/terms-protocol)。数据仅供内部使用，不得转播（见第 2(e) 条）。" + +`doublezero-edge-connect` 是一个桥接工具，它接入 **DoubleZero Edge 二进制组播**，并在本地以两种数据源的形式重新提供服务： + +1. **Solana shred 转发** — 去重（可选签名验证）的 shred 分发到一个或多个本地 UDP 目的地，可直接供验证节点或 RPC 使用。 +2. **标准化市场数据** — Edge 交易所数据源经过解码、精度校正，并通过 `ws://host:8081` 上的单一 JSON WebSocket 重新提供服务。 + +两者运行在同一个容器中，使用相同的一行安装命令。根据您的链上授权启用所需的数据源。 + +``` + ┌─ UDP datagrams ──▶ validator / RPC +DZ Edge multicast ──▶ doublezero-edge-connect ─┤ + (binary) (dedup · decode · normalize) └─ WebSocket (JSON) ──▶ trading engine + ws://host:8081 +``` + +--- + +## 系统要求 + +- **Linux/amd64** 主机，具有在链上已授权目标环境的公共 IPv4 地址。 +- **Docker**（一行安装命令会在缺失时自动安装）。 +- **GRE 连接** — 在云服务商处允许 IP 协议 47；在 AWS 上需禁用 ENI 源/目标检查。 +- **DoubleZero 访问密钥**：一个 `DZ_` 前缀的 base64 令牌或密钥对文件路径，通过 [DoubleZero 入驻流程](setup.md)获取。 + +--- + +## 步骤 1：安装并运行 + +一条命令即可准备主机并启动桥接容器。它会加入 DoubleZero 网络，并启动您的授权所涵盖的所有数据源 — shred 转发和/或 `:8081` 上的市场数据 WebSocket： + +=== "Mainnet-beta" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect | bash + ``` + +=== "Testnet" + + ```bash + curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + ``` + +=== "Devnet（私有）" + + ```bash + # 需要具有 read:packages 权限的 GHCR 令牌 + DZ_GHCR_TOKEN= curl -fsSL https://get.doublezero.xyz/connect-devnet | bash + ``` + +脚本执行内容： + +1. 检查主机是否为 Linux/amd64，确认 Docker 已安装（缺失时提示安装）。 +2. 为 GRE 隧道准备主机内核：加载 `tun`/`ip_gre`，提高 `net.core.rmem_max`，提醒防火墙和云服务商规则。 +3. 加载您的访问密钥（如未设置 `DZ_SECRET`，会提示输入一次）。 +4. 运行桥接容器（`--network host`、`NET_ADMIN`/`NET_RAW`、`/dev/net/tun`）并执行 `doublezero connect multicast`。 + +!!! tip "非交互式安装" + 在管道命令前设置 `DZ_SECRET=DZ_…` 即可完全无人值守运行 — 无需任何确认提示。 + +--- + +## 步骤 2：配置 + +所有配置均通过**管道命令前设置的环境变量**完成，没有配置文件。 + +```bash +DZ_SECRET=DZ_… VAR=value curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +### 安装程序变量 + +| 变量 | 默认值 | 用途 | +|------|--------|------| +| `DZ_SECRET` | *(交互式提示)* | `DZ_` 前缀的 base64 令牌**或**密钥对文件路径。令牌注入容器且不写入磁盘；文件以只读方式挂载。 | +| `DZ_ENV` | 取决于脚本 | `mainnet-beta` \| `testnet` \| `devnet`。 | +| `DZ_IMAGE` | 取决于脚本 | 覆盖容器镜像。 | +| `DZ_NAME` | `doublezero-edge-connect` | 容器名称。 | +| `DZ_FEEDS` | *(全部)* | 以逗号分隔的交易所列表，用于缩小市场数据摄入范围（例如 `VenueA,VenueB`）。不影响 Solana shred 转发。 | +| `DZ_ASSUME_YES` | `0` | 跳过确认提示（例如 Docker 安装提示）。 | +| `DZ_GHCR_TOKEN` | — | **仅限 Devnet** — 具有 `read:packages` 权限的 GHCR 令牌（devnet 镜像为私有）。 | +| `DZ_GHCR_USER` | `malbeclabs` | **仅限 Devnet** — 用于登录的 GHCR 用户名。 | + +### 桥接变量 + +安装程序会将**任何非空的**桥接变量直接传递到容器中。常用变量： + +| 变量 | 默认值 | 用途 | +|------|--------|------| +| `DZ_IFACE` | `doublezero1` | 监听的网络接口。 | +| `DZ_RECV_BUF` | — | UDP 接收缓冲区覆盖值（字节）。 | +| `METRICS_BIND` | *(空 / 关闭)* | 启用 Prometheus `/metrics` 端点（例如 `127.0.0.1:9090`）。 | +| `RUST_LOG` | `info` | 日志级别（`debug`、`warn` 等）。 | +| `DZ_SHRED_FORWARD` | — | 转发 shred 的本地 UDP 目的地 — 参见 [Solana Shred 转发](#solana-shred-转发)。 | +| `WS_BIND` | `0.0.0.0:8081` | 市场数据 WebSocket 绑定地址 — 参见[市场数据 WebSocket](#市场数据-websocket)。 | +| `WS_MAX_CLIENTS` | `64` | 最大并发 WebSocket 客户端数。 | +| `WS_INPUT_COINS` | *(空 / 关闭)* | 为列出的交易对启用公共 WebSocket 后备源（例如 `BTC,ETH`）。 | + +**示例：** + +```bash +# 将 shred 转发到本地验证节点/RPC： +DZ_SECRET=DZ_… DZ_SHRED_FORWARD=127.0.0.1:20000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# 非交互式，testnet： +DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect-testnet | bash + +# 缩小市场数据到特定交易所，详细日志，非默认 WS 端口： +DZ_SECRET=DZ_… DZ_FEEDS=VenueA,VenueB RUST_LOG=debug WS_BIND=0.0.0.0:9000 \ + curl -fsSL https://get.doublezero.xyz/connect | bash + +# 启用指标和公共 WS 后备源： +DZ_SECRET=DZ_… METRICS_BIND=127.0.0.1:9090 WS_INPUT_COINS=BTC,ETH \ + curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +!!! note + 由于安装程序只转发**非空**值，您无法通过一行安装命令传递空覆盖值（例如 `WS_BIND=""` 以禁用 WebSocket 输出）。此类场景请使用手写的 `docker run` — 参见[自托管](#高级自托管)。 + +--- + +## Solana Shred 转发 + +桥接工具加入 `edge-solana-*` shred 组播组，并将每个数据报分发到一个或多个本地 UDP 目的地 — 直接从 Edge 网络为您的验证节点或 RPC 提供数据。当您的授权中存在这些组播组时，它会在发现时自动激活。 + +```bash +# 默认（仅去重，转发到本地端口 20000）： +DZ_SHRED_FORWARD=127.0.0.1:20000 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash + +# 带签名验证： +DZ_SHRED_DEDUP_MODE=sigverify \ + DZ_SHRED_RPC_URL=https://api.mainnet-beta.solana.com \ + DZ_SHRED_FORWARD=127.0.0.1:20000 \ + DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +| 变量 | 默认值 | 用途 | +|------|--------|------| +| `DZ_SHRED_FORWARD` | `127.0.0.1:20000` | 转发 shred 的目的地（可重复设置）。 | +| `DZ_SHRED_DEDUP_MODE` | `dedup` | `dedup`（每个 shred 一份副本）、`sigverify`（+ ed25519 验证）、`none`（所有数据报）。 | +| `DZ_SHRED_RPC_URL` | — | Solana RPC 端点；`sigverify` 模式必需。 | +| `DZ_SHRED_DEDUP_WINDOW_SLOTS` | `512` | 去重窗口大小。 | + +详见 [Shred 转发](https://github.com/malbeclabs/doublezero-edge-connect/blob/main/docs/shred-forwarding.md)了解完整管道和注意事项。 + +--- + +## 市场数据 WebSocket + +打开到 `ws://:8081` 的 WebSocket 连接并读取 JSON 帧。您将接收所有已授权交易所的数据。可通过可选的 `subscribe` 消息将流缩小到特定交易所和交易对。 + +任何支持 WebSocket + JSON 的引擎只需一个轻量级（约 50–100 行）适配器即可消费数据。二进制组播、每个交易所的双端口拆分以及清单/精度握手都保留在桥接内部；消费者唯一需要对接的接口就是 WebSocket JSON。 + +### 连接生命周期 + +每次新连接时，桥接会： + +1. **重放当前合约定义** — 每个已知交易对一条 `instrument` 消息 — 确保消费者在收到首个报价前已获得精度信息。 +2. **重放每个交易对的最新深度快照**（如果逐笔委托数据源处于活跃状态）。 +3. **流式推送** `quote` / `trade` / `midpoint` / `depth` 消息，到达后分发给所有已连接的消费者。 + +``` +connect → instrument (×N) → depth (×M, latest books) → quote → trade → depth → … +``` + +### 消息类型 + +每条消息都是一个带有 `type` 字段标签的 JSON 对象： + +| `type` | 含义 | +|--------|------| +| `instrument` | 合约/精度定义。 | +| `quote` | 最优买卖更新（完整状态）。 | +| `trade` | 成交记录（最新成交）。 | +| `midpoint` | 衍生中间价。 | +| `depth` | 完整订单簿深度快照。 | +| `status` | 交易所级别的数据源健康状态变化。 | + +消费者**必须忽略未知的 `type` 值和未知字段**（前向兼容性）。 + +#### `instrument` + +```json +{"type":"instrument","venue":"ExampleVenue","symbol":"SOL","price_exponent":-2,"qty_exponent":-2} +``` + +在连接时发送，定义变更时也会发送。`price_exponent` 和 `qty_exponent` 以十的幂次表示交易所的最小价格变动和最小数量步长。 + +#### `quote` + +```json +{ + "type": "quote", + "venue": "ExampleVenue", + "symbol": "SOL", + "bid": 184.20, "ask": 184.21, + "bid_size": 12.5, "ask_size": 8.0, + "bid_n": 3, "ask_n": 2, + "source_ts_ns": 1781019263715344015, + "recv_ts_ns": 1781019263715501230, + "kernel_rx_ts_ns": 1781019263715300010, + "ws_send_ts_ns": 1781019263715600440 +} +``` + +每条 `quote` 都是**完整状态** — 丢失的消息会在下一条报价时自动恢复，无需重新同步。四个时间戳分解了端到端延迟： + +``` +source_ts_ns → kernel_rx_ts_ns → recv_ts_ns → ws_send_ts_ns → (consumer recv) + venue book wire arrival post-decode WS hand-off +``` + +`0` 是"不可用"的哨兵值 — 将其视为缺失值，而非 1970 年。 + +#### `trade` + +```json +{ + "type": "trade", + "venue": "ExampleVenue", "symbol": "SOL", + "price": 184.20, "size": 3.5, + "aggressor_side": "buy", + "trade_id": 987654, "cumulative_volume": 12500.0, + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`aggressor_side` 为 `"buy"`、`"sell"` 或 `"unknown"`。成交记录是时间点事件，重新连接时不会重放。 + +#### `depth` + +```json +{ + "type": "depth", + "venue": "MboVenue", "symbol": "SOL", + "bids": [[184.20, 12.5], [184.19, 4.0]], + "asks": [[184.21, 8.0], [184.22, 6.5]], + "source_ts_ns": ..., "recv_ts_ns": ..., + "kernel_rx_ts_ns": ..., "ws_send_ts_ns": ... +} +``` + +`bids` 按价格从高到低排序；`asks` 按价格从低到高排序。每条 `depth` 都是**完整快照** — 直接替换，不要合并。 + +#### `status` + +```json +{"type":"status","venue":"ExampleVenue","state":"down","stale_ms":30000,"ts_ns":...} +``` + +当交易所的报价组播静默时发出（`state:"down"`），恢复时发出（`state:"ok"`）。用于在 UI 中将交易所置灰。报价推送不受状态限制 — 数据源在下一条报价时自动恢复。 + +### 订阅 + +默认情况下您会接收所有数据。发送控制消息以缩小流范围： + +```json +{"method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +{"method":"unsubscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +省略某个字段将匹配任意值（`{"symbol":"SOL"}` = 所有交易所的 SOL）。`venue` 匹配不区分大小写。 + +**服务器确认：** + +```json +{"channel":"subscription_response","method":"subscribe","subscription":{"venue":"ExampleVenue","symbol":"SOL"}} +``` + +错误返回 `{"channel":"error","error":""}`。 + +### 心跳与活性检测 + +- 服务器每 20 秒发送一次 **WebSocket Ping**；合规客户端自动回复 Pong。 +- 60 秒无活动的客户端将被关闭并清理。 +- 应用层保活：`{"method":"ping"}` → `{"channel":"pong"}`。 + +### 消费者代码骨架 + +```python +import json, websocket + +def on_message(ws, frame): + msg = json.loads(frame) + t = msg.get("type") + if t == "instrument": + register_instrument(msg["venue"], msg["symbol"], + msg["price_exponent"], msg["qty_exponent"]) + elif t == "quote": + on_top_of_book(msg["venue"], msg["symbol"], + msg["bid"], msg["ask"], + msg["bid_size"], msg["ask_size"]) + elif t == "trade": + on_trade(msg["venue"], msg["symbol"], + msg["price"], msg["size"], msg["aggressor_side"]) + elif t == "depth": + replace_book(msg["venue"], msg["symbol"], + msg["bids"], msg["asks"]) + # unknown types: silently ignore (forward compatibility) + +ws = websocket.WebSocketApp("ws://localhost:8081", on_message=on_message) +ws.run_forever() +``` + +### 输入源与 WebSocket 后备机制 + +Edge 组播数据源始终在线。可选的**公共 WebSocket 后备源**可在 Edge 数据源中断时填补缺口： + +```bash +# 为 BTC 和 ETH 启用后备源： +WS_INPUT_COINS=BTC,ETH DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +两个数据源在共享仲裁器内按 `(venue, symbol, source_ts)` 逐 tick 竞争。在稳态下 Edge 源获胜（亚毫秒 vs. 互联网上的数十毫秒）；当 Edge 出现间隙时，公共副本进行填补。无论某次更新由哪个源送达，WebSocket 输出都是一致的。 + +--- + +## 管理容器 + +```bash +# 查看实时日志 +sudo docker logs -f doublezero-edge-connect + +# 检查隧道状态 +sudo docker exec -it doublezero-edge-connect doublezero status + +# 检查设备延迟 +sudo docker exec -it doublezero-edge-connect doublezero latency + +# 停止并移除 +sudo docker stop doublezero-edge-connect && sudo docker rm doublezero-edge-connect +``` + +!!! note "无 TLS" + 桥接工具面向可信/本地网络。如果对外暴露 WebSocket 端点，请在反向代理处终止 TLS。 + +--- + +## 监控（Prometheus 指标） + +指标端点**默认关闭**。使用 `METRICS_BIND` 启用： + +```bash +METRICS_BIND=127.0.0.1:9090 DZ_SECRET=DZ_… curl -fsSL https://get.doublezero.xyz/connect | bash +``` + +然后抓取： + +```bash +curl -s localhost:9090/metrics | grep '^dz_' +``` + +关键指标： + +| 指标 | 追踪内容 | +|------|----------| +| `dz_feed_up{venue}` | 该交易所组播在线时为 `1`，静默时为 `0`。 | +| `dz_datagrams_received_total{venue}` | 每个交易所的摄入量。 | +| `dz_emit_total{venue,kind}` | 去重后按类型广播的消息数。 | +| `dz_quotes_dropped_total{venue}` | 被抑制的过期/重复报价数。 | +| `dz_ws_clients` | 当前已连接的 WebSocket 客户端数。 | +| `dz_ws_messages_sent_total{kind}` | 转发给客户端的消息数。 | +| `dz_ws_client_lagged_total` | 为保护数据源而淘汰慢速客户端的次数。 | + +同一绑定地址上还提供 `GET /healthz` 存活探针。 + +--- + +## 高级：自托管 + +容器可在 GHCR 上获取： + +| 环境 | 镜像 | 标签 | +|------|------|------| +| Mainnet-beta | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:mainnet-beta` | +| Testnet | `ghcr.io/malbeclabs/doublezero-edge-connect` | `:testnet` | +| Devnet（私有） | `ghcr.io/malbeclabs/doublezero-edge-connect-devnet` | `:latest` | + +手动运行（适用于安装程序无法转发的选项，如 `WS_BIND=""`）： + +```bash +docker run --rm --network host --cap-add NET_ADMIN --device /dev/net/tun \ + -e DZ_SECRET=DZ_… \ + -e DZ_SHRED_FORWARD=127.0.0.1:20000 \ + -e WS_BIND=0.0.0.0:8081 \ + -e METRICS_BIND=127.0.0.1:9090 \ + ghcr.io/malbeclabs/doublezero-edge-connect:mainnet-beta +``` + +**从源码构建：** + +```bash +git clone https://github.com/malbeclabs/doublezero-edge-connect +cd doublezero-edge-connect +cargo build --release +cargo test + +./target/release/doublezero-edge-connect \ + --iface doublezero1 \ + --ws-bind 0.0.0.0:8081 +``` + +建议为突发数据源设置更大的内核接收缓冲区： + +```bash +sudo sysctl -w net.core.rmem_max=268435456 +``` + +--- + +## 限制与背压 + +| 限制 | 默认值 | 超出时的行为 | +|------|--------|------------| +| 并发客户端数 (`WS_MAX_CLIENTS`) | 64 | 新连接被拒绝。 | +| 每个客户端的订阅数 (`WS_MAX_SUBS`) | 256 | `subscribe` 被拒绝并返回错误。 | +| 每客户端每分钟入站控制消息数 (`WS_MAX_INBOUND_PER_MIN`) | 600 | 客户端被断开连接。 | +| 广播缓冲区 (`WS_BROADCAST_CAPACITY`) | 4096 | 慢速客户端**丢弃最旧的消息**（永远不会阻塞数据源）。 | + +由于每条 `quote` 和 `depth` 都是完整状态，消费者在背压下丢失消息后会在下一条消息时自动恢复 — 无需重新同步握手。 + +--- + +## 故障排除 + +### 本地端口未收到 shred + +- 确认您的访问已在链上获得 `edge-solana-*` shred 组的授权。 +- 验证隧道是否正常：`sudo docker exec -it doublezero-edge-connect doublezero status` +- 检查日志中的加入错误：`sudo docker logs -f doublezero-edge-connect` +- 确认 `DZ_SHRED_FORWARD` 指向一个可达的本地 UDP 目的地。 + +### 未收到某个交易所的消息 + +- 验证隧道是否正常：`sudo docker exec -it doublezero-edge-connect doublezero status` +- 检查日志中的加入错误：`sudo docker logs -f doublezero-edge-connect` +- 确认您的访问已在链上获得该交易所的授权。 +- 使用 `DZ_FEEDS=` 缩小摄入范围以隔离问题。 + +### WebSocket 已连接但未收到报价 + +- `instrument` 消息始终先到达；报价在参考数据握手完成后才会跟进。连接后请等待 10–20 秒再判断数据是否缺失。 +- 检查指标中的 `dz_feed_up{venue}` — `0` 表示组播在您的主机上处于静默状态。 +- 验证防火墙规则是否允许 `doublezero1` 接口上的组播 UDP。 + +### `dz_ws_client_lagged_total` 值过高 + +您的消费者读取速度慢于桥接发布速度。请使用 `WS_BROADCAST_CAPACITY` 增大广播缓冲区，减少每条消息的处理时间，或添加专用读取线程。 + +### 容器立即退出 + +- 桥接工具需要 `--network host` 和 `/dev/net/tun` 设备；没有这些标志的普通 `docker run` 会失败。 +- 请使用一行安装命令或[自托管](#高级自托管)中展示的完整 `docker run` 命令。 + +### GRE 隧道无法建立 + +请参阅[故障排除](troubleshooting.md)，并确保在云服务商处已允许 IP 协议 47。在 AWS 上，需为主机禁用 ENI 源/目标检查。 \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 5c8c602..606ad9c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -69,6 +69,7 @@ nav: - Validator Rewards: Validator Rewards.md - Other Multicast Connection: Other Multicast Connection.md - Edge Subscriber Connection: Edge Subscriber Connection.md + - Edge Connection: Edge Market Data Connection.md - Troubleshooting: troubleshooting.md - Shelby: - Shelby Connection: Shelby Permissioned Connection.md