diff --git a/STYLE_GUIDE.md b/STYLE_GUIDE.md index 52efbe0..1f44808 100644 --- a/STYLE_GUIDE.md +++ b/STYLE_GUIDE.md @@ -89,7 +89,7 @@ The Operate section covers topics related to running Sei infrastructure. This is **Contents include:** - **Node Operations**: Overview, Seictl setup, statesync, snapshot sync, node types, troubleshooting, API configuration, validators, oracle price feeder -- **Advanced Operations**: Configuration & monitoring, Giga SS Store migration, frozen nodes, technical reference +- **Advanced Operations**: Configuration & monitoring, Giga SS Store migration, frozen nodes, ingress nodes, technical reference ## Style Guidelines diff --git a/docs.json b/docs.json index 8574242..49daef1 100644 --- a/docs.json +++ b/docs.json @@ -368,6 +368,7 @@ "node/advanced-config-monitoring", "node/giga-storage-migration", "node/frozen-nodes", + "node/ingress-nodes", "node/technical-reference" ] } diff --git a/node/index.mdx b/node/index.mdx index 9c42cb4..efecc2b 100644 --- a/node/index.mdx +++ b/node/index.mdx @@ -451,6 +451,9 @@ seid query staking validator $(seid keys show --bech val -a) Serve history from before an upgrade with a read-only node on the outgoing release. + + Serve EVM transaction submission and WebSocket traffic from a low-footprint full node. + Detailed technical specifications, API endpoints, and commands for node operators. diff --git a/node/ingress-nodes.mdx b/node/ingress-nodes.mdx new file mode 100644 index 0000000..0fc45c2 --- /dev/null +++ b/node/ingress-nodes.mdx @@ -0,0 +1,197 @@ +--- +title: 'Run ingress nodes for transaction submission' +sidebarTitle: 'Ingress Nodes' +description: 'Run a low-footprint seid full node for EVM transaction submission and WebSocket subscriptions, for example next to high-frequency trading systems.' +keywords: ['sei node', 'ingress node', 'ingress mode', 'ingress-only node', 'seid start --ingress', 'transaction submission', 'high-frequency trading', 'HFT', 'market maker', 'EVM RPC', 'WebSocket', 'low footprint', 'state sync'] +--- + +An ingress node is a full node that runs with a small memory and disk footprint. It serves EVM JSON-RPC and WebSocket traffic for the most recent 1000 blocks, which covers transaction submission and live subscriptions. The name comes from transaction ingress, the point where your transactions enter the network. + +Run an ingress node when you want your own endpoint for transaction submission and need less memory and disk than a full RPC node uses. For example, a high-frequency trading (HFT) operator can run one next to its trading systems in each region where it trades. + + + No tagged `seid` release includes ingress mode yet. A release without it refuses to start with `--ingress` and reports `unknown flag: --ingress`. The same release ignores `ingress = true` in `app.toml` and starts a regular full node. Check the [`sei-chain` release notes](https://github.com/sei-protocol/sei-chain/releases) for the first release that includes ingress mode. + + +## How ingress mode works + +You turn on ingress mode with `seid start --ingress`. At startup, `seid` overrides a fixed set of settings in `app.toml` and `config.toml`. The node still executes every block and computes the same app hash as any other full node. P2P networking and the mempool work as on any full node, so the node gossips the transactions that you submit to its peers. + +An ingress node serves: + +- EVM JSON-RPC over HTTP on port `8545` and over WebSocket on port `8546`. Ingress mode turns on both servers. +- The Tendermint RPC on port `26657`, without transaction queries. +- State and EVM transaction receipts for the last 1000 blocks. + +Ingress mode turns off: + +- The REST API on port `1317`. +- The transaction index. The Tendermint RPC methods `tx`, `tx_search`, `block_search`, and `broadcast_tx_commit` return errors. +- Snapshot production for state sync. Other nodes cannot state sync from an ingress node. +- Gas simulation in `eth_sendRawTransaction`. The node broadcasts each transaction without simulating it first. +- State-store snapshots, state proofs, trace baking, and the supply invariant check. + +Ingress mode also keeps only the latest memIAVL snapshot and writes memIAVL snapshot trees one at a time. It shrinks the FlatKV caches and memtables too. For every setting that it changes, see [Settings that ingress mode overrides](#settings-that-ingress-mode-overrides). + +### History on an ingress node + +The node keeps state and receipts for the last 1000 blocks. It keeps blocks for longer. The node does not prune blocks inside the evidence window, so it keeps at least as many blocks as the `max_age_num_blocks` evidence parameter. To read the parameter, run: + +```bash +curl -s http://localhost:26657/consensus_params | jq -r .consensus_params.evidence.max_age_num_blocks +``` + +A request that needs state or receipts from an older height fails. The error names the earliest height that the node still serves. Send these requests to a full RPC node, an [archive node](/node/node-types), or an [RPC provider](/learn/rpc-providers). This applies to `eth_getLogs` backfills, `debug_trace*` calls on older transactions, and indexers that read history. + +## Before you begin + +You need: + +- A `seid` release that includes ingress mode. +- A node in `full` mode. Ingress mode refuses validator and seed mode. It also refuses to start with Autobahn enabled, that is, with a non-empty `autobahn-config-file` in `config.toml`. You cannot combine ingress mode with a non-zero `freeze-height`. +- An empty data directory. State sync runs only on a node without local state. To reuse a host, first follow [Clean up](/node/statesync#clean-up) in the state sync guide. +- A host sized for your workload. An ingress node needs less memory and disk than the full node [hardware requirements](/node#hardware-requirements). Its disk use still grows with the current state of the network and with the blocks in the evidence window. Measure a test node under your workload before you size production hosts. + +## Set up an ingress node + +These steps set up an ingress node on Sei Mainnet (`pacific-1`). For Sei Testnet, use the chain ID `atlantic-2`. + + + + + +Install a `seid` release that includes ingress mode. The [node setup guide](/node) covers the prebuilt binary, a build from source, and Docker. + +Initialize the node without `--mode`. The default mode is `full`, which ingress mode requires. + +```bash +seid init --chain-id pacific-1 +``` + + + + + +Configure state sync with the script in [Statesync script](/node/statesync#statesync-script). Do not start `seid` at the end of that guide. You start it in ingress mode in the next steps. + + + + + +Create a systemd service that starts `seid` with the `--ingress` flag. Run these commands as the user that runs `seid`: + +```bash +sudo tee /etc/systemd/system/seid.service > /dev/null << EOF +[Unit] +Description=Sei ingress node +After=network-online.target + +[Service] +User=$USER +ExecStart=$(which seid) start --ingress +Restart=always +RestartSec=3 +LimitNOFILE=65535 + +[Install] +WantedBy=multi-user.target +EOF + +sudo systemctl daemon-reload +sudo systemctl enable seid +``` + +Instead of the flag, you can set `ingress = true` in `app.toml`. Put the key in the base configuration at the top of the file, before the first `[section]` header: + +```toml app.toml +# Run this full node in ingress mode. +ingress = true +``` + + + Prefer the flag. A `seid` release without ingress mode refuses to start with the flag, so you notice the problem at once. The same release ignores the `app.toml` key and starts a regular full node. + + + + + + +Start `seid` and follow the log: + +```bash +sudo systemctl start seid +sudo journalctl -fu seid +``` + +The node restores a state sync snapshot and then follows the chain. If it exits right after it starts, look for one of these errors in the log: + +- `--ingress requires Tendermint mode "full"`: set `mode = "full"` in `config.toml`. +- `--ingress is not supported with Autobahn`: leave `autobahn-config-file` empty in `config.toml`. +- `--ingress cannot be combined with --freeze-height`: set `freeze-height = 0` in `app.toml`, and do not pass the `--freeze-height` flag. + + + + + +Check that the node has caught up and that the EVM RPC answers. The first command prints `false` after the node has caught up: + +```bash +curl -s http://localhost:26657/status | jq -r .sync_info.catching_up +# false + +curl -s -X POST http://localhost:8545 -H 'Content-Type: application/json' \ + --data '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' +# {"jsonrpc":"2.0","id":1,"result":"0x531"} +``` + +`0x531` is the Sei Mainnet EVM chain ID, `1329`. On Sei Testnet, the result is `0x530`. + +Then check the ports that the node listens on: + +```bash +ss -ltn | grep -E ':(1317|8545|8546) ' +``` + +The output lists ports `8545` and `8546`, and it has no line for port `1317`. `seid init` turns on the REST API in `app.toml`, so a regular full node listens on port `1317`. If the output lists that port, the node did not start in ingress mode. Check that your `seid` release includes ingress mode and that the service passes `--ingress`. + + + + + +## Use ingress nodes for high-frequency trading + +Latency-sensitive operators, such as market makers and HFT firms, often want their own transaction submission and WebSocket endpoints in each region where they trade. Ingress mode lowers the memory and disk that each of these nodes needs. When you plan such a setup, follow these guidelines: + +- **Run each node near the systems that use it.** Put an ingress node in the same region or data center as the systems that submit transactions through it. Their requests then do not cross regions. +- **Submit and subscribe on your own node.** Send signed transactions with `eth_sendRawTransaction`. Subscribe to `newHeads` and `logs` with `eth_subscribe` on port `8546`. For client examples, see [WebSocket connections](/evm/evm-parity/websocket). +- **Set gas limits close to what your transactions use.** An ingress node does not simulate a transaction before it broadcasts it, so it reports the gas limit as the estimated gas. Block proposers count each EVM transaction at its estimated gas when they fill a block. A gas limit far above actual use takes up more of the block's gas budget than the transaction needs. +- **Keep the node well connected.** The node gossips your transactions to its peers through the mempool, like any full node. It does not send them to validators directly, and it does not give them priority. Keep `broadcast = true` under `[mempool]` in `config.toml`, which is the default. To check the peer count, run `curl -s http://localhost:26657/net_info | jq -r .n_peers`. +- **Restrict access.** The EVM servers listen on all network interfaces. Allow only your own systems through the firewall to ports `8545`, `8546`, and `26657`. Keep port `26656` open for P2P. +- **Plan for restarts and upgrades.** Run at least two ingress nodes in each region, so that one serves traffic while the other restarts. An ingress node executes every block, so upgrade it at each upgrade height, like any full node. See [Update procedures](/node/node-operators#update-procedures). + +Ingress mode does not add subscription types or earlier block signals. Its WebSocket subscriptions are the same as on any other full node. + +## Settings that ingress mode overrides + +Ingress mode sets these values at startup. While ingress mode is on, the values in your configuration files for these settings have no effect. + +| File and section | Setting | Value in ingress mode | +| --- | --- | --- | +| `app.toml`, base configuration | `min-retain-blocks` | `1000` | +| `app.toml`, `[state-sync]` | `snapshot-interval` | `0` | +| `app.toml`, `[state-commit]` | `sc-enable-state-proof` | `false` | +| `app.toml`, `[state-commit]` | `sc-hash-logger-blocks-to-retain` | `1000` | +| `app.toml`, `[state-commit]` | `sc-snapshot-writer-limit` | `1` | +| `app.toml`, `[state-store]` | `ss-enable` | `true` | +| `app.toml`, `[state-store]` | `ss-keep-recent` | `1000` | +| `app.toml`, `[state-store]` | `ss-snapshot-enable` | `false` | +| `app.toml`, `[evm]` | `http_enabled` | `true` | +| `app.toml`, `[evm]` | `ws_enabled` | `true` | +| `app.toml`, `[evm]` | `enable_simulation` | `false` | +| `app.toml`, `[evm]` | `trace_bake_enabled` | `false` | +| `app.toml`, `[api]` | `enable` | `false` | +| `app.toml`, `[wasm]` | `memory_cache_size` | `16` (MiB) | +| `app.toml`, `[light_invariance]` | `supply_enabled` | `false` | +| `config.toml`, `[tx-index]` | `indexer` | `["null"]` | + +Ingress mode also tunes two stores in ways that no `app.toml` setting controls. memIAVL keeps only its latest snapshot. FlatKV uses smaller caches, memtables, and write backlogs, and it keeps only one old snapshot. diff --git a/node/node-types.mdx b/node/node-types.mdx index 1bd64b2..9331f9e 100644 --- a/node/node-types.mdx +++ b/node/node-types.mdx @@ -11,6 +11,8 @@ You can run a few types of nodes on the Sei network. Each type has a different p - Frozen nodes: these full nodes stop one block before an upgrade height. They keep running the outgoing release and serve queries for the heights before the upgrade. To run one, see [Run frozen nodes](/node/frozen-nodes). +- Ingress nodes: these full nodes serve EVM transaction submission and WebSocket subscriptions with a small memory and disk footprint. They keep state and receipts for only the last 1000 blocks. To run one, see [Run ingress nodes](/node/ingress-nodes). + - State sync nodes: these nodes serve snapshot data that other nodes use to bootstrap onto the chain. To enable this type of node, set a non-zero `snapshot-interval` under `[state-sync]` in `app.toml`. The `enable=true` setting under `[statesync]` in `config.toml` does the opposite: it makes a node bootstrap from the snapshots of other nodes. - Validator nodes: these nodes propose and sign blocks to secure the chain. To enable this type of node, set `mode=validator` in `config.toml`. Because Sei uses proof of stake, you must have enough delegation to join the active set. diff --git a/node/technical-reference.mdx b/node/technical-reference.mdx index 92c06a4..70997ba 100644 --- a/node/technical-reference.mdx +++ b/node/technical-reference.mdx @@ -48,6 +48,31 @@ The experimental `SEI_CONFIG_MANAGER` environment variable selects how `seid` lo SEI_CONFIG_MANAGER=v2 seid start ``` +#### Ingress mode (`--ingress`) + +Ingress mode runs a full node that serves EVM traffic over the last 1000 blocks with a small memory and disk footprint. Use the `--ingress` start flag or a top-level `ingress = true` key in `app.toml`. No tagged `seid` release includes ingress mode yet. For setup steps and deployment guidance, see [Run ingress nodes](/node/ingress-nodes). + +At startup, ingress mode overrides a fixed set of settings: + +- The EVM HTTP and WebSocket servers stay on. +- The REST API, the transaction index, state proofs, snapshot production for state sync, state-store snapshots, gas simulation in `eth_sendRawTransaction`, trace baking, and the supply invariant check are off. +- The node keeps state, receipts, and hash logs for the last 1000 blocks. It keeps blocks for at least the evidence window. +- P2P networking and the mempool do not change. + +```bash +# Start a full node in ingress mode +seid start --ingress +``` + +```toml +# app.toml — top-level key in the base configuration, not under any [section] +ingress = true +``` + + + Ingress mode is supported only for full nodes. In validator or seed mode, startup fails with `--ingress requires Tendermint mode "full", got ""`. Ingress mode also does not run with Autobahn. When `autobahn-config-file` is set in `config.toml`, startup fails with `--ingress is not supported with Autobahn`. You also cannot combine ingress mode with a non-zero `freeze-height`. The node then fails with `--ingress cannot be combined with --freeze-height`. + + #### Freeze mode (`--freeze-height`) As of v6.6.3, you can put a full node into read-only freeze mode at a specified block height. Use the `--freeze-height` start flag or the corresponding `freeze-height` field in `app.toml`. Freeze mode exists for historical RPC nodes. A node frozen at an upgrade height keeps running the pre-upgrade binary and keeps serving the state that binary produced. It does not shut down at the boundary or execute the upgrade block with code that no longer matches that state. For setup steps and the freeze heights of each network, see [Run frozen nodes](/node/frozen-nodes).