Postgres Data Handler for DeSo Blockchain

Note: This guide assumes you are already familiar with running a DeSo node. In this documentation, we introduce two new concepts—the state consumer interface and the Postgres data handler—and explain the minimal container setup required to extract on-chain state from your DeSo node into a PostgreSQL database.

---

Overview

The Postgres Data Handler project implements a state consumer interface that reads a state change file (generated by a DeSo node), decodes the on-chain state change entries, and applies those changes (inserts, updates, deletes) to a Postgres database. This solution is designed specifically to persist on-chain state in Postgres, using batching and transaction management for consistency and performance.

In a typical deployment, you will need three separate containers:

1. DeSo Node Container
   This container runs your DeSo node. It synchronizes with the network and—when properly configured—writes a state change file. The state change file is the source of truth for on-chain state changes and must be accessible to the data handler.

2. Postgres Data Handler Container
   This container implements the state consumer interface. It reads and decodes the state change file generated by your DeSo node, processes the state change entries using batch operations, and synchronizes the on-chain state to a Postgres database.

3. Postgres Instance Container
   This is your PostgreSQL database container. It stores the on-chain state as maintained by the Postgres data handler.

---

Key Concepts

1. State Consumer Interface

• Purpose:
  This is a Go interface that abstracts the process of reading a binary state change file and invoking handlers to process on-chain state.
• How It Works:
  • Reads a file (generated by your DeSo node) that contains encoded state change entries.
  • Decodes each entry and triggers operations (insert, update, delete) on the target datastore.
• Benefits:
  • Data-store agnostic design (other backends could be supported in the future).
  • Allows batch processing and ensures consistency during state synchronization.

2. Postgres Data Handler

• Purpose:
  The Postgres data handler is a concrete implementation of the state consumer interface that applies state changes directly to a PostgreSQL database.
• Core Features:
  • Processes state change entries in batches.
  • Uses an LRU cache and transaction savepoints to optimize performance.
  • Supports configurable batching (e.g., BATCH_BYTES and THREAD_LIMIT) and optional mempool syncing.
• Outcome:
  The on-chain state—such as posts, profiles, likes, NFTs, and transactions—is effectively maintained as queryable rows in a Postgres database.

---

Container Setup

For a complete setup, you will need to deploy three containers. Here is a high-level summary of the required configuration for each:

DeSo Node Container

• Purpose:
  Runs the DeSo node that synchronizes with the blockchain.
• Key Configuration:
  • STATE_CHANGE_DIR
    Specifies the directory where the node writes the state change file. Example: /state-changes
  • Other networking/environment variables as required by your node.

Postgres Data Handler Container

• Purpose:
  Consumes the state change file and applies state changes to Postgres.
• Key Configuration:
  • STATE_CHANGE_DIR
    Must match the directory where the DeSo node writes its state change file.
  • CONSUMER_PROGRESS_DIR
    Directory for storing consumer progress (i.e., the last processed state change offset).
  • Database Connection Details:
    Set DB_HOST, DB_PORT, DB_NAME, DB_USERNAME, DB_PASSWORD, and optionally READONLY_USER_PASSWORD so that the handler can connect to the Postgres instance.
  • Batching and Synchronization Settings:
    Variables such as BATCH_BYTES, THREAD_LIMIT, and SYNC_MEMPOOL allow you to tune performance.

Postgres Instance Container

• Purpose:
  Serves as the PostgreSQL database backend that holds the on-chain state.
• Key Configuration:
  • The container must expose standard Postgres connection settings.
  • Credentials and database names provided must match those specified in the Postgres data handler configuration.
  • Storage and resource configurations should be set as needed for your anticipated workload.

For a working example of container orchestration using Docker Compose, please refer to the local.docker-compose.yml file in our GitHub repository.

---

Deployment Recommendations

Rather than focusing on a specific orchestration tool, here are some best practices for deploying these containers:

• Container Orchestration:
  You can use Docker Compose for local development or Kubernetes with tools like Pulumi for production. The focus should be on ensuring the three containers are networked together correctly.

• Networking:
  Make sure that the Postgres Data Handler container has access to both the state change file (shared via a volume or network file share from the DeSo node) and the Postgres database.

• Configuration Management:
  Use environment variables for all sensitive keys and configuration details. This will allow you to manage different settings for local development versus production deployments.

• Monitoring and Logging:
  Integrate logging (using glog or another logging tool) and monitoring (e.g., DataDog) to keep an eye on the health of your containers and the state synchronization process.

---

Contributing

Contributions, feature enhancements, and bug fixes are welcome. If you have any suggestions or encounter issues, please open an issue or submit a pull request.

Have more questions?

DeepWiki (powered by Devin AI) provides up-to-date documentation you can talk to for this repo, click the button below to try it out.

---

Happy syncing!