From 5fd9517de519b369d5a683737318975ce820b01e Mon Sep 17 00:00:00 2001
From: neriumpete <pavlovskipetko@protonmail.com>
Date: Wed, 15 Apr 2026 10:44:09 +0300
Subject: [PATCH 1/2] Add developer docs: overview, architecture, and demo
 guide

---
 docs/SUMMARY.md      |  16 ++
 docs/architecture.md | 184 ++++++++++++++++++++++
 docs/demo-guide.md   | 367 +++++++++++++++++++++++++++++++++++++++++++
 docs/overview.md     |  78 +++++++++
 4 files changed, 645 insertions(+)
 create mode 100644 docs/SUMMARY.md
 create mode 100644 docs/architecture.md
 create mode 100644 docs/demo-guide.md
 create mode 100644 docs/overview.md

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
new file mode 100644
index 0000000..7cfc40e
--- /dev/null
+++ b/docs/SUMMARY.md
@@ -0,0 +1,16 @@
+# Substreams Data Service
+
+## Getting Started
+
+- [Overview](overview.md)
+- [Demo Guide](demo-guide.md)
+
+## Reference
+
+- [Architecture](architecture.md)
+- [MVP Scope](mvp-scope.md)
+- [MVP Implementation Sequencing](mvp-implementation-sequencing.md)
+- [Provider Persistence Boundary](provider-persistence-boundary.md)
+- [Operator Auth](operator-auth.md)
+- [Contracts](contracts.md)
+- [Agent Workflow](agent-workflow.md)
diff --git a/docs/architecture.md b/docs/architecture.md
new file mode 100644
index 0000000..735a277
--- /dev/null
+++ b/docs/architecture.md
@@ -0,0 +1,184 @@
+# Architecture
+
+## Components
+
+### Consumer Sidecar (`consumer/sidecar/`)
+
+Runs alongside a Substreams client. Exposes a gRPC interface the client uses to manage sessions and report usage.
+
+**Key RPCs:**
+
+| RPC | What it does |
+|-----|-------------|
+| `InitSession` | Creates a session, resolves payer/signer, validates the collector |
+| `ReportUsage` | Aggregates usage, creates RAVs, sends to provider; detects `NeedMoreFunds` |
+| `EndSession` | Terminates the session and cleans up |
+
+`payment_session_manager.go` manages the bidirectional stream to the Provider Gateway.
+
+> **MVP note:** The consumer-facing Substreams-compatible proxy endpoint (MVP-007, MVP-017) is not yet implemented. Clients currently integrate via the `sds_sink` wrapper.
+
+---
+
+### Provider Gateway (`provider/gateway/`)
+
+The public-facing payment control plane. Consumers connect here to initiate sessions and stream payment updates.
+
+**Key RPCs:**
+
+| RPC | What it does |
+|-----|-------------|
+| `StartSession` | Initialises session, validates consumer, checks escrow balance |
+| `PaymentSession` | Bidirectional stream: handles usage, tracks funds, requests RAVs at thresholds, enforces low-funds termination |
+| `SubmitRAV` | Validates and stores a RAV with EIP-712 signature verification |
+| `GetSessionStatus` | Operator read surface for session inspection |
+
+Low-funds handling: the provider detects when the escrow balance is insufficient mid-stream and sends `NeedMoreFunds` to the consumer sidecar, which propagates this to the client.
+
+---
+
+### Provider Plugin Gateway (`provider/plugin/`)
+
+Registers auth, session, and usage-metering plugins with Firehose Core via the `BorrowWorkerRequest` / `Event` interfaces. Always runs colocated with the Provider Gateway — they communicate via local in-process notification, not gRPC.
+
+**Three plugin services:**
+
+| Service | Interface | What it does |
+|---------|-----------|-------------|
+| Auth | `BorrowWorkerRequest` | Validates session credentials on each Firehose request |
+| Session | `BorrowWorkerRequest` | Propagates session ID, applies per-session quotas |
+| Usage (Metering) | `Event` | Collects byte-level usage data and forwards to billing |
+
+---
+
+### Oracle (`cmd/sds/oracle serve`)
+
+Provider discovery service. Returns eligible providers and pricing recommendations to consumer sidecars. Currently backed by a manually curated whitelist (not permissionless). Available in the oracle reflex configuration.
+
+> **Status:** Partially implemented. The command is runnable but the full provider discovery flow is in progress (MVP-005).
+
+---
+
+### Horizon (`horizon/`)
+
+The cryptographic core. Implements EIP-712 typed data signing and verification for RAVs and Receipts.
+
+Key types: `Receipt`, `RAV`, signature domain.
+
+`aggregator.go` merges receipts into RAVs with strict validation (collectionId, payer, receiver uniqueness). The `devenv/` subdirectory handles local Anvil chain setup and Horizon/TAP contract deployment for integration tests.
+
+---
+
+### Repository (`provider/repository/`)
+
+Pluggable persistence layer with two backends:
+
+| Backend | When to use |
+|---------|------------|
+| `inmemory.go` | Development and demo (no DSN needed, state lost on restart) |
+| `psql/` | Production-grade; PostgreSQL with golang-migrate schema management |
+
+The backend is selected by the `--repository-dsn` flag: omit it for in-memory, provide a `psql://` DSN for PostgreSQL.
+
+**Schema:** `sessions`, `workers`, `usage_events`, `quota_usage`, `ravs` tables.
+
+See [`docs/provider-persistence-boundary.md`](provider-persistence-boundary.md) for the detailed boundary between runtime session state and collectible RAV state.
+
+---
+
+## Network topology
+
+```
+┌────────────────────────────────────────────────────────────────┐
+│ Consumer host                                                   │
+│                                                                 │
+│   Substreams Client ──── Consumer Sidecar (:9002)              │
+│                              │                                  │
+└──────────────────────────────┼──────────────────────────────────┘
+                               │ gRPC (plaintext in dev, TLS in prod)
+┌──────────────────────────────┼──────────────────────────────────┐
+│ Provider host                │                                  │
+│                              ▼                                  │
+│              Provider Gateway (:9001)                           │
+│                      │   (local notification)                   │
+│              Plugin Gateway (:9003) ◄──── Firehose Core        │
+│                                               (:10016 Substreams│
+│                                                :10015 Firehose) │
+└────────────────────────────────────────────────────────────────┘
+```
+
+**Ports at a glance:**
+
+| Port | Service |
+|------|---------|
+| 9001 | Provider Gateway (gRPC) |
+| 9002 | Consumer Sidecar (gRPC) |
+| 9003 | Plugin Gateway (internal — Firehose only) |
+| 9004 | Oracle (oracle mode only) |
+| 10015 | Firehose gRPC (TLS) |
+| 10016 | Substreams Tier1 gRPC (TLS) |
+| 5432 | PostgreSQL |
+| 6379 | Redis |
+| 8081 | pgweb (database UI) |
+| 58545 | Local devenv RPC (Anvil) |
+
+---
+
+## Key design decisions
+
+**Plugin Gateway ↔ Provider Gateway is local notification, not gRPC.**
+These two components are always colocated. The separation exists for security boundary reasons (plugin-facing vs payment-facing), but there is no reason to split them across hosts. A post-MVP improvement task exists to make the separation cleaner, but it is deferred.
+
+**Session-local funding.**
+Funds are scoped per session, not globally across concurrent streams. If a consumer runs multiple concurrent streams, each session has its own budget. Aggregate exposure tracking across concurrent sessions is an explicit MVP non-goal.
+
+**Pricing is provider-authoritative.**
+The oracle provides pricing metadata, but the provider handshake response is the binding pricing authority.
+
+**Two persistence backends.**
+In-memory for testing and demos; PostgreSQL for durable production state. The in-memory backend loses all state on restart — acceptable for demos, not for production.
+
+**TLS by default.**
+The `--plaintext` flag exists for local development only. Production deployments always use TLS. Firehose Core uses a self-signed TLS cert on its gRPC ports (note the `*` suffix in `firecore.config.yaml`).
+
+---
+
+## Repo layout
+
+```
+cmd/sds/          CLI entrypoints (consumer sidecar, provider gateway, oracle, devenv, tools)
+consumer/
+  sidecar/        Consumer payment sidecar (session init, RAV signing, usage reporting)
+provider/
+  gateway/        Public-facing payment gateway (session management, RAV validation)
+  auth/           Firehose plugin — authentication service
+  session/        Firehose plugin — session management + quotas
+  usage/          Firehose plugin — usage metering
+  plugin/         Plugin gateway (local, not gRPC — registers above services with firecore)
+  repository/     Persistence (in-memory or PostgreSQL)
+sidecar/          Shared components (session types, escrow/collector queriers, pricing, transport)
+horizon/          Core RAV/Receipt types, EIP-712 signing, signature verification
+  devenv/         Local Anvil chain + contract deployment for testing
+internal/         Operator auth utilities
+proto/            Protobuf service definitions
+pb/               Generated Go protobuf code
+contracts/        Solidity ABIs + bytecode (embedded for test deployment)
+devel/            Dev tooling (reflex configs, wrapper scripts, migrate.sh, firecore config)
+docs/             Architecture and design docs (you are here)
+plans/            MVP backlog and gap analysis
+test/integration/ Integration tests (Docker-based, real chain)
+```
+
+---
+
+## Key files for onboarding
+
+| File | Why read it |
+|------|-------------|
+| `README.md` | Quick start, dev setup overview |
+| `AGENTS.md` | Coding conventions, CLI patterns, domain types, concurrency rules |
+| `docs/mvp-scope.md` | Stable target definition, non-goals, MVP assumptions |
+| `plans/mvp-implementation-backlog.md` | Active task tracker with done/not_started/in_progress status |
+| `plans/mvp-gap-analysis.md` | Current-state assessment — what's done, what's missing |
+| `docs/provider-persistence-boundary.md` | What goes in the DB and why |
+| `docs/operator-auth.md` | Operator authentication contract (roles, bearer tokens) |
diff --git a/docs/demo-guide.md b/docs/demo-guide.md
new file mode 100644
index 0000000..9edd71c
--- /dev/null
+++ b/docs/demo-guide.md
@@ -0,0 +1,367 @@
+# Demo Guide
+
+This guide walks you through running the full Substreams Data Service stack locally — a real end-to-end paid Substreams session on a local chain with real contract interactions.
+
+By the end you will have:
+- A local Anvil chain with deployed TAP/Horizon contracts and pre-funded escrow
+- A running Provider Gateway, Consumer Sidecar, and Firehose Core instance
+- A Substreams client streaming data through the full payment loop
+
+---
+
+## Prerequisites
+
+Install the following before starting.
+
+### Go 1.25+
+
+```bash
+go version  # must be >= 1.25
+```
+
+### Docker
+
+Docker is used for PostgreSQL, Redis, and pgweb. Make sure `docker compose` (v2) is available:
+
+```bash
+docker compose version
+```
+
+### reflex
+
+Used to orchestrate the dev stack with auto-restart on code changes:
+
+```bash
+go install github.com/cespare/reflex@latest
+```
+
+### dummy-blockchain
+
+A lightweight fake blockchain used by Firehose Core in the dev environment:
+
+```bash
+go install github.com/streamingfast/dummy-blockchain@latest
+```
+
+### firecore
+
+The Firehose Core binary. Follow the [firecore installation instructions](https://github.com/streamingfast/firehose-core) for your platform.
+
+> **Tip:** After installing, verify with `firecore --version`.
+
+### direnv (optional but recommended)
+
+Automatically loads `.envrc` when you enter the directory:
+
+```bash
+brew install direnv   # macOS
+# or: https://direnv.net/docs/installation.html
+```
+
+---
+
+## 1. Clone and set up
+
+```bash
+git clone https://github.com/graphprotocol/substreams-data-service
+cd substreams-data-service
+
+# Check out the MVP scope branch (Juan's branch — has the latest reflex configs and MVP work)
+git checkout juanmardefago/mvp-scope
+```
+
+### Set up your PATH (with direnv)
+
+Create an `.envrc` file in the repo root:
+
+```bash
+cat > .envrc << 'EOF'
+PATH_add "$(pwd)/devel"
+PATH_add "$(go env GOPATH)/bin"
+EOF
+
+direnv allow
+```
+
+This adds the `devel/` wrapper scripts and your Go bin to `$PATH`, so you can run `sds` and `sds_sink` directly.
+
+> **Without direnv:** prefix commands with `./devel/` (e.g. `./devel/sds devenv`) and make sure `$GOPATH/bin` is in your `$PATH`.
+
+---
+
+## 2. Run the full stack (recommended)
+
+The easiest way to run everything is with `reflex`, which starts all services and restarts them automatically when you change Go or SQL files:
+
+```bash
+reflex -c .reflex
+```
+
+That's it. Reflex orchestrates the following in order:
+
+| Step | What runs | Where |
+|------|-----------|-------|
+| 1 | Docker Compose — PostgreSQL, Redis, pgweb | Background |
+| 2 | Database migrations | One-shot |
+| 3 | `sds devenv` — Anvil chain + contract deployment | Foreground |
+| 4 | Consumer Sidecar | `:9002` |
+| 5 | Provider Gateway | `:9001` |
+| 6 | Firehose Core | `:10016` (Substreams), `:10015` (Firehose) |
+
+Wait until you see log output from all services before proceeding to step 3.
+
+**Expected output (abridged):**
+
+```
+Starting Docker Compose (PostgreSQL, Redis, pgweb)...
+Starting Devenv...
+[devenv] Anvil started on http://localhost:58545
+[devenv] Contracts deployed:
+[devenv]   GraphTallyCollector: 0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9
+[devenv]   PaymentsEscrow:      0xfc7487a37ca8eac2e64cba61277aa109e9b8631e
+[devenv]   SubstreamsDataService: 0x37478fd2f5845e3664fe4155d74c00e1a4e7a5e2
+[devenv] Demo escrow funded. Ready.
+Starting Consumer Sidecar ...
+Starting Provider Gateway (with PostgreSQL)...
+Restarting firehose-core instance...
+Starting Firehose Core
+  Substreams: sds_sink -e https://localhost:10016 --insecure
+```
+
+> **pgweb** is available at [http://localhost:8081](http://localhost:8081) — you can inspect the PostgreSQL database in your browser as sessions and RAVs are created.
+
+---
+
+## 3. Stream data through the payment loop
+
+Once the stack is running, use the `sds_sink` wrapper to run a Substreams package against it:
+
+```bash
+# Stream the map_clocks module from the common package, starting from block 1
+sds_sink run common@v0.1.0 map_clocks -s 1
+```
+
+Or, starting from the latest block:
+
+```bash
+sds_sink run common@v0.1.0 map_clocks -s -1
+```
+
+`sds_sink` connects to the Consumer Sidecar on `:9002`, which initiates a paid session with the Provider Gateway, which in turn authorises the Firehose Core connection. You are now streaming data through the full payment loop.
+
+**What you should see:**
+
+```
+[consumer-sidecar] InitSession called
+[consumer-sidecar] Session started: <session-id>
+[provider-gateway] StartSession: consumer=0xe908... escrow_balance=10000 GRT
+[provider-gateway] PaymentSession opened
+[firehose] block 1 streamed
+[firehose] block 2 streamed
+...
+[provider-gateway] RAV threshold reached — requesting RAV
+[consumer-sidecar] Signing RAV for <amount> units
+[provider-gateway] RAV accepted and stored
+...
+```
+
+---
+
+## 4. Manual setup (alternative to reflex)
+
+If you prefer to run each service in a separate terminal, or if reflex is not available, follow these steps in order.
+
+### Terminal 1: Docker + database migrations + devenv
+
+```bash
+# Start PostgreSQL, Redis, and pgweb
+docker compose up -d
+
+# Wait for PostgreSQL to be healthy, then apply migrations
+./devel/migrate.sh up
+
+# Start Anvil + deploy contracts + fund demo escrow
+./devel/sds devenv
+```
+
+Leave this terminal running. The devenv process holds the Anvil chain; killing it loses all chain state.
+
+**Devenv output to confirm success:**
+
+```
+Contracts deployed successfully.
+Demo state prepared — escrow funded.
+RPC available at http://localhost:58545
+```
+
+### Terminal 2: Consumer Sidecar
+
+```bash
+DLOG=".*=debug" ./devel/sds consumer sidecar \
+  --grpc-listen-addr=:9002 \
+  --plaintext \
+  --signer-private-key=0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281 \
+  --collector-address=0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9
+```
+
+Wait for: `Consumer Sidecar listening on :9002`
+
+### Terminal 3: Provider Gateway
+
+```bash
+DLOG=".*=debug" ./devel/sds provider gateway \
+  --repository-dsn="psql://dev-node:changeme@localhost:5432/dev-node?sslmode=disable" \
+  --grpc-listen-addr=:9001 \
+  --plaintext \
+  --service-provider=0xa6f1845e54b1d6a95319251f1ca775b4ad406cdf \
+  --collector-address=0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9 \
+  --escrow-address=0xfc7487a37ca8eac2e64cba61277aa109e9b8631e \
+  --rpc-endpoint=http://localhost:58545
+```
+
+Wait for: `Provider Gateway listening on :9001`
+
+> **In-memory mode:** Omit `--repository-dsn` to use the in-memory backend (no Docker needed, but state is lost on restart).
+
+### Terminal 4: Firehose Core
+
+```bash
+firecore -c devel/firecore.config.yaml -d ./devel/.firehose start
+```
+
+Wait for: `Substreams Tier1 listening on :10016`
+
+### Terminal 5: Stream data
+
+```bash
+sds_sink run common@v0.1.0 map_clocks -s 1
+```
+
+---
+
+## Demo accounts and contract addresses
+
+All addresses and keys in the dev environment are **deterministic** — the same every time `sds devenv` runs. Do not use these in production.
+
+### Contract addresses
+
+| Contract | Address |
+|----------|---------|
+| GraphTallyCollector | `0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9` |
+| PaymentsEscrow | `0xfc7487a37ca8eac2e64cba61277aa109e9b8631e` |
+| SubstreamsDataService | `0x37478fd2f5845e3664fe4155d74c00e1a4e7a5e2` |
+
+### Test accounts
+
+Each account starts with 10 ETH and 10,000 GRT on the local chain.
+
+| Role | Address | Private Key |
+|------|---------|-------------|
+| Service Provider | `0xa6f1845e54b1d6a95319251f1ca775b4ad406cdf` | — |
+| Payer | `0xe90874856c339d5d3733c92ea5acadc6014b34d5` | — |
+| Demo Signer | `0x82b6f0bbbab50f0ddc249e5ff60c6dc64d55340e` | `0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281` |
+
+---
+
+## Oracle mode (alternative reflex config)
+
+The default `.reflex` config runs in **direct mode** — the consumer sidecar connects directly to the provider gateway with no discovery step.
+
+An **oracle mode** config is also available. In oracle mode, the consumer sidecar contacts the oracle first to discover available providers and pricing before initiating a session.
+
+```bash
+reflex -c .reflex.oracle
+```
+
+This adds:
+- Oracle service on `:9004`
+- Consumer sidecar uses `devel/consumer-sidecar.oracle.yaml` for configuration
+
+> **Note:** The oracle implementation is partial (MVP-005). Oracle mode works for local dev but the full permissionless provider discovery flow is not yet complete.
+
+---
+
+## Database
+
+pgweb provides a browser UI for the PostgreSQL database:
+
+**URL:** [http://localhost:8081](http://localhost:8081)
+
+Tables of interest:
+
+| Table | What's in it |
+|-------|-------------|
+| `sessions` | All sessions — status, consumer, provider, timestamps |
+| `workers` | Active worker assignments per session |
+| `usage_events` | Raw metered usage events from the plugin gateway |
+| `quota_usage` | Per-session quota consumption |
+| `ravs` | Accepted RAVs — the collectible payment records |
+
+---
+
+## Database migrations
+
+```bash
+./devel/migrate.sh up          # Apply all pending migrations
+./devel/migrate.sh down        # Roll back one migration
+./devel/migrate.sh version     # Show current schema version
+./devel/migrate.sh new <name>  # Create a new migration file pair
+```
+
+---
+
+## Running integration tests
+
+Integration tests use Docker (real chain, real contracts). Make sure Docker is running:
+
+```bash
+go test ./test/integration/...
+```
+
+Unit tests (no Docker needed):
+
+```bash
+go test ./...
+```
+
+---
+
+## Troubleshooting
+
+**`firecore` not found**
+
+Make sure `firecore` is installed and on your `$PATH`. See the [firecore repository](https://github.com/streamingfast/firehose-core) for installation instructions.
+
+**`dummy-blockchain` not found**
+
+```bash
+go install github.com/streamingfast/dummy-blockchain@latest
+```
+
+Ensure `$(go env GOPATH)/bin` is in your `$PATH`.
+
+**Port already in use**
+
+Check which process is using the port:
+
+```bash
+lsof -i :<port>
+```
+
+Kill the process or change the `--grpc-listen-addr` flag on the relevant service.
+
+**`migrate.sh up` fails with "already up to date"**
+
+This is fine — it means migrations have already been applied. The reflex config suppresses this error automatically.
+
+**Consumer sidecar shows `NeedMoreFunds` immediately**
+
+The demo escrow was not funded. Restart `sds devenv` — it re-deploys and re-funds the escrow on each run.
+
+**Firehose Core exits immediately**
+
+The Plugin Gateway (Provider Gateway side) must be running before Firehose Core starts. In manual mode, make sure Terminal 3 is ready before starting Terminal 4.
+
+**pgweb shows empty tables**
+
+No sessions have been created yet. Run `sds_sink` to initiate a paid stream and watch the tables populate.
diff --git a/docs/overview.md b/docs/overview.md
new file mode 100644
index 0000000..d9505e4
--- /dev/null
+++ b/docs/overview.md
@@ -0,0 +1,78 @@
+# Overview
+
+The Substreams Data Service (SDS) is the payment infrastructure layer for paid Substreams data streaming on The Graph Network.
+
+It implements the [TAP (Timeline Aggregation Protocol) V2](https://docs.thegraph.com/tap) payment flow on top of Horizon smart contracts — handling session initiation, streaming receipts, RAV (Receipt Aggregate Voucher) signing and aggregation, and on-chain settlement.
+
+## What it is not
+
+SDS is **not** a Substreams node. It does not serve blockchain data. The actual Substreams/Firehose data plane is handled by [Firehose Core](https://github.com/streamingfast/firehose-core). SDS components run **alongside** it, acting as the payment and metering layer.
+
+## Two sides
+
+SDS has two distinct sides — one for consumers, one for providers.
+
+### Consumer side
+
+The **consumer sidecar** runs alongside any Substreams client application. It:
+
+- Initiates paid sessions with a provider
+- Signs RAVs (payment receipts) as usage accumulates
+- Monitors escrow balance and signals when funds are low
+- Terminates sessions cleanly
+
+### Provider side
+
+The **provider stack** runs alongside a Firehose/Substreams node. It consists of three components that work together:
+
+- **Provider Gateway** — public-facing payment control plane; validates sessions, tracks usage, requests RAVs
+- **Plugin Gateway** — internal Firehose plugin; handles auth, session propagation, and usage metering inside Firehose Core
+- **Repository** — durable persistence for sessions, usage events, and accepted RAVs (in-memory for dev, PostgreSQL for production)
+
+There is also an **Oracle** service for provider discovery — partially implemented, available in the oracle reflex configuration.
+
+## Where it fits in The Graph ecosystem
+
+```
+Substreams Client
+    └── Consumer Sidecar (SDS)          ← payment management
+            └── Provider Gateway (SDS)   ← payment validation
+                    └── Plugin Gateway (SDS)  ← Firehose auth/session/metering
+                            └── Firehose Core ← actual blockchain data
+```
+
+The Graph Network's escrow contracts (Horizon) sit on-chain. Funds flow from consumer to provider via RAV aggregation and on-chain `collect()` calls — SDS coordinates this without touching the data path directly.
+
+## Payment flow summary
+
+1. Consumer sidecar calls `StartSession` on the provider gateway
+2. Provider validates the consumer and checks escrow balance
+3. Data streams through Firehose — the plugin gateway meters usage and forwards billing events
+4. As usage accumulates, the provider requests RAVs from the consumer sidecar at deterministic thresholds
+5. The consumer sidecar signs RAVs and sends them back; the provider stores them
+6. If funds run low, the provider sends a `NeedMoreFunds` signal; the session terminates
+7. The provider operator later collects accepted RAVs on-chain via the settlement CLI (post-MVP)
+
+## Current state (MVP)
+
+The core payment loop is implemented and working end-to-end in the development environment:
+
+- Horizon V2 / TAP signing, verification, and RAV aggregation
+- Consumer sidecar (InitSession, ReportUsage, EndSession)
+- Provider gateway (StartSession, PaymentSession, SubmitRAV)
+- Provider plugins integrated with Firehose Core
+- Low-funds detection and session termination
+- Deterministic RAV request thresholds
+- Local devenv (Anvil + contract deployment)
+- PostgreSQL and in-memory repository backends
+
+Work still in progress (post-Juan OOO):
+
+- Collection CLI — on-chain RAV settlement
+- Funding CLI — consumer escrow top-up workflows
+- Consumer-facing Substreams endpoint/proxy (so clients connect to the sidecar directly)
+- Oracle — full provider discovery flow
+- Authenticated operator surfaces
+- Observability hardening (metrics)
+
+For a detailed status breakdown, see [`plans/mvp-gap-analysis.md`](../plans/mvp-gap-analysis.md).

From da301fb2ef64b2e8a8ef1a98d4badebf5d71db87 Mon Sep 17 00:00:00 2001
From: neriumpete <pavlovskipetko@protonmail.com>
Date: Wed, 15 Apr 2026 10:56:21 +0300
Subject: [PATCH 2/2] Fix demo guide: update commands for Juan's latest
 changes, add missing docs to SUMMARY

---
 docs/SUMMARY.md    |   2 +
 docs/demo-guide.md | 141 +++++++++++++++++++++++++++++++--------------
 2 files changed, 99 insertions(+), 44 deletions(-)

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 7cfc40e..3513057 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -11,6 +11,8 @@
 - [MVP Scope](mvp-scope.md)
 - [MVP Implementation Sequencing](mvp-implementation-sequencing.md)
 - [Provider Persistence Boundary](provider-persistence-boundary.md)
+- [Provider Runtime Compatibility](provider-runtime-compatibility.md)
 - [Operator Auth](operator-auth.md)
 - [Contracts](contracts.md)
+- [Codebase Reading Guide](codebase-reading-guide.md)
 - [Agent Workflow](agent-workflow.md)
diff --git a/docs/demo-guide.md b/docs/demo-guide.md
index 9edd71c..0ab6b44 100644
--- a/docs/demo-guide.md
+++ b/docs/demo-guide.md
@@ -47,7 +47,20 @@ go install github.com/streamingfast/dummy-blockchain@latest
 
 The Firehose Core binary. Follow the [firecore installation instructions](https://github.com/streamingfast/firehose-core) for your platform.
 
-> **Tip:** After installing, verify with `firecore --version`.
+> **Note:** The published `ghcr.io/streamingfast/dummy-blockchain:v1.7.7` Docker image may be stale for the current SDS plugin contract. For integration test (`TestFirecore`) validation, build local `firehose-core` and `dummy-blockchain` images from sibling checkouts — see the [runtime compatibility doc](provider-runtime-compatibility.md). For the `reflex -c .reflex` demo flow using a local `firecore` binary, the above is not required.
+
+```bash
+firecore --version  # verify it's installed
+```
+
+### substreams CLI
+
+Used to stream data through the running stack:
+
+```bash
+# Installation: https://docs.substreams.dev/how-to-guides/installing-the-cli
+substreams --version  # verify it's installed
+```
 
 ### direnv (optional but recommended)
 
@@ -66,7 +79,7 @@ brew install direnv   # macOS
 git clone https://github.com/graphprotocol/substreams-data-service
 cd substreams-data-service
 
-# Check out the MVP scope branch (Juan's branch — has the latest reflex configs and MVP work)
+# Check out the MVP scope branch
 git checkout juanmardefago/mvp-scope
 ```
 
@@ -83,7 +96,7 @@ EOF
 direnv allow
 ```
 
-This adds the `devel/` wrapper scripts and your Go bin to `$PATH`, so you can run `sds` and `sds_sink` directly.
+This adds the `devel/` wrapper scripts and your Go bin to `$PATH`, so you can run `sds` directly.
 
 > **Without direnv:** prefix commands with `./devel/` (e.g. `./devel/sds devenv`) and make sure `$GOPATH/bin` is in your `$PATH`.
 
@@ -104,11 +117,11 @@ That's it. Reflex orchestrates the following in order:
 | 1 | Docker Compose — PostgreSQL, Redis, pgweb | Background |
 | 2 | Database migrations | One-shot |
 | 3 | `sds devenv` — Anvil chain + contract deployment | Foreground |
-| 4 | Consumer Sidecar | `:9002` |
-| 5 | Provider Gateway | `:9001` |
+| 4 | Consumer Sidecar (direct provider ingress) | `:9002` |
+| 5 | Provider Gateway | `:9001` (control plane), `:9003` (plugin) |
 | 6 | Firehose Core | `:10016` (Substreams), `:10015` (Firehose) |
 
-Wait until you see log output from all services before proceeding to step 3.
+Wait until you see the Firehose Core startup banner before proceeding to step 3.
 
 **Expected output (abridged):**
 
@@ -116,16 +129,14 @@ Wait until you see log output from all services before proceeding to step 3.
 Starting Docker Compose (PostgreSQL, Redis, pgweb)...
 Starting Devenv...
 [devenv] Anvil started on http://localhost:58545
-[devenv] Contracts deployed:
-[devenv]   GraphTallyCollector: 0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9
-[devenv]   PaymentsEscrow:      0xfc7487a37ca8eac2e64cba61277aa109e9b8631e
-[devenv]   SubstreamsDataService: 0x37478fd2f5845e3664fe4155d74c00e1a4e7a5e2
+[devenv] Contracts deployed.
 [devenv] Demo escrow funded. Ready.
-Starting Consumer Sidecar ...
+Starting Consumer Sidecar (direct provider ingress) ...
 Starting Provider Gateway (with PostgreSQL)...
 Restarting firehose-core instance...
-Starting Firehose Core
-  Substreams: sds_sink -e https://localhost:10016 --insecure
+ Consumer sidecar ingress: http://localhost:9002
+ Provider control plane:   http://localhost:9001
+ Substreams tier1:         https://localhost:10016?insecure=true
 ```
 
 > **pgweb** is available at [http://localhost:8081](http://localhost:8081) — you can inspect the PostgreSQL database in your browser as sessions and RAVs are created.
@@ -134,35 +145,39 @@ Starting Firehose Core
 
 ## 3. Stream data through the payment loop
 
-Once the stack is running, use the `sds_sink` wrapper to run a Substreams package against it:
+Once the stack is running, use the `substreams` CLI to run a package against the consumer sidecar on `:9002`:
 
 ```bash
-# Stream the map_clocks module from the common package, starting from block 1
-sds_sink run common@v0.1.0 map_clocks -s 1
+# Stream 20 blocks from block 0 and exit
+substreams run common@v0.1.0 map_clocks \
+  -e localhost:9002 \
+  --plaintext \
+  -s 0 \
+  -t +20
 ```
 
-Or, starting from the latest block:
+Or use the interactive GUI mode (streams continuously):
 
 ```bash
-sds_sink run common@v0.1.0 map_clocks -s -1
+substreams gui common@v0.1.0 map_clocks \
+  -e localhost:9002 \
+  --plaintext
 ```
 
-`sds_sink` connects to the Consumer Sidecar on `:9002`, which initiates a paid session with the Provider Gateway, which in turn authorises the Firehose Core connection. You are now streaming data through the full payment loop.
+The consumer sidecar on `:9002` is the Substreams-compatible ingress. It initiates a paid session with the Provider Gateway, which authorises the upstream Firehose connection. You are streaming data through the full payment loop.
 
 **What you should see:**
 
 ```
-[consumer-sidecar] InitSession called
-[consumer-sidecar] Session started: <session-id>
+[consumer-sidecar] session started
 [provider-gateway] StartSession: consumer=0xe908... escrow_balance=10000 GRT
 [provider-gateway] PaymentSession opened
-[firehose] block 1 streamed
-[firehose] block 2 streamed
+[firehose] processing block {"block_number": 1}
+[firehose] processing block {"block_number": 2}
 ...
-[provider-gateway] RAV threshold reached — requesting RAV
-[consumer-sidecar] Signing RAV for <amount> units
+[provider-gateway] rav_request issued at threshold
+[consumer-sidecar] RAV signed and returned
 [provider-gateway] RAV accepted and stored
-...
 ```
 
 ---
@@ -199,6 +214,7 @@ RPC available at http://localhost:58545
 ```bash
 DLOG=".*=debug" ./devel/sds consumer sidecar \
   --grpc-listen-addr=:9002 \
+  --config=devel/consumer-sidecar.direct.yaml \
   --plaintext \
   --signer-private-key=0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281 \
   --collector-address=0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9
@@ -212,16 +228,18 @@ Wait for: `Consumer Sidecar listening on :9002`
 DLOG=".*=debug" ./devel/sds provider gateway \
   --repository-dsn="psql://dev-node:changeme@localhost:5432/dev-node?sslmode=disable" \
   --grpc-listen-addr=:9001 \
+  --plugin-listen-addr=:9003 \
   --plaintext \
   --service-provider=0xa6f1845e54b1d6a95319251f1ca775b4ad406cdf \
   --collector-address=0x1d01649b4f94722b55b5c3b3e10fe26cd90c1ba9 \
   --escrow-address=0xfc7487a37ca8eac2e64cba61277aa109e9b8631e \
-  --rpc-endpoint=http://localhost:58545
+  --rpc-endpoint=http://localhost:58545 \
+  --data-plane-endpoint="https://localhost:10016?insecure=true"
 ```
 
 Wait for: `Provider Gateway listening on :9001`
 
-> **In-memory mode:** Omit `--repository-dsn` to use the in-memory backend (no Docker needed, but state is lost on restart).
+> **In-memory mode (no Docker needed):** Replace `--repository-dsn="psql://..."` with `--repository-dsn="inmemory://"`. State is lost on restart.
 
 ### Terminal 4: Firehose Core
 
@@ -234,7 +252,11 @@ Wait for: `Substreams Tier1 listening on :10016`
 ### Terminal 5: Stream data
 
 ```bash
-sds_sink run common@v0.1.0 map_clocks -s 1
+substreams run common@v0.1.0 map_clocks \
+  -e localhost:9002 \
+  --plaintext \
+  -s 0 \
+  -t +20
 ```
 
 ---
@@ -257,17 +279,20 @@ Each account starts with 10 ETH and 10,000 GRT on the local chain.
 
 | Role | Address | Private Key |
 |------|---------|-------------|
-| Service Provider | `0xa6f1845e54b1d6a95319251f1ca775b4ad406cdf` | — |
-| Payer | `0xe90874856c339d5d3733c92ea5acadc6014b34d5` | — |
+| Service Provider | `0xa6f1845e54b1d6a95319251f1ca775b4ad406cdf` | `0x41942233cf1d78b6e3262f1806f8da36aafa24a941031aad8e056a1d34640f8d` |
+| Payer | `0xe90874856c339d5d3733c92ea5acadc6014b34d5` | `0xe4c2694501255921b6588519cfd36d4e86ddc4ce19ab1bc91d9c58057c040304` |
+| User1 | `0x90353af8461a969e755ef1e1dbadb9415ae5cb6e` | `0xdd02564c0e9836fb570322be23f8355761d4d04ebccdc53f4f53325227680a9f` |
+| User2 | `0x9585430b90248cd82cb71d5098ac3f747f89793b` | `0xbc3def46fab7929038dfb0df7e0168cba60d3384aceabf85e23e5e0ff90c8fe3` |
+| User3 | `0x37305c711d52007a2bcfb33b37015f1d0e9ab339` | `0x7acd0f26d5be968f73ca8f2198fa52cc595650f8d5819ee9122fe90329847c48` |
 | Demo Signer | `0x82b6f0bbbab50f0ddc249e5ff60c6dc64d55340e` | `0x0bba7d355d1750fce9756af7887e826e8071a56d9d8e327f546b1f34c78f9281` |
 
 ---
 
 ## Oracle mode (alternative reflex config)
 
-The default `.reflex` config runs in **direct mode** — the consumer sidecar connects directly to the provider gateway with no discovery step.
+The default `.reflex` config runs in **direct mode** — the consumer sidecar connects directly to the provider gateway using config from `devel/consumer-sidecar.direct.yaml`.
 
-An **oracle mode** config is also available. In oracle mode, the consumer sidecar contacts the oracle first to discover available providers and pricing before initiating a session.
+An **oracle mode** config is also available. In oracle mode, the consumer sidecar contacts the oracle first to discover available providers before initiating a session.
 
 ```bash
 reflex -c .reflex.oracle
@@ -275,7 +300,18 @@ reflex -c .reflex.oracle
 
 This adds:
 - Oracle service on `:9004`
-- Consumer sidecar uses `devel/consumer-sidecar.oracle.yaml` for configuration
+- Consumer sidecar uses `devel/consumer-sidecar.oracle.yaml`
+
+When streaming in oracle mode, packages that do not declare `package.network` need an explicit network flag:
+
+```bash
+substreams run common@v0.1.0 map_clocks \
+  -e localhost:9002 \
+  --plaintext \
+  --network mainnet \
+  -s 0 \
+  -t +20
+```
 
 > **Note:** The oracle implementation is partial (MVP-005). Oracle mode works for local dev but the full permissionless provider discovery flow is not yet complete.
 
@@ -302,20 +338,20 @@ Tables of interest:
 ## Database migrations
 
 ```bash
-./devel/migrate.sh up          # Apply all pending migrations
-./devel/migrate.sh down        # Roll back one migration
-./devel/migrate.sh version     # Show current schema version
-./devel/migrate.sh new <name>  # Create a new migration file pair
+./devel/migrate.sh up            # Apply all pending migrations
+./devel/migrate.sh down          # Roll back one migration
+./devel/migrate.sh version       # Show current schema version
+./devel/migrate.sh new <name>    # Create a new migration file pair
 ```
 
 ---
 
-## Running integration tests
+## Running tests
 
 Integration tests use Docker (real chain, real contracts). Make sure Docker is running:
 
 ```bash
-go test ./test/integration/...
+go test ./test/integration/... -v
 ```
 
 Unit tests (no Docker needed):
@@ -324,6 +360,8 @@ Unit tests (no Docker needed):
 go test ./...
 ```
 
+For the full Firehose Core runtime integration test (`TestFirecore`), see [Provider Runtime Compatibility](provider-runtime-compatibility.md).
+
 ---
 
 ## Troubleshooting
@@ -338,7 +376,11 @@ Make sure `firecore` is installed and on your `$PATH`. See the [firecore reposit
 go install github.com/streamingfast/dummy-blockchain@latest
 ```
 
-Ensure `$(go env GOPATH)/bin` is in your `$PATH`.
+Ensure `$(go env GOPATH)/bin` is in your `$PATH`. In Firehose Core logs, look for:
+```
+auth plugin instantiation {"plugin_kind": "sds"}
+processing block {"block_number": ...}
+```
 
 **Port already in use**
 
@@ -348,7 +390,7 @@ Check which process is using the port:
 lsof -i :<port>
 ```
 
-Kill the process or change the `--grpc-listen-addr` flag on the relevant service.
+Kill the process or change the `--grpc-listen-addr` / `--plugin-listen-addr` flag on the relevant service.
 
 **`migrate.sh up` fails with "already up to date"**
 
@@ -360,8 +402,19 @@ The demo escrow was not funded. Restart `sds devenv` — it re-deploys and re-fu
 
 **Firehose Core exits immediately**
 
-The Plugin Gateway (Provider Gateway side) must be running before Firehose Core starts. In manual mode, make sure Terminal 3 is ready before starting Terminal 4.
+The Plugin Gateway (Provider Gateway side, `:9003`) must be running before Firehose Core starts. In manual mode, make sure Terminal 3 is up before starting Terminal 4. In logs, look for:
+```
+errors about unknown `sds` plugin kind/scheme → your firecore binary is too old
+```
+
+**Oracle mode: `oracle-backed ingress requires a v3/v4 Substreams request`**
+
+Your `substreams` CLI is too old. Upgrade to a version that supports v3/v4 package context.
+
+**Oracle mode: `either <substreams_package.network> or <requested_network> is required`**
+
+Your package does not embed a network. Pass `--network mainnet` (or the appropriate network) to the `substreams` CLI.
 
 **pgweb shows empty tables**
 
-No sessions have been created yet. Run `sds_sink` to initiate a paid stream and watch the tables populate.
+No sessions have been created yet. Run a `substreams` command against `:9002` to initiate a paid stream and watch the tables populate.