Add btcd-compatible JSON-RPC/WebSocket endpoint (phase A) - #838
Open
eynhaender wants to merge 6 commits into
Open
Add btcd-compatible JSON-RPC/WebSocket endpoint (phase A)#838eynhaender wants to merge 6 commits into
eynhaender wants to merge 6 commits into
Conversation
New endpoints/test_btcd_rpc.py covering the btcd JSON-RPC/websocket compatibility interface, modeled on test_bitcoind_rpc.py's request/response pattern and test_electrum_subscriptions.py's persistent-connection/ notification-reading pattern. No new pip dependency: the websocket client (WebSocketConnection) is a minimal stdlib-only RFC 6455 implementation (handshake, masked text frames, multi-frame message reassembly), matching this suite's existing reference for hand-rolled wire protocol over a library (see Electrum's raw-socket client). Tests are split into two kinds: - Real assertions (no marker) — authenticate, session, notifyblocks/stopnotifyblocks, chain methods over http post, deprecated methods and stop staying rejected, response envelope behavior. A failure here is a regression. - @pytest.mark.xfail(strict=False) — development targets for what isn't implemented yet: chain methods bridged into the ws connection itself (phase B), the wired not-yet-implemented stubs (notifynewtransactions, loadtxfilter, rescanblocks, ...), and the not-yet-wired phase B/C methods (getinfo, getdifficulty, createrawtransaction, ...). Expected to fail now; flips to XPASS once implemented, which is the cue to tighten the assertion and drop the marker. conftest.py: --btcd-host/--btcd-port/--btcd-username/--btcd-password options and btcd_config fixture. utils.py: DEFAULT_BTCD_PORT. README.md: btcd section (config sample, options, coverage, debug/trigger env vars) alongside the existing per-endpoint documentation.
New test/protocols/btcd/{btcd_rpc.cpp,btcd_setup_fixture.{hpp,cpp}}, in the style of test/protocols/bitcoind: a real websocket connection to a mock ten-block store, covering authenticate, session, notifyblocks/stopnotifyblocks, every not_implemented stub, response id echo across multiple requests on one connection, and unknown-method handling (json-rpc error reply, connection stays alive for the next request).
Adds channel_btcd (channel_http + per-connection auth state, preselects the json-rpc request parser for websocket frames) and protocol_btcd_rpc (inherits protocol_bitcoind_rpc for the standard chain handlers, adds a second dispatcher for btcd's session/notification/filter/admin extension methods over dispatch_websocket). Live: authenticate (validates against configured btcd.username/password, or no-op success if none configured), session (returns the channel identifier), notifyblocks/stopnotifyblocks (chase::organized/reorganized -> blockconnected/blockdisconnected, positional [hash, height, time] per real btcd wire format). Stubbed not_implemented: stop (no secure remote-shutdown path exists), notifynewtransactions/stopnotifynewtransactions (no mempool in v4), loadtxfilter/rescanblocks (phase B), the deprecated notifyreceived/ notifyspent/rescan family. Extends interfaces/btcd.hpp with a stop method. Wiring: channels.hpp, protocols.hpp, sessions.hpp, server_node.hpp/.cpp (start_btcd chained after bitcoind), parser.cpp ([btcd] config section mirroring [bitcoind]), builds/gnu/Makefile.am. Auth over websocket: basic auth is a per-http-request header, which ws data frames cannot carry (only the upgrade handshake has headers, and that request never reaches the auth check -- see channel_http's error::upgraded short-circuit). channel_btcd::unauthorized() bypasses the inherited per-request check for websocket traffic; the plain http post path keeps it unchanged. Enforcement over ws moves into dispatch_websocket instead: every method other than authenticate is rejected until the connection has authenticated. handle_authenticate's reject path also no longer calls stop() synchronously alongside the error send -- that raced the async write and could close the connection before the error response flushed. It's now threaded through as the SEND completion reason, so the close only happens once the write actually finishes. Known gap: standard chain methods (getblockcount etc.) remain reachable only via plain http post on the same endpoint, not yet bridged into the ws dispatcher -- see the phase B TODO on protocol_btcd_rpc.hpp.
conn fixture now authenticates up front when the server has credentials configured, matching what a real client must do before calling anything else over ws (every method but authenticate is now rejected until then). Without this, most of the suite degraded to xfail/failed once credentials were actually configured on the server, since none of the individual tests called authenticate themselves. http_rpc() now sends HTTP Basic Auth when credentials are configured, for the same reason on the plain-post path (which keeps the normal per-request check, unlike ws). test_response_id_matches_request no longer hardcodes absolute request ids 0/1: the fixture's own authenticate call consumes an id first when credentials are configured, so only the relative ordering (each id is exactly one more than the last) is a stable guarantee. Verified against a live bs_btcd instance with btcd.username/password configured: 19 passed, 2 skipped, 20 xfailed, 0 failures.
…ake stage).
session_handshake<protocol_btcd_auth, protocol_btcd_rpc> (tried in the two
preceding commits, now dropped) modeled btcd's auth on electrum/p2p's version
negotiation, but that pattern assumes the handshake can only complete by
reading a real message. btcd's doesn't: with no credential configured it
completes without reading anything, leaving a real async gap between
"handshake decided it's done" and protocol_btcd_rpc actually subscribing its
handlers. Harmless for websocket (the upgrade round-trip masks it), but a
plain HTTP client can and reliably did land a request in that gap. Real btcd
has no such gap because it has no handshake protocol at all.
Revert session_btcd to session_server<protocol_btcd_rpc> (matching bitcoind
and real btcd) and move 'authenticate' back into protocol_btcd_rpc as an
ordinary extension method, but with the real semantics this time:
- Digest-based credential check (sha256("Basic " + base64(user:pass)), same
scheme as config::credential) via options_.authorized(), instead of a
plaintext username/password comparison.
- btcd.username/btcd.password collapse into the same 'btcd.credential =
user:pass[:method,...]' shape bitcoind already uses, so a credential can be
scoped to a method allowlist -- channel_btcd::permitted() enforces that
scope for both http and ws (ws basic-auth-on-upgrade and in-band
authenticate are two alternative ways to satisfy it, matching real btcd's
own "authenticate is disallowed once basic auth is already established").
- send_btcd_error/send_btcd_rpc thread a close_reason through to
handle_complete, so a failed authenticate's error reaches the client before
the connection closes, without racing the async send.
docs/btcd-endpoint.md documents why the handshake variant was reverted.
Boost.Test (640 cases) and the Python endpoint suite pass against a live
server both with and without a configured credential.
Some endpoint tests (e.g. test_blockconnected_notification) wait on a real external event -- an actual new block -- up to --subscription-timeout, which made a normal pytest run slow and made CI/local runs flaky when no block happened to arrive in time. Mark those @pytest.mark.slow and skip them by default; --run-slow opts back in for a real (if longer) end-to-end check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds phase A of a
btcd-compatible endpoint (JSON-RPC 1.0 over WebSocket, port 8334 by convention), following the pattern established by thebitcoindendpoint. Full design rationale, phased scope, and an lnd/btcwallet compatibility analysis are indocs/btcd-endpoint.md.protocol_btcd_rpcinheritsprotocol_bitcoind_rpc(all standard chain handlers — getblockcount, getblock, getrawtransaction, sendrawtransaction, etc. — come along unchanged) and adds a second dispatcher for btcd's session/notification/filter/admin extension methods, routed throughnetwork::protocol_http's existingdispatch_websocketoverride point.channel_btcdpreselects the same json-rpc request parser for websocket frames that the http post path already uses, so both transports share one parsing/dispatch model.Implemented (live)
authenticate— validates against configuredbtcd.username/passwordwhen set; no-op success otherwise.session— returns the channel identifier.notifyblocks/stopnotifyblocks— subscribes tonode::chase::organized/reorganized, pushesblockconnected/blockdisconnectednotifications with the real btcd positional wire format ([hash, height, time], verified againstbtcsuite/btcd/btcjson/chainsvrwsntfns.go— not the filtered variant, which isloadtxfilter's, separately scoped).Auth over websocket
Basic auth is a per-HTTP-request header, which websocket data frames structurally cannot carry (only the upgrade handshake has headers, and that request never reaches the auth check at all — see
channel_http'serror::upgradedshort-circuit).channel_btcd::unauthorized()bypasses the inherited per-request check for websocket traffic; the plain HTTP POST path (e.g. the inherited chain methods) keeps that check unchanged, identical tobitcoind.Enforcement over ws instead happens in
dispatch_websocket: every method other thanauthenticateis rejected until the connection has authenticated. A failedauthenticatesends a json-rpc error and then closes the connection — deferred to the write-completion callback rather than called synchronously alongside the send, so it can't race the outgoing bytes.Deliberately
not_implementedstop— no secure remote-shutdown path exists; unconditional, no gating config (there's no conditional branch to gate).notifynewtransactions/stopnotifynewtransactions— no mempool in v4.loadtxfilter/rescanblocks— scheduled for phase B (also the two methods most load-bearing for a reallnd/btcwalletclient, per the doc's compatibility analysis).notifyreceived/stopnotifyreceived/notifyspent/stopnotifyspent/rescan.Known gap
Standard chain methods are currently reachable only via plain HTTP POST on the same endpoint (unchanged from
bitcoind) — not yet over the websocket connection itself.dispatch_websocketonly routes to the btcd-only dispatcher; the inherited handlers' response path caches the originalhttp::requestfor header derivation, which doesn't exist for a ws frame, so bridging them needs its own care. Flagged as a phase B TODO inprotocol_btcd_rpc.hppand the design doc rather than silently deferred.Test plan
C++ (
test/protocols/btcd/)nix-gnu-debug-static).btcd_rpc_tests).not_implementedstub, response id echo across multiple requests on one connection, unknown-method handling (json-rpc error reply, connection stays alive).Python (
endpoints/test_btcd_rpc.py)test_bitcoind_rpc.py/test_electrum_subscriptions.py. No new pip dependency — the websocket client is a small stdlib-only RFC 6455 implementation, matching this suite's existing preference for hand-rolled wire protocol.@pytest.mark.xfail(strict=False)development targets for everything not yet implemented/bridged —pytest -m xfail -rxshows the live checklist.bs_btcdinstance twice: once with no credentials configured (19 passed, 2 skipped, 20 xfailed, including a realblockconnectedpush notification received and validated during the run), and again after configuringbtcd.username/btcd.passwordand rebuilding/restarting the server (same result: 19 passed, 2 skipped, 20 xfailed, 0 failures) — the second run is what caught and confirmed the fix for the auth-over-websocket issue described above.