diff --git a/docs/architecture.md b/docs/architecture.md index c36ae26..bd04393 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -51,7 +51,7 @@ flowchart TB → **[Detailed documentation](server.md)** -The server implements six RPC methods — `getHealth`, `getNetwork`, `getLatestLedger`, `sendTransaction`, `getTransaction`, and `traceTransaction` — and the K semantics answer all of them. +The server implements eight RPC methods — `getHealth`, `getNetwork`, `getLatestLedger`, `sendTransaction`, `getTransaction`, `getTransactions`, `getLedgers`, and `traceTransaction` — and the K semantics answer all of them. `sendTransaction` always returns `PENDING` and clients poll `getTransaction` for the result — matching the Stellar RPC async pattern even though the transaction executes synchronously. See [server.md](server.md) for details. @@ -91,11 +91,12 @@ All of the server's input and output artifacts live in one directory, the *io di | `metadata.json` | persistent | the K semantics | `{"latest_ledger": N}` — the server ledger counter, bumped by 1 per committed transaction. | | `receipts/receipt_.json` | persistent | the semantics (on success) or the server (on failure) | one stored receipt per transaction, keyed by tx hash, answering `getTransaction`. Each is `{status, ledger, createdAt, envelopeXdr, resultXdr, resultMetaXdr}`. | | `traces/trace_.jsonl` | persistent | the semantics | one execution trace per transaction, keyed by tx hash — the instruction-level records, one JSON object per line. `traceTransaction` returns this file's contents. | +| `ledgers/ledger_.json` | persistent | the server (on success) | one record per closed ledger — `{sequence, txHash, closedAt, hash, headerXdr, metadataXdr}` — the ledger→transaction index behind `getTransactions` and `getLedgers`. Written in Python because the header artifacts are XDR, which K cannot construct. | | `requests/request_.json` | persistent | the server | an archive of each incoming JSON-RPC request, numbered by a monotonic counter, kept for debugging. | | `request.json` | transient | the server | the request envelope for the call in flight (`method`, `id`, `now`, and method-specific fields). The semantics remove it once they respond. | | `response.json` | transient | the semantics | the JSON-RPC response (`{jsonrpc, id, result}`) for the most recent call. The server reads it back; it is absent when a transaction gets stuck. | -Receipts, traces, and request archives are split into one file per item — keyed by tx hash, or numbered — so that no single file grows without bound as the chain advances. The server creates the `receipts/`, `traces/`, and `requests/` directories before the semantics run, because the K file-system hooks open files with POSIX `open()`, which does not create parent directories. +Receipts, traces, ledger records, and request archives are split into one file per item — keyed by tx hash, ledger sequence, or a counter — so that no single file grows without bound as the chain advances. The server creates the `receipts/`, `traces/`, `ledgers/`, and `requests/` directories before the semantics run, because the K file-system hooks open files with POSIX `open()`, which does not create parent directories. The world state stays in KORE (rather than a JSON snapshot) because an uploaded wasm module is a `ModuleDecl` that the semantics cannot reconstruct from bytes — only `wasm2kast` (Python) can produce it. The receipts and the ledger counter, by contrast, are plain data and live in JSON files, which the semantics read and write directly via the file-system hooks. diff --git a/docs/node-semantics.md b/docs/node-semantics.md index 6f72b79..20f9dfb 100644 --- a/docs/node-semantics.md +++ b/docs/node-semantics.md @@ -17,6 +17,7 @@ The semantics communicate with the Python process through files in the working d | `metadata.json` | K ↔ K | `{"latest_ledger": N}` — the ledger counter | | `receipts/receipt_.json` | K → Python | one stored receipt per transaction, keyed by tx hash | | `traces/trace_.jsonl` | K → Python | one execution trace per transaction (per-instruction records), keyed by tx hash | +| `ledgers/ledger_.json` | Python → K | one record per closed ledger (`sequence`, `txHash`, `closedAt`, and the ledger-header XDR artifacts), written by the server and read back to serve `getTransactions`/`getLedgers` | --- @@ -39,7 +40,8 @@ insert-handleRequestFile → handleRequestFile ▼ #dispatchMethod(method, request) ← routes on the "method" field │ - ├─ getHealth / getNetwork / getLatestLedger / getTransaction / traceTransaction → #respond(...) + ├─ getHealth / getNetwork / getLatestLedger / getTransaction / + │ getTransactions / getLedgers / traceTransaction → #respond(...) │ └─ sendTransaction → #runTx → run steps → #finalizeTx → record receipt + bump ledger → #respond(...) @@ -62,6 +64,7 @@ If `request.json` is absent, `insert-handleRequestFile` does not fire and K halt - `getNetwork` → `{ "friendbotUrl": null, "passphrase": ..., "protocolVersion": ... }` (passphrase/version come from the request, keeping the semantics network-agnostic) - `getLatestLedger` → reads `metadata.json` and returns `{ "id": <64 zeros>, "protocolVersion": ..., "sequence": }` - `getTransaction` → reads the hash's `receipts/receipt_.json` file; returns the stored receipt merged with the current `latestLedger`/`latestLedgerCloseTime`, or `{ "status": "NOT_FOUND", ... }` when the file is absent +- `getTransactions` / `getLedgers` → walk the per-ledger index files `ledgers/ledger_.json` from the envelope's `startSeq` up to the latest ledger, taking at most `limit` records (`#txInfos` / `#ledgerInfos`), and format the history page (`#txHistoryPage` / `#ledgerHistoryPage`). Parameter validation and cursor resolution happen in the server; the response `cursor` (`#pageCursor`) names the page's last record when the page is full — a TOID for transactions, the plain sequence for ledgers — and is empty otherwise. Serialization matches real stellar-rpc, including its quirks: per-transaction `createdAt` is a JSON number (unlike singular `getTransaction`), per-ledger `ledgerCloseTime` is a decimal string, and empty `resultXdr`/`resultMetaXdr` stubs are omitted (`#optXdrEntry`) `#respond(ID, RESULT)` is the shared terminal: it writes the JSON-RPC envelope to `response.json`, removes `request.json`, and sets the exit code to 0. @@ -105,7 +108,8 @@ The trace is not part of the receipt — the executing steps already appended it - `#getJSON(key, obj[, default])`, `#getString(key, obj)`, `#getInt(key, obj)` — read a field - `#concatJSONs(a, b)` — append object entries (used to merge `latestLedger` fields into a stored receipt) -- `#receiptFile(hash)`, `#traceFile(hash)` — build the per-transaction file paths (`receipts/receipt_.json`, `traces/trace_.jsonl`) +- `#receiptFile(hash)`, `#traceFile(hash)`, `#ledgerFile(seq)` — build the per-item file paths (`receipts/receipt_.json`, `traces/trace_.jsonl`, `ledgers/ledger_.json`) +- `#readJSONFile(path)`, `#asInt(json)`, `#lengthJSONs(list)`, `#lastIntIn(key, list)` — small conveniences for the history methods These complement the **order-sensitive** step decoders below. diff --git a/docs/notes.md b/docs/notes.md index f1a0b3d..b358388 100644 --- a/docs/notes.md +++ b/docs/notes.md @@ -22,7 +22,7 @@ State lives in the io dir as `state.kore` (KORE world state) and `metadata.json` ## Tests (`src/tests/integration/`) -- `test_server.py` drives the running HTTP server end-to-end. It exercises the read-only methods, `sendTransaction` + `getTransaction`, ledger increments, the full lifecycle (create → upload wasm → deploy → invoke), and the `traceTransaction` flows. `test_call_tx_with_args` deploys `args.wat` and calls functions with `bool`, `u32`, `i32`, `u64`, `i64`, `u128`, `i128`, and `symbol` arguments, exercising the `scval_to_json` / `#decodeArg` pipeline. +- `test_server.py` drives the running HTTP server end-to-end. It exercises the read-only methods, `sendTransaction` + `getTransaction`, the history methods (`getTransactions`/`getLedgers` response shapes, pagination, and parameter validation against the official spec and the Go protocol structs), ledger increments, the full lifecycle (create → upload wasm → deploy → invoke), and the `traceTransaction` flows. `test_call_tx_with_args` deploys `args.wat` and calls functions with `bool`, `u32`, `i32`, `u64`, `i64`, `u128`, `i128`, and `symbol` arguments, exercising the `scval_to_json` / `#decodeArg` pipeline. - `test_integration.py` and `test_unit.py` hold small sanity checks. Run with `make test` (requires `make kdist-build` first). @@ -36,3 +36,4 @@ The tests do not yet cover `bytes` / `address` SCVal arguments or `SCVec` / `SCM - `resultXdr` / `resultMetaXdr` are empty stubs (contract return values not surfaced). - `SCVec` / `SCMap` contract arguments are not yet encoded. - `simulateTransaction`, `getEvents`, `getLedgerEntries`, `getFeeStats`, and TTL/footprint operations are not implemented. +- `getTransactions` / `getLedgers` serve only ledgers with an index file under `ledgers/`; io-dirs created before the ledger index existed resume fine, but their earlier ledgers do not appear in the history. diff --git a/docs/server.md b/docs/server.md index 0af41c5..690e076 100644 --- a/docs/server.md +++ b/docs/server.md @@ -14,6 +14,7 @@ class StellarRpcServer: state_file: Path # io_dir / 'state.kore' receipts_dir: Path # io_dir / 'receipts' — receipt_.json per transaction traces_dir: Path # io_dir / 'traces' — trace_.jsonl per transaction + ledgers_dir: Path # io_dir / 'ledgers' — ledger_.json per closed ledger requests_dir: Path # io_dir / 'requests' — request_.json archive ``` @@ -28,7 +29,7 @@ server = StellarRpcServer(io_dir=Path('out')) server.handle_rpc('sendTransaction', {'transaction': xdr}) ``` -For `sendTransaction` it builds the request envelope with `encoder.build_tx_request` and runs it with `interpreter.run`; for the read-only methods (`getHealth`, `getNetwork`, `getLatestLedger`, `getTransaction`, `traceTransaction`) it builds a small envelope and runs it. In every case the *content* of the response is produced by the semantics (`node.md`), not by Python — the one exception is the failure fallback (below). Each call is logged to stderr. +For `sendTransaction` it builds the request envelope with `encoder.build_tx_request` and runs it with `interpreter.run`; for the read-only methods (`getHealth`, `getNetwork`, `getLatestLedger`, `getTransaction`, `getTransactions`, `getLedgers`, `traceTransaction`) it builds a small envelope and runs it. For the history methods (`getTransactions`, `getLedgers`) the server also validates the pagination parameters — limit range, `startLedger` bounds, `cursor`/`startLedger` exclusivity — and resolves the cursor to the first ledger sequence to serve, because the JSON-RPC parameter-error path lives here. In every case the *content* of the response is produced by the semantics (`node.md`), not by Python — the exceptions are the failure fallback (below) and the per-ledger XDR artifacts (`_record_closed_ledger`), which K cannot construct. Each call is logged to stderr. --- @@ -51,7 +52,7 @@ At construction the server prepares the *io dir*, where `state.kore` lives at `i - **`state.kore` absent** — `interpreter.empty_config()` produces the initial idle K configuration (a blank-slate state with no accounts, contracts, or storage) and writes it; `metadata.json` is seeded with `{"latest_ledger": 0}`. - **`state.kore` present** — it is used as-is, and `metadata.json` is seeded only if missing. This lets you resume a previous session (ledger counter and stored receipts included) or start against a pre-built state. -In both cases the server creates the `receipts/`, `traces/`, and `requests/` directories if they do not already exist, because the K file-system hooks write into them but cannot create them. +In both cases the server creates the `receipts/`, `traces/`, `ledgers/`, and `requests/` directories if they do not already exist, because the K file-system hooks write into them but cannot create them. Once the socket is bound, `serve` logs three lines to stderr: whether it is starting from a fresh state (an empty io-dir) or resuming an existing one (with the latest ledger), the io-dir path, and the listening address. Instruction tracing is always on, so every transaction the semantics run produces a trace. (Tracing only produces records for contract invocations.) @@ -71,6 +72,8 @@ per successful transaction: write receipts/receipt_.json, bump latest_ledger in metadata.json, and write response.json → NodeInterpreter persists the new state.kore + → the server writes ledgers/ledger_.json (the ledger→tx index entry, + with the ledger-header XDR artifacts) per failed (stuck) transaction: → no response.json is produced; state.kore and metadata.json are left unchanged @@ -143,6 +146,42 @@ All methods are answered by the K semantics and follow the [Stellar RPC specific `resultXdr` and `resultMetaXdr` are currently empty stubs. The receipt carries no trace — use `traceTransaction` with the same hash to fetch it. +### `getTransactions` + +`getTransactions` returns the transactions in a ledger range, in chain order. Params: `startLedger` (number; mutually exclusive with a cursor), `pagination` `{cursor, limit}` (limit 1–200, default 50), and `xdrFormat` (`base64` only; `json` is rejected with `-32602`). The records are joined from the per-ledger index files (`ledgers/ledger_.json`) and the stored receipts. Failed transactions never close a ledger, so they do not appear in the history. + +```json +{ + "transactions": [ + { "status": "SUCCESS", "txHash": "<64-char hex>", "applicationOrder": 1, "feeBump": false, + "envelopeXdr": "", "ledger": 5, "createdAt": 1716000000 } + ], + "latestLedger": 5, "latestLedgerCloseTimestamp": 1716000000, + "oldestLedger": 0, "oldestLedgerCloseTimestamp": 0, + "cursor": "" +} +``` + +Serialization follows real stellar-rpc: ledger sequences and the top-level close times are JSON numbers, and per-transaction `createdAt` is a number too (unlike the singular `getTransaction`, where it is a string — an upstream quirk). `resultXdr`/`resultMetaXdr` are omitted while the receipts carry empty stubs. Each ledger holds exactly one transaction on this node, so `applicationOrder` is always `1`. The `cursor` is a TOID-style stringified integer (`ledger << 32 | applicationOrder << 12`) naming the page's last transaction when the page is full, and empty otherwise; resume by passing it as `pagination.cursor` (without `startLedger`). + +### `getLedgers` + +`getLedgers` returns the closed ledgers in a range and takes the same parameters as `getTransactions`. Each record comes straight from the ledger's index file; `headerXdr` (a `LedgerHeaderHistoryEntry`) and `metadataXdr` (a `LedgerCloseMeta`) are built by the server when the ledger closes, since K cannot construct XDR, and the ledger `hash` is the SHA-256 of the header XDR — unique per ledger and chained through `previousLedgerHash`. + +```json +{ + "ledgers": [ + { "hash": "<64-char hex>", "sequence": 5, "ledgerCloseTime": "1716000000", + "headerXdr": "", "metadataXdr": "" } + ], + "latestLedger": 5, "latestLedgerCloseTime": 1716000000, + "oldestLedger": 0, "oldestLedgerCloseTime": 0, + "cursor": "" +} +``` + +Per-ledger `ledgerCloseTime` is a *string* holding a decimal number (matching real stellar-rpc's Go `,string` encoding), while the top-level close times are numbers. The `cursor` is the last returned ledger sequence, stringified, under the same full-page rule as `getTransactions`. + --- ## Failure fallback @@ -161,4 +200,4 @@ komet-node [--host HOST] [--port PORT] [--io-dir DIR] |---|---|---| | `--host` | `localhost` | Bind address | | `--port` | `8000` | Port | -| `--io-dir` | a fresh temp dir | Directory holding every artifact (`state.kore`, `metadata.json`, `receipts/`, `traces/`, `requests/`) | +| `--io-dir` | a fresh temp dir | Directory holding every artifact (`state.kore`, `metadata.json`, `receipts/`, `traces/`, `ledgers/`, `requests/`) | diff --git a/src/komet_node/kdist/node.md b/src/komet_node/kdist/node.md index b8f1db2..515faa1 100644 --- a/src/komet_node/kdist/node.md +++ b/src/komet_node/kdist/node.md @@ -350,6 +350,207 @@ exists for that hash. requires notBool #fileExists( #traceFile( HASH ) ) ``` +############################################################################### +## getTransactions / getLedgers + +The history methods are served from the per-ledger index files `ledgers/ledger_.json` +that the Python server writes whenever a transaction closes a ledger. Each file carries the +ledger's `sequence`, the `txHash` of the transaction that closed it, its `closedAt` unix +time, and the ledger-header artifacts (`hash`, `headerXdr`, `metadataXdr`) — the latter are +XDR, which only Python can construct. Parameter validation (limit range, `startLedger` +bounds, `cursor`/`startLedger` exclusivity, resolving the cursor) also happens in the +server, because it needs the JSON-RPC error path; the envelope arrives here with a +validated `startSeq` (first ledger sequence to serve) and `limit`, and these rules only +collect the records and format the response. + +Serialization notes, matching real stellar-rpc (the Go protocol structs win over the +OpenRPC doc): + - ledger sequences and both top-level close-time fields are JSON numbers, and the + close-time keys differ between the methods (`latestLedgerCloseTimestamp` here, + `latestLedgerCloseTime` on getLedgers) — an upstream quirk, kept as is; + - per-transaction `createdAt` is a JSON number, unlike the singular getTransaction; + - per-ledger `ledgerCloseTime` is a string holding a decimal number (Go `,string`); + - `resultXdr`/`resultMetaXdr` are `omitempty`: omitted while receipts carry empty stubs; + - `cursor` names the page's last record when the page is full (a TOID for transactions: + `ledger << 32 | applicationOrder << 12`), and is empty otherwise. + +```k + rule #dispatchMethod( "getTransactions", REQ ) + => #respond( #getJSON( "id", REQ ), #getTransactionsResult( + #getInt( "startSeq", REQ ), + #getInt( "limit", REQ ), + #getInt( "latest_ledger", #readJSONFile( "metadata.json" ) ) + )) + ... + + + rule #dispatchMethod( "getLedgers", REQ ) + => #respond( #getJSON( "id", REQ ), #getLedgersResult( + #getInt( "startSeq", REQ ), + #getInt( "limit", REQ ), + #getInt( "latest_ledger", #readJSONFile( "metadata.json" ) ) + )) + ... + + + syntax JSON ::= #getTransactionsResult( Int, Int, Int ) [function, symbol(getTransactionsResult)] + | #txHistoryPage( JSONs, Int, Int ) [function, symbol(txHistoryPage)] + // ------------------------------------------------------------------------------------------------- + rule #getTransactionsResult( START, LIMIT, LL ) => #txHistoryPage( #txInfos( START, LIMIT, LL ), LIMIT, LL ) + + rule #txHistoryPage( TXS, LIMIT, LL ) => { + "transactions" : [ TXS ], + "latestLedger" : LL, + "latestLedgerCloseTimestamp" : #ledgerCloseTimeOf( LL ), + "oldestLedger" : 0, + "oldestLedgerCloseTimestamp" : 0, + "cursor" : #pageCursor( TXS, LIMIT, "ledger", 4294967296, 4096 ) + } + + syntax JSON ::= #getLedgersResult( Int, Int, Int ) [function, symbol(getLedgersResult)] + | #ledgerHistoryPage( JSONs, Int, Int ) [function, symbol(ledgerHistoryPage)] + // ------------------------------------------------------------------------------------------- + rule #getLedgersResult( START, LIMIT, LL ) => #ledgerHistoryPage( #ledgerInfos( START, LIMIT, LL ), LIMIT, LL ) + + rule #ledgerHistoryPage( LS, LIMIT, LL ) => { + "ledgers" : [ LS ], + "latestLedger" : LL, + "latestLedgerCloseTime" : #ledgerCloseTimeOf( LL ), + "oldestLedger" : 0, + "oldestLedgerCloseTime" : 0, + "cursor" : #pageCursor( LS, LIMIT, "sequence", 1, 0 ) + } +``` + +Both collectors walk the ledger sequence upwards from `startSeq` to the latest ledger, +taking at most `limit` records. Ledgers without an index file are skipped: the genesis +ledger 0 never has one, and neither do ledgers closed by io-dirs predating the index. +Failed transactions never closed a ledger, so they do not appear in the history. + +```k + syntax JSONs ::= #txInfos( Int, Int, Int ) [function, symbol(txInfos)] + // ---------------------------------------------------------------------- + rule #txInfos( _, LIMIT, _ ) => .JSONs requires LIMIT <=Int 0 + rule #txInfos( SEQ, LIMIT, LL ) => .JSONs requires LIMIT >Int 0 andBool SEQ >Int LL + rule #txInfos( SEQ, LIMIT, LL ) => #txInfos( SEQ +Int 1, LIMIT, LL ) + requires LIMIT >Int 0 andBool SEQ <=Int LL andBool notBool #fileExists( #ledgerFile( SEQ ) ) + rule #txInfos( SEQ, LIMIT, LL ) + => ( #txInfoOf( SEQ, #getString( "txHash", #readJSONFile( #ledgerFile( SEQ ) ) ) ) + , #txInfos( SEQ +Int 1, LIMIT -Int 1, LL ) ) + requires LIMIT >Int 0 andBool SEQ <=Int LL andBool #fileExists( #ledgerFile( SEQ ) ) + + syntax JSONs ::= #ledgerInfos( Int, Int, Int ) [function, symbol(ledgerInfos)] + // ------------------------------------------------------------------------------ + rule #ledgerInfos( _, LIMIT, _ ) => .JSONs requires LIMIT <=Int 0 + rule #ledgerInfos( SEQ, LIMIT, LL ) => .JSONs requires LIMIT >Int 0 andBool SEQ >Int LL + rule #ledgerInfos( SEQ, LIMIT, LL ) => #ledgerInfos( SEQ +Int 1, LIMIT, LL ) + requires LIMIT >Int 0 andBool SEQ <=Int LL andBool notBool #fileExists( #ledgerFile( SEQ ) ) + rule #ledgerInfos( SEQ, LIMIT, LL ) + => ( #ledgerInfoOf( #readJSONFile( #ledgerFile( SEQ ) ) ) + , #ledgerInfos( SEQ +Int 1, LIMIT -Int 1, LL ) ) + requires LIMIT >Int 0 andBool SEQ <=Int LL andBool #fileExists( #ledgerFile( SEQ ) ) +``` + +One transaction record, in the field set and order of the Go `TransactionInfo` struct. +The receipt provides the status, the envelope, and the creation time; the ledger sequence +comes from the index. Each ledger holds exactly one transaction on this node, so +`applicationOrder` is always 1, and fee-bump envelopes are not supported, so `feeBump` is +always false. + +```k + syntax JSON ::= #txInfoOf( Int, String ) [function, symbol(txInfoOf)] + | #txInfoFrom( Int, String, JSON ) [function, symbol(txInfoFrom)] + // ------------------------------------------------------------------------------- + rule #txInfoOf( SEQ, HASH ) => #txInfoFrom( SEQ, HASH, #readJSONFile( #receiptFile( HASH ) ) ) + + rule #txInfoFrom( SEQ, HASH, RCPT ) => { #concatJSONs( + ( "status" : #getJSON( "status", RCPT ), + "txHash" : HASH, + "applicationOrder" : 1, + "feeBump" : false, + "envelopeXdr" : #getJSON( "envelopeXdr", RCPT ), + .JSONs ), + #concatJSONs( + #optXdrEntry( "resultXdr", #getJSON( "resultXdr", RCPT, "" ) ), + #concatJSONs( + #optXdrEntry( "resultMetaXdr", #getJSON( "resultMetaXdr", RCPT, "" ) ), + ( "ledger" : SEQ, + "createdAt" : #asInt( #getJSON( "createdAt", RCPT ) ), + .JSONs ) + ) + ) + )} + + // An entry for an optional (`omitempty`) XDR field: omitted when the stored value is + // an empty stub or the receipt predates the field. + syntax JSONs ::= #optXdrEntry( JSONKey, JSON ) [function, symbol(optXdrEntry)] + // ------------------------------------------------------------------------------ + rule #optXdrEntry( _, "" ) => .JSONs + rule #optXdrEntry( _, null ) => .JSONs + rule #optXdrEntry( KEY, V ) => ( KEY : V, .JSONs ) [owise] + + // One ledger record, straight from the index file. + syntax JSON ::= #ledgerInfoOf( JSON ) [function, symbol(ledgerInfoOf)] + // ---------------------------------------------------------------------- + rule #ledgerInfoOf( L ) => { + "hash" : #getJSON( "hash", L ), + "sequence" : #asInt( #getJSON( "sequence", L ) ), + "ledgerCloseTime" : Int2String( #asInt( #getJSON( "closedAt", L ) ) ), + "headerXdr" : #getJSON( "headerXdr", L ), + "metadataXdr" : #getJSON( "metadataXdr", L ) + } +``` + +Shared helpers for the history methods. + +```k + syntax String ::= #ledgerFile( Int ) [function, symbol(ledgerFile)] + // ------------------------------------------------------------------- + rule #ledgerFile( SEQ ) => "ledgers/ledger_" +String Int2String(SEQ) +String ".json" + + syntax JSON ::= #readJSONFile( String ) [function, symbol(readJSONFile)] + // ------------------------------------------------------------------------ + rule #readJSONFile( FILE ) => String2JSON( {#readFile( FILE )}:>String ) + + // Coerce a JSON number-or-decimal-string to Int (receipts store some ints as strings). + syntax Int ::= #asInt( JSON ) [function, symbol(asInt)] + // ------------------------------------------------------- + rule #asInt( I:Int ) => I + rule #asInt( S:String ) => String2Int( S ) + + // The close time recorded for a ledger, or 0 when it has no index file (genesis). + syntax Int ::= #ledgerCloseTimeOf( Int ) [function, symbol(ledgerCloseTimeOf)] + // ------------------------------------------------------------------------------ + rule #ledgerCloseTimeOf( SEQ ) => #asInt( #getJSON( "closedAt", #readJSONFile( #ledgerFile( SEQ ) ) ) ) + requires #fileExists( #ledgerFile( SEQ ) ) + rule #ledgerCloseTimeOf( SEQ ) => 0 + requires notBool #fileExists( #ledgerFile( SEQ ) ) + + // The paging token to return: the position of the page's last record when the page is + // full (more records may follow), the empty string otherwise. The position is the value + // of KEY in the last record, scaled as POS *Int MULT +Int OFFSET — the TOID encoding + // for getTransactions (MULT 2^32, OFFSET 1 << 12), the plain sequence for getLedgers + // (MULT 1, OFFSET 0). The server resolves cursors back to a start sequence. + syntax JSON ::= #pageCursor( JSONs, Int, JSONKey, Int, Int ) [function, symbol(pageCursor)] + // ------------------------------------------------------------------------------------------- + rule #pageCursor( RECORDS, LIMIT, KEY, MULT, OFFSET ) + => Int2String( #lastIntIn( KEY, RECORDS ) *Int MULT +Int OFFSET ) + requires #lengthJSONs( RECORDS ) ==Int LIMIT + rule #pageCursor( RECORDS, LIMIT, _, _, _ ) => "" + requires #lengthJSONs( RECORDS ) =/=Int LIMIT + + syntax Int ::= #lengthJSONs( JSONs ) [function, symbol(lengthJSONs)] + // -------------------------------------------------------------------- + rule #lengthJSONs( .JSONs ) => 0 + rule #lengthJSONs( ( _, REST ) ) => 1 +Int #lengthJSONs( REST ) + + // The value of KEY in the last object of a non-empty list, as an Int. + syntax Int ::= #lastIntIn( JSONKey, JSONs ) [function, symbol(lastIntIn)] + // ------------------------------------------------------------------------- + rule #lastIntIn( KEY, ( J:JSON, .JSONs ) ) => #getInt( KEY, J ) + rule #lastIntIn( KEY, ( _:JSON, J:JSON, REST:JSONs ) ) => #lastIntIn( KEY, ( J, REST ) ) +``` + ############################################################################### # Step decoding diff --git a/src/komet_node/ledger.py b/src/komet_node/ledger.py new file mode 100644 index 0000000..cc27206 --- /dev/null +++ b/src/komet_node/ledger.py @@ -0,0 +1,76 @@ +"""Construction of the per-ledger XDR artifacts served by ``getLedgers``. + +K cannot build XDR, so when a transaction closes a ledger the server materialises the +ledger's header artifacts here and stores them in ``ledgers/ledger_.json`` for the +history methods (``getLedgers``/``getTransactions``) to read back. +""" + +from __future__ import annotations + +import hashlib +from typing import Final + +from stellar_sdk import xdr + +_PROTOCOL_VERSION: Final = 22 +_ZERO_HASH: Final = b'\x00' * 32 + + +def build_ledger_artifacts( + sequence: int, + close_time: int, + previous_hash: bytes, + envelope_xdr: str, +) -> tuple[str, str, str]: + """Build the ``(hash, headerXdr, metadataXdr)`` artifacts for a newly closed ledger. + + ``headerXdr`` is a base64 ``LedgerHeaderHistoryEntry`` and ``metadataXdr`` a base64 + ``LedgerCloseMeta`` (v0), per the getLedgers spec. komet-node has no consensus, + buckets, or fee pool, so every header field except the sequence, close time, + previous-ledger hash, and the transaction set is zeroed. The ledger hash is the + SHA-256 of the header XDR — unique per ledger and chained to the previous ledger + through ``previous_ledger_hash`` — which is enough for clients that treat it as an + opaque identifier. + """ + header = xdr.LedgerHeader( + ledger_version=xdr.Uint32(_PROTOCOL_VERSION), + previous_ledger_hash=xdr.Hash(previous_hash), + scp_value=xdr.StellarValue( + tx_set_hash=xdr.Hash(_ZERO_HASH), + close_time=xdr.TimePoint(xdr.Uint64(close_time)), + upgrades=[], + ext=xdr.StellarValueExt(v=xdr.StellarValueType.STELLAR_VALUE_BASIC), + ), + tx_set_result_hash=xdr.Hash(_ZERO_HASH), + bucket_list_hash=xdr.Hash(_ZERO_HASH), + ledger_seq=xdr.Uint32(sequence), + total_coins=xdr.Int64(0), + fee_pool=xdr.Int64(0), + inflation_seq=xdr.Uint32(0), + id_pool=xdr.Uint64(0), + base_fee=xdr.Uint32(100), + base_reserve=xdr.Uint32(5_000_000), + max_tx_set_size=xdr.Uint32(1), + skip_list=[xdr.Hash(_ZERO_HASH)] * 4, + ext=xdr.LedgerHeaderExt(v=0), + ) + ledger_hash = hashlib.sha256(header.to_xdr_bytes()).digest() + entry = xdr.LedgerHeaderHistoryEntry( + hash=xdr.Hash(ledger_hash), + header=header, + ext=xdr.LedgerHeaderHistoryEntryExt(v=0), + ) + meta = xdr.LedgerCloseMeta( + v=0, + v0=xdr.LedgerCloseMetaV0( + ledger_header=entry, + tx_set=xdr.TransactionSet( + previous_ledger_hash=xdr.Hash(previous_hash), + txs=[xdr.TransactionEnvelope.from_xdr(envelope_xdr)], + ), + tx_processing=[], + upgrades_processing=[], + scp_info=[], + ), + ) + return ledger_hash.hex(), entry.to_xdr(), meta.to_xdr() diff --git a/src/komet_node/server.py b/src/komet_node/server.py index 6399575..280aa30 100644 --- a/src/komet_node/server.py +++ b/src/komet_node/server.py @@ -13,6 +13,7 @@ from stellar_sdk import Network from komet_node.interpreter import NodeInterpreter +from komet_node.ledger import build_ledger_artifacts from komet_node.transaction import TransactionEncoder if TYPE_CHECKING: @@ -20,6 +21,13 @@ _PROTOCOL_VERSION: Final = '22' +# Pagination bounds for the history methods (getTransactions/getLedgers), per the spec: +# limit ranges from 1 to 200 and defaults to 50. +_MAX_PAGE_LIMIT: Final = 200 +_DEFAULT_PAGE_LIMIT: Final = 50 +# getTransactions cursors are TOID-style: (ledger sequence << 32) | (application order << 12). +_TOID_LEDGER_SHIFT: Final = 32 + # Only sendTransaction executes a transaction. traceTransaction is a read-only lookup of the # trace stored on a previously executed transaction's receipt (see _read_only_envelope). _TX_METHODS: Final = ('sendTransaction',) @@ -43,10 +51,12 @@ class StellarRpcServer: - ``metadata.json`` — ``{"latest_ledger": N}`` - ``receipts/receipt_.json`` — one stored receipt per transaction - ``traces/trace_.jsonl`` — one execution trace per transaction + - ``ledgers/ledger_.json`` — one record per closed ledger (tx hash, close + time, and the ledger-header XDR artifacts), serving getTransactions/getLedgers - ``requests/request_.json`` — an archive of each incoming JSON-RPC request - Splitting receipts, traces, and requests into per-item files keeps any single file from - growing without bound as the chain advances. + Splitting receipts, traces, ledgers, and requests into per-item files keeps any single + file from growing without bound as the chain advances. """ interpreter: NodeInterpreter @@ -85,8 +95,9 @@ def __init__( # directories, so the directories must exist before the semantics run. self.receipts_dir = self.io_dir / 'receipts' self.traces_dir = self.io_dir / 'traces' + self.ledgers_dir = self.io_dir / 'ledgers' self.requests_dir = self.io_dir / 'requests' - for directory in (self.receipts_dir, self.traces_dir, self.requests_dir): + for directory in (self.receipts_dir, self.traces_dir, self.ledgers_dir, self.requests_dir): directory.mkdir(exist_ok=True) # Continue the request archive numbering past anything a previous run left behind, so # resuming an io-dir never overwrites its earlier request files. @@ -198,6 +209,7 @@ def handle_rpc(self, method: str | None, params: dict[str, Any], request_id: Any response = self.interpreter.run(self.state_file, self.io_dir, envelope, program_steps) if response is None: return json.dumps(self._failure_response(request_id, envelope, now)) + self._record_closed_ledger(envelope, now) return response read_only_envelope = self._read_only_envelope(method, params, request_id, now) @@ -230,8 +242,96 @@ def _read_only_envelope( if not isinstance(tx_hash, str): return _error_str(request_id, -32602, "Invalid params: 'hash' (string) is required") return {**base, 'hash': tx_hash} + if method in ('getTransactions', 'getLedgers'): + return self._history_envelope(method, params, request_id, base) return None + def _history_envelope( + self, method: str, params: dict[str, Any], request_id: Any, base: dict[str, Any] + ) -> dict[str, Any] | str: + """Validate getTransactions/getLedgers params and build the request envelope. + + Parameter validation lives here rather than in K because it needs the JSON-RPC + error path and the latest-ledger bound, both of which the server already owns for + the other methods. The envelope carries the resolved first ledger sequence to + serve (``startSeq``) and the page ``limit``; the semantics collect the records + and format the response. + """ + + def invalid(message: str) -> str: + return _error_str(request_id, -32602, f'Invalid params: {message}') + + xdr_format = params.get('xdrFormat', 'base64') + if xdr_format == 'json': + return invalid("xdrFormat 'json' is not supported; use 'base64'") + if xdr_format not in ('', 'base64'): + return invalid("'xdrFormat' must be 'base64'") + + pagination = params.get('pagination') or {} + if not isinstance(pagination, dict): + return invalid("'pagination' must be an object") + limit = pagination.get('limit', _DEFAULT_PAGE_LIMIT) + if not _is_int(limit) or not 1 <= limit <= _MAX_PAGE_LIMIT: + return invalid(f"'limit' must be an integer between 1 and {_MAX_PAGE_LIMIT}") + + cursor = pagination.get('cursor') + start_ledger = params.get('startLedger') + if cursor is not None: + if start_ledger is not None: + return invalid("'startLedger' and 'cursor' are mutually exclusive") + if not isinstance(cursor, str) or not cursor.isdigit(): + return invalid("'cursor' must be a cursor string from a previous response") + # The cursor names the last record already returned: a TOID for transactions + # (ledger << 32 | order << 12; one transaction per ledger here), the plain + # ledger sequence for ledgers. Either way the next page starts one ledger on. + position = int(cursor) + if method == 'getTransactions': + position >>= _TOID_LEDGER_SHIFT + start_seq = position + 1 + else: + if start_ledger is None: + start_ledger = 0 # like real stellar-rpc: unset means "from the oldest ledger" + if not _is_int(start_ledger): + return invalid("'startLedger' must be a number") + latest_ledger = self._latest_ledger() + if start_ledger != 0 and not 0 <= start_ledger <= latest_ledger: + return invalid( + f"'startLedger' must be between the oldest ledger 0 and the latest ledger {latest_ledger}" + ) + start_seq = start_ledger + return {**base, 'startSeq': start_seq, 'limit': limit} + + def _latest_ledger(self) -> int: + metadata = json.loads((self.io_dir / 'metadata.json').read_text()) + return int(metadata.get('latest_ledger', 0)) + + def _record_closed_ledger(self, envelope: dict[str, Any], now: str) -> None: + """Materialise ``ledgers/ledger_.json`` for the ledger a transaction just closed. + + The K semantics own the ledger counter and receipts; this per-ledger record + additionally carries the ledger-header artifacts (``hash``, ``headerXdr``, + ``metadataXdr``) that only Python can construct (XDR), and is what the semantics + read to serve getTransactions and getLedgers. Written only on the success path — + a failed transaction closes no ledger. + """ + sequence = self._latest_ledger() # the semantics bumped it just before responding + previous_hash = b'\x00' * 32 # the genesis ledger's hash + previous_file = self.ledgers_dir / f'ledger_{sequence - 1}.json' + if previous_file.exists(): + previous_hash = bytes.fromhex(json.loads(previous_file.read_text())['hash']) + ledger_hash, header_xdr, metadata_xdr = build_ledger_artifacts( + sequence, int(now), previous_hash, envelope['envelopeXdr'] + ) + record = { + 'sequence': sequence, + 'txHash': envelope['txHash'], + 'closedAt': int(now), + 'hash': ledger_hash, + 'headerXdr': header_xdr, + 'metadataXdr': metadata_xdr, + } + (self.ledgers_dir / f'ledger_{sequence}.json').write_text(json.dumps(record)) + def _archive_request(self, method: str | None, params: dict[str, Any], request_id: Any) -> None: """Write each incoming JSON-RPC call to its own ``requests/request_.json`` file. @@ -250,8 +350,7 @@ def _failure_response(self, rpc_id: Any, envelope: dict[str, Any], now: str) -> world state is left unchanged. We record a FAILED receipt so a later getTransaction finds it, without bumping the ledger. """ - metadata = json.loads((self.io_dir / 'metadata.json').read_text()) - ledger = metadata.get('latest_ledger', 0) + ledger = self._latest_ledger() tx_hash = envelope['txHash'] # This FAILED receipt mirrors the SUCCESS receipt the semantics build in @@ -276,6 +375,11 @@ def _failure_response(self, rpc_id: Any, envelope: dict[str, Any], now: str) -> return {'jsonrpc': '2.0', 'id': rpc_id, 'result': result} +def _is_int(value: Any) -> bool: + """True for actual JSON numbers only — bool is an int subclass in Python, so exclude it.""" + return isinstance(value, int) and not isinstance(value, bool) + + def _next_request_index(requests_dir: Path) -> int: """Return the next free index for ``requests/request_.json``. diff --git a/src/tests/integration/test_server.py b/src/tests/integration/test_server.py index 3e5cfc3..d1de08a 100644 --- a/src/tests/integration/test_server.py +++ b/src/tests/integration/test_server.py @@ -498,3 +498,211 @@ def builder() -> TransactionBuilder: # All four transactions, including the non-Void invocation, advanced the ledger. assert _rpc(server.port(), 'getLatestLedger', {})['result']['sequence'] == 4 + + +# --------------------------------------------------------------------------- +# getTransactions / getLedgers (transaction history) +# +# Ground truth: the official OpenRPC spec (stellar-docs, methods/getTransactions.json and +# methods/getLedgers.json) and the Go serialization structs from stellar/go-stellar-sdk +# protocols/rpc, which is what real stellar-rpc emits. Notable serialization traps asserted +# below: +# - ledger sequences and the top-level close-time fields are JSON numbers, +# - per-transaction `createdAt` in getTransactions is a JSON *number* (upstream quirk; +# the singular getTransaction returns it as a string), +# - per-ledger `ledgerCloseTime` in getLedgers is a *string* (Go int64 `,string`), +# - XDR fields are `omitempty`: real base64 XDR or absent, never empty strings. +# --------------------------------------------------------------------------- + + +def _rpc_result(port: int, method: str, params: dict[str, Any]) -> dict[str, Any]: + """Call an RPC method and return its result, failing the test on a JSON-RPC error.""" + response = _rpc(port, method, params) + assert 'error' not in response, f'{method} returned an error: {response["error"]}' + return response['result'] + + +def _is_hex64(value: Any) -> bool: + return isinstance(value, str) and len(value) == 64 and all(c in '0123456789abcdef' for c in value) + + +def _send_create_accounts(server: StellarRpcServer, count: int) -> list[tuple[str, str]]: + """Submit ``count`` successful create-account transactions; return (hash, envelopeXdr) pairs. + + Each successful transaction closes its own ledger, so after this call the latest ledger + is ``count`` and transaction ``i`` (1-based) sits alone in ledger ``i``. + """ + keypair = Keypair.random() + account = Account(keypair.public_key, sequence=0) + sent: list[tuple[str, str]] = [] + for _ in range(count): + envelope = ( + TransactionBuilder(account, Network.TESTNET_NETWORK_PASSPHRASE) + .append_create_account_op(destination=keypair.public_key, starting_balance='1000') + .set_timeout(30) + .build() + ) + envelope.sign(keypair) + xdr_str = envelope.to_xdr() + send_res = _rpc(server.port(), 'sendTransaction', {'transaction': xdr_str}) + assert send_res['result']['status'] == 'PENDING' + tx_hash = send_res['result']['hash'] + assert _rpc(server.port(), 'getTransaction', {'hash': tx_hash})['result']['status'] == 'SUCCESS' + sent.append((tx_hash, xdr_str)) + return sent + + +def test_get_transactions_spec_shape(server: StellarRpcServer) -> None: + """getTransactions returns the response shape of GetTransactionsResponse (Go SDK).""" + before = int(time.time()) + sent = _send_create_accounts(server, 3) + after = int(time.time()) + + result = _rpc_result(server.port(), 'getTransactions', {'startLedger': 1}) + + # All six top-level fields lack `omitempty` in the Go struct, so all must be present. + required_keys = { + 'transactions', + 'latestLedger', + 'latestLedgerCloseTimestamp', + 'oldestLedger', + 'oldestLedgerCloseTimestamp', + 'cursor', + } + assert required_keys <= result.keys(), f'missing keys: {required_keys - result.keys()}' + + assert type(result['latestLedger']) is int + assert result['latestLedger'] == 3 + assert type(result['latestLedgerCloseTimestamp']) is int + assert type(result['oldestLedger']) is int + assert 0 <= result['oldestLedger'] <= 1 + assert type(result['oldestLedgerCloseTimestamp']) is int + assert isinstance(result['cursor'], str) + + txs = result['transactions'] + assert isinstance(txs, list) + # All three transactions, in chain order (ascending ledger, then application order). + assert [tx['txHash'] for tx in txs] == [tx_hash for tx_hash, _ in sent] + for i, tx in enumerate(txs, start=1): + assert tx['status'] == 'SUCCESS' + assert _is_hex64(tx['txHash']) + assert type(tx['applicationOrder']) is int + assert tx['applicationOrder'] == 1 # one transaction per ledger on this node + assert tx['feeBump'] is False + assert type(tx['ledger']) is int + assert tx['ledger'] == i + # Upstream quirk: createdAt is a JSON number here (int64 without `,string` in Go), + # unlike getTransaction (singular) where it is a string. + assert type(tx['createdAt']) is int + assert before <= tx['createdAt'] <= after + assert tx['envelopeXdr'] == sent[i - 1][1] + # omitempty: XDR fields carry real base64 XDR or are absent — never empty strings. + for optional in ('resultXdr', 'resultMetaXdr'): + if optional in tx: + assert isinstance(tx[optional], str) and tx[optional] != '' + + # xdrFormat: 'base64' is the default and must be accepted; unknown values are rejected. + with_format = _rpc_result(server.port(), 'getTransactions', {'startLedger': 1, 'xdrFormat': 'base64'}) + assert [tx['txHash'] for tx in with_format['transactions']] == [tx_hash for tx_hash, _ in sent] + assert _rpc(server.port(), 'getTransactions', {'startLedger': 1, 'xdrFormat': 'bogus'})['error']['code'] == -32602 + + # The limit for getTransactions ranges from 1 to 200. + bad_limit = _rpc(server.port(), 'getTransactions', {'startLedger': 1, 'pagination': {'limit': 201}}) + assert bad_limit['error']['code'] == -32602 + + +def test_get_transactions_pagination(server: StellarRpcServer) -> None: + """A limited page returns a cursor from which the next page resumes without overlap.""" + sent = _send_create_accounts(server, 3) + + page1 = _rpc_result(server.port(), 'getTransactions', {'startLedger': 1, 'pagination': {'limit': 2}}) + assert [tx['txHash'] for tx in page1['transactions']] == [sent[0][0], sent[1][0]] + cursor = page1['cursor'] + assert isinstance(cursor, str) and cursor != '' + + # Resume from the cursor; startLedger must be omitted on cursor requests. + page2 = _rpc_result(server.port(), 'getTransactions', {'pagination': {'cursor': cursor, 'limit': 2}}) + assert [tx['txHash'] for tx in page2['transactions']] == [sent[2][0]] + + +def test_get_transactions_invalid_params(server: StellarRpcServer) -> None: + port = server.port() + # startLedger beyond the latest ledger (0 on a fresh chain) is out of retention range. + assert _rpc(port, 'getTransactions', {'startLedger': 999})['error']['code'] == -32602 + # startLedger and cursor are mutually exclusive. + both = _rpc(port, 'getTransactions', {'startLedger': 1, 'pagination': {'cursor': '1'}}) + assert both['error']['code'] == -32602 + # startLedger must be a number. + assert _rpc(port, 'getTransactions', {'startLedger': 'one'})['error']['code'] == -32602 + + +def test_get_ledgers_spec_shape(server: StellarRpcServer) -> None: + """getLedgers returns the response shape of GetLedgersResponse (Go SDK).""" + _send_create_accounts(server, 2) + + result = _rpc_result(server.port(), 'getLedgers', {'startLedger': 1}) + + required_keys = { + 'ledgers', + 'latestLedger', + 'latestLedgerCloseTime', + 'oldestLedger', + 'oldestLedgerCloseTime', + 'cursor', + } + assert required_keys <= result.keys(), f'missing keys: {required_keys - result.keys()}' + + assert type(result['latestLedger']) is int + assert result['latestLedger'] == 2 + assert type(result['latestLedgerCloseTime']) is int + assert type(result['oldestLedger']) is int + assert 0 <= result['oldestLedger'] <= 1 + assert type(result['oldestLedgerCloseTime']) is int + assert isinstance(result['cursor'], str) + + ledgers = result['ledgers'] + assert isinstance(ledgers, list) + assert [ledger['sequence'] for ledger in ledgers] == [1, 2] + hashes = set() + for ledger in ledgers: + assert type(ledger['sequence']) is int + assert _is_hex64(ledger['hash']) + hashes.add(ledger['hash']) + # Per-ledger close time is a STRING containing a decimal number (Go int64 `,string`). + assert isinstance(ledger['ledgerCloseTime'], str) + assert ledger['ledgerCloseTime'].isdigit() + # headerXdr is a base64 LedgerHeaderHistoryEntry for this ledger. + header = xdr.LedgerHeaderHistoryEntry.from_xdr(ledger['headerXdr']) + assert header.header.ledger_seq.uint32 == ledger['sequence'] + # metadataXdr is a base64 LedgerCloseMeta union for this ledger. + meta = xdr.LedgerCloseMeta.from_xdr(ledger['metadataXdr']) + meta_header = getattr(meta, f'v{meta.v}').ledger_header + assert meta_header.header.ledger_seq.uint32 == ledger['sequence'] + # Ledger hashes identify ledgers and must be unique. + assert len(hashes) == 2 + + # The limit for getLedgers ranges from 1 to 200. + bad_limit = _rpc(server.port(), 'getLedgers', {'startLedger': 1, 'pagination': {'limit': 201}}) + assert bad_limit['error']['code'] == -32602 + + +def test_get_ledgers_pagination(server: StellarRpcServer) -> None: + """A limited page returns a cursor from which the next page resumes without overlap.""" + _send_create_accounts(server, 2) + + page1 = _rpc_result(server.port(), 'getLedgers', {'startLedger': 1, 'pagination': {'limit': 1}}) + assert [ledger['sequence'] for ledger in page1['ledgers']] == [1] + cursor = page1['cursor'] + assert isinstance(cursor, str) and cursor != '' + + page2 = _rpc_result(server.port(), 'getLedgers', {'pagination': {'cursor': cursor, 'limit': 1}}) + assert [ledger['sequence'] for ledger in page2['ledgers']] == [2] + + +def test_get_ledgers_invalid_params(server: StellarRpcServer) -> None: + port = server.port() + # startLedger beyond the latest ledger (0 on a fresh chain) is out of retention range. + assert _rpc(port, 'getLedgers', {'startLedger': 999})['error']['code'] == -32602 + # startLedger and cursor are mutually exclusive. + both = _rpc(port, 'getLedgers', {'startLedger': 1, 'pagination': {'cursor': '1'}}) + assert both['error']['code'] == -32602