Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion cmd/livepeer/testdata/cli/redeemer-help.golden
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ Flags:
-k, --keystore-file string Encrypted geth redemption account JSON file (env: LIVEPEER_REDEEMER_KEYSTORE_FILE)
--keystore-password-file string Owner-only file containing exact password bytes (env: LIVEPEER_REDEEMER_KEYSTORE_PASSWORD_FILE)
--rpc-url-file string (env: LIVEPEER_REDEEMER_RPC_URL_FILE)
--auth-webhook-file string (env: LIVEPEER_REDEEMER_AUTH_WEBHOOK_FILE)
--auth-webhook-headers-file string (env: LIVEPEER_REDEEMER_AUTH_WEBHOOK_HEADERS_FILE)
--chain-id uint (env: LIVEPEER_REDEEMER_CHAIN_ID)
--controller-address string (env: LIVEPEER_REDEEMER_CONTROLLER_ADDRESS)
--redeemer-tx-timeout duration Transaction inclusion timeout (env: LIVEPEER_REDEEMER_REDEEMER_TX_TIMEOUT) (default 5m0s)
Expand All @@ -35,5 +37,7 @@ Flags:
Use "livepeer-redeemer [command] --help" for more information about a command.

Environment Variables:
LIVEPEER_REDEEMER_KEYSTORE_PASSWORD Keystore decryption password
LIVEPEER_REDEEMER_KEYSTORE_PASSWORD Keystore decryption password
LIVEPEER_REDEEMER_RPC_URL
LIVEPEER_REDEEMER_AUTH_WEBHOOK
LIVEPEER_REDEEMER_AUTH_WEBHOOK_HEADERS
3 changes: 3 additions & 0 deletions configs/redeemer/config.example.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ RedeemerDB = "tickets.sqlite"
KeystoreFile = "/run/secrets/livepeer-redeemer-keystore.json"
KeystorePasswordFile = "/run/secrets/livepeer-redeemer-password"
RPCURLFile = "/run/secrets/livepeer-rpc-url"
# Optional per-request authorization; sends request headers and the ticket.
# AuthWebhookFile = "/run/secrets/livepeer-redeemer-auth-url"
# AuthWebhookHeadersFile = "/run/secrets/livepeer-redeemer-auth-headers"
# Files containing credentials must be owner-only. Password bytes are exact.
# ChainID = 42161
# Controller = "0xD8E8328501E9645d16Cf49539efC04f734606ee4"
Expand Down
1 change: 1 addition & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ openssl rand -hex 32 | tr -d '\n' > runner-bootstrap.secret
Set `BootstrapSecretFile` to its absolute path and use the same credential for
initial runner registration. Secret-manager mounts must preserve exact bytes;
URL files require a valid URL without a trailing newline.
Redeemer webhook URL and headers files must be owner-only regular files.

## Ethereum keystores

Expand Down
30 changes: 30 additions & 0 deletions docs/http-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,36 @@ parameters. This field is not covered by the payer's signature; submitters must
preserve the value issued by the orchestrator. Protect ticket submissions with
TLS and endpoint authentication in your deployment; see [redeemer setup](redeemer.md).

### Redeemer authorization webhook

When [configured](redeemer.md#authorization-webhook), the redeemer checks each submitted ticket.
It sends the webhook a JSON POST, for example:

```json
{
"headers": {"Authorization": ["Bearer caller-credential"]},
"ticket": {"sig": "0x...", "recipient_rand_hash": "0x..."}
}
```

The example shows selected [ticket fields](#redeemer-http). The webhook receives
all those fields except `recipient_rand`.

The webhook must return HTTP 200 with a JSON decision, for example:

```json
{"status": 200}
```

| Decision `status` | Result | Client response |
| --- | --- | --- |
| 200 | Approve the ticket. | 202 Accepted if validation and storage succeed. |
| 409 | Deny the ticket. | 403 Forbidden |
| Other 400–599 | Deny the ticket. | Same status |

A redirect, timeout, non-200 response, or invalid decision returns 502. The
timeout is five seconds.

## Accounting events

Optional Kafka accounting emits `create_signed_ticket` JSON events with
Expand Down
1 change: 1 addition & 0 deletions docs/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ gauges; timestamps use Unix seconds. Most event families appear on first use.
| Payments | `payment_operations_total{operation,outcome}` for challenge/receive/refresh/charge; `payment_operation_duration_seconds{operation}` for challenge/receive/refresh; `payment_tickets_total{kind="accepted\|winning"}`, `payment_credit_eth_total`, `payment_charge_eth_total`, `payment_billed_seconds_total`, `payment_fixed_charges_total`, with app identity. Credit is accepted expected ticket value; charges are successfully debited service usage. |
| Signer | `signer_payments_total{type,outcome}`, `signer_payment_duration_seconds{type}`, `signer_tickets_total{type}`, `signer_sign_operations_total{kind,outcome}`. Signing kinds are ticket, segment, state, and orchestrator_info. Partial signing before a failed request remains visible. Successful payment/ticket totals follow required outbox persistence, even if response delivery is interrupted. |
| Authorization | `signer_auth_webhook_requests_total{status,outcome}`, `signer_auth_webhook_duration_seconds`, `signer_auth_cache_hits_total`. Status zero means no webhook response; outcomes distinguish success, denied, and error. |
| Redeemer authorization | `redeemer_auth_webhook_requests_total{status,outcome}`, `redeemer_auth_webhook_duration_seconds`. Status is the upstream HTTP status, or zero without a response; outcomes are `success`, `denied`, and `error`. |
| Outbox | Existing `signer_kafka_pending_events`, `signer_kafka_pending_bytes`, `signer_kafka_oldest_event_age_seconds`, `signer_kafka_outbox_healthy`, `signer_kafka_publish_errors_total`, `signer_kafka_enqueue_errors_total`, and `signer_kafka_storage_errors_total`. New `signer_kafka_delivered_events_total` and `signer_kafka_last_delivery_timestamp_seconds` follow successful local acknowledgement; delivery remains at least once. |
| Redemption queue | `redemption_queue_tickets{state}`, `redemption_oldest_ticket_timestamp_seconds{state}`: parameter-expiry `waiting`, clock-eligible `eligible`, `in_flight`, `awaiting_finality`, rejected remote submissions (`rejected`), invalid or reverted local attempts (`blocked`), or `unknown` without a fresh clock observation. Each logical ticket counts once, regardless of signatures or replacements. |
| Transactions | `redemption_unresolved_transactions{phase}`, `redemption_oldest_transaction_timestamp_seconds{phase}` include historical retries/replacements until the identity is confirmed, reverted, or superseded, or its ticket is settled (`confirmed`/`used`). Ticket expiry or invalidation alone does not remove an unresolved identity. `sending` appears as `broadcast`. |
Expand Down
14 changes: 14 additions & 0 deletions docs/redeemer.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,20 @@ through your deployment or a proxy. Use a separate persistent database for each
redeemer; see [configuration](configuration.md) and [recovery](payment-recovery.md).
`--config` takes a local TOML or JSON file path, as with other components.

## Authorization webhook

To authorize ticket submissions:

```toml
AuthWebhookFile = "/run/secrets/livepeer-redeemer-auth-url"
# AuthWebhookHeadersFile = "/run/secrets/livepeer-redeemer-auth-headers"
```

The URL must be absolute HTTP or HTTPS without a username, password, or fragment.
The optional headers file contains comma-separated `Header: value` entries. See
[credential files](configuration.md#credential-files) and the
[HTTP contract](http-api.md#redeemer-authorization-webhook).

## Connect an orchestrator

Add these settings to an onchain orchestrator's configuration, using its
Expand Down
3 changes: 2 additions & 1 deletion nodeconfig/config.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
// Package nodeconfig supplies shared CLI policy, data-directory defaults, and storage paths.
// Package nodeconfig supplies shared configuration types, CLI policy,
// data-directory defaults, and storage paths.
package nodeconfig

import (
Expand Down
89 changes: 89 additions & 0 deletions nodeconfig/headers.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
package nodeconfig

import (
"bytes"
"encoding/csv"
"errors"
"io"
"maps"
"net/http"
"slices"
"strings"
)

// Headers parses comma-separated Header: value entries, with CSV quoting for
// commas. Names are case-insensitive; repeated names retain all values.
type Headers http.Header

func validHeader(name, value string) bool {
if name == "" {
return false
}
for _, c := range name {
if !((c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || (c >= '0' && c <= '9') || strings.ContainsRune("!#$%&'*+-.^_`|~", c)) {
return false
}
}
for _, c := range value {
if (c < 32 && c != '\t') || c == 127 {
return false
}
}
return true
}

func (h *Headers) UnmarshalText(text []byte) error {
parsed := make(http.Header)
input := strings.TrimSpace(string(text))
if input != "" {
reader := csv.NewReader(strings.NewReader(input))
reader.TrimLeadingSpace = true
reader.FieldsPerRecord = -1
entries, err := reader.Read()
if err != nil {
return errors.New("invalid comma-separated webhook headers")
}
if _, err := reader.Read(); err != io.EOF {
return errors.New("webhook headers must be one comma-separated record")
}
for _, entry := range entries {
name, value, ok := strings.Cut(entry, ":")
name, value = strings.TrimSpace(name), strings.TrimSpace(value)
if !ok || !validHeader(name, value) {
return errors.New("expected webhook headers in Header: value form")
}
parsed.Add(name, value)
}
}
*h = Headers(parsed)
return nil
}

// MarshalText returns CSV entries sorted by header name.
func (h Headers) MarshalText() ([]byte, error) {
var entries []string
for _, name := range slices.Sorted(maps.Keys(h)) {
if !validHeader(name, "") {
return nil, errors.New("invalid webhook header name")
}
for _, value := range h[name] {
if !validHeader(name, value) {
return nil, errors.New("invalid webhook header name or value")
}
entries = append(entries, name+": "+value)
}
}
if len(entries) == 0 {
return nil, nil
}
var buffer bytes.Buffer
writer := csv.NewWriter(&buffer)
if err := writer.Write(entries); err != nil {
return nil, err
}
writer.Flush()
if err := writer.Error(); err != nil {
return nil, err
}
return bytes.TrimSuffix(buffer.Bytes(), []byte("\n")), nil
}
2 changes: 1 addition & 1 deletion signer/headers_test.go → nodeconfig/headers_test.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package signer
package nodeconfig

import (
"net/http"
Expand Down
61 changes: 44 additions & 17 deletions redeemer/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"net"
"net/http"
"net/url"
"os"
"os/signal"
"sync"
"syscall"
Expand All @@ -33,23 +34,27 @@ import (
type Params struct {
ConfigFile string `name:"config" configfile:"optional-default" basepath:"source" default:"config.toml" boa:"noconfig" descr:"Configuration file; empty disables discovery"`
nodeconfig.Settings
Listen boa.HTTPBind `default:"127.0.0.1:8941" descr:"Ticket submission listener"`
MetricsListen boa.HTTPBind `default:"127.0.0.1:8942" descr:"Metrics and probe listener"`
Account *common.Address `descr:"Dedicated redemption wallet; must not be an orchestrator"`
RedeemerDB string `file:"optional" descr:"Winning-ticket SQLite database"`
KeystoreFile string `basepath:"source" optional:"true" descr:"Encrypted geth redemption account JSON file"`
KeystorePassword *string `secret:"true" optional:"true" descr:"Keystore decryption password"`
KeystorePasswordFile string `basepath:"source" secretfor:"KeystorePassword" descr:"Owner-only file containing exact password bytes"`
RPCURL *url.URL `name:"rpc-url" secret:"true" optional:"true"`
RPCURLFile string `basepath:"source" name:"rpc-url-file" secretfor:"RPCURL"`
ChainID *uint64 `min:"1"`
Controller *common.Address `name:"controller-address"`
RedeemerTxTimeout time.Duration `default:"5m" descr:"Transaction inclusion timeout"`
RedeemerMaxTxReplacements int `default:"1" min:"0" descr:"Maximum fee replacements per transaction"`
RedeemerMaxFeePerGas *uint256.Int `descr:"Optional maximum fee in wei per gas"`
MaxPendingTickets int `default:"100000" min:"1" descr:"Maximum unsettled winning tickets"`
MaxTicketsPerRecipient int `default:"10000" min:"1" descr:"Maximum unsettled tickets per recipient"`
PrintConfig bool `optional:"true" boa:"noconfig,noenv" descr:"Print audited redacted TOML configuration"`
Listen boa.HTTPBind `default:"127.0.0.1:8941" descr:"Ticket submission listener"`
MetricsListen boa.HTTPBind `default:"127.0.0.1:8942" descr:"Metrics and probe listener"`
Account *common.Address `descr:"Dedicated redemption wallet; must not be an orchestrator"`
RedeemerDB string `file:"optional" descr:"Winning-ticket SQLite database"`
KeystoreFile string `basepath:"source" optional:"true" descr:"Encrypted geth redemption account JSON file"`
KeystorePassword *string `secret:"true" optional:"true" descr:"Keystore decryption password"`
KeystorePasswordFile string `basepath:"source" secretfor:"KeystorePassword" descr:"Owner-only file containing exact password bytes"`
RPCURL *url.URL `name:"rpc-url" secret:"true" optional:"true"`
RPCURLFile string `basepath:"source" name:"rpc-url-file" secretfor:"RPCURL"`
AuthWebhook *url.URL `name:"auth-webhook" optional:"true" secret:"true"`
AuthWebhookFile string `basepath:"source" name:"auth-webhook-file" secretfor:"AuthWebhook"`
AuthWebhookHeaders nodeconfig.Headers `name:"auth-webhook-headers" optional:"true" secret:"true"`
AuthWebhookHeadersFile string `basepath:"source" name:"auth-webhook-headers-file" secretfor:"AuthWebhookHeaders"`
ChainID *uint64 `min:"1"`
Controller *common.Address `name:"controller-address"`
RedeemerTxTimeout time.Duration `default:"5m" descr:"Transaction inclusion timeout"`
RedeemerMaxTxReplacements int `default:"1" min:"0" descr:"Maximum fee replacements per transaction"`
RedeemerMaxFeePerGas *uint256.Int `descr:"Optional maximum fee in wei per gas"`
MaxPendingTickets int `default:"100000" min:"1" descr:"Maximum unsettled winning tickets"`
MaxTicketsPerRecipient int `default:"10000" min:"1" descr:"Maximum unsettled tickets per recipient"`
PrintConfig bool `optional:"true" boa:"noconfig,noenv" descr:"Print audited redacted TOML configuration"`
}

func (p Params) Validate() error {
Expand All @@ -68,6 +73,23 @@ func (p Params) Validate() error {
if err := destination.ValidateURL(p.RPCURL); err != nil {
return errors.New("redeemer requires a valid RPC URL")
}
if p.AuthWebhook != nil || p.AuthWebhookFile != "" || len(p.AuthWebhookHeaders) > 0 || p.AuthWebhookHeadersFile != "" {
if err := destination.ValidateURL(p.AuthWebhook); err != nil || p.AuthWebhook.User != nil || p.AuthWebhook.Fragment != "" {
return errors.New("auth webhook must be an absolute HTTP or HTTPS URL without credentials or fragment")
}
}
if _, err := p.AuthWebhookHeaders.MarshalText(); err != nil {
return fmt.Errorf("invalid auth webhook headers: %w", err)
}
for _, path := range []string{p.AuthWebhookFile, p.AuthWebhookHeadersFile} {
if path == "" {
continue
}
info, err := os.Stat(path)
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0077 != 0 {
return errors.New("auth webhook secret files must be owner-only regular files")
}
}
if p.RedeemerTxTimeout <= 0 || p.RedeemerMaxTxReplacements < 0 || p.MaxPendingTickets <= 0 || p.MaxTicketsPerRecipient <= 0 {
return errors.New("invalid redemption timeout, replacement count or queue capacity")
}
Expand Down Expand Up @@ -198,6 +220,11 @@ func Serve(parent context.Context, p Params, logOut io.Writer) (result error) {
})
probe.Handle("GET /metrics", metrics.Handler())
handler := redemption.Handler(store)
if p.AuthWebhook != nil {
auth := newAuthWebhook(p.AuthWebhook, p.AuthWebhookHeaders, logger, metrics)
defer auth.client.CloseIdleConnections()
handler = redemption.HandlerWithAuthorization(store, auth.authorize)
}
servers := []*http.Server{
{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { metrics.HTTP(w, r, handler.ServeHTTP) }), BaseContext: func(net.Listener) context.Context { return ctx }, ReadHeaderTimeout: 5 * time.Second, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 30 * time.Second, MaxHeaderBytes: 1 << 16, MaxHeaderValueCount: 128},
{Handler: probe, ReadHeaderTimeout: 5 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 30 * time.Second, MaxHeaderBytes: 1 << 16, MaxHeaderValueCount: 128},
Expand Down
Loading
Loading