Skip to content

Hold GET /discovery producers to the published schema #64

Open

Description

@eliteprox

Problem

docs/discovery.schema.json defines the GET /discovery response, but very little code has to match it.

Only two served responses are validated. discovery/schema_test.go checks handwritten documents and the example in docs/http-api.md. Beyond that, only the orchestrator inventory (TestDiscoveryLocalInventory) and one standalone discovery body (TestDiscoveryPreservesWholeRecords) are checked against the schema. Nothing else that serves or accepts this document is tested.

Decode accepts documents the schema rejects. discovery/shared.Decode disagrees with the schema on what an entry looks like:

Document Schema (TestDiscoverySchema) Decode (TestDecodeValidation)
[{"address":"https://orch.example"}], with no runners array Invalid Accepted, as the omitted case
[{"nodes":[],"address":"https://orch.example"}] Invalid, as the metadata with address case Accepted as an orchestrator entry. Decode treats an entry as metadata only when address is empty.

Other modules have to copy the encoder. discoveryResponse and discoveryNode in discovery/server.go are unexported. Any other module that serves GET /discovery has to copy them. It can forward runner records as the upstream bytes, but the envelope around them becomes a second implementation, and this repo's schema tests never run against it.

Consumers can't pin the schema from a release. The schema landed on main in #49 and is not in github.com/livepeer/node v1.0.6. A consumer that wants to validate against it has to pin a commit that includes the file.

Expected differences

  • Future timestamps. ClockSkew in discovery/shared/inventory.go rejects a stamp more than 30 seconds ahead. That is a runtime check, and the schema description already says timestamp freshness is not a schema constraint. Keep this exception, and keep it explicit.
  • New fields. The schema sets additionalProperties: true, so new optional fields still validate. This issue is about bodies the schema rejects, and about a second encoder that never runs these tests. It is not about new fields.

Done when

  • Decode accepts the documents the discovery/schema_test.go corpus marks valid and rejects the ones it marks invalid. The only named exception is a stamp more than ClockSkew in the future.
  • A test in this repo fails when a GET /discovery body written here does not validate. The two handler tests already cover their fixtures. The gap is the set of documents Decode accepts.
  • Another module can write the envelope without copying discoveryResponse. Export the builder, or the server path that writes the body, and name it in docs/http-api.md next to the schema.

Out of scope

  • Do not load the JSON Schema validator in the running process. Publish JSON Schema for discovery responses #49 kept it in tests on purpose.
  • Do not build runner records from the schema. discoveryRunner.MarshalJSON must keep returning the upstream bytes, so unknown fields and the original numeric tokens survive.
  • Do not move the schema out of this repo. Callers parse this response, so its wire format is defined here.

Activity

  1. j0sh commented on Nov 12, 2026

    @j0sh
    Collaborator

    PR #82 has merged, addressing the demonstrated raw-forwarding and client interoperability gaps. CI already ran the schema fixtures and handler tests; the change strengthens their coverage rather than adding another workflow.

    The agreed direction is leniency by default, tightening validation where malformed or ambiguous data is likely to cause unexpected client behavior. We are prioritizing broad correctness rules rather than an exhaustive runtime implementation of the schema.

    Deliberate input tolerance is preserved: malformed runners are skipped individually, valid siblings survive, empty inventories succeed, and existing envelope/metadata tolerance and timestamp-skew checks remain. The existing failure when a source advertises runners but none are usable also remains. Consequently, the original acceptance criterion requiring Decode to reject every schema-invalid document is too strict and should not be the goal.

    The merged changes:

    • Require the documented casing and types for known fields on the raw-forwarding path. Reject aliases that Unicode case folding would make other clients interpret as known fields, including nested fields. For example, capacity_used and capacity_uſed must not be forwarded together with conflicting values.
    • Reject duplicate keys, invalid Unicode, and non-RFC3339 observation timestamps. Keep the published nullability rules for raw records, since explicit null can break clients that accept an omitted field but expect a string or number when present.
    • Preserve complete valid runner records, unknown extensions, and exact numeric tokens. Signer's existing typed normalization, including interpretable nulls and quoted numeric prices, remains unchanged; these stricter checks apply where discovery would otherwise forward the original bytes unchanged.
    • Extend existing schema, forwarding, and signer tests, including root and nested Unicode alias regressions. Race tests, vet, and PR CI passed. No runtime JSON Schema validator was introduced.

    The checks run when upstream inventories are refreshed; cached HTTP reads do not repeat validation.

    The builder API request still needs more detail. Server.ServeHTTP is already public and lets another application serve the inventory collected by a configured discovery Server without copying the encoder. Exporting DiscoveryResponse would mainly expose that same server-owned inventory; it would not let a caller supply arbitrary records from its own database or collector.

    Before adding another public API, we need a concrete consumer example: where its inventory comes from, why the existing handler is insufficient, and the inputs and output it needs. If the consumer owns its inventory, a separate envelope-building function may be useful. If it is embedding the existing collector/server, the current handler may already cover the use case. That clarification remains open rather than adding a speculative builder API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions