Skip to content

fix(signing): harden trust and request atomicity - #999

Open
bokelley wants to merge 1 commit into
mainfrom
codex/security-signing-atomicity
Open

fix(signing): harden trust and request atomicity#999
bokelley wants to merge 1 commit into
mainfrom
codex/security-signing-atomicity

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

  • make replay and idempotency claims atomic across concurrent requests
  • prevent PostgreSQL idempotency pool self-deadlock with distinct lock ownership
  • bound signing-related HTTP fetches and preserve canonical-origin/SSRF checks
  • harden key rotation, JWKS refresh, digest verification, and brand key-origin enforcement
  • retain compatibility fallbacks with explicit deprecation warnings

Why

The security audit identified race conditions in replay/idempotency handling and trust-boundary gaps around remote signing metadata. Concurrent requests could pass non-atomic checks, while unbounded fetches and permissive key-origin behavior weakened failure containment.

Validation

  • 705 focused signing/idempotency tests passed (10 skipped)
  • independent security review against current origin/main

Compatibility

Digest verification is required by default, and missing brand key_origins now fails closed. Legacy replay/idempotency backends continue through warned process-local compatibility locks; deployments should migrate to atomic implementations for cross-process guarantees. PostgreSQL idempotency users must configure a distinct lock pool.

Comment thread tests/conformance/decisioning/test_pg_idempotency_backend.py Fixed
Comment thread tests/test_server_idempotency.py Fixed
Comment thread tests/test_server_idempotency.py Fixed
Comment thread src/adcp/signing/replay.py Fixed
Comment thread tests/test_server_idempotency.py Fixed
Comment thread tests/conformance/signing/test_jwks.py Fixed

@KonstantinMirin KonstantinMirin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — PR #999

Overview — Read as a Codex security-CLI pass, this behaves like one: the primitives it adds are sound, and the failures cluster into three recognisable shapes rather than scattering. Worth keeping as-is — the atomic-claim design in InMemoryReplayStore, the indexed expiry heap and the _NoItemsDict test, and the single-flight threading.Lock in CachingJwksResolver, which is proven by a test that fails on revert.

The three shapes, all one root: a local security property tightened without the surface it depends on.

  • Capability detected by shape instead of by contract. type(self.backend).hold is not IdempotencyBackend.hold is a correct-looking duck-type probe that a delegating wrapper always satisfies. LazyBackend defines both methods as forwarders, so the probe answers "native" for whatever it wraps: webhook replay dedup off and silent, request path NotImplementedError. This one is a regression against main (1, 11).
  • Fail-closed flips landed without threading the surface they need. expected_key_origins is None collapsed with {} (4); covers_content_digest flipped to "required" against the pinned 3.1.8 schema default (5). Each is right read alone and rejects traffic the spec says must verify.
  • Bounds that count the wrong quantity or invent a number the spec does not have. A byte budget that never bounds decompression (9); cache lifetimes 2× and 4× the ceilings they derive from (14).

The reason none of it surfaced is one finding, and it is the one to fix first: 31 one-at-a-time reverts leave the suite green (16) — including test_concurrent_deliveries_have_exactly_one_first_seen passing verbatim against the pre-PR non-atomic get-then-put sequence. Until a revert turns something red, this class of change cannot be told apart from a working one, and the same three shapes will land again on the next pass.

Should fix

1. Backend capability probed by method identity — LazyBackend defeats it

IdempotencyStore._hold (store.py:144) and WebhookDedupStore._put_if_absent (webhook_dedup.py:99-101) decide whether a backend is atomic with type(self.backend).hold is not IdempotencyBackend.hold. LazyBackend always defines hold and put_if_absent as forwarding wrappers (lazy.py:114, :118), so the probe reports "native atomic backend" for whatever it wraps, including the legacy backends the compatibility path exists for. The inner call then reaches IdempotencyBackend.hold's raise NotImplementedError (backends.py:114).

# LegacyBackend (get/put/delete_expired only) behind LazyBackend:
                              PR head            main
webhook check_and_record  ->  True True True      True False False
deprecation warnings      ->  []                  []
entries written           ->  0                   1
request path              ->  NotImplementedError OK

Webhook replay dedup is off and silent: check_and_record's except Exception: ... return True (webhook_dedup.py:126-159) swallows the NotImplementedError and reports first-seen on every delivery. The request path raises NotImplementedError on every idempotent call. Both contradict the PR body's "Legacy replay/idempotency backends continue through warned process-local compatibility locks".

Root cause: there is no capability contract, only a same-object method-identity probe living in the layer that consumes the backend. IdempotencyBackend.hold is declared -> Any with a runtime NotImplementedError instead of an @abstractmethod returning AbstractAsyncContextManager[None], which is also why mypy is happy. Make the capability a declared, delegable property of the backend contract (a supports_atomic_hold() predicate, or a runtime_checkable Protocol) and have LazyBackend answer for the backend it resolves. The same root drives a DRY problem: the keyed-lock fallback is written three times in this diff (backends.py:170-181, store.py:152-165, webhook_dedup.py:114-124), and the two store copies keep per-instance lock tables, so two stores over one backend do not serialize against each other. Once the capability is a contract, wrap a legacy backend in a _ProcessLocalHold adapter at construction and both store bodies collapse to one path.

2. The supervising task shadows operation, so IDEMPOTENCY_CONFLICT returns a Task repr to the buyer

store.py:217 binds operation = getattr(handler, "__name__", "handler"). store.py:293 rebinds the same local: operation = asyncio.create_task(_execute_locked()). _execute_locked closes over that cell and reads it at store.py:254, after the rebind.

                  PR head                                    main
exc.operation ->  Task                                       str
str(exc)      ->  <Task pending name='Task-3' coro=<...      create_media_buy: idempotency_key
                  running at /private/tmp/.../wt-999/src/     reused with a different payload
                  adcp/server/idempotency/store.py:253> ...

Any authenticated buyer that reuses one idempotency_key with a changed payload gets the deployment's absolute source path and Python install path in the error message, and the spec-defined IDEMPOTENCY_CONFLICT loses the operation field that dispatch and audit layers key on (exceptions.py:381). Rename the task (supervised, execution_task) and assert exc.operation == "create_media_buy" on the conflict path — the existing conflict tests check only the exception type (tests/test_server_idempotency.py:679,967), which is why this path is green.

3. Every handler exception is ERROR-logged as a post-cancellation failure

store.py:293-296 attaches _finish_supervised_operation unconditionally, and the callback (store.py:71-81) logs at ERROR whenever task.exception() is not None. That is the normal terminal state for any handler exception the caller already received through await asyncio.shield(operation), including the spec-defined IDEMPOTENCY_CONFLICT above. Three ordinary handler ValueErrors produce three ERROR records claiming cancellation with a full traceback each.

So a buyer sending malformed input drives ERROR-level log volume at request rate, the message asserts something false, and the operator signal the callback exists to provide ("work continued past a client disconnect") is buried in noise. The callback cannot know whether the awaiter was cancelled; the awaiter can. Record a flag when await asyncio.shield(...) raises CancelledError and log only in that case. A test asserting no ERROR record on the conflict path pins it — the file already has the caplog idiom in test_put_failure_logs_warning_and_returns_handler_result.

4. The key-origin fail-closed flip landed without the webhook surface, so brand.json webhook receivers reject everything with the wrong purpose

verifier.py:576 stopped skipping the step-7 check when expected_key_origins is None, and :614 passes expected_key_origins or {}. verify_webhook_signature (webhook_verifier.py:141-160) builds its inner VerifyOptions with neither expected_key_origins nor signing_purpose, and WebhookVerifyOptions has no field for either.

# resolver with jwks_source = "brand_json" (the spec's canonical webhook trust anchor):
PR head -> webhook_signature_key_origin_missing
           "identity.key_origins.request_signing declaration missing"
main    -> accepted (with the warning that named the affordance)

Two defects. Every webhook is rejected for a receiver conforming to the exported BrandSourcedJwksResolver protocol, and the emitted purpose is request_signing where AdCP 3.1.8 names webhook_signing for webhook delivery (get-adcp-capabilities-response.jsonidentity.key_origins.webhook_signing; security.mdx:1442).

Root cause: None ("the SDK caller supplied no map") was collapsed with {} ("capabilities were observed and declared no map"). The spec's reject predicate is a statement about the counterparty's capabilities document — security.mdx:1108 @ 3.1.8 says "If the agent declares signing without a corresponding identity.key_origins.{purpose} entry, reject" — while the schema says of an absent map: "Absent means the operator has not declared a separation scheme; receivers SHOULD assume shared-origin." The pre-PR code drew exactly that distinction, and the deleted docstring documented the affordance ("pass an empty dict if the operator advertises no map"). Keep the two cases distinct, have verify_from_agent_url pass resolution.key_origins or {} so the capabilities-derived path fails closed where the spec requires, add expected_key_origins and posture to WebhookVerifyOptions, and pass signing_purpose="webhook_signing" from verify_webhook_signature. Cite security.mdx:1108 @ 3.1.8 on the rejecting branch. No test caught this: every webhook test uses a bare StaticJwksResolver with no jwks_source.

5. covers_content_digest default flipped to "required", against the 3.1.8 schema default and this repo's own migration guide

verifier.py:105. The pinned schema defines the default: dist/schemas/3.1.8/protocol/get-adcp-capabilities-response.jsonproperties.request_signing.properties.covers_content_digest.default == "either". agent_resolver.py:767 builds VerifierCapability(supported=True) when the caller omits capability, so verify_from_agent_url inherits the flip.

vector positive/001-basic-post.json  "Basic POST, Ed25519, no content-digest coverage"
  expected_outcome.success: True
  verified with VerifierCapability(supported=True)
  -> request_signature_components_incomplete (step 6)

The vectors pass only because the harness hard-codes cap_data.get("covers_content_digest", "either") (test_verifier_vectors.py:50), so the graded artifact still encodes the spec default while the SDK no longer does. docs/request-signing-migration.md:123 says "Don't enable covers_content_digest="required" yet" and :191 tells adopters to stay on "either" when intermediaries touch bodies.

Root cause: VerifierCapability is both the block a verifier advertises on get_adcp_capabilities (its own docstring says so) and the local enforcement policy, so hardening one falsifies the other. A seller rendering capabilities from the wire model now advertises either and enforces required. Restore "either" as the dataclass default and express the strict posture as a separately named opt-in, or split the wire mirror from a VerifierPolicy that defaults strict. Either way docs/request-signing-migration.md moves in the same commit. This rejects previously-accepted spec-legal traffic through the production helper, so it needs a deliberate call on whether it ships under fix without !.

6. InMemoryReplayStore.remember silently drops the write at cap

replay.py:93-98: when either cap is reached and the key is new, remember returns without recording.

s = InMemoryReplayStore(per_keyid_cap=2, global_cap=10)
s.remember('k','n1',60); s.remember('k','n2',60); s.remember('k','n3',60)
s.seen('k','n3')   # PR head: False    main: True

remember is on the public ReplayStore Protocol (replay.py:34) and seen then remember is the documented legacy composition that verifier._claim_replay_nonce and _NamespacedReplayStore still use, so a caller composing them accepts that nonce for the rest of the TTL. AdCP 3.1.8 security.mdx:1322 is explicit about this failure mode: "On cap exceeded, verifiers MUST reject new signatures from that keyid with request_signature_rate_abuse — NOT silently evict … Silent eviction is the dangerous mode: it creates replay windows exactly when the verifier is under attack." Before this PR remember always recorded and the cap was enforced by at_capacity. Either keep recording over cap, or make the refusal observable (raise, or change the Protocol's return type) so a two-step caller cannot mistake "not recorded" for "recorded". test_renewal_heap_remains_bounded currently pins the silent drop rather than flagging it.

7. Cold-path revocation freshness raises outside the SignatureVerificationError family and erases the spec's grace window

revocation_fetcher.py:452-455 rejects a successfully fetched, signed list the instant next_update has passed, with RevocationListParseError and zero skew tolerance — while the sibling updated check two lines above allows 60 s. On the cold path (_current_list is None) _ensure_fresh does not catch it, so it escapes __call__, escapes the unguarded options.revocation_checker(keyid) call in verify_request_signature, and reaches the adopter as a type unauthorized_response_headers cannot map (it takes SignatureVerificationError).

cold checker, list next_update 14:15Z, now 14:16Z (1 min past, inside grace)
-> RevocationListParseError: revocation list next_update '2026-04-18T14:15:00Z' is already expired
   (not a SignatureVerificationError -> 500, not 401 + request_signature_revocation_stale)

AdCP 3.1.8 puts the boundary elsewhere: security.mdx:1333 — "verifiers that have not refreshed within next_update + grace MUST reject new request-signed mutations with request_signature_revocation_stale". At the spec's 1-minute polling floor any fetch latency across the boundary trips this. Root cause: a freshness decision placed in the parse/validate layer, which owns neither the grace window nor the wire code. The static-list path at verifier.py:279 answers the same question correctly with REQUEST_SIGNATURE_REVOCATION_STALE at step 9, so the diff now has two answers. Install the fetched list and let _ensure_fresh's grace check own the rejection.

8. The cached JWKS failure object is re-raised, so request N gets request M's wire code and a traceback that grows 2 frames per request

jwks.py:471 and :612 do raise self._last_failure or SignatureVerificationError(...), re-raising the instance stored at :491/:499 and :644/:652.

first refresh fails with SSRFValidationError -> request_signature_jwks_untrusted (3 tb frames)
200 further calls inside the 30s cooldown:
  codes: {'request_signature_jwks_untrusted'}   same object: True
  tb frames: 403                                upstream fetches: 0

Two consequences. The wire code is a per-request judgment and request_signature_jwks_untrusted denotes an SSRF verdict on this resolution, so requests whose key resolved fine from cache are rejected with a verdict about an earlier request. And during a publisher outage at production rate the traceback on one long-lived shared object grows without bound, so every handler that formats it writes a linearly growing record. The fallback branch already builds the right per-request code. Cache (code, message) and construct a fresh SignatureVerificationError(...) at each raise, carrying the prior failure as __cause__.

9. The new byte budget does not bound decompression

_bounded_http.read_limited_bytes / async_read_limited_bytes (_bounded_http.py:30, :50) count accumulated decoded bytes from iter_bytes/aiter_bytes. httpx's content decoder inflates each raw chunk in full before iter_bytes re-chunks the output, so a compressed body walks straight through the budget. The content-length pre-check is the encoded length and is absent under chunked transfer.

203 KB gzip stream decoding to 200 MB, read with limit=1024
-> raised: response exceeds 1024 bytes (received at least 1025)
-> peak RSS: 295 MB

None of the four converted fetchers overrides Accept-Encoding, so httpx advertises gzip on all of them: jwks.py:377 / :547, brand_jwks.py:641, agent_resolver.py:269, revocation_fetcher.py:284 / :331. Every one of those reads from a counterparty-controlled origin, and with the real defaults (256 KiB brand.json, 1 MiB JWKS) the amplification budget is far larger than in this repro. Send Accept-Encoding: identity on these fetches (they are small JSON/JWS documents) or bound the raw stream via iter_raw and decode with zlib.decompressobj().decompress(chunk, max_length=remaining). Add a content-encoding: gzip case to tests/conformance/signing/test_bounded_fetches.py, which today covers identity-encoded chunking only.

10. One process-wide replay store shares its global_cap across every counterparty, and the namespace key is not canonicalized

agent_resolver.py:585 creates one process-wide InMemoryReplayStore() (global_cap=1_000_000) and :760-763 wraps it per identity in _NamespacedReplayStore(..., str(httpx.URL(resolved_agent_url))). The wrapper prefixes the keyid, so it isolates the per-keyid cap. It does not isolate the global cap, which claim and at_capacity both check.

shared = InMemoryReplayStore(global_cap=3)
a.claim('kid1', n0..n2) -> ['claimed','claimed','claimed']
b.claim('kid1','fresh') -> capacity        # b's first-ever request
b.at_capacity('kid1')   -> True            # -> REQUEST_SIGNATURE_RATE_ABUSE

The one bound that constrains a hostile counterparty is the resource every other counterparty in the process depends on. Second half: the namespace string is resolution.agent_entry.get("url"), taken from the counterparty's own brand.json, normalized only by str(httpx.URL(...)). That is not an origin canonicalization, so query and fragment spellings mint distinct namespaces and one kid gets N × per_keyid_cap live nonces:

'https://a.evil/mcp'      -> 5   'https://a.evil/mcp?x=1' -> 5   'https://a.evil/mcp#f' -> 5
total live entries under one kid, per_keyid_cap=5: 15

Give each namespace its own budget (per-namespace global_cap, or one store per resolved identity in a bounded LRU), and derive the namespace from a canonicalized origin using _idna_canonicalize.canonicalize_host, which the repo already owns. Replay state ownership belongs with the verifier rather than as import-time global state in the resolver, so the test seam is a parameter rather than monkeypatch.setattr on a private module attribute (tests/test_verify_from_agent_url.py:303).

11. One capability, three detection mechanisms, an unused Protocol, and a warning that fires per request

replay.py:39 adds AtomicReplayStore(ReplayStore, Protocol) with claim. It appears in zero call sites and zero tests, and it is not @runtime_checkable, so it cannot be used for the check it was added for. The capability is instead detected three ways: getattr(store, "claim", None) in verifier.py:377, a second getattr probe in _NamespacedReplayStore.claim (agent_resolver.py:607-612) with no warning, and type(x).m is not Base.m in store.py:144.

Consequences beyond the duplication. verifier.py:381 writes result: ReplayClaimResult = claim(...) — an annotation-only assertion about a value from adopter code, which is what the Protocol was for; a store whose claim() returns True verifies the identical signature three times in a row with seen/remember never called and no warning. And because _NamespacedReplayStore always exposes claim, wrapping any store in it makes _claim_replay_nonce take the atomic branch unconditionally, suppressing the non-atomic-backend signal. Meanwhile verifier.py:385-391 warns on every verification where its two siblings added in this same diff warn once (store.py:139/149/157, webhook_dedup.py:90/104/113):

RuntimeWarnings emitted for 500 verifications: 500

warnings.warn(..., stacklevel=2) costs stack introspection per request, and an adopter running -W error::RuntimeWarning gets a 100% failure rate after crypto verification has already succeeded.

Mark AtomicReplayStore @runtime_checkable and narrow with isinstance (or add one _supports_atomic_claim(store) helper in replay.py), route _NamespacedReplayStore through the same helper instead of carrying its own copy, and mirror the once-flag on the warning. Declaring _NamespacedReplayStore against ReplayStore/AtomicReplayStore also gets it type-checked, which the bare duck-typed class is not.

12. PgBackend.hold holds a lock-pool connection for the whole handler, including cache-hit replays

backends.py:466-479 checks out a lock_pool connection and opens a transaction around the yield, and store.py:239-243 takes the hold before the cache lookup, so a pure replay (cache hit, no handler run) consumes a lock connection and an advisory lock. The class docstring example (backends.py:251-252) sizes both pools at max_size=10, which makes 10 the ceiling on concurrent idempotent requests; the 11th waits out the psycopg_pool timeout and raises PoolTimeout, a 5xx to a buyer who retries with the same key. asyncio.shield makes it worse: a disconnected client's operation keeps its lock connection checked out to completion.

Three further mismatches. backends.py:317 describes lock_pool as "reserved for advisory-lock transactions", but get/put inside a hold run on that connection (:418-420, :456-458), so it carries the wrapped path's cache traffic. The held connection travels through a per-instance ContextVar compared against asyncio.current_task(), and put_if_absent and delete_expired do not participate, so inside one hold block two of four data methods run in the lock transaction and two run on separate pool connections, with nothing in the signature saying which. And the lock_pool is pool check at :339-340 is a spot check: a second pool the adopter also uses for handler SQL reproduces the deadlock it claims to prevent.

Root cause: hold yields nothing, so the connection is passed by an implicit channel instead of an explicit handle. Have hold yield a bound session the store threads into get/put/put_if_absent, and annotate the ABC's hold as AbstractAsyncContextManager[<handle>] instead of Any so a backend that ignores the handle fails type-check. Read the cache once outside the lock and acquire the hold only on a miss. Also: lock_pool is now a required keyword (backends.py:334), so PgBackend(pool=pool) raises TypeError on upgrade and docs/handler-authoring.md:632 still shows exactly that call — update the doc in this PR, and decide whether a required-kwarg break ships under fix(...).

13. client._origin re-rolls host comparison instead of using the repo's canonicalizer

src/adcp/signing/client.py:95 compares hosts with parsed.hostname.lower(). _idna_canonicalize.py exists for this and its module docstring enumerates the callsites that canonicalize host strings for comparison; jwks.py, ip_pinned_transport.py, revocation_fetcher.py, key_origins.py, brand_jwks.py and etld.py all use it, and the two commits immediately before this PR (b7ef1dfc, 827e4f4e) fixed this same class of bug.

_origin("https://sellér.example.com") -> ('https', 'sellér.example.com', 443)
_origin(httpx.Request(...).url)       -> ('https', 'xn--sellr-esa.example.com', 443)
match: False

An operator who passes a unicode expected_origin has every signed request refused, blamed on a redirect that never happened. Fail-closed, so not exploitable, but it is a silent misdiagnosed outage for a legitimate config. Route both sides of _origin through canonicalize_host, which already short-circuits IP literals. The same primitive fixes the namespace half of finding 10. Related: the invariant "a hook signs for exactly one origin" now lives in one of the two signing request hooks. ADCPClient._sign_outgoing_request (src/adcp/client.py:968-1049) is the semantically equivalent hook the SDK's own transports install (client.py:581mcp.py:370,1020, a2a.py:263) and it has no origin binding. Those paths pin follow_redirects=False, so the redirect-capture vector is not live there today; the structural problem is that the next change to either hook drifts. Build ADCPClient's hook through install_signing_event_hook(..., expected_origin=...) and delete the duplicate body.

14. New cache-lifetime and count defaults exceed or invent the 3.1.8 bounds they derive from

Three uncited constants, all new and exported in this diff:

  • DEFAULT_JWKS_MAX_AGE_SECONDS = 3600.0 (jwks.py:49). security.mdx:1457 @ 3.1.8: "JWKS cache TTL bounded above by the revocation-list polling interval (floor 1 min, ceiling 30 min)". 3600 s is 2× the ceiling and 4× the spec's own cacheMaxAge: 15 * 60 * 1000 example, so a compromised kid added to revoked_kids keeps verifying on the JWKS path for up to an hour. Introducing a max-age at all is the right call — :1457 also says verifiers MUST NOT pin one snapshot for a task's lifetime — the number is what needs fixing.
  • DEFAULT_MAX_STALE_SECONDS = 3600.0 (brand_jwks.py:100) stacks on DEFAULT_MAX_AGE_SECONDS = 3600.0 for up to 7200 s of trust in an expired brand.json snapshot. security.mdx:1103 @ 3.1.8: "Cache TTL on a successful fetch MUST be bounded above by the JWKS revocation polling interval (so a key rotation cannot be masked by a stale brand.json)".
  • DEFAULT_MAX_SCHEMA_IDS = 32 (references.py:45, enforced at :589). The normative format_schema fetch contract (docs/creative/canonical-formats.mdx:217-231 @ 3.1.8) enumerates its bounds — $ref depth ≤ 8, total $ref count ≤ 256, compiled keyword count ≤ 10 000, 1 MiB streaming cap, ≤ 5 s timeout — and has no $id bound. A schema with 200 $refs and 40 $ids satisfies every listed bound and is now INVALID_SCHEMA, which per :230 means the buyer loses the product declaration. Two conformant SDKs then disagree on whether the same digest-pinned document resolves.

Derive the first two from the configured revocation polling interval so the spec's "bounded above by" relation holds by construction, and cite security.mdx:1457 / :1103 @ 3.1.8 at each constant. For the third: the cost being defended is _validate_schema_id_value's _resolve_public_https call per $id, and $id must already be same-origin or the AAO catalog, so at most two distinct hosts per document — memoize the host resolution instead of lowering the acceptance bar, or raise the default to the spec's own 256.

15. The post-crypto over-cap rejection is stamped step="9a", and the step-9a citation was deleted

verifier.py:346-351 raises REQUEST_SIGNATURE_RATE_ABUSE with step="9a" after crypto verify. AdCP 3.1.8 security.mdx:1324 names a different step: "step 9a remains the cheap amplification guard, step 13 is the authoritative enforcement point. A verifier whose atomic insert returns over-cap MUST reject the request with request_signature_rate_abuse". step is adopter-facing through middleware.verify_starlette_request, so it should name the step that actually failed. That paragraph is also the grounding for the whole atomic-claim design and it is cited nowhere in the diff, while verifier.py:289-291 removed the one citation that was there ("Step 9a (per spec, after adcp#2342): the per-keyid cap runs between JWKS resolution and crypto verify"), leaving the ordering constraint the early check exists to satisfy undocumented. Stamp the post-crypto rejection step=13, keep the code, and restore the ordering citation on the early at_capacity guard.

16. Most of the changed behavior survives deletion with the suite green

31 one-at-a-time reverts were run against the 706-test selection for the touched areas. The suite stayed green for all of the following, so none of this behavior is fenced:

  • test_concurrent_deliveries_have_exactly_one_first_seen passes verbatim against the pre-PR get-then-put sequence (confirmed: 20 tasks, True count 1 either way). MemoryBackend.get/put take an uncontended asyncio.Lock, so no coroutine ever suspends and the 20 tasks serialize by scheduling accident. Give the backend a real suspension point between read and write, or assert MemoryBackend.put_if_absent returns False for the second caller directly.
  • Deleting the native branch in _hold, deleting it in _put_if_absent, and replacing the ABC's documented hold+get+put composition with return True are each green. Nothing detects a silent downgrade from a PgBackend's cross-process guarantee to per-process locking, which is the load-bearing decision of this PR. Assert the absence of the compatibility DeprecationWarning on the supported backends, and cover a hold-only custom backend through the ABC default.
  • _keyid()return keyid (namespacing removed) is green. The namespacing is what justifies the process-wide shared default store; nothing asserts two agent URLs can claim the same (kid, nonce).
  • Deleting both JWKS cooldown raise blocks is green. The behavior is observable: with the guard, a JWKS-origin outage costs 2 upstream fetches across 5 inbound requests; without it, 5.
  • All four of BrandJsonJwksResolver's new fail-closed branches mutate cleanly (unbounded stale-serve restored, can_refresh re-anchored on fetched_at, _sync_selector's clearing removed, _do_refresh's _last_error recording removed). test_authz_stale_on_error_is_bounded covers the authorization resolver; the JWKS resolver serves key material and DEFAULT_MAX_STALE_SECONDS appears in no test.
  • Deleting the claim == "capacity" block (a full replay cache then accepts) and dropping the global-cap term from at_capacity are both green. Both sites emit a normative wire code.
  • A claim() returning True (wrong type, right arity) verifies the identical signature three times with no warning. AtomicReplayStore and ReplayClaimResult appear in zero test files, while this repo does pin protocol conformance elsewhere (test_pg_idempotency_backend.py::test_satisfies_idempotency_backend_protocol).
  • The content-length pre-check in both readers deletes green; _index_jwks_keys's non-string-kid and non-list-keys rejections delete green ({"keys": {}} then yields an empty primed cache that satisfies the cooldown for max_age seconds); the base_url-fallback origin binding and the expected_origin-with-no-scheme ValueError both delete green; the async _ensure_fresh 304 change deletes green while its proven sync twin does not; weakening put's active[1] is asyncio.current_task() check to if active is not None: deletes green.
  • Removing the SSRF re-validation on the capabilities redirect target (agent_resolver.py:249-255) is green — grep finds no 3xx / Location / max_redirects case for that fetcher anywhere. Note this branch is unreachable at DEFAULT_CAPABILITIES_MAX_REDIRECTS = 0 (hop 0 == max_redirects raises the limit error first) and only live when an adopter raises the limit, which is itself untested.

The two Postgres files skip locally for lack of psycopg and run in the pg-conformance job, so this is what they assert rather than whether they run: test_concurrent_claim_same_nonce_has_one_winner's one-winner outcome already follows from the (keyid, nonce) primary key plus ON CONFLICT … WHERE expires_at <= now() RETURNING 1 without pg_advisory_xact_lock — the advisory lock serializes the capacity leg, which no test races, and PgReplayStore.claim's "capacity" return has no PG test. test_concurrent_wrapped_calls_execute_once is single-process, so the process-local fallback satisfies it and it does not demonstrate the cross-process guarantee lock_pool was introduced for. Reasoned from the SQL, not measured — worth the author's confirmation.

Notes

  • DEFAULT_GRACE_MULTIPLIER = 2.0 with the comment "Spec recommends 2×" (revocation_fetcher.py:76) contradicts pinned 3.1.8, which says grace = 4× the previous polling interval in both places it appears (security.mdx:721, :1333). The line is untouched context in this diff, so out of scope here — it compounds finding 7, which removes the grace window entirely on the cold path. Worth its own ticket.
  • tests/test_pg_idempotency_backend.py::test_delete_expired_defaults_to_wall_clock failed once in ~20 full runs (before <= cutoff_dt.timestamp() <= after, float round-trip through datetime.fromtimestamp). The assertion is pre-existing and this PR touched only the constructor line in that test, so it is out of scope for this diff.
  • PgBackend.hold, PgBackend.put_if_absent and PgReplayStore.claim were not exercised against a live Postgres in this review — the advisory-lock and ON CONFLICT … WHERE expires_at <= now() logic in finding 12 is reviewed by reading only.

Comment thread src/adcp/server/idempotency/webhook_dedup.py Outdated
Comment thread src/adcp/server/idempotency/store.py Outdated
Comment thread src/adcp/server/idempotency/store.py Outdated
Comment thread src/adcp/server/idempotency/store.py Outdated
Comment thread src/adcp/signing/verifier.py Outdated
Comment thread src/adcp/server/idempotency/backends.py
Comment thread src/adcp/signing/client.py Outdated
Comment thread src/adcp/signing/jwks.py Outdated
Comment thread src/adcp/signing/brand_jwks.py Outdated
Comment thread src/adcp/canonical_formats/references.py Outdated
@aao-ipr-bot

aao-ipr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

⚠️ Argus review could not complete

The automated review encountered an issue (possibly reached max turns, timed out, or failed to post the final gh pr review). A human reviewer should take this PR.

View workflow run

This is an automated message from the Argus AI review workflow.

@bokelley
bokelley force-pushed the codex/security-signing-atomicity branch from 1f7c701 to 63ae795 Compare August 5, 2026 02:10
@aao-ipr-bot

aao-ipr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

⚠️ Argus review could not complete

The automated review encountered an issue (possibly reached max turns, timed out, or failed to post the final gh pr review). A human reviewer should take this PR.

View workflow run

This is an automated message from the Argus AI review workflow.

@bokelley
bokelley force-pushed the codex/security-signing-atomicity branch from 63ae795 to b152b84 Compare August 5, 2026 02:21

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. Two blockers: a breaking public-API change shipped under a non-breaking commit prefix, and a cross-tenant denial-of-verification introduced by the new shared replay store. The signing/trust hardening itself is sound — SSRF re-checks per hop, streaming body caps, fail-closed key-origin, the atomic replay claim, and the 304-no-longer-extends-trust removal all check out. It's the two items below that gate.

MUST FIX (blocking)

1. PgBackend(lock_pool=...) is a breaking change to a public export, shipped under fix(signing):. backends.pyPgBackend.__init__ now takes a required lock_pool with no default and raises ValueError if lock_pool is pool. PgBackend is public (from adcp.server.idempotency import PgBackend); every existing PgBackend(pool=pool) caller now hits TypeError: missing required keyword-only argument 'lock_pool'. The lazy resolver also now calls await app.get_idempotency_lock_pool(), a new required method on the adopter's app object. The commit is fix(signing): harden trust and key lifecycle — no !, empty body, no BREAKING CHANGE: footer. Under release-please this bumps a patch and hides a hard break. ad-tech-protocol-expert flagged the same divergence independently. The migration note already exists in the PR body ("PostgreSQL idempotency users must configure a distinct lock pool") — the missing piece is the semver signal. Retitle to fix(signing)!: (or feat!:) and add a BREAKING CHANGE: footer naming the lock_pool requirement. One character plus a footer.

2. Shared process-wide default replay store + global_cap == per_keyid_cap → cross-tenant denial-of-verification. agent_resolver.py (_DEFAULT_REPLAY_STORE = InMemoryReplayStore(), _default_replay_store_for_origin) wraps every resolved origin as _NamespacedReplayStore over one shared backend; the namespace only prefixes the kid. replay.py InMemoryReplayStore defaults global_cap=1_000_000 == per_keyid_cap=1_000_000, and both at_capacity(keyid) (the cheap step-9a pre-check) and claim() reject on len(self._entries) >= global_cap across all namespaces. Failure scenario (security-reviewer HIGH, and reachable non-adversarially): one enrolled counterparty signing unique-nonce requests fills its own per-key cap, which — because the two caps are equal — simultaneously exhausts the global cap; from that point every verification for every other counterparty in the process raises REQUEST_SIGNATURE_RATE_ABUSE before crypto verify, until entries expire. This crosses the tenant boundary the per-key cap previously preserved. Fix: give each canonical origin its own InMemoryReplayStore in _default_replay_store_for_origin (cleanest — matches the per-origin namespacing intent), or default global_cap to a large multiple of per_keyid_cap and don't surface a global-cap rejection as fatal RATE_ABUSE for a key under its own per-key cap.

Things I checked

  • SSRF on the client.stream() rewrite: agent_resolver._fetch_capabilities and brand_jwks._fetch_brand_json rebuild the IP-pinned transport and re-run the SSRFValidationError check on each redirect hop; follow_redirects=False throughout. security-reviewer confirmed.
  • _bounded_http.py caps are not bypassable: content-length-too-large rejected up front, spoofed-small length still caught by the per-chunk len(body) > limit, chunked oversize stopped mid-stream (tests assert 2 of 3 chunks read), non-identity Content-Encoding rejected before iterating. Accept-Encoding: identity on all four fetchers.
  • Replay atomicity: InMemoryReplayStore.claim and PgReplayStore.claim (advisory lock + ON CONFLICT ... WHERE expires_at <= now() RETURNING 1) are atomic with replayed-before-capacity precedence, so a replay is never masked as rate-abuse. The legacy seen/remember fallback re-reads seen and fails closed on a silent cap-drop.
  • Key-origin: verify_from_agent_url passes resolution.key_origins or {}; _maybe_check_key_origin short-circuits only on None, so an empty map reaches check_key_origin_consistency and a brand declaring no origins is rejected, not trusted. Matches the fail-closed claim.
  • Freshness: can_serve_stale hard-bounded by min(expires_at + max_stale, fetched_at + DEFAULT_MAX_AGE_SECONDS) — no config extends trust past 30 min. _slide_next_update removal means a 304 carries no JWS and cannot extend the signed next_update; correct fail-closed posture.
  • covers_content_digest default stays "either" per schemas/cache/3.1/protocol/get-adcp-capabilities-response.json:1193; webhook profile hard-codes "required". No enum/default drift. AtomicReplayStore / ReplayClaimResult exports are additive typing surface, no wire change.
  • Idempotency store rework (store.py): asyncio.shield(create_task(_execute_locked())) with _SUPERVISED_OPERATIONS holding strong refs runs the handler exactly once and releases the hold lock even when the caller is cancelled; _finish_supervised_operation consumes the terminal exception — no orphaned-task warnings. put commits only on success.
  • replay.py indexed min-heap: _remove_expiry arbitrary-removal (pop-last, place-at-hole, sift up-or-down) and _swap_heap keep _heap_positions consistent; _counts pairs one increment with one decrement per path and cannot drift from _entries.

Follow-ups (non-blocking — file as issues)

  • Async JWKS cooldown re-check (code-reviewer Major). jwks.pyAsyncCachingJwksResolver.__call__ performs the expired-but-within-cooldown fail-closed raise only before acquiring self._lock, not inside it after the re-read. The sync CachingJwksResolver does it inside the lock. Against a down JWKS endpoint, queued verifications each issue their own serialized outbound fetch instead of failing closed cheaply — the cooldown's anti-storm purpose is defeated in the async path only. Security posture (never serving a stale key) is preserved; impact is availability amplification and sync/async divergence. Move the raise into the async with self._lock block to mirror the sync path. Not a blocker, but fix before this lands.
  • Direct verify_request_signature(expected_key_origins=None) now skips the brand-json key-origin check with no warning (verifier.py _maybe_check_key_origin, the PR removed the prior UserWarning). Moot for verify_from_agent_url (passes or {}), but a direct caller with a brand-sourced resolver loses the misconfig signal. Restore a one-time warning on the source == "brand_json" and expected_key_origins is None path.
  • _purge_expired bounded at _SWEEP_BATCH=16 can transiently false-reject at the global cap after a large batch of short-TTL entries expires (self-heals at −15/call). Pre-existing bounded-sweep tradeoff, not a regression; worth a comment or a larger purge budget when over cap.
  • Spec citations unverifiable in-repo. security.mdx:1103/1324/1457 @ AdCP 3.1.8 are prose in the upstream adcontextprotocol/adcp repo, not vendored here, and this SDK's schema cache tops out at 3.1.0-rc.13. Confirm the 30-minute revocation ceiling and step numbers (9a/12/13) against the tagged upstream security.mdx before merge — the caps fail toward refetch so interop is safe either way, but the cited line numbers are load-bearing in the comments.

Minor nits (non-blocking)

  1. First-use origin binding is nondeterministic when expected_origin and base_url are both unset. client.py install_signing_event_hook._hook — two concurrent first-use requests to different origins can each observe bound_origin is None. No cross-origin capture (each signs only its own origin), but document expected_origin as the supported production configuration and treat the first-use fallback as best-effort.
  2. Handler now runs in a child task (store.py, create_task), so contextvars the handler sets aren't visible to the caller after the wrapped call returns. Values set before the call still reach the handler. Worth a one-line docstring note.

Fix the two blockers — the !/footer on the commit and the shared-store cross-tenant cap — and the async JWKS re-check, and this is safe to merge. The hardening is the right shape.

@bokelley
bokelley force-pushed the codex/security-signing-atomicity branch from b152b84 to 0b81fc2 Compare August 5, 2026 04:23
Comment thread src/adcp/signing/replay.py Fixed
@bokelley
bokelley force-pushed the codex/security-signing-atomicity branch 2 times, most recently from 656ad26 to 8c40bd3 Compare August 5, 2026 04:28
@bokelley
bokelley force-pushed the codex/security-signing-atomicity branch from 8c40bd3 to 91b9d22 Compare August 5, 2026 04:34

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes — one thing, and it's the semver marker, not the code. The hardening itself is sound: atomic replay claim, bounded streaming fetches, fail-closed key-origin and 304 handling all hold up. The block is that a breaking public-constructor change is riding in under fix(signing): with no ! and no BREAKING CHANGE: footer, so release-please will cut it as a patch and the changelog won't surface the break.

MUST FIX (blocking)

Breaking public-API change shipped without the semver signal. src/adcp/server/idempotency/backends.pyPgBackend.__init__ now takes lock_pool: Any with no default (and raises ValueError if lock_pool is pool). Every existing PgBackend(pool=pool) call site — the exact form documented in docs/handler-authoring.md before this PR — now raises TypeError: missing 1 required keyword-only argument: 'lock_pool'. PgBackend is public (from adcp.server.idempotency import PgBackend), so this is on the wire for adopters.

This is a release-please + conventional-commits repo. A required-constructor-arg break landing as fix: bumps a patch and buries the break. code-reviewer: Blocker, same finding. Fix is mechanical — retitle to fix(idempotency)!: (the scope is also wrong; the diff is mostly server/idempotency, not signing) or add a BREAKING CHANGE: footer, and keep the migration line you already have in the PR body's Compatibility section. Same marker covers the two other intended behavior breaks in this PR — verify_from_agent_url now fails closed on missing brand key_origins, and digest verification defaults required — both of which are correct hardening but are contract changes adopters must be told about via the changelog, not just the PR body.

Flip the marker and I approve on the next pass. No code change required for the block.

Things I checked

  • PgBackend.hold + ContextVar connection pinning (active[1] is asyncio.current_task()) correctly reuses the advisory-lock connection for nested get/put; distinct-lock_pool assertion prevents the self-deadlock. Detached _execute_locked task inherits the context copy so pinning survives the asyncio.shield.
  • InMemoryReplayStore heap rewrite: _push_expiry/_remove_expiry/_sift_*/_swap_heap keep _entries_expiry_heap_heap_positions consistent; old _sweep_for_keyid fully removed, no dangling heap entries. claim returns replayed before capacity, so a legitimate retry is never misreported as rate abuse.
  • Bounded readers (_bounded_http.py): non-identity content-encoding rejected before the size check, content-length lie caught by the streaming cutoff at limit+1, chunked/no-length bounded by the loop. SSRF/IP-pinning + per-hop redirect re-validation preserved across the stream() refactor (security-reviewer, confirmed).
  • Origin binding in install_signing_event_hook: no await between the bound_origin is None check and assignment — no TOCTOU; cross-origin redirect raises before the signature is emitted.
  • Revocation 304: _slide_next_update removal means an unauthenticated 304 no longer extends signed next_update; grace still computes off the signed value. Correct fail-closed posture (ad-tech-protocol-expert: matches RFC 5280 §5.1.2.5 nextUpdate non-extension).
  • security-reviewer: no High — no reachable replay bypass, SSRF regression, or origin-check bypass on the production path.

Follow-ups (non-blocking — file as issues)

  • Async JWKS cooldown divergence. src/adcp/signing/jwks.py AsyncCachingJwksResolver.__call__ evaluates the expired-and-cooldown-not-elapsed raise only outside the lock; the sync resolver does it inside _refresh_lock. Under N concurrent verifications against an expired cache with a failing upstream, the async path issues N sequential fetches ignoring the 30s cooldown where sync fails closed after one. Fails closed, so availability only — but mirror the sync in-lock raise.
  • Default helper replay cap is process-global, not per-namespace. agent_resolver.py _DEFAULT_REPLAY_STORE is one module-global store; _NamespacedReplayStore partitions the key but global_cap counts across all origins. A single provisioned signer flooding valid nonces can trip the global cap and fail-close every other counterparty. Still a net improvement over the prior replay_store=None, but scope the cap per namespace or document wiring an explicit per-tenant store for multi-tenant verifiers.
  • Confirm the spec anchors before the diagnostics ship. verifier.py emits step=13 for the authoritative capacity rejection and cites security.mdx:1324/1457/1103 @ AdCP 3.1.8. ad-tech-protocol-expert couldn't verify from this repo whether 3.1.8 defines a step 13 (vs reusing 9a) or whether identity.key_origins is MUST vs SHOULD for brand.json posture. If it's SHOULD, the fail-closed flip rejects conformant brands that haven't published the map. Confirm against the 3.1.8 tag.

Minor nits (non-blocking)

  1. Removed warning hides the low-level footgun. verifier.py _maybe_check_key_origin still returns early on expected_key_origins is None regardless of jwks_source, and this PR deletes the UserWarning that previously fired for brand_json + None. The production helper is fixed (passes ... or {}), but a direct verify_request_signature caller with a brand-json resolver and default None now skips the origin check silently. Keep the one-time warning.
  2. _purge_expired batch bound can over-count at global_cap. replay.py sweeps ≤16 expired entries per call while at_capacity/claim compare len(self._entries) (including unpurged-expired rows) to global_cap. Transient early rate_abuse under short-TTL bursts; self-heals. Worth a comment.
  3. get_idempotency_lock_pool() is documented-only. lazy.py's docstring example now calls it but LazyBackend doesn't require it structurally. Fine, just note adopters who copied the _resolve() pattern must add it.

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The atomicity and trust-boundary hardening is the right shape — replay/idempotency claims go atomic, remote-signing fetches get bounded during streaming, and missing brand key_origins fails closed. Fail-closed beats fail-open, and the concurrency machinery holds up under scrutiny. Blocking on one thing: a breaking public-API change shipped without the semver signal release-please needs.

MUST FIX (blocking)

Breaking public constructor under a fix: prefix. src/adcp/server/idempotency/backends.py:375PgBackend.__init__ now takes lock_pool as a required keyword-only param with no default. PgBackend(pool=pool) — the exact call in the class docstring at L200-L201 and in every existing adopter — now raises TypeError: missing 1 required keyword-only argument: 'lock_pool'. PgBackend is a public export (adcp.server.idempotency.__init__:84).

The commit is fix(signing): harden trust and request atomicity — no !, no BREAKING CHANGE: footer, and no CHANGELOG/MIGRATION file in the diff. Under release-please the conventional-commit type is load-bearing: a fix: cuts a patch and lands this in the changelog as a bugfix, not a breaking change. The migration requirement ("PostgreSQL idempotency users must configure a distinct lock pool") lives only in the PR body, which release-please does not read — so adopters upgrading get a construction-time crash with no changelog signal that they must add a lock_pool.

This is the block. Two ways to clear it:

  • Retitle to fix(idempotency)!: (or add a BREAKING CHANGE: footer carrying the lock_pool migration note), so the major bump and changelog entry are correct. code-reviewer also notes the scope should be idempotency, not signing — the change lives under server/idempotency/** — worth fixing in the same retitle.
  • Or give lock_pool a default of None and fall back to the existing single-pool path with the deprecation warning, keeping PgBackend(pool=pool) working.

Either is fine. The current state — breaking diff under a non-breaking prefix — is not.

Things I checked

  • store.py supervised-task machinery: asyncio.shield + _SUPERVISED_OPERATIONS strong-ref + done-callback consumes the terminal exception without leaking, re-raises CancelledError, and lets the lock-holding task run to completion so no concurrent same-key execution. Correct.
  • Pg hold() advisory-xact-lock with the _active_connection ContextVar keyed on current_task(): get/put reuse the checked-out connection because they run inside the same _execute_locked task; binding is reset in finally. lock_pool is pool is rejected at L378 to prevent the handler/lock-pool deadlock. Right shape.
  • replay.py indexed min-heap (_remove_expiry pop-last / move-into-hole / sift-up-if-smaller-than-parent-else-down, plus the position == len short-circuit): invariant holds, _entries/_counts/_heap_positions stay consistent.
  • SSRF posture survives the client.get()client.stream() switch: _bounded_http rejects non-identity content-encoding before reading, sends Accept-Encoding: identity, short-circuits on oversized declared content-length, and enforces the byte cap independently while streaming. Redirect targets in agent_resolver._fetch_capabilities rebuild an IP-pinned, SSRF-checked transport per hop. (security-reviewer: SSRF fully preserved.)
  • Key-origin fail-closed: verify_from_agent_url passes resolution.key_origins or {} — an empty map, never None — so a brand-sourced resolver with no declared map reaches check_key_origin_consistency({}) and fails closed with request_signature_key_origin_missing. The removed UserWarning only fired on the is None path that already returned. No check dropped.
  • _claim_replay_nonce legacy fallback fails closed: re-checks store.seen() after remember() and returns capacity on a silent drop; invalid claim result → RATE_ABUSE.
  • covers_content_digest wire default is untouched at verifier.py:112 ("either") — only the docstring was rewritten; webhook profile still hard-codes "required". (ad-tech-protocol-expert: no schema-default drift.)
  • Dropping _slide_next_update on 304: a 304 carries no fresh JWS, so it correctly no longer extends the signed next_update boundary. Tightens toward fail-closed. (ad-tech-protocol-expert: correct revocation posture.)

Follow-ups (non-blocking — file as issues)

  • can_serve_stale nullifies DEFAULT_MAX_STALE_SECONDS at defaults. brand_jwks.py caps stale_deadline at min(snap.expires_at + self._max_stale, snap.fetched_at + DEFAULT_MAX_AGE_SECONDS). With the default max_age=900 and expires_at == fetched_at + 900, the second term equals expires_at, so the grace window collapses to zero — the new max_stale=900 never extends trust, and the effective ceiling is 15 min, not the 30 the inline comment and security.mdx:1103 claim. The JWKS side (DEFAULT_JWKS_MAX_AGE_SECONDS=1800) encodes the full ceiling correctly; brand.json doesn't. It errs safe (never more permissive, still fails closed) so it's not a block — but the shipped parameter is dead config. Fix: cap at snap.fetched_at + self._max_age + self._max_stale, and read self._max_age rather than the module constant so a raised max_age_seconds isn't silently pinned at 900. Both security-reviewer (Low) and ad-tech-protocol-expert (material) flagged this.
  • Shared process-wide default replay store couples global_cap across counterparties. agent_resolver._DEFAULT_REPLAY_STORE is one InMemoryReplayStore (global_cap 1M) partitioned only by origin-namespaced keyid. Per-keyid isolation is correct and the length-prefixed namespace (f"{len(ns)}:{ns}{keyid}") is collision-safe. But global_cap counts entries across all namespaces: an onboarded-but-hostile counterparty flooding validly-signed unique nonces can drive at_capacity() true for every other tenant sharing the process (→ request_signature_rate_abuse). Self-heals after TTL; security-reviewer rates it Medium. Also note per_keyid_cap == global_cap == 1M, so at defaults the global cap adds no protection against a single key while carrying the cross-tenant downside. Give each origin its own bounded store, or enforce the global bound per namespace.
  • Async JWKS resolver drops the in-lock cooldown re-check the sync path has. jwks.py AsyncCachingJwksResolver.__call__ checks the expired-cache-within-cooldown guard only before acquiring self._lock; the sync CachingJwksResolver re-checks it inside _refresh_lock. On an expired cache with a failing JWKS host, N concurrent async callers that pass the pre-lock check each fire a fresh live fetch, bypassing the cooldown — a self-inflicted thundering herd where the sync path fails fast. Still fails closed. Mirror the sync guard inside the lock.

Minor nits (non-blocking)

  1. Removed key-origin UserWarning goes quiet on the raw API. verifier.py _maybe_check_key_origin no longer warns when a direct VerifyOptions(expected_key_origins=None) caller skips the brand-json check. The factory path (verify_from_agent_url) is safe since it passes {}, but direct VerifyOptions users lose the operator-log nudge. Worth a changelog line.
  2. Handler now runs in a detached asyncio.create_task. store.py _wrapped runs the handler under a copy of the request's contextvars, so contextvars a handler sets (OTel spans, request-id) are no longer visible to outer middleware. Correct for the shield-across-cancellation goal, but a behavior change worth a note in the PR body.
  3. Per-instance default_factory=InMemoryReplayStore on WebhookVerifyOptions. A receiver rebuilding options per request gets a fresh empty store each time — replay protection silently disabled while appearing configured. Documented in the VerifyOptions docstring; strictly better than the prior None default. Consider a one-time warning when a default-constructed store is used.

Clear the semver signal on the PgBackend constructor and this is ready. The rest is sound — good hardening across the board.

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on one thing: a breaking public-API change shipped under a non-breaking fix: prefix. The security hardening itself is right — atomic replay claims, bounded streaming fetches, fail-closed key-origins. Fix the semver signal and this ships.

Blocker

PgBackend gains a required lock_pool param under a fix: commit with no breaking marker. src/adcp/server/idempotency/backends.py:372__init__ now takes a required keyword-only lock_pool (and ValueErrors if lock_pool is pool). PgBackend is public — adcp.server.idempotency.__all__ carries it (__init__.py:84), and the prior docs constructed it PgBackend(pool=pg_pool). Every existing adopter on that call site now gets TypeError: __init__() missing 1 required keyword-only argument: 'lock_pool' at construction. The commit is fix(signing): harden trust and key lifecycle — no !, empty body, no BREAKING CHANGE: footer, and the scope (signing) doesn't even cover server/idempotency. release-please reads that as a patch bump and the changelog won't flag the constructor break for adopters upgrading within the 7.0.0 RC line.

The distinct-lock-pool design is correct — sharing the advisory-lock pool with handler SQL deadlocks under saturation, and the lock_pool is pool guard is the right shape. This is purely a labeling defect. Two ways out:

  • Relabel the squash commit/PR title as breaking — feat(idempotency)!: or a BREAKING CHANGE: footer — and land the migration note (PgBackend(pool=...)PgBackend(pool=..., lock_pool=...)) in the changelog. The PR body already has the prose; it just needs to reach release-please.
  • Or give lock_pool a safe default and warn when it collapses to pool.

code-reviewer: Blocker, same call.

Things I checked

  • SSRF re-pinning survives the streaming rewrite. agent_resolver._fetch_capabilities redirect branch now lives inside the stream() context but still rebuilds build_async_ip_pinned_transport(url) per hop and fails closed on SSRFValidationError. security-reviewer: defended.
  • Byte-cap is not spoofable. _bounded_http.read_limited_bytes treats Content-Length as an early fast-reject only; the streaming len(body) > limit counter is authoritative, and Accept-Encoding: identity + _reject_encoded_response closes the decompression-bomb vector. test_bounded_fetches.py asserts stream.read == 2 — it stops mid-stream, not after buffering.
  • Legacy replay fallback fails closed. verifier._claim_replay_nonce re-seen()s after remember() and returns capacity on a silent cap-drop → request_signature_rate_abuse, no replay window. InMemoryReplayStore.claim and PgReplayStore.claim (per-keyid pg_advisory_xact_lock) are race-free.
  • key_origins fails closed on brand_json. verify_from_agent_url passes resolution.key_origins or {}; an empty map reaches check_key_origin_consistency and raises request_signature_key_origin_missing at step 7. Error codes (REQUEST_SIGNATURE_REPLAYED, REQUEST_SIGNATURE_RATE_ABUSE) confirmed in errors.py, webhook remap table intact.
  • Idempotency detached-task path is correct. store.py returns the handler result via await asyncio.shield(execution_task); _SUPERVISED_OPERATIONS holds a strong ref, the done-callback consumes the terminal exception idempotently, tenant/scope keys are captured in the closure. No concurrent duplicate execution, no credential path.
  • Replay min-heap bookkeeping is sound. _heap_positions stays consistent across remember/claim/_expire_one/_purge_expired; _remove_expiry handles the last-element pop correctly. No fail-open — eviction only touches entries whose stored expiry < now.
  • Revocation 304 fail-closed is spec-right. Dropping _slide_next_update is correct: a 304 authenticates no new signed next_update, so it can't extend a cryptographic freshness boundary; _ensure_fresh still serves within grace and a re-signed 200 (new ETag) resumes normally.

Follow-ups (non-blocking — file as issues)

  • Shared global_cap is a cross-counterparty DoS surface. agent_resolver.py:1110 — the default verify_from_agent_url store is one process-wide InMemoryReplayStore; _NamespacedReplayStore only prefixes the kid, so all counterparties share the 1M global_cap and _entries table. One authenticated high-volume signer can fill it and starve replay verification for every other origin in the process (security-reviewer: Medium — DoS only, no fail-open). For hostile-multi-tenant deployments, wire a shared PgReplayStore (per-keyid cap only) or allocate per-origin stores with a bounded origin count. At minimum document it on verify_from_agent_url.
  • DEFAULT_MAX_STALE_SECONDS is neutralized by its own clamp. brand_jwks.py can_serve_stale clamps to min(snap.expires_at + self._max_stale, snap.fetched_at + DEFAULT_MAX_AGE_SECONDS); since _compute_lifetime already bounds expires_at at fetched_at + max_age, the second term always wins and the new stale grace is dead code — effective brand.json trust ceiling is 15 min, not the 30 the comment cites. ad-tech-protocol-expert: sound-with-caveats. Errs conservative (refuses stale earlier), so not a block, but the knob you just added does nothing under default config — clamp to self._max_age + self._max_stale.
  • step=13 on the capacity raise is unverified against the wire profile. verifier.py emits request_signature_rate_abuse at both step=\"9a\" (early-out) and step=13 (atomic claim). ad-tech-protocol-expert couldn't confirm step 13 against security.mdx:1324 (spec lives in adcontextprotocol/adcp, not vendored). Confirm the spec enumerates a step 13 for the atomic insert before conformance harnesses keying on (code, step) diverge.
  • Hung handler pins a lock_pool connection + advisory lock indefinitely. store.py:666 detaches the task on cancellation; a handler that hangs holds the PgBackend advisory-lock connection for its full runtime. Enough hung keys exhaust the (deliberately small) lock_pool. Worth a handler-timeout / max-in-flight bound on _SUPERVISED_OPERATIONS.
  • Document the verify_from_agent_url replay-default flip. Default moved from None (replay off) to an installed process-wide store. Security win, but it's a behavior change with no signature change — call it out in the changelog so callers who relied on the None default aren't surprised, and so multi-process operators don't take false confidence from a per-process store.

Minor nits (non-blocking)

  1. Sync/async JWKS cooldown re-check divergence. jwks.py — sync CachingJwksResolver.__call__ re-checks the expired+cooldown guard inside _refresh_lock; async AsyncCachingJwksResolver.__call__ checks it only before acquiring self._lock. Both fail closed, but under concurrent expiry the async path lets a few queued waiters each re-fetch instead of raising. Consistency only.
  2. _purge_expired over-counts against global_cap. replay.py sweeps at most _SWEEP_BATCH (16) expired entries per call, but remember/at_capacity compare len(self._entries) — including not-yet-swept expired rows — against global_cap. A burst yields transient spurious capacity rejections until the heap catches up. Self-correcting; drain the expired heap-top fully before the cap comparison if you want it tight.

Relabel the break and this is safe to merge — the trust-boundary work is the right shape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants