Add outbound activity observer

CHANGES.md

@@ -48,6 +48,20 @@ To be released.
     `idempotency` strategy, as direct processing already did, instead of always
     deduplicating them per origin.  Activities recorded by workers of older
     versions are still recognized.  [[#1290], [#1299]]
+ -  Added application-facing outbound observers for sends, forwarding,
+    replication, and queued retries, independently of OpenTelemetry. Optional
+    callbacks report preparation, delivery handoffs, send/HTTP results, and
+    worker actions with correlation IDs and preparation failure details.
+    Applications can opt in to redacted headers, bounded streaming response
+    capture, and privacy-filtered sender payloads. Inbox reports now preserve
+    original request context and snapshot request cancellation without changing
+    processing outcomes. Malformed optional queue metadata no longer prevents
+    delivery; consumers warn and continue without its correlation facts. Error
+    diagnostic reads have a one-second deadline.
+    The public `Federatable` interface now requires `setOutboundObservers()`,
+    breaking source compatibility for custom implementations. Update custom
+    mocks, decorators, and wrappers to implement this method.
+    [[#1316], [#1359] by Jiwon Kwon\]
  -  Added the `activity-origin-mismatch` reason to rejected
     `InboxAuthentication` decisions in `onRequestFinished` reports.  Inbox
     deliveries rejected because the activity's ID does not have the same origin
@@ -87,9 +101,11 @@ To be released.
 [#1289]: https://github.com/fedify-dev/fedify/issues/1289
 [#1290]: https://github.com/fedify-dev/fedify/issues/1290
 [#1299]: https://github.com/fedify-dev/fedify/pull/1299
+[#1316]: https://github.com/fedify-dev/fedify/issues/1316
 [#1318]: https://github.com/fedify-dev/fedify/issues/1318
 [#1328]: https://github.com/fedify-dev/fedify/pull/1328
 [#1353]: https://github.com/fedify-dev/fedify/pull/1353
+[#1359]: https://github.com/fedify-dev/fedify/pull/1359
 
 ### @fedify/cli
 
@@ -312,6 +328,12 @@ To be released.
 
 ### @fedify/testing
 
+ -  Added outbound observer registration support to mock federations and
+    dispatch IDs to recorded sends. Tests can inspect `dispatchId` through
+    `context.getSentActivities()` and `federation.sentActivities` to assert
+    application correlation. Mock sends reject invalid dispatch IDs before
+    recording, using production's 1,024-byte UTF-8 limit. Mock sends do not
+    invoke outbound observer callbacks. [[#1316], [#1359] by Jiwon Kwon\]
  -  Mock inbox contexts made by `createInboxContext()` now have the
     `authentication` property, which can be given as an option and defaults
     to evidence that nothing was authenticated.  [[#1290], [#1299]]

changes.d/fedify/outbound-observers.md

@@ -0,0 +1,19 @@
+---
+links:
+  '#1316': https://github.com/fedify-dev/fedify/issues/1316
+  '#1359': https://github.com/fedify-dev/fedify/pull/1359
+---
+ -  Added application-facing outbound observers for sends, forwarding,
+    replication, and queued retries, independently of OpenTelemetry. Optional
+    callbacks report preparation, delivery handoffs, send/HTTP results, and
+    worker actions with correlation IDs and preparation failure details.
+    Applications can opt in to redacted headers, bounded streaming response
+    capture, and privacy-filtered sender payloads. Inbox reports now preserve
+    original request context and snapshot request cancellation without changing
+    processing outcomes. Malformed optional queue metadata no longer prevents
+    delivery; consumers warn and continue without its correlation facts. Error
+    diagnostic reads have a one-second deadline.
+    The public `Federatable` interface now requires `setOutboundObservers()`,
+    breaking source compatibility for custom implementations. Update custom
+    mocks, decorators, and wrappers to implement this method.
+    [[#1316], [#1359] by Jiwon Kwon]

changes.d/testing/outbound-observers.md

@@ -0,0 +1,11 @@
+---
+links:
+  '#1316': https://github.com/fedify-dev/fedify/issues/1316
+  '#1359': https://github.com/fedify-dev/fedify/pull/1359
+---
+ -  Added outbound observer registration support to mock federations and
+    dispatch IDs to recorded sends. Tests can inspect `dispatchId` through
+    `context.getSentActivities()` and `federation.sentActivities` to assert
+    application correlation. Mock sends reject invalid dispatch IDs before
+    recording, using production's 1,024-byte UTF-8 limit. Mock sends do not
+    invoke outbound observer callbacks. [[#1316], [#1359] by Jiwon Kwon]

claude-plugin/skills/actor/SKILL.md

@@ -20,7 +20,7 @@ Walk through:
 3.  Chaining `.setKeyPairsDispatcher()` to supply RSA and Ed25519 keys.
 4.  Optionally configuring `mapActorAlias()` for handle-based or
     fixed-path aliases.
-5.  Making the actor discoverable via WebFinger. Fedify handles
+5.  Making the actor discoverable via WebFinger.  Fedify handles
     `/.well-known/webfinger` automatically once the actor dispatcher
     is registered.
 

claude-plugin/skills/docs/SKILL.md

@@ -19,7 +19,7 @@ Fetch and explain Fedify documentation about “$ARGUMENTS”.
 
     The documentation index is at <https://fedify.dev/llms.txt>.
 
-2.  Summarise the key points with runnable TypeScript examples.
+2.  Summarize the key points with runnable TypeScript examples.
 
 3.  Mention related pages the user might also want to read.
 

claude-plugin/skills/fedify/SKILL.md

@@ -244,7 +244,66 @@ deliveries have no retry.
 > those must be distinct activities.  Use a fresh UUID or counter in the
 > fragment.
 
-See <https://fedify.dev/manual/send.md>.
+### Observing outbound delivery
+
+Register callbacks with `federation.setOutboundObservers()` or the builder's
+matching method.  Registration replaces the whole set; `{}` clears it.  Active
+sends keep the callbacks selected when they began.  Observers use the existing
+context and work without OpenTelemetry.
+
+Choose callbacks by the question you need to answer:
+
+| Callback         | Reports                                                                    |
+| ---------------- | -------------------------------------------------------------------------- |
+| `onBroadcast`    | One send or forward invocation, including preparation and its result.      |
+| `onDelivery`     | Each destination's plan, handoff, or preparation failure.                  |
+| `onAttempt`      | The outcome of one concrete send, including each retry or gateway attempt. |
+| `onHttpExchange` | Each actual HTTP request and its response or fetch error.                  |
+| `onWorker`       | The local result and scheduling actions of a fanout or outbox worker.      |
+
+`onBroadcast` and `onDelivery` also cover automatic portable replication.
+Broadcasts have a `broadcastId`; destinations have a `deliveryId`.  An
+`onDelivery` event can refer to an unknown broadcast when it comes from legacy
+fanout work.  These IDs group reports; they do not prove remote receipt.
+The event.activity reference is a snapshot of known id and type IRI strings; it
+may be empty on started and can gain facts later.
+
+For broadcasts, narrow `event.stage` before reading `event.result.type`.
+For deliveries, narrow `event.stage` and, for handoffs, `event.result.type`.
+Attempt reports have a fresh `attemptId` and an `outcome`; their
+`httpExchangeCount` counts actual fetches, including redirects and signature
+challenges.  A failure before the first fetch has a count of zero.  Worker
+reports distinguish normal local completion from an escaping error.  Their
+`actions` record enqueues, retries, deferrals, direct handoffs, abandonment,
+and failed drops.  A completed worker does not imply remote success or queue
+acknowledgment.
+
+Header capture is off by default.  Set `requestHeaders` or `responseHeaders`
+with an allowlist and byte limit to add redacted header snapshots to exchange
+reports.  `responseBodyCapture` separately invokes `onBody` with a bounded
+stream and deadline; read it incrementally and honor its cancellation signal.
+Use `senderPayload: true` only when the attempt report needs a privacy-filtered
+projection of the outgoing activity.  These options do not change the bytes
+sent or signed.  Treat all captured evidence as sensitive application data.
+
+Callbacks are awaited, and callback or logging errors do not replace delivery
+results.  A callback that never settles can still delay the send.  If you
+record events in the background, handle rejection, shutdown, and persistence
+in the application.
+
+Use the `dispatchId` send or forward option to carry an application correlation
+ID through queues.  Event `provenance` preserves known source facts; `local`
+describes the current execution.  Optional `trace` contains the operation's
+own span IDs when tracing is enabled.  Older queue messages can lack these
+facts.  Consumers warn and omit malformed queued provenance so delivery can
+continue.  Invalid `dispatchId` values on new sends raise an error before
+preparation; the limit is 1,024 UTF-8 bytes.  Reusing an ID does not
+deduplicate sends.
+
+For typed examples, report variants, and capture limits, see the
+[sending manual].
+
+[sending manual]: https://fedify.dev/manual/send.md
 
 
 Vocabulary imports

claude-plugin/skills/fep/SKILL.md

@@ -15,7 +15,7 @@ explain how to implement it with Fedify.
 Normalization
 -------------
 
-Normalise the identifier first: strip any leading `FEP-` or `fep-` prefix
+Normalize the identifier first: strip any leading `FEP-` or `fep-` prefix
 and lowercase the result to get the bare four-character hex id (e.g. `8fcf`).
 Call that `$ID` in the steps below.  If the result does not match
 `^[0-9a-f]{4}$`, stop and ask the user to provide a valid FEP identifier.
@@ -39,7 +39,7 @@ cat "$FEP_DIR/fep/$ID/fep-$ID.md"
 Summary
 -------
 
-Summarise: status, what problem it solves, and what extensions it defines
+Summarize: status, what problem it solves, and what extensions it defines
 (new JSON-LD terms, HTTP endpoints, or activity shapes).
 
 

claude-plugin/skills/migration/SKILL.md

@@ -16,7 +16,8 @@ Migration workflow
 1.  Fetch *CHANGES.md* from the repo to identify breaking changes between
     the versions in question:
     `https://raw.githubusercontent.com/fedify-dev/fedify/main/CHANGES.md`
-2.  List every breaking change that affects the user's code range.
+2.  List the breaking changes between those versions that affect the user's
+    code.
 3.  For each breaking change, show the old API, the new API, and a concrete
     before/after code snippet.
 4.  Search the user's codebase for usages of deprecated symbols and suggest

docs/manual/context.md

@@ -84,6 +84,13 @@ The `Context` object has properties to get the base URL of the current request:
 For `RequestContext`, there is an additional property named
 `~RequestContext.url` that contains the full URL of the current request.
 
+Outgoing observations snapshot these values under `event.local`. For work
+caused by an inbox request, `event.provenance.inboxRequest.context` retains the
+original URL facts through derived contexts and queues. Inbox completion reports
+preserve the original values under `report.local` and separately snapshot
+request cancellation; neither changes the lifetime of accepted queue work. See
+[outgoing source and correlation](./send.md#original-source-and-current-context).
+
 [^1]: If no canonical origin is explicitly configured, it is the same as the
       `Context.origin`. See also the [*Explicitly setting the canonical origin*
       section](./federation.md#explicitly-setting-the-canonical-origin) in the

docs/manual/inbox.md

@@ -206,6 +206,9 @@ federation
   .setInboxListeners("/users/{identifier}/inbox", "/inbox")
   .onRequestFinished(async (ctx, report) => {
     await saveReport(report);
+    if (report.requestCancellation.type === "aborted") {
+      console.log("Request aborted", report.requestCancellation.reason);
+    }
   });
 ~~~~
 
@@ -215,6 +218,29 @@ covers failures while preparing document loaders or resolving a portable
 recipient.  Inbox collection requests, unmatched routes, programmatic
 `routeActivity()` calls, and queue workers do not invoke it.
 
+Since Fedify 2.5.0, routed HTTP inbox reports include `report.inboxRequestId`
+and `report.requestId`, including preparation failures. The former is a fresh
+server-generated ID for each HTTP delivery, even for the same Activity. The
+latter is an informational, externally supplied or generated logging label;
+it is not unique. These fields are absent when no routed source is known.
+
+`report.local` preserves the original `origin`, `host`, `hostname`, and
+`canonicalOrigin`, even when a later context's URL changes. It is absent when
+no routed source is known. `report.requestCancellation` snapshots the native
+request signal immediately before report construction: `not-aborted`, or
+`aborted` with the exact `reason`, including non-Error values. These envelopes
+are immutable; the reason object itself is not cloned or frozen.
+
+Request cancellation is separate from processing outcome. A processed or
+enqueued `202` can coexist with an aborted signal. The callback still runs once
+when processing settles; an abort after the snapshot does not change it or
+trigger another report. Observation does not cancel HTTP processing or pass its
+signal to accepted queue work or outgoing descendants.
+
+Outgoing work using the handler's context retains this source through derived
+contexts and queues. See
+[outgoing source and correlation](./send.md#original-source-and-current-context).
+
 `report.payload` is either `unavailable` or `parsed`, whose `value` contains the
 original JSON.  A parsed JSON `null` is distinct from an unavailable body.
 `report.activity` contains the `Activity` obtained by the existing processing
@@ -963,6 +989,9 @@ federation.setInboxListeners("/{identifier}/inbox", "/inbox")
 })
 ~~~~
 
+To record forwarding preparation, delivery attempts, and HTTP results, see
+[observing outgoing activities](./send.md#observing-outgoing-activities).
+
 > [!NOTE]
 > The `~InboxContext.forwardActivity()` method does not guarantee that the
 > forwarded activity is successfully delivered to the recipient, since

docs/manual/mq.md

@@ -1297,6 +1297,38 @@ Optimized performance
 :   Backend-specific optimizations for retry logic.
 
 
+Observing outgoing workers
+--------------------------
+
+*This API is available since Fedify 2.5.0.*
+
+Register `onWorker` through `Federatable.setOutboundObservers()` to observe
+local fanout and outbox execution, including calls to
+`Federation.processQueuedTask()`. Reports include the current task, concrete
+send count, local result, and actual child enqueue, retry, deferral, or
+abandonment actions. They receive the current worker context and data, with
+known delivery identities and source metadata restored from the queued message.
+Inbox and custom-task processing use separate mechanisms.
+
+A completed worker report means the local path returned normally; it can include
+a handled send failure or abandonment. Reports retain caught preparation errors
+under `preparationFailures`, even when another gateway succeeds. Actions also
+retain their contributing preparation failures, with affected routes and
+operation phases. An accepted enqueue can survive a later worker failure, and a
+rejected enqueue or batch does not prove the backend accepted nothing. Reports
+do not establish backend acknowledgment or future native retry execution. Use
+attempt observers for concrete send results and queue depth for messages still
+waiting. See
+[worker execution and actions](./send.md#worker-execution-and-actions) for
+report variants and a typed example.
+
+Metadata callbacks are awaited without a default timeout. Optional
+[response-body capture](./send.md#streaming-response-bodies) instead has
+explicit byte/time limits; several selected exchanges can accumulate waiting
+time within one worker execution. Applications own archival and must handle
+recording work that outlives the capture signal or a backend deadline.
+
+
 Queue depth reporting
 ---------------------