Skip to content

Collector: a client that goes away no longer cancels ingest - #164

Merged
Karanvir1729 merged 1 commit into
mainfrom
fix/collector-disconnect
Nov 4, 2026
Merged

Karanvir1729 merged 1 commit into
mainfrom
fix/collector-disconnect

Conversation

@Karanvir1729

Copy link
Copy Markdown
Collaborator

Summary

  • ingest() registered its counting with waitUntil only after awaiting the R2 write. A client that reset its connection during that write (about 150 ms; for example a browser being shut down) cancelled the handler, so the batch stayed in R2 but never reached the dashboard's counters, the export index or forwarding. A reset before the write lost the batch altogether.
  • handle() now passes the whole of ingest() to opts.waitUntil from its first line. The response is unchanged and other routes are not touched.

Evidence

Local workerd with the real entry point (src/index.ts). R2's timing is simulated in the two reset rows, where the client resets 100 ms in. Each cell is batches stored in R2 / counted on the dashboard:

Client behaviour Batches Before After
Waits for the reply 20 20 / 20 20 / 20
Closes normally 10 10 / 10 10 / 10
Resets, write landed but not yet acknowledged 10 10 / 0 10 / 10
Resets before the write starts 10 0 / 0 10 / 10
Resets with half the body sent 5 0 / 0 0 / 0

A batch sent twice still counts once. A plain close (FIN) does not cancel the handler under workerd, so reproducing this needs a connection reset (socket.resetAndDestroy() in Node).

In production the cancellation itself can't be observed, because logging is off and the error was swallowed. What points to it:

  • One app had 13 of 198 page loads in R2 and not in the dashboard. All 13 were solo visits by one automated browser, and every batch of the other 185 page loads was counted.
  • Replaying those 16 batches locally over an open connection counted all 13 page loads, so their content is not the cause.
  • Cloudflare recorded 72 client disconnects on the collector from 20:00 to 03:00 UTC, all during that browser's active hours.
  • Across all apps since Nov 2, 7 of 355 page loads with a session event are missing from the dashboard.

Test plan

  • cd telemetry-worker && npx vitest run: 363 pass. The two new tests in test/index.test.ts failed before the fix.
  • The local workerd reset test above, before and after. The harness is not committed.
  • After deploy, compare the R2-fed admin sessions with the dashboard's sessions rows for new page loads. The gap should close.

Deploying

Not deployed. The live worker runs 931b6f45, so deploying from main after this merges also ships e8767ebf and 22590994, which change text in pages.ts and studio.ts. There are no migrations. Counts already missed are not backfilled.

🤖 Generated with Claude Code

…could reach R2 and never be counted)

ingest() registered its counting with waitUntil only after awaiting the R2 write. A client that reset its connection
during that write (about 150 ms; a browser being shut down) cancelled the handler, so the batch stayed in R2 without
reaching the dashboard's counters, the export index or forwarding. A reset before the write lost the batch altogether.
waitUntil now holds ingest from its first line; the response is unchanged.

Under local workerd with the real entry point, 10 batches per case, the client resetting 100 ms in, R2's timing modelled:
  write landed, ack pending: before 10 stored, 0 counted; after 10 stored, 10 counted
  write not started yet:     before  0 stored, 0 counted; after 10 stored, 10 counted
No change when the client waits or closes normally; half a body still stores nothing; a repeated batch still counts once.
Production itself is not observed (logging is off, the error was swallowed): one app showed 13 of 198 page loads in R2
and not in D1, all solo visits by an automated browser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Karanvir1729
Karanvir1729 merged commit 6949d46 into main Nov 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant