fix(cf-workers): size aws-chunked streamed PUTs so the egress leg has a Content-Length

[alukach]

Candidate fix for intermittent 520s on streamed uploads through the Workers runtime. See source-cooperative/data.source.coop#206 for the production evidence.

What I'm changing

WorkerBackend::forward streams PUT bodies through without buffering them into WASM memory. A bare ReadableStream body makes the Workers runtime send the subrequest with Transfer-Encoding: chunked and drop Content-Length, so the branch wraps the stream in a FixedLengthStream to force a real Content-Length — except that fixed_body_length() returned None for any content-encoding: aws-chunked body, exempting exactly the shape AWS SDKs and the CLI send by default.

The exemption was justified in-comment by "aws-chunked bodies are sized by S3 from x-amz-decoded-content-length". That is true for the object: S3 uses it to size the payload it reconstructs after de-chunking. It says nothing about the HTTP leg, which was left unsized.

Content-Length is the correct size for an aws-chunked body too. It counts the bytes actually placed on the wire — chunk framing and trailer included — e.g. 10094598 encoded vs 10094463 decoded, a 135-byte gap of framing plus the CRC32 trailer. Sizing the leg does not reframe the body, so the chunk framing still reaches S3 untouched. It is forwarded unsigned (build_streaming_forward: "the transfer framing is the runtime's to manage, so signing it risks a mismatch"), so changing the framing cannot invalidate the SigV4 signature.

Why this is a candidate and not a confirmed fix

On data.source.coop, ~0.6% of aws-chunked PUTs (25/4,059 in a 15-minute production sample) fail as a Cloudflare-minted 520 on the worker's own egress leg — server: cloudflare, no x-amz-* headers, a 16-byte error code: 520 body, dead in 34–701 ms. S3 never produces an HTTP response at all.

That rate is the problem for this change's own theory. The failures are Poisson (CV of inter-failure gaps = 0.99) and independent of body size (failed p50 6.56 MB vs succeeded 6.76 MB) and of colo (0.43% vs 0.82% across the only two in the sample). A structurally identical request failing 0.6% of the time is a race, not a protocol defect — if an unsized chunked leg were simply rejected, the rate would be ~100%, not 0.6%. S3 evidently accepts this wire shape thousands of times per window.

So the missing Content-Length can be at most a predisposing condition, never the trigger. The surviving form of the argument is second-order: a request with a known Content-Length is replayable, whereas a chunked body streaming from a live inbound socket is not — so a transient egress event (a stale pooled keep-alive connection, a reset) that would otherwise be retried invisibly instead surfaces as an empty response. Pooled-connection races are Poisson, size-independent, and kill you just after headers, which fits. That is a plausible mechanism, not a demonstrated one, and the 520 is minted inside Cloudflare's egress so S3's side of the connection is not observable from the worker.

This PR may therefore be a no-op for the bug. #141 (stacked on this branch) is the experiment that tells you: it captures the pipe_to rejection, and a 520 arriving with no rejection is connection-side evidence for this change, while one arriving with a rejection means the fault is body-side and this change is irrelevant. Landing this without that instrumentation risks a green soak that nobody can attribute.

Independent of whether it fixes the 520s, the sizing itself is a correctness improvement: an unsized leg is a strictly worse thing to hand an origin than a sized one, at no cost.

How I did it

• crates/core/src/backend/mod.rs — new pub fn streamed_put_body_length(&HeaderMap) -> Option<u64>: the old helper minus the aws-chunked early return, so Content-Length now sizes both body shapes. Returns None only when there is no usable Content-Length. Doc comment records why the encoded length is the right number and why this does not disturb the framing.
• crates/cf-workers/src/backend.rs — drops the local fixed_body_length, calls the core helper, and rewrites the stale rationale comment on the PUT branch.
• Why the move. crates/cf-workers is absent from the workspace default-members, so cargo test never compiles it and a #[cfg(test)] module there would be dead (the existing one in cors.rs never runs). The function is pure header inspection with no wasm dependencies, so hosting it in crates/core — next to the existing free pub fn create_builder — is what makes it testable at all. No visibility widening was needed elsewhere.
• No docs/ change: the sizing behaviour is not documented there (docs/reference/operations.md covers aws-chunked only for conditional-write precondition forwarding).
• Cargo.lock deliberately left untouched; it carries an unrelated stale 0.7.1→0.7.2 version correction.

Test plan

Bug-first, per CLAUDE.md. The three tests were added against the unfixed helper and aws_chunked_put_is_sized_by_encoded_content_length failed as intended (left: None, right: Some(10094598)) before the early return was removed — so the test demonstrably detects the bug rather than merely passing.

• cargo test — 161 core + all suites pass, 0 failures
• cargo fmt
• cargo clippy --all-targets — clean (one pre-existing too_many_arguments warning, unrelated)
• cargo check -p multistore-cf-workers --target wasm32-unknown-unknown

Not covered here, and needed before this can be called confirmed:

• Attribute the failure with feat(cf-workers): log the streamed PUT body pipe's rejection #141 first — a 520 with no pipe rejection is connection-side (supports this PR); a 520 with one is body-side (this PR is irrelevant to the bug).
• Framing assertion against a stub origin — that an aws-chunked PUT's subrequest now carries Content-Length and no Transfer-Encoding: chunked, and that the bytes arrive byte-identical with framing and CRC32 trailer intact. This needs a workerd/miniflare harness, which does not exist in this repo.
• Soak. At a 0.6% baseline a small clean run proves nothing — (1-0.006)^100 ≈ 55%. Roughly 1,000 consecutive clean multi-MB aws-chunked PUTs puts the odds of a broken fix still looking clean near 0.2%.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B3x2KzUtojpwPvDKmSgvht

[claude]

Claude finished @alukach's task in 1m 38s —— View job

---

✅ No blocking issues — safe to merge.

Reviewed the diff (crates/core/src/backend/mod.rs, crates/cf-workers/src/backend.rs, Cargo.lock) against origin/main.

Correctness

• Core fix is sound: streamed_put_body_length (crates/core/src/backend/mod.rs:228) drops the aws-chunked early-return that previously exempted exactly the body shape that needed sizing. The doc comment's claim that Content-Length counts the encoded (framed) bytes rather than the decoded payload is correct and matches how x-amz-decoded-content-length is defined vs. Content-Length in the SigV4 streaming spec.
• Traced that Content-Length actually reaches this function on the aws-chunked path: AWS_CHUNKED_FORWARD_HEADERS (crates/core/src/proxy.rs:104) doesn't list content-length, but build_streaming_forward (crates/core/src/proxy.rs:1247-1248) copies it from original_headers unconditionally, after signing, unsigned — so the header the new function keys off is present for both the aws-chunked and plain PUT paths. No dead code path here.
• crates/cf-workers/src/backend.rs correctly swaps in the shared helper and rewrites the stale rationale comment; HeaderMap import is still used elsewhere in the file (no leftover unused import).
• No other backend (lambda, etc.) has an equivalent stream-body-sizing branch, so there's no sibling caller left un-fixed.
• Cargo.lock version bump (0.7.1→0.7.2) matches the workspace Cargo.toml version — a legitimate drift correction, not scope creep.

Tests

• The three new tests in crates/core/src/backend/mod.rs cover: plain PUT, aws-chunked PUT (the bug case), and the no-Content-Length fallback. Per the PR description these were run against the unfixed helper first and failed as expected — consistent with the repo's bug-first testing convention.
• Note: I wasn't able to execute cargo test / cargo check -p multistore-cf-workers --target wasm32-unknown-unknown myself in this environment (Bash execution isn't available here), so I verified correctness by tracing the header flow through proxy.rs instead of running the suite.

Simplify (ponytail): none — the new function is a minimal header lookup, the move to crates/core is justified (it's the only crate where #[cfg(test)] actually compiles), and the doc comment length is proportionate to a newly-pub function per this repo's CLAUDE.md convention.

Branch: fix/size-aws-chunked-streamed-put

---

💰 Estimated review cost: $0.44 · 1m37s · 26 turns