Skip to content

fix(js-sdk): release inflight concurrency slot on body end, not headers - #1717

Open
devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
devin/1787165477-inflight-body-end
Open

fix(js-sdk): release inflight concurrency slot on body end, not headers#1717
devin-ai-integration[bot] wants to merge 4 commits into
mainfrom
devin/1787165477-inflight-body-end

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #1666.

limitConcurrency released its semaphore slot as soon as fetcher resolved with the response headers, so streaming responses (logs, command output) stopped counting against the cap while their bodies were still occupying HTTP/2 streams — the cap could overshoot into ERR_HTTP2_TOO_MANY_CONCURRENT_STREAMS. This resolves the existing TODO in inflight.ts.

Now the slot is held until the body ends:

const release = await sem.acquire(signal, streaming, onQueuedAbort)
let response: Response
try {
  response = await fetcher(input, init)
} catch (err) {
  release()  // fetch rejection still frees the slot immediately
  throw err
}
if (!hasBodyToTrack(response, method)) {
  release()  // no bytes will ever arrive: nothing to wait for
  return response
}
return releaseOnBodyEnd(response, release)

No-byte responses release immediately (hasBodyToTrack): null-body statuses (204/205/304), a body a mock/interceptor already read or locked (bodyUsed/locked — also prevents getReader() throwing past the wrapper and leaking the slot), HEAD responses, and content-length: 0. These are exactly the shapes openapi-fetch returns without touching the body, so a slot waiting on one would deadlock the cap.

Reserved unary capacity (consolidating the fix from the now-closed #1695): long-lived Connect streams (Content-Type: application/connect+… — background commands, PTYs, watchDir) would otherwise hold slots for their whole lifetime and starve the unary calls that end them (commands.kill, sendStdin, list, pty.*) on the same envd RPC semaphore. limitConcurrency(fetcher, max, { reserved }) caps streaming requests at max - reserved; when max is too small to carve out the reservation (e.g. max=1), unary gets a matching overflow slot beyond max, so a stream can never wedge its own teardown. createEnvdRpcFetch reserves 1 slot. Unary waiters are woken before streaming ones.

Queued-abort messaging: a request aborted while still waiting for a slot now rejects with a TimeoutError naming the env var that configured the cap (E2B_API_INFLIGHT_REQUESTS / E2B_ENVD_INFLIGHT_REQUESTS / E2B_ENVD_RPC_INFLIGHT_REQUESTS) instead of echoing the caller's requestTimeoutMs abort — raising the request timeout only lengthens the hang.

releaseOnBodyEnd:

  • swaps the body for a passthrough ReadableStream reading through body.getReader() that releases (idempotently, via the semaphore's slot handle) when the underlying stream is drained, errors, or is cancelled. Reading through the reader — rather than handing the stream itself to new Response(...) — keeps foreign/cross-realm streams working, matching the repo's class-agnostic conventions in is.ts;
  • also observes reader.closed, so a source that terminates between pulls (e.g. an abort erroring the stream while a chunk sits in the queue with no read pending) still releases the slot;
  • the wrapped new Response(passthrough, response) gets url / redirected / type carried over via Object.defineProperties, since the constructor only copies status/statusText/headers;
  • if the global Response can't represent the original (ponyfill edge cases), it falls back to the original response and releases the slot right away instead of breaking the request.

Tests (15 in tests/api/inflight.test.ts, run in both node and workerd) cover: slot held after headers until the body is read, release on body cancel / error / error-between-pulls, immediate release for 204, content-length: 0 (asserting bodyUsed === false), HEAD, and locked/used bodies, reserved streaming capacity with unary admission, the max=1, reserved=1 overflow, the env-var-naming queued abort, and field preservation on the wrapped response.

No user-facing API change — behavior-only fix; changeset included (patch for e2b). Python SDK has no equivalent inflight cap, so no parity change is needed.

Link to Devin session: https://app.devin.ai/sessions/66bc0b6c5e204bb98db02aacffea68fe
Requested by: @mishushakov

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@cla-bot cla-bot Bot added the cla-signed label Aug 19, 2026
@changeset-bot

changeset-bot Bot commented Aug 19, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: bac0b8e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
e2b Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Package Artifacts

Built from 5f158d0. Download artifacts from this workflow run.

JS SDK (e2b@2.42.1-devin-1787165477-inflight-body-end.0):

npm install ./e2b-2.42.1-devin-1787165477-inflight-body-end.0.tgz

CLI (@e2b/cli@2.16.4-devin-1787165477-inflight-body-end.0):

npm install ./e2b-cli-2.16.4-devin-1787165477-inflight-body-end.0.tgz

Python SDK (e2b==2.42.0+devin.1787165477.inflight.body.end):

pip install ./e2b-2.42.0+devin.1787165477.inflight.body.end-py3-none-any.whl

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be39cf12e3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

controller.close()
return
}
controller.enqueue(result.value)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Observe source errors that occur between pulls

When the source yields a chunk and then errors before the downstream consumer requests another chunk, enqueue fills the passthrough queue so no reader.read() remains pending; consequently neither this catch nor cancel invokes done(). This occurs, for example, when a streaming response is paused after a chunk and its request signal is then aborted, and it leaves the semaphore slot occupied even though the HTTP/2 stream has terminated, eventually blocking queued requests. Observe rejection of reader.closed (while retaining the current EOF behavior) so asynchronous source errors also release the slot.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in 22860fa. releaseOnBodyEnd now also observes reader.closed (void reader.closed.then(done, done)), so a source that terminates between pulls — e.g. an abort erroring the stream while a chunk sits in the queue with no read pending — still releases the slot. EOF behavior is unchanged (done is idempotent via releaseOnce). Added a regression test: limitConcurrency releases when the body errors between pulls.

@mishushakov

Copy link
Copy Markdown
Member

@devin-ai-integration check comments

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this reintroduces an envd RPC deadlock, and open PR #1695 already fixes it

The premise checks out — I confirmed the base really does release on headers, and that holding to body end is the right shape for #1666. But the naive version of this change wedges the envd RPC stack, and #1695 (still open, same issue, same objective) already carries fixes for every point below. These two should be consolidated rather than landed side by side.

1. Open streams starve the unary calls that end them (blocking)

createEnvdRpcFetch hands one process-wide semaphore (module-level cache, shared across every sandbox) to the whole Connect transport. Long-lived server-streaming calls — commands.run({ background: true }) and commands.connect (rpc.start/rpc.connect), pty.start/pty.connect, files.watchDir — now hold a slot for the stream's entire lifetime. The unary calls that end those streams (commands.kill, sendStdin, sendSignal, closeStdin, list, pty.sendInput/sendSignal) go through the same this.rpc.*, so they queue behind the streams they exist to tear down.

Live against a real sandbox with E2B_ENVD_RPC_INFLIGHT_REQUESTS=1 and one sleep 60 background command open:

call (requestTimeoutMs: 5000) this PR base d55ddb8
commands.list fails at 5002 ms OK in 137 ms
commands.sendStdin fails at 5002 ms reached the server in 88 ms
commands.kill fails at 5000 ms OK in 131 ms

(The base sendStdin "failure" is the legitimate SandboxError 13: stdin not enabled or closed — it still proves the request left the process.) Same repro, same numbers as on #1695. The only escape is the caller's own abort signal; nothing in the SDK breaks the cycle.

At the default cap of 2000 you need many concurrent streams to hit this, but the semaphore is process-wide across all sandboxes, anyone who lowers the documented knob hits it immediately, and item 4 below makes the pool shrink monotonically. #1695 solves it by classifying Connect streaming requests off the application/connect+… Content-Type and reserving slots that streams cannot occupy, with an overflow slot so a cap of 1 still lets teardown through.

2. The failure surfaces as advice to raise requestTimeoutMs, which makes it worse

Every starved call above reported:

TimeoutError [canceled] The operation was aborted due to timeout: This error is
likely due to exceeding 'requestTimeoutMs'. You can pass the request timeout
value as an option when making the request.

The request never left the process, so raising requestTimeoutMs only lengthens the hang. TASTE: "Timeout errors must tell the user which knob to turn." Before this change queue waits were transient and the message was merely imprecise; now a queue wait can last as long as a stream, so the message needs to name the concurrency env var. That requires threading the knob name into limitConcurrency, which is what #1695's LimitConcurrencyOptions.envVarName does.

3. content-length: 0 and HEAD work only by accident

See the inline comments — the if (!body) guard misses both, and they survive today only because the default queuing strategy's highWaterMark of 1 triggers one eager pull.

4. An abandoned stream permanently lowers the cap, and nothing says so

There is no FinalizationRegistry here, so a body that is neither drained nor cancelled never returns its slot. That is reachable from the public surface: files.read(path, { format: 'stream' }) and volume.read(..., { format: 'stream' }) hand the stream to the user. wrapStreamWithConnectionCleanup already documents the sibling case ("a consumer that holds the stream but stops reading ... is reclaimed server-side, not by this timer") — the inflight slot is precisely the part that is not reclaimed server-side. Verified offline: one unread non-empty body permanently starves a cap of 1.

TASTE treats docstrings as API and asks every public method to document its failure modes, so this needs saying in two places:

  • the three cap docstrings still read "max number of API requests that can be in flight at once" (getApiInflightLimit, getEnvdInflightLimit, getEnvdRpcInflightLimit). After this change they bound how many response bodies can be open at once — a materially different quantity that users tune against.
  • the changeset should tell users that streams they obtain must be drained or cancelled. As written it reads as a pure internal accounting fix.

One small correction to the changeset text: aligning the release point makes the cap accurate, but with the default cap far above the dispatcher's connections, it does not by itself prevent ERR_HTTP2_TOO_MANY_CONCURRENT_STREAMS.

What checked out

  • packages/js-sdk lint, typecheck, and prettier --check are clean; the 8 new/updated tests/api/inflight.test.ts cases pass.
  • Full unit project: 1 failed | 461 passed | 30 skipped. The failure is the pre-existing tests/sandbox/network.test.ts > firewall transform injects headers (404: template 'httpbin' not found), unrelated to this diff.
  • The getReader()-instead-of-piping choice is correct and well argued — a foreign stream really would fail the platform brand check, and it matches the existing wrapStreamWithConnectionCleanup pattern rather than inventing a new one.
  • Carrying url/redirected/type across the wrapper is more careful than needed (nothing in src/, openapi-fetch, or @connectrpc/connect-web reads them) and is the right instinct anyway; the accompanying test is real.
  • The catch around new Response(...) with a fallback to the original response is a good instinct, and releasing the slot on that path is right.
  • Confirming the premise: on the base, headers really do free the slot, so #1666 is a real defect and not a stale TODO.

Nothing in this diff reaches the exported surface (limitConcurrency is not re-exported from src/index.ts), so the parity, options-object, and naming rules don't apply. The TASTE rules that do bite are the timeout-message rule (item 2) and the docstring rules (item 4).

Open in Web View Automation 

Sent by Cursor Automation: /check SDK complies with TASTE.md

Comment thread packages/js-sdk/src/api/inflight.ts Outdated
*/
function releaseOnBodyEnd(response: Response, release: () => void): Response {
const body = response.body
if (!body) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This misses two of the three shapes whose bodies the HTTP client never touches.

openapi-fetch@0.14.1 returns without reading the body on exactly status === 204 || method === 'HEAD' || headers.get('Content-Length') === '0' (dist/index.mjs:173). Only 204 (and 205/304) give response.body === null; I verified on node 22 that a 200 or 500 with Content-Length: 0 has a non-null body, both synthetic and over the wire. And this is a live path, not a hypothetical — sandbox/filesystem/index.ts:556 exists for it: "When the file is empty, the response body is skipped and res.data is undefined." So files.read() on an empty file, and any non-2xx with Content-Length: 0 (the case that produced the isRunning() bug TASTE calls out), take the tracked path.

Those cases do release today, but only by accident. The default queuing strategy has highWaterMark: 1, so the ReadableStream constructor issues exactly one eager pull with no consumer attached; on an empty source that pull sees done: true and releases. Measured:

source consumer pull calls
empty none 1 → releases
non-empty none 1, then the queue is full → never releases
empty, highWaterMark: 0 none 0 → never releases

Nothing in the suite pins that, so switching to a byte strategy or highWaterMark: 0 later would silently deadlock every empty-body response. Worth checking explicitly, the way #1695's bodyToTrack() does:

if (!body || response.bodyUsed || body.locked) return releaseAndReturn()
if (method.toUpperCase() === 'HEAD') return releaseAndReturn()
if (response.headers.get('content-length') === '0') return releaseAndReturn()

The HEAD case needs the request method, which means reading it off init?.method ?? input.method up in limitConcurrency.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bac0b8e. hasBodyToTrack(response, method) now short-circuits explicitly on the same conditions openapi-fetch does — !body || bodyUsed || body.locked, HEAD (method read off init?.method ?? input.method), and content-length: 0 — releasing the slot immediately instead of relying on the constructor's one eager pull. New tests pin content-length: 0 (asserting bodyUsed === false and the body unlocked afterwards), HEAD with a non-zero advertised Content-Length, and locked/used bodies.

// `getReader` rather than piping the stream itself into the new Response:
// a foreign stream (cross-realm, ponyfill) would fail the platform's brand
// check, while a native wrapper reading through the reader always passes.
const reader = body.getReader()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getReader() throws TypeError if the body is already locked or consumed, and this call sits outside any try, after the slot has been taken. So the throw escapes limitConcurrency to the caller and the slot is never released.

Verified with a fetcher whose response body already has a reader: the first call throws, and the next call on a cap of 1 starves indefinitely (still queued at 500 ms). Nothing in src/ locks a body before the fetch returns, but createRuntimeFetch late-binds globalThis.fetch specifically so that mocks and instrumentation can replace it, and those are exactly what would hand back a spoken-for body. Folding bodyUsed/locked into the guard above covers it — if we could not read the body to its end anyway, there is nothing to wait for.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bac0b8e. limitConcurrency now checks response.bodyUsed || response.body.locked (together with the other no-byte shapes) before any getReader() call, and releases the slot immediately in that case — a mock/interceptor that already spoke for the body can no longer throw past the wrapper or leak the slot. Regression test: limitConcurrency releases immediately when the body is already locked or used.

Comment thread packages/js-sdk/src/api/inflight.ts Outdated
@@ -73,10 +70,95 @@ export function limitConcurrency(
const signal =
init?.signal ?? (isRequestLike(input) ? input.signal : undefined)
const release = await sem.acquire(signal)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth threading the env var name that configured max into limitConcurrency here, so a request aborted while still queued can say so.

As it stands the abort surfaces through abortReason(signal), which echoes the caller's request-timeout abort: TimeoutError ... likely due to exceeding 'requestTimeoutMs'. That was tolerable when slots turned over at header speed; now a queue wait can last as long as a stream, and the advice is actively wrong — the request never reached the network, so a larger requestTimeoutMs just lengthens the hang. TASTE asks timeout errors to name the knob to turn, and the knob here is E2B_ENVD_RPC_INFLIGHT_REQUESTS / E2B_ENVD_INFLIGHT_REQUESTS / E2B_API_INFLIGHT_REQUESTS, which this layer currently has no way to name.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in bac0b8e. limitConcurrency takes options.envVarName (wired as E2B_API_INFLIGHT_REQUESTS / E2B_ENVD_INFLIGHT_REQUESTS / E2B_ENVD_RPC_INFLIGHT_REQUESTS from the respective fetch factories), and a request aborted while still queued now rejects with a TimeoutError naming that env var and its current value — telling the caller to raise the cap or close open streams instead of pointing at requestTimeoutMs. Test: limitConcurrency names the env var when a queued request aborts.

start(controller) {
controllerRef = controller
// One chunk fills the passthrough queue so no read is pending
// when the source errors later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new Response(null, { status: 204 }) is the one bodiless shape that already works structurally (body === null). The interesting cases are the ones where the body exists but the HTTP client never reads it — a 200 or 500 with content-length: 0, and a HEAD response — since those are what openapi-fetch short-circuits on. Both currently pass only because of the constructor's one eager pull, so a case like

new Response('', { status: 200, headers: { 'content-length': '0' } })

driven through createClient({ fetch: limited }) (asserting bodyUsed === false on the first response, then that a second request is admitted) would pin the behaviour against a later change of queuing strategy. A companion negative test — an unread non-empty body does not release — would document the abandoned-stream contract from item 4 of the review at the same time.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in bac0b8e: limitConcurrency releases immediately for content-length: 0 responses drives a 200 with content-length: 0 through the wrapper and asserts bodyUsed === false / body unlocked on the first response before a second request is admitted, so the release no longer depends on the queuing strategy's eager pull. A HEAD counterpart covers the other short-circuit shape. The abandoned non-empty-body contract is pinned by the existing queue test (headers arrive, slot stays held until the body is read) and the reserved-capacity test, where an unread streaming body keeps its slot until the source closes.

@devin-ai-integration

devin-ai-integration Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Re-tested commit bac0b8e end-to-end (local streaming server through the SDK's real buildDispatchedFetch/undici path; streaming simulated with content-type: application/connect+json, unary as plain GETs — no live E2B key available).

Reserved unary capacity & overflow (the review's starvation scenario):

=== cap=2 reserved=1 (streaming vs unary split) ===
PASS  second streaming request NOT dispatched (streamingLimit=1)
PASS  unary admitted while stream open and 2nd stream queued
PASS  freed slot wakes queued UNARY first; streaming waiters still parked
PASS  cancelling stream admits queued stream (FIFO)

=== cap=1 reserved=1 (kill can't be wedged behind the stream it ends) ===
PASS  unary "kill" completes while the only slot is held by an open stream (overflow)
PASS  second stream still NOT dispatched (streaming budget stays 1)

Queued abort now points at the cap:

PASS  queued request rejects with TimeoutError
PASS  message: "Request was aborted while queued for an in-flight slot under
      'E2B_TEST_INFLIGHT' (currently 1). That cap counts open response bodies ..."
No-byte release + regression (all passed)
=== cap=1, no-byte responses release immediately ===
PASS  200 + content-length:0 releases immediately (bodyUsed=false, body unlocked)
PASS  HEAD releases immediately
PASS  locked-body response (mock grabbed reader) does not throw, releases slot
PASS  consumed-body (bodyUsed=true) response does not throw, releases slot

=== regression: v1 scenarios ===
PASS  cap enforced through body lifetime (3rd req queued until consume/cancel; server peak = 2)
PASS  8MiB body sha256-identical; status/headers/url/redirected preserved
PASS  204 / abort-after-headers / errored-body all release; no deadlocks

Unit tests: 15/15 in both node and workerd (cloudflare) configs. Earlier v1 run (pre-review-fixes) also included a differential run proving the pre-fix code leaks past the cap on unread streaming bodies while the fix does not.

Not covered: live-sandbox repro of the original starvation (no E2B_API_KEY in the test environment); the scenario shape was reproduced locally instead.

Tested by Devin — session link

@mishushakov

Copy link
Copy Markdown
Member

@devin-ai-integration check comments

…ately

Streaming Connect RPCs can no longer wedge the unary calls (kill,
sendStdin, list) that end them: the envd RPC semaphore reserves one
unary slot, with overflow when the cap is too small to carve it out.
Responses carrying no bytes (content-length: 0, HEAD, locked/used
bodies) release their slot immediately instead of relying on the
stream's eager pull, and queued-abort errors name the inflight env var.

Co-Authored-By: mish@e2b.dev <mish@e2b.dev>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Re: the blocking starvation finding — addressed in bac0b8e, porting the reserved-unary-capacity design from #1695 (which is now closed, so consolidation happens here):

  • limitConcurrency accepts { envVarName, reserved }. Connect streaming requests (Content-Type: application/connect+…) are limited to max - reserved, so unary/control RPCs (commands.kill, sendStdin, list, pty.*) always have headroom on the shared envd RPC semaphore. When the cap is too small to carve the reservation out (e.g. max=1), unary gets a matching overflow slot beyond max, so a stream can never wedge its own teardown. Unary waiters are preferred when the queue holds both kinds.
  • createEnvdRpcFetch reserves 1 slot; all three fetch factories pass their env-var name for queued-abort messaging.
  • No-byte responses (content-length: 0, HEAD, locked/used bodies) now release explicitly rather than via the eager pull, per the inline comments.

Tests cover the reserved cap (stream queues while unary is admitted), the max=1, reserved=1 overflow, the queued-abort message, and each no-byte shape. This PR retains url/redirected/type preservation on the wrapped response and the reader.closed observation for sources that terminate between pulls.

@mishushakov

Copy link
Copy Markdown
Member

@devin-ai-integration what's our Python story here

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

what's our Python story here

No parity change needed — there is no counterpart to mirror. This PR fixes the JS-only limitConcurrency semaphore, which exists because the SDK caps requests going into the undici dispatcher; the Python SDK has no equivalent in-flight cap anywhere under packages/python-sdk/e2b (no E2B_*_INFLIGHT_REQUESTS knobs, no semaphore).

Python's transports run on pyqwest/reqwest, which manages HTTP/2 stream limits itself: reqwest/hyper negotiates SETTINGS_MAX_CONCURRENT_STREAMS per connection and opens additional connections when a connection's streams are saturated, rather than erroring like undici's ERR_HTTP2_TOO_MANY_CONCURRENT_STREAMS. The E2B_MAX_CONNECTIONS note in packages/python-sdk/e2b/api/__init__.py documents this: reqwest doesn't cap concurrent connections, so that env var is no longer read there. The same goes for the follow-on pieces in this PR (reserved unary capacity, queued-abort messaging) — they only exist to protect the JS-side semaphore, which Python doesn't have.

If we ever want a client-side concurrency budget in Python (e.g. to bound total open streams per process), that would be a new feature — happy to open an issue for it, but nothing here regresses or diverges Python behavior.

@mishushakov

Copy link
Copy Markdown
Member

@devin-ai-integration can we implement this on top of undici

negotiates SETTINGS_MAX_CONCURRENT_STREAMS per connection and opens additional connections when a connection's streams are saturated, rather than erroring

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

can we implement this on top of undici

Good news: undici 8 already does this, and we already prefer it (undici8 on Node ≥ 22.19.0, see getUndiciPackageCandidates). I verified both source and behavior:

  • undici 8.10 gates h2 dispatch on the remote settings (client-h2.js: busy() returns true when kRunning >= kMaxConcurrentStreams, updated from the server's SETTINGS frame). A saturated client reports need-drain, so the Agent/Pool fills a connection up to the server's SETTINGS_MAX_CONCURRENT_STREAMS, then opens the next connection (up to connections), then queues — it never throws ERR_HTTP2_TOO_MANY_CONCURRENT_STREAMS.
  • undici 7 (fallback for older Node) has no such gate, but its default pipelining: 1 caps it at 1 stream per connection, so it also queues rather than erroring — just underutilizes h2.

Empirical probe (local h2 TLS server advertising maxConcurrentStreams: 3, 8 concurrent requests with long-lived bodies):

connections=1 connections=4
undici 7.29 1 stream dispatched, 7 queued, all served after draining, 0 errors 4 dispatched (1/conn), 4 queued, 0 errors
undici 8.10 3 dispatched (= server limit), 5 queued, 0 errors all 8 dispatched across 4 conns, 0 errors

So the transport layer already "opens additional connections when saturated, rather than erroring" on the path modern Node takes. What it can't do is distinguish request classes: when every connection is saturated by long-lived Connect streams, a kill/sendStdin unary just queues inside undici indefinitely — same starvation, one layer down, and silent. That's what the SDK-level cap buys us on top: reserved unary capacity so teardown RPCs always dispatch, an actionable error naming the env var instead of an unbounded queue, and identical semantics on runtimes that never load undici (workerd, deno, bun, the global-fetch fallback).

We could push the reservation into a custom undici Dispatcher (compose our own clients with class-aware admission), but it would only cover the Node+undici path and reimplement pool scheduling for the same math. Happy to open a follow-up issue if you want to explore that, or to bump pipelining on the undici-7 path so older Node gets real multiplexing too.

(Probe scripts run against the exact versions pinned in packages/js-sdk: undici 7.29.0 and undici8 = npm:undici@8.10.0.)

@mishushakov

Copy link
Copy Markdown
Member

@devin-ai-integration what would you do? can we drop undici 7 altogether in our stack?

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

what would you do? can we drop undici 7 altogether in our stack?

Yes, I think you can — with one caveat about the supported Node range.

Why undici 7 is there: the SDK's engines are >=20.18.1 <21 || >=22, while undici@8 requires Node >=22.19.0 (its engines field; that's why #1575 introduced the undici8 alias as an optional dep with the runtime version check in getUndiciPackageCandidates). undici 7 exists solely to cover Node 20.18.1–22.18.

What I'd do:

  1. Land this PR as is. The body-lifetime cap + reserved unary slot is orthogonal: it protects every runtime, including workerd/deno/bun and the global-fetch fallback where no undici is loaded at all, and it's what gives you an actionable error instead of undici's silent internal queueing when all connections are stream-saturated.
  2. Drop undici 7 in a follow-up, tied to an engines bump. Node 20 hit EOL in April 2026, so requiring >=22.19.0 now only asks Node 22 users to be on a ~year-old patch release. Concretely: remove the undici dep and the undici8 alias, depend on undici@^8 directly (keep it optional if you want installs to survive on out-of-range Node), delete the version-gated candidate list in undici.ts, and keep the existing global-fetch fallback for when undici can't load. That also deletes the v7-only quirk of 1-stream-per-connection multiplexing.
  3. Treat the engines bump as breaking. Narrowing supported Node versions should ride a major (or at minimum a loudly-flagged minor) of e2b, not this patch-level fix — another reason to do it separately.

Happy to open the follow-up PR/issue if you want — it's a small, mostly-deletion change plus a changeset and CI matrix update (the workflows still test on Node versions that would fall out of range).

@mishushakov

Copy link
Copy Markdown
Member

lets keep this pr as is and follow up with PR for 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inflight Concurrency Cap Bypass on Streaming Bodies

1 participant