Skip to content

Real-Codex test fixtures are never stopped: one test file peaks at 92 app-servers and 12.6k threads #547

Description

@jbellis

Summary

The tests that use a real Codex start one codex app-server per test and never stop it. They all keep running until the test file finishes. One run of CodexAcpClient.test.ts peaks at 92 app-server processes holding about 12,600 threads. Each fixture also leaves a temporary CODEX_HOME directory in the system temp folder after the run.

Measurements

Clean checkout of main at 7fee150, npm ci, Linux x64:

npx vitest run src/__tests__/CodexACPAgent/CodexAcpClient.test.ts
  • 105 tests pass in about 20 seconds.
  • Sampling once a second during the run: at most 92 codex app-server processes at the same time, with 12,596 threads between them.
  • After the run no app-server is left, but running this file together with mcp-session.test.ts left 113 new codex-acp-codex-home-* directories in /tmp, one per fixture.

Cause

  • createTestFixture() in src/__tests__/acp-test-utils.ts starts a real app-server with startCodexConnection and returns no way to stop it. Its temporary CODEX_HOME is removed only from the process exit handler, which runs in the Vitest worker. When the worker is gone by the time Codex exits, nothing removes the directory.
  • CodexAcpClient.test.ts calls createTestFixture() in beforeEach, so every test in the file starts an app-server, including tests that never use it. Several tests also create a second fixture of their own. The file's afterEach only calls vi.unstubAllEnvs().
  • mcp-session.test.ts does the same in its beforeEach.

The processes only end because Codex exits when its stdin closes, which happens when the Vitest worker exits.

Consequences

On a host with other work running, the thread count is enough to reach the process limit. In a downstream fork with an identical createTestFixture(), running npm test on a host with a 32,768-task cgroup limit led to three things:

  • spawn EAGAIN from Vitest's worker pool and from startCodexConnection.
  • 40-second timeouts in whichever real-Codex tests ran at that moment.
  • 167 Codex processes, with 5,397 threads between them, still running half an hour later. Their parent was init.

We did not reproduce these orphaned processes on main under normal load. They appear to need the resource pressure the tests create.

Suggested fix

  • Give the real-Codex fixture a dispose() that stops codexConnection.process, waits for it to exit, and removes its CODEX_HOME.
  • Call it from afterEach for every fixture a test creates.
  • Create the real-Codex fixture only in the tests that need one, not in beforeEach for the whole file.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions