Skip to content

scripts: stop executor close waiting on inherited stdout - #532

Open
Jay (jay23606) wants to merge 10 commits into
microsoft:devfrom
jay23606:pr-scout/503-track-adversarial-executor-bugs-timeout-
Open

Jay (jay23606) wants to merge 10 commits into
microsoft:devfrom
jay23606:pr-scout/503-track-adversarial-executor-bugs-timeout-

Conversation

@jay23606

Copy link
Copy Markdown
Contributor

Bug

ExecutorClient.close() closed the executor's stdout stream from the calling thread while the reader thread was still blocked in readline() on that same stream. BufferedReader.close() waits for the blocked read, so when an external wrapper exits but a child it started keeps the inherited stdout pipe open, close() waits until that child exits. The two-second reader join never ran, because the stream close blocked first. A timed-out executor therefore delayed failure classification and summary emission by however long the child kept the pipe open.

Change

  • scripts/nvx_tools/adversarial.py: close() no longer closes stdout. The reader thread closes it in a finally after its last read, so the close never races a blocked read. stdin and stderr are closed as before, and the reader join stays bounded at two seconds.
  • scripts/test_adversarial.py: adds test_external_executor_close_ignores_inherited_stdout. The wrapper exits at once and leaves a 10-second grandchild holding stdout. The test asserts close() returns in under 5 seconds.

Validation

Before the fix, the new test fails with close() blocked 10.04s waiting on inherited stdout. After the fix it passes.

  • cd scripts && python -m unittest test_adversarial.CopilotContainmentTests.test_external_executor_close_ignores_inherited_stdout: OK
  • cd scripts && python -m unittest test_adversarial: OK (skipped=6)
  • ruff format --check scripts/nvx_tools/adversarial.py scripts/test_adversarial.py: 2 files already formatted
  • ruff check scripts/nvx_tools/adversarial.py scripts/test_adversarial.py: All checks passed!

Not run here: Pyright strict over scripts/, and the Windows CI jobs. The test uses # type: ignore[arg-type] on the Popen call, which Pyright may flag.

Regression test: fails on base twice (/close() blocked \d+.\d+s waiting on inherited stdout/), passes on fix twice (cd scripts && python -m unittest test_adversarial.CopilotContainmentTests.test_external_executor_close_ignores_inherited_stdout)

Fixes #503

Copilot AI balanced review requested due to automatic review settings October 13, 2026 20:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused fix addresses the deadlock scenario and includes appropriate regression coverage.

0 open findings

What changed in this PR

Prevents executor cleanup from blocking on stdout inherited by child processes.

Changes:

  • Moves stdout closure to the reader thread after reading finishes.
  • Adds regression coverage ensuring close() remains bounded.
File Description
scripts/​nvx_tools/​adversarial.py Avoids racing stdout closure with a blocked reader.
scripts/​test_adversarial.py Tests inherited-stdout cleanup behavior.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI balanced review requested due to automatic review settings October 14, 2026 13:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation directly addresses the blocking race and includes cross-platform regression coverage.

0 open findings

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 14, 2026 13:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The surviving reader can indefinitely enqueue output after closure, causing unbounded memory growth.

1 open finding

🧠 Review effort: Balanced

Comment thread scripts/nvx_tools/adversarial.py
Copilot AI balanced review requested due to automatic review settings October 14, 2026 14:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

🧠 Review effort: Lite

Comment thread scripts/nvx_tools/adversarial.py
Comment thread scripts/test_adversarial.py
Copilot AI balanced review requested due to automatic review settings October 14, 2026 15:17
@ppenna

Copy link
Copy Markdown
Contributor

Jay (@jay23606) please rebase. no merge commits.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reader kept queuing lines from stdout after close() had returned, so a
child that outlived the join could grow the queue without bound. Once close()
has started, the reader keeps draining the pipe but retains nothing.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 15, 2026 11:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused changes address blocking cleanup and post-close queue growth, with regression coverage and no identified blocking issues.

2 open findings
1 resolved since last review

🧠 Review effort: Balanced

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 15, 2026 16:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The timing regression test permits the original blocking behavior because its threshold exceeds the child’s lifetime.

2 open findings
1 resolved since last review

🧠 Review effort: Balanced

Comment thread scripts/test_adversarial.py Outdated
The grandchild must outlive the five-second bound, or a blocking close()
would still pass. Also bound close() time in the noisy-child regression test.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 15, 2026 17:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused changes address the blocking cleanup and prior queue-growth concern, with regression coverage and no identified blocking issues.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced

…ys open

A descendant that keeps the stdout pipe open kept the reader thread and its
descriptor alive after close(). Start every executor in its own session on
POSIX, and kill its group when the reader is still blocked after the join.
On Windows, taskkill /T covers the tree while the root is still running.

Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 15, 2026 18:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Windows tree cleanup normally runs after the wrapper exits, leaving inherited-pipe readers and handles alive.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced

Comment on lines +590 to +593
if self._process.poll() is None:
with contextlib.suppress(OSError, subprocess.TimeoutExpired):
subprocess.run(
["taskkill.exe", "/PID", str(self._process.pid), "/T", "/F"],
@ppenna

Copy link
Copy Markdown
Contributor

Jay (@jay23606) please remove all merge commits and resolve all copilot comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Track adversarial executor bugs: timeout cleanup blocks on inherited stdout

3 participants