Skip to content

test: reduce CI flakiness in Ray-backed and smoke tests#253

Draft
Copilot wants to merge 3 commits intomainfrom
copilot/improve-test-reliability
Draft

test: reduce CI flakiness in Ray-backed and smoke tests#253
Copilot wants to merge 3 commits intomainfrom
copilot/improve-test-reliability

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 23, 2026

Summary

Tighten test reliability for failures seen on main: intermittent Ray worker/actor startup crashes in unit tests and transient external HTTP failures in the tutorial smoke test. The changes target test isolation and failure handling without weakening the underlying assertions.

Changes

  • ZMQ test fixture isolation

    • Replace per-test os.environ mutation for PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY with scoped DI settings overrides.
    • Apply the change consistently across shared, unit, and integration ZMQ fixtures to avoid cross-test/process races when Ray workers start in parallel.
  • Targeted reruns for Ray-backed flakes

    • Add limited reruns to the Ray-backed unit tests that intermittently fail during actor/worker startup:
      • test_multiprocessing_channel
      • test_state_backend_init
      • test_state_backend_init_with_existing_job
    • Keep reruns narrow to preserve signal and avoid broad pipeline inflation.
  • Smoke test handling for external network instability

    • Retry tutorial execution once on transient network failures.
    • If the failure remains clearly external (ConnectError, ReadTimeout, RemoteProtocolError, etc.), skip instead of failing the pipeline.
    • Preserve normal failures as hard failures.
testing_settings = Settings.model_validate({"flags": {"zmq_pubsub_proxy": zmq_pubsub_proxy}})
with override_settings(testing_settings):
    yield ZMQConnector

Copilot AI and others added 3 commits April 23, 2026 19:47
Agent-Logs-Url: https://github.com/plugboard-dev/plugboard/sessions/9de3125e-8da3-432a-a1b1-73188d656af3

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>
Agent-Logs-Url: https://github.com/plugboard-dev/plugboard/sessions/9de3125e-8da3-432a-a1b1-73188d656af3

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>
Agent-Logs-Url: https://github.com/plugboard-dev/plugboard/sessions/9de3125e-8da3-432a-a1b1-73188d656af3

Co-authored-by: toby-coleman <13170610+toby-coleman@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants