fix: enforce training timeout for silent subprocesses

[Tyagiquamar]

Problem

LocalProcessRunner.run_training can hang forever even though a timeout is provided. The deadline was only checked after an output line arrived from the training subprocess, so any subprocess that stays silent past the deadline blocks the main thread inside readline() indefinitely.

Realistic triggers: loading a large dataset before the first log line, Keras/PyTorch training with progress output disabled, or a hung torchrun rendezvous during DDP startup.

Root cause

In plexe/execution/training/local_runner.py, the timeout was enforced inside the output-pump loop:

for line in iter(process.stdout.readline, ""):
    ...
    elapsed = time.time() - start_time
    if elapsed > timeout:
        process.kill()

If the child produces no lines, control never returns to the check: readline() waits on the pipe forever and the documented contract (timeout: Max training time (seconds)) is not upheld.

Fix

Replace the per-line check with a threading.Timer watchdog that kills the process at the deadline regardless of output flow. The watchdog raises the same subprocess.TimeoutExpired as before, so callers still get the identical TrainingError("Training timed out after N seconds").

Testing

• New regression test tests/unit/execution/training/test_local_runner_timeout.py simulates a silent child via mocked Popen. On main it hangs (no result within 15 s for timeout=3); with this fix it fails fast (~3 s) with TrainingError: Training timed out after 3 seconds.
• python -m pytest tests/unit -q -n 2: 107 passed, 15 skipped (main baseline on same machine: 106 passed, 15 skipped)
• ruff check and black --check: clean on changed files