Fix IterablePackingDataset workers to use spawn

[zupengwang]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Fixes #9649.

IterablePackingDataset currently creates worker queues and processes from the platform default multiprocessing context. On Linux this defaults to fork, so workers can inherit already-initialized CUDA or distributed state and deadlock during training.

This change uses a dedicated spawn context for the queues and workers. The worker entry point receives only the template, queues, and strict-mode flag, which keeps multiple spawned workers from serializing previously started Process objects. A two-worker regression test checks both the start method and CUDA isolation.

Experiment results

• Baseline regression on Linux: failed with fork; Python 3.12 also emitted the multiprocessing deadlock warning.
• python -m unittest -v tests.general.test_packing (Python 3.10): passed.
• python -m pytest -q tests/general/test_packing.py (Python 3.12): passed.
• pre-commit run --all-files: all hooks passed.
• Real Qwen2-0.5B template data-flow check with two packing workers:
  • baseline: fork, packed sequence lengths [116, 116]
  • modified: spawn, packed sequence lengths [116, 116]
• Dual RTX 3090 NCCL check: both ranks initialized CUDA before constructing the dataset; each rank started two spawn packing workers, produced [116, 116], and exited successfully.

[tastelikefeet]

Thanks for the contribution! Based on your code, we've attempted a fix for both the template pickling and CUDA initialization issues — could you check whether this works for your use case: https://github.com/modelscope/ms-swift/pull/9979/changes

[zupengwang]

Thanks — I tested #9979 at c43ce75b (merged as af807e3c and unchanged on current main).

For the original Qwen3.5 streaming-packing use case, the fix works: with parent CUDA initialized, Qwen3.5-0.8B multimodal packing using num_proc=2 selected spawn for both workers; the workers had CUDA uninitialized and were not in a bad fork. Packing content, order, and lengths matched the pre-fix baseline exactly. A separate parent-CUDA + NCCL test also completed without deadlock. #9979 additionally fixes the live-template-model pickling failure that remains in #9876.

I found one generic regression and two lifecycle gaps, though:

• Template.__getstate__ always removes model/dummy_model. A module-level Template subclass whose encode() needs a picklable self.model works on Fix IterablePackingDataset workers to use spawn #9876, but Fix the compatibility of py3.14 and fix qwen3-vl training bug #9979 silently drops every sample with strict=False; with strict=True the workers exit and the parent blocks on Queue.get().
• Local Template classes remain unpicklable under spawn.
• If a later worker fails to start, already-started workers are not terminated or joined.

The current tests also do not exercise spawn with num_proc=2 — the multi-process test uses the platform-default fork on Linux.

So #9979 fixes the original Qwen3.5 case, but I would not yet call it a complete replacement for all Template subclasses. Would you prefer the model-dependent-template and worker-error handling to be addressed in a follow-up?