fix(ci): serialize convergence model config preparation

[athitten]

What does this PR do ?

Prepares the resolved convergence recipe and Hugging Face model config once on node 0 before distributed training starts.

Changelog

• Resolve the shared finetune config only on node 0.
• Warm and validate the Hugging Face model config before launching torchrun.
• Make other nodes wait for a job-specific readiness marker with a timeout.
• Fail before training when config resolution or model config preparation fails.

Before your PR is "Ready for review"

Pre checks:

• Read and followed the contributor guidelines.
• Ran 90 focused CI unit tests in the failing job's container.
• Ran a CPU-only Gemma 4 cold-cache launcher preflight in the failing job's container.
• Ran bash -n and git diff --check.
• No documentation update is required for this internal CI launcher change.

Full convergence validation requires the original 4-node, 32-H100 topology.

Additional Information

• Failing NeMo-CI job: https://gitlab-master.nvidia.com/dl/JoC/nemo-ci/-/jobs/438631838
• The failure occurred when one rank read the shared cold Hugging Face cache without a usable model_type; the remaining ranks later timed out in NCCL.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[athitten]

/ok to test fe3c0b4