Enable multiple runners for b200-multinode-slurm#771
Conversation
| if [[ "${{ runner.name }}" == mi355x-amds* ]] || [[ "${{ runner.name }}" == b200-dgxc-slurm* ]]; then | ||
| echo "[Slurm] Cleaning up jobs with name: ${{ runner.name }} for user: $USER ..." | ||
| scancel -u "$USER" --name="${{ runner.name }}" || true | ||
| while [ -n "$(squeue -u "$USER" --name='${{ runner.name }}' --noheader --format='%i')" ]; do | ||
| squeue -u "$USER" --name="${{ runner.name }}" |
There was a problem hiding this comment.
This change wouldn't break the existing implementation. All the runners are under the same linux username for B200 as well. It just adds an additional check to make sure we are checking for jobs with the runner name only under the current user, just to ensure jobs run with the same name by any other user are not disturbed.
There was a problem hiding this comment.
ahh gotcha! can u launch an test run to help validate that this works?
There was a problem hiding this comment.
sure! triggered a test run here - https://github.com/SemiAnalysisAI/InferenceX/actions/runs/22244986816
There was a problem hiding this comment.
@functionstackx @cquil11 the test run is completed
|
thanks! @cquil11 can help review & merge this PR |
2 participants
No description provided.