fix(ci): 🐛 let the C++ MPI test variants oversubscribe under OpenMPI 5 - #324
Conversation
The mpi ctest variants spawn `mpiexec -n <n>` inside the test, so the `--map-by :OVERSUBSCRIBE` the neighbouring Python steps pass on the command line cannot reach them; only the environment can. Both justfile recipes already exported OMPI_MCA_rmaps_base_oversubscribe, which is the OpenMPI 4 spelling and is ignored by 5: measured against 5.0.8, `mpiexec -n 4 --host localhost:2` still fails with it set and succeeds with PRTE_MCA_rmaps_default_mapping_policy. No effect while monoprop_MPI_TEST_PROCS is "2", which every runner can place. It matters for a branch that widens the sweep: at 4 ranks the ubuntu-x86 and macOS lanes fail with prte-rmaps-base:alloc-error while arm places it, in both test.yml and the coverage job. Assisted-by: ClaudeCode:claude-opus-5
|
Docs preview: https://pr-324.monoprop-docs.pages.dev |
robertodr
left a comment
There was a problem hiding this comment.
thanks good catch! Does OpenMPI 4 honor the PRTE_MCA_* version?
Don't know sorry |
Co-authored-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com> Signed-off-by: Roberto Di Remigio Eikås <robertodr@users.noreply.github.com>
I checked, it does not. There was a breaking change in runtime between the versions |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #324 +/- ##
=======================================
Coverage 97.70% 97.70%
=======================================
Files 14 14
Lines 742 742
Branches 98 98
=======================================
Hits 725 725
Misses 12 12
Partials 5 5
Flags with carried forward coverage won't be shown. Click here to find out more. |
|



🤖 AI text below 🤖
Summary
The C++
mpictest variants spawn their ownmpiexec -n <n>from inside the test, so the--map-by :OVERSUBSCRIBEthat the neighbouring Python steps pass on the command line never reaches them — only the environment can. The variable both justfile recipes already export,OMPI_MCA_rmaps_base_oversubscribe, is the OpenMPI 4 spelling and OpenMPI 5 ignores it.Measured against OpenMPI 5.0.8,
mpiexec -n 4 --host localhost:2 hostname:rc=1— refuses to placeOMPI_MCA_rmaps_base_oversubscribe=1rc=1— still refusesPRTE_MCA_rmaps_default_mapping_policy=:oversubscriberc=0, 4 ranksThis is latent on
main, wheremonoprop_MPI_TEST_PROCSis"2"and every runner can place 2 ranks — CI stays green either way. It bites any branch that widens the sweep: at 4 ranks theubuntu-26.04andmacos-15lanes fail withprte-rmaps-base:alloc-error/ "not enough slots", whileubuntu-26.04-armplaces it. Bothtest.ymland theCollect coverage [mpi:on]job are affected, since the coverage recipe runs the samectest --label-regex mpi.Found while rebasing #296, which widens the sweep to
"2;4"deliberately — at R=2 the peer plan resolves only one peer, so everyfor k in [0, f)in the sparse transport stays a single iteration and the multi-peer paths never run.Changes
.github/workflows/test.yml: setPRTE_MCA_rmaps_default_mapping_policy(and keep the OpenMPI 4 name) on the Run C++ unit tests under MPI step, which previously set no environment at all.justfile: add the same export totest-mpiand tocode-coverage-collect, both of which runctest --label-regex mpiafter exporting only the OpenMPI 4 name.Checklist
mainruns the sweep at 2 ranks where the bug is invisible; perf(mpi): 🧭 GF(2)-linear rank routing — messages per rank flat in R #296 exercises it at 4.docs/,CONTRIBUTING.md) if needed — n/aCHANGELOG/ release notes updated if applicable — n/aAI/LLM disclosure