Skip to content

Fix MPI submesh injection across parent ownership boundaries - #858

Draft
gthyagi wants to merge 2 commits into
underworldcode:developmentfrom
gthyagi:bugfix/submesh-parent-ownership
Draft

gthyagi wants to merge 2 commits into
underworldcode:developmentfrom
gthyagi:bugfix/submesh-parent-ownership

Conversation

@gthyagi

@gthyagi gthyagi commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Submesh injection can lose shared-node values when a parent DOF owner retains no extracted-region cells. Local copies reach another rank's parent ghosts, then owner synchronization overwrites them with untouched values. In the annulus workflow this zeroed four inner-wall topography values before gravity assembly.

Reduce matched values and their multiplicities through the parent field DM before synchronizing owners. Averaging shared contributions preserves replace/add semantics and leaves unmatched parent values unchanged. Production transfer uses distributed PETSc vectors without gathering global coordinates.

The original CI failure exposed a second cause: if an extracted region initially occupies one MPI rank, the filtered DM reports isDistributed() == False. The Mesh constructor then redistributed the child, invalidating rank-local parent matches. Preserve the parent-derived partition when constructing region and surface submeshes; ordinary mesh distribution is unchanged. Surface extraction now checks for a globally empty face stratum, allowing legitimate empty ranks while still rejecting missing/empty surfaces. Independent child repartitioning remains outside coordinate-coincidence injection.

Validation of the updated branch on an isolated Mac PETSc 3.25.0 int32/Open MPI 5.0.10 build:

  • Deterministic single-rank-region P1/P2 replace/add cases first fail on the original PR with the same CI errors: replace 5.0, add 4.0. Default-partition controls pass.
  • Patched serial suite: 20 passed and one pre-existing expected failure in embedded 1D on-surface evaluation.
  • Two MPI ranks: 23 tests passed on each rank, covering injection, region extraction/restriction, starved-rank cell sizes, subsequent mesh construction and surface extraction.
  • Four MPI ranks: all 10 focused cases passed on each rank, including empty parent/child ranks, re-extraction, unchanged exterior values and collective rejection of a globally empty surface.
  • Style gate, test formatting, Python parsing, diff checks and CI test reachability pass. No new benchmark models were run; the shared Mac installation is unchanged. Fresh Linux/MPICH CI validation is pending.

Earlier Gadi validation applies to the unchanged owner-reduction implementation in feature commit 86cb3cf1, before the partition-preservation follow-up:

  • Fresh PETSc int32 build: 18 UW3 tests in serial and on each of 16 MPI ranks; 270 annulus components passed.
  • Actual h=1/32 annulus meshes with 1h/2h/3h buffers reproduce the old 2h failure; corrected serial/MPI16 transfers agree within 1.8e-15.
  • Six complete Stokes–topography–Poisson regressions restore every saved source value, preserve Stokes fields to roundoff and agree between serial/MPI potential coefficients within 1.01e-11. Audited evidence.

Underworld development team with AI support from Claude Code

Reduce source values and multiplicities through the parent field DM before owner synchronisation. This preserves spatial injection without global coordinate gathering, including add mode and untouched exterior values. A parent owner may retain no extracted-region cells even when another rank has the corresponding source node.

Add independent P1/P2 vector replace/add regressions. Reproducer: annulus 2h buffers at h=1/32 on 16 ranks lose four inner-wall source values despite correct saved submesh topography. Static compilation and diff checks pass; matching versioned int32 installation, serial/MPI regression gates and affected scientific reruns remain pending on Gadi.
Avoid implicit child redistribution when a filtered region or surface occupies only one MPI rank. Preserve the parent-derived point/DOF mappings and retain the existing parent-owner reduction. Make surface emptiness validation collective so ranks outside a valid surface do not fail before PETSc extraction.

Add deterministic P1/P2 replace/add coverage with the region concentrated on one parent rank, plus empty-rank re-extraction/cell-size and surface-label tests. The four new transfer cases fail on the original PR with errors 5.0/4.0 before this change.

Validation on an isolated rebuilt Mac PETSc 3.25 int32/Open MPI runtime: 20 serial passed, one existing expected failure; 23 passed on each of two ranks; 10 passed on each of four ranks. Python parsing, Black test formatting, deprecated-pattern gate, CI test reachability and diff checks pass. Shared runtime and benchmark results unchanged. Next: verify the fresh Linux/MPICH CI run.

Underworld development team with AI support from Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant