Make serial submesh construction linear in the number of points - #5349
Merged
pbrubeck merged 12 commits intoAug 19, 2026
Conversation
Submesh(mesh, comm=COMM_SELF) cost O(P^2) in the number of mesh points P. On three ranks a UnitSquareMesh(181, 181) needed 8.9 seconds to build one serial submesh. Four times more cells cost about ten times more time. Every preconditioner that builds a subdomain matrix through firedrake.preconditioners.matis pays this cost, because local_mesh builds one serial submesh per process. The cost was in submesh_correct_entity_classes. A submesh on COMM_SELF has no neighbour, so it has no ghost points and no owned points, and every point is core. The function set that state one point at a time. DMLabelSetValue moves the stratum into a hash set and destroys its index set. The DMLabelHasPoint of the next iteration reads the hash set back, sorts it, and rebuilds the index set and the bit array. Each point therefore cost O(k log k) in the size k of the core stratum so far. Set the three strata in bulk instead. This is the same fix as the one in PR firedrakeproject#5326 for the coarse and fine cell maps. submesh_create also built a temporary label that marked every cell, and passed it to DMPlexFilter. DMPlexFilter selects every cell by itself when it gets no label, so the label was unnecessary. Submesh now leaves label_name as None for a codim-0 submesh of the whole mesh. A codim-1 submesh still passes the "depth" label, because DMPlexFilter with no label selects cells, not facets. Measured on three ranks, for submesh_correct_entity_classes alone: cells before after 717 34 ms 0.12 ms 2787 320 ms 0.43 ms 11059 3214 ms 1.85 ms 22023 8907 ms 3.80 ms The whole submesh build at 11059 cells falls from 3256 ms to 71 ms. Both paths produce the same submesh. On UnitSquareMesh(128, 128) with three ranks, the submesh built with a label and the submesh built without one have the same chart, the same subpoint index set, and the same pyop2_core, pyop2_owned, pyop2_ghost and Face Sets strata. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
connorjward
requested changes
Aug 17, 2026
pbrubeck
force-pushed
the
haldaas/submesh-entity-classes
branch
from
August 18, 2026 13:19
91a8d82 to
5cb9b41
Compare
connorjward
reviewed
Aug 18, 2026
connorjward
reviewed
Aug 18, 2026
connorjward
left a comment
Contributor
There was a problem hiding this comment.
Why did a lot of the code from fe68986 get removed? It looked relevant to the actual fix.
Contributor
The majority of the old diff just moved a block of code into an if-statement in order to avoid passing a DMLabel to DMPlexFilter for a very minor performance win that did not extended well to codim != 0. That wasn't fixing the nonlinear cost, but gave a tiny speedup by making the constant in front of the linear term smaller. |
pbrubeck
force-pushed
the
haldaas/submesh-entity-classes
branch
from
August 19, 2026 08:35
e7efe93 to
db87f54
Compare
connorjward
requested changes
Aug 19, 2026
connorjward
left a comment
Contributor
There was a problem hiding this comment.
This is definitely better than before. I like having all the parsing live in one place.
pbrubeck
reviewed
Aug 19, 2026
pbrubeck
reviewed
Aug 19, 2026
pbrubeck
reviewed
Aug 19, 2026
Co-authored-by: Pablo Brubeck <brubeck@protonmail.com>
connorjward
previously approved these changes
Aug 19, 2026
pbrubeck
reviewed
Aug 19, 2026
pbrubeck
reviewed
Aug 19, 2026
pbrubeck
reviewed
Aug 19, 2026
Co-authored-by: Pablo Brubeck <brubeck@protonmail.com>
pbrubeck
reviewed
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Submesh(mesh, comm=COMM_SELF)cost O(P^2) in the number of mesh points P.On three ranks a
UnitSquareMesh(181, 181)needed 8.9 seconds to build oneserial submesh. Four times more cells cost about ten times more time.
Cellwise BDDCPC pays this cost, because local_mesh builds one serial submesh per process.
The cost was in
submesh_correct_entity_classes. A submesh onCOMM_SELFhasno neighbour, so it has no ghost points and no owned points, and every
point is core. The function set that state one point at a time.
DMLabelSetValuemoves the stratum into a hash set and destroys its indexset. The
DMLabelHasPointof the next iteration reads the hash set back,sorts it, and rebuilds the index set and the bit array. Each point
therefore cost O(k log k) in the size k of the core stratum so far.
Set the three strata in bulk instead. This is the same fix as the one in
PR #5326 for the coarse and fine cell maps.
submesh_createalso built a temporary label that marked every cell, andpassed it to
DMPlexFilter.DMPlexFilterselects every cell by itself whenit gets no label, so the label was unnecessary.
Submeshnow leaveslabel_name as None for a codim-0 submesh of the whole mesh. A codim-1
submesh still passes the "depth" label, because
DMPlexFilterwith no labelselects cells, not facets.
Measured on three ranks, for
submesh_correct_entity_classesalone:The whole submesh build at 11059 cells falls from 3256 ms to 71 ms.
Both paths produce the same submesh. On
UnitSquareMesh(128, 128)withthree ranks, the submesh built with a label and the submesh built without
one have the same chart, the same subpoint index set, and the same
pyop2_core,pyop2_owned,pyop2_ghostand Face Sets strata.Co-Authored-By: Claude Opus 5 noreply@anthropic.com