Skip to content

Make serial submesh construction linear in the number of points - #5349

Merged
pbrubeck merged 12 commits into
firedrakeproject:mainfrom
haldaas:haldaas/submesh-entity-classes
Aug 19, 2026
Merged

Make serial submesh construction linear in the number of points#5349
pbrubeck merged 12 commits into
firedrakeproject:mainfrom
haldaas:haldaas/submesh-entity-classes

Conversation

@haldaas

@haldaas haldaas commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Description

Submesh(mesh, comm=COMM_SELF) cost O(P^2) in the number of mesh points P.
On three ranks a UnitSquareMesh(181, 181) needed 8.9 seconds to build one
serial submesh. Four times more cells cost about ten times more time.

Cellwise BDDCPC pays this cost, because local_mesh builds one serial submesh per process.

The cost was in submesh_correct_entity_classes. A submesh on COMM_SELF has
no neighbour, so it has no ghost points and no owned points, and every
point is core. The function set that state one point at a time.
DMLabelSetValue moves the stratum into a hash set and destroys its index
set. The DMLabelHasPoint of the next iteration reads the hash set back,
sorts it, and rebuilds the index set and the bit array. Each point
therefore cost O(k log k) in the size k of the core stratum so far.

Set the three strata in bulk instead. This is the same fix as the one in
PR #5326 for the coarse and fine cell maps.

submesh_create also built a temporary label that marked every cell, and
passed it to DMPlexFilter. DMPlexFilter selects every cell by itself when
it gets no label, so the label was unnecessary. Submesh now leaves
label_name as None for a codim-0 submesh of the whole mesh. A codim-1
submesh still passes the "depth" label, because DMPlexFilter with no label
selects cells, not facets.

Measured on three ranks, for submesh_correct_entity_classes alone:

cells   before     after
  717    34 ms   0.12 ms
 2787   320 ms   0.43 ms
11059  3214 ms   1.85 ms
22023  8907 ms   3.80 ms

The whole submesh build at 11059 cells falls from 3256 ms to 71 ms.

Both paths produce the same submesh. On UnitSquareMesh(128, 128) with
three ranks, the submesh built with a label and the submesh built without
one have the same chart, the same subpoint index set, and the same
pyop2_core, pyop2_owned, pyop2_ghost and Face Sets strata.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

haldaas and others added 2 commits August 13, 2026 14:06
Submesh(mesh, comm=COMM_SELF) cost O(P^2) in the number of mesh points P.
On three ranks a UnitSquareMesh(181, 181) needed 8.9 seconds to build one
serial submesh. Four times more cells cost about ten times more time.

Every preconditioner that builds a subdomain matrix through
firedrake.preconditioners.matis pays this cost, because local_mesh builds
one serial submesh per process.

The cost was in submesh_correct_entity_classes. A submesh on COMM_SELF has
no neighbour, so it has no ghost points and no owned points, and every
point is core. The function set that state one point at a time.
DMLabelSetValue moves the stratum into a hash set and destroys its index
set. The DMLabelHasPoint of the next iteration reads the hash set back,
sorts it, and rebuilds the index set and the bit array. Each point
therefore cost O(k log k) in the size k of the core stratum so far.

Set the three strata in bulk instead. This is the same fix as the one in
PR firedrakeproject#5326 for the coarse and fine cell maps.

submesh_create also built a temporary label that marked every cell, and
passed it to DMPlexFilter. DMPlexFilter selects every cell by itself when
it gets no label, so the label was unnecessary. Submesh now leaves
label_name as None for a codim-0 submesh of the whole mesh. A codim-1
submesh still passes the "depth" label, because DMPlexFilter with no label
selects cells, not facets.

Measured on three ranks, for submesh_correct_entity_classes alone:

    cells   before     after
      717    34 ms   0.12 ms
     2787   320 ms   0.43 ms
    11059  3214 ms   1.85 ms
    22023  8907 ms   3.80 ms

The whole submesh build at 11059 cells falls from 3256 ms to 71 ms.

Both paths produce the same submesh. On UnitSquareMesh(128, 128) with
three ranks, the submesh built with a label and the submesh built without
one have the same chart, the same subpoint index set, and the same
pyop2_core, pyop2_owned, pyop2_ghost and Face Sets strata.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread firedrake/cython/dmcommon.pyx
@pbrubeck
pbrubeck requested a review from connorjward August 18, 2026 13:18
@pbrubeck
pbrubeck force-pushed the haldaas/submesh-entity-classes branch from 91a8d82 to 5cb9b41 Compare August 18, 2026 13:19
Comment thread firedrake/mesh.py
Comment thread firedrake/cython/dmcommon.pyx

@connorjward connorjward left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did a lot of the code from fe68986 get removed? It looked relevant to the actual fix.

@pbrubeck

pbrubeck commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Why did a lot of the code from fe68986 get removed? It looked relevant to the actual fix.

The majority of the old diff just moved a block of code into an if-statement in order to avoid passing a DMLabel to DMPlexFilter for a very minor performance win that did not extended well to codim != 0. That wasn't fixing the nonlinear cost, but gave a tiny speedup by making the constant in front of the linear term smaller.

@pbrubeck
pbrubeck force-pushed the haldaas/submesh-entity-classes branch from e7efe93 to db87f54 Compare August 19, 2026 08:35
@pbrubeck
pbrubeck requested a review from connorjward August 19, 2026 08:37

@connorjward connorjward left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely better than before. I like having all the parsing live in one place.

Comment thread firedrake/cython/dmcommon.pyx
Comment thread firedrake/cython/dmcommon.pyx Outdated
Comment thread firedrake/cython/dmcommon.pyx Outdated
Comment thread firedrake/cython/dmcommon.pyx Outdated
Comment thread firedrake/mesh.py Outdated
Co-authored-by: Pablo Brubeck <brubeck@protonmail.com>
@pbrubeck
pbrubeck requested a review from connorjward August 19, 2026 09:53
connorjward
connorjward previously approved these changes Aug 19, 2026
Comment thread firedrake/cython/dmcommon.pyx Outdated
Comment thread firedrake/cython/dmcommon.pyx
Comment thread firedrake/mesh.py Outdated
Co-authored-by: Pablo Brubeck <brubeck@protonmail.com>
Comment thread firedrake/mesh.py Outdated
@pbrubeck
pbrubeck merged commit 00e43da into firedrakeproject:main Aug 19, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants