Load-balance, active-box windowing, block-structured AMR - #1628
Open
sbryngelson wants to merge 795 commits into
Open
Load-balance, active-box windowing, block-structured AMR#1628sbryngelson wants to merge 795 commits into
sbryngelson wants to merge 795 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
| D = ((gs_min(lp) - 1.0_wp)*cvs(lp))/((gs_min(vp) - 1.0_wp)*cvs(vp)) | ||
|
|
||
| #ifdef MFC_SIMULATION | ||
| if (relax .and. load_weight_wrt) then |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 61.67% 62.39% +0.72%
==========================================
Files 84 96 +12
Lines 21619 29068 +7449
Branches 3196 4778 +1582
==========================================
+ Hits 13334 18138 +4804
- Misses 6093 7753 +1660
- Partials 2192 3177 +985 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
sbryngelson
changed the title
Member
Author
|
2 tasks
Fixed per-rank work (fine_work imbalance 1.004): per-GCD peak VRAM 49.9 GiB (np=1) -> 56.4 (np=2) -> device OOM (np=4 and np=8) - about 6.5 GiB of replicated per-global-entity device memory per np doubling, ~90 MB per global box per rank. freg/creg are confirmed global-indexed but only the ~1 GiB class; the remainder is unattributed and must be found by code audit before pricing any S-track increment. W8 added to the invariants: the device-memory ceiling, not wire bytes, is what kills weak scaling first (np=4 on this node). Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
Triple-sourced (code audit, VRAM traces, exact per-slot arithmetic): at np>=2 a rank's owned slots are a shifting SFC window of the GLOBAL index space plus received migration slots, so the never-decrementing local-index high-water ratchets the store capacity at 210.6 MiB per slot while the compaction gate (cap > 3*nlive) never fires. np=1 traces sit flat from startup - a single-rank window structurally cannot exhibit the term, which is why a month of np-invariant-window measurements never saw it. Corrects the previous revision's freg/creg guess (np-delta ~90 MiB only). Promotes the already-designed fix - device-side store remap + local-index derivation (cap == live at every reconcile) - to the first S-track increment: it is the np>=4 unblocker. Also confirmed: migration spack/ rpack are host-side (per-block bytes x global block count), I4's item. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…ompletes) The store-capacity ratchet ran over GLOBAL slot indices: at np>=2 a rank's owned set is a shifting SFC window plus received migration slots, so amr_loc_n climbed run-long and the S0 weak-scaling sweep device-OOMed at np=4 at fixed per-rank work. Four mechanisms, each forced by a measurement: (1) in-place index re-densification at every reconcile (s_amr_st_move_slot, ascending-source order, no realloc) - np=2 and np=4 VRAM now plateau flat with live 72/rank at both; (2) growth increments capped at 16 slots (the proportional +25% transient was itself store-scaled and tipped a 59.5 GiB card over 64); (3) early-free of consumed old slots during the rebuild (last_use per old block; np=4 plateau 57.3 -> 51.9 GiB); (4) stash-only replica slots (received old blocks only touch their amr_stor_st half - full slots doubled the migration storm's cost). Adds the [amr-cap] invariant line (live/cap per reconcile, rank_time_wrt-gated). The full device-side remake was built first and refuted: its staging transient OOMed init, and dropping growth's device->host round trip broke the rebuild carry-forward's host reads - both churn goldens caught the NaN. Gates: 27DEC5B6 + D127EC91 pass, AMR subset 67/67, S0 np=2 rc=0 (peak 55.3 GiB), np=4 rc=0 (peak 63.6 GiB - margin-thin; pool q_prim/rhs before growing the operating point). Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…re-sequences Action plan: the first complete weak-scaling pair (post-fix S0 table) - memory weak scaling holds (live 72/rank, flat VRAM at np=2 and np=4) and TIME is the measured blocker (wall 2.59x per np-doubling at fixed per-rank work, split unmeasured -> phase-diff the np=2/np=4 pair before building anything); W4/S2 baselines now two-point curves (ntag 393.6 -> 787.3 MiB/rank/regrid). Ladder consequences: P1 q_prim/rhs pooling promoted to next memory lever (np=4 margin 0.4 GiB), I1 unchanged but gains two assertable invariants (stash-only replicas, [amr-cap] flatness), np=8 arm and np=1 re-run owed, churn-goldens-first is policy for slot/stash/exchange changes. Endstate: W8 invariant HOLDING, W4 row carries the measured doubling, ladder row 0.4 updated (stale freg/creg guess removed). Implementation reference sec. 5 rewritten to the landed lifecycle (stash-only flavor + upgrade, capped growth, in-place re-densification with the ascending-order safety argument, [amr-cap]); the growth host round trip is documented as a load-bearing host-coherence CONTRACT (the rebuild carry-forward's host reads - the refuted device-side remake NaN'd both churn goldens through it); sec. 9 free-before-allocate note updated to the landed last_use early-free; sec. 11.3 weaknesses 1-2 updated with the 2026-08-21 outcomes. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…egistered M4 rules M3 (attribution of the np=2 -> np=4 time gap) turned out to cost zero runs: the passing gate logs carried full phase budgets. The +413 s delta splits into three families: wait/skew 38% (reflux +105 s pure wait, imb 1.81 - the sink pattern returns at np=4; coarse +44 s imb 2.85), rhs per-call 27% (ms/call 17.3 -> 31.3 in the mean, mechanism open), regrid scaling 27% (migration +43, build +38 incl. rb:gath +17). The plan pre-registers three discriminators with decision rules before any building: M4a same-card vs cross-card GCD pinning for the rhs doubling (no rhs work until it lands), M4b per-rank work imbalance for the skew source (balancer gated on imb > 1.15), M4c np=8 + np=1 table completion (np=8 OOM promotes P1 pooling to immediate). Increment order: I1 validator now (M4-independent), P1 q_prim/rhs pooling with the carry-forward device conversion in the same increment, then one M4-directed front. Standing do-not list: no reflux optimization, no operating-point growth pre-pooling, no P2 re-litigation, no multi-node before clean np=8, churn-goldens-first, watch CI on the new device paths. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…lus the L0 sites New module m_amr_xchg_audit: a 30-id site registry over the 35 physical MPI call sites (fypp twins share ids), always-cheap per-site message/word/tag-range counters recorded at the MPI call itself, a per-family global send==recv conservation assert at finalize (collective, before the amr early-return since L0 families fire with amr=F), and a [amr-xa] report under rank_time_wrt. Also: a reconcile-time assert that no stash-only replica slot survives reconcile (a survivor would reach the solver with no q_prim/rhs). This is the I1a half of the exchange validator (amr_plan_based_exchange.md, split recorded there); I1b adds MFC_DEBUG per-xfer identity headers and the destination-coverage tiling assert, gated by a seeded-bug tripwire. Gate: churn goldens 27DEC5B6/D127EC91 pass, AMR subset 67/67 (in 584 s with the newly engaged test GPU pool, -j 8 --gpus 0-7, vs ~20 min serialized on one GCD), no behavior change. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…d allocator state Three GCD arrangements (baseline / NUMA-spread / one-per-MCM) gave identical walls (593/605/581 s), the same rank-2 straggler (353/357/358 s of rhs), and per-rank VRAM fingerprints that reproduce on whatever GCD the rank lands on - no sick silicon, no NUMA effect, no MCM pairing. The cliff is bounded: 62.5 GiB fast vs 63.0-63.6 slow (97.7 vs 98.4% device-full). Standing hypothesis: cumulative per-slot q_prim/rhs alloc/free churn ratchets the OpenMP runtime's retained device pool to the ceiling (probe m4mem.sh pre-registered; knobs verified present in the runtime binary). np=1 re-run matches history exactly; np=8 fails with all eight cards at 62.3-63.7 GiB, firing the pre-registered rule: P1 q_prim/rhs pooling is promoted to immediate. The uniform +36% rhs floor at np=4 survives every tested hypothesis and is recorded as a named open question. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
… by the consumer audit m4mem 40-step probe: with LIBOMPTARGET_MEMORY_MANAGER_THRESHOLD=0 the rank-2 straggler vanishes (358.6 -> 152.3 s rhs), the uniform +36% floor vanishes with it (all ranks ~18.5 ms/call, the np=1/np=2 level), the reflux shadow collapses (176 -> 15 s), and wall drops 621.6 -> 405.4 s: the np-doubling factor falls 2.59x -> ~1.59x from one env var. Cross-node reproduction exact. np=8 still fails with the knob (peaks to 63.4) - retention and live footprint are separate terms, so P1 pooling stands per its rule. The pooling increment is redesigned by the consumer-lifetime audit: batch-shaped shared scratch aligned with P2 (not single-block), per-slot rhs kept for L0 tiles only, conditional q_prim; the m_rhs.fpp:772 stale-psi read is unreachable for AMR (checker chain) and filed as an upstream candidate. LIBOMPTARGET_INFO probe closed the micro-model: ~4 descriptor-class (16-120 B) device allocs per launch, concentrated in hllc/weno/conversion regions, 64:1 present-table hit ratio - the allocator-pressure mechanism is measured at every link. Re-audit cadence section added per user directive. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…ult (P4 arm identical) Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
The fused per-block fine advance leaves no cross-block q_prim/rhs lifetime, so fine slots share one slot-shaped scratch pair instead of carrying ~2x105 MiB per live slot - the np>=8 live-footprint blocker and the churn that fed the libomptarget retention plateau. L0 tiles keep per-slot rhs (all owned tiles' rhs coexist across the MPI reflux point) and per-slot q_prim exactly when the m_rhs copy-out gate writes it. Stage routines take the target arrays as dummies; the full-vs-stash discriminator moves from allocated(q_prim) to allocated(x_cb). Scratch is allocated on every rank at the one point per mode where mbuf* are final (module init, or after the tile mbuf union). AMR subset 67/67 incl. churn goldens 27DEC5B6/D127EC91. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…ds through np=8), np=8 phase budget, audit gaps G-A/G-B, increment 3=rb:gath 4=T1 Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…; both families in one chunked plan pg:recv (the level>=2 parent gather's blocking per-box MPI_RECV, the unconverted half of R1) is 99.2 s at S0 np=8 vs rb:wait 58.9 s; at np=4 it is essentially all of rb:gath. The old level-1-only scope was matched-point-specific. Both families now go through one chunked plan-then-execute; increment 1 validates the plan against the I1a XA counters before any batching. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
… path against it (gather-batching step 1) s_amr_build_gather_plan computes, before the rebuild box loop, every level-1 box's contributor ranks and message sizes and every level>=2 box's parent source and size, from the same replicated caches the per-box path reads. Five always-on asserts in the per-box gathers verify the inline derivation matches the plan box by box; exchange behavior is unchanged. Step 2's chunked exchange may trust the plan only because these asserts prove it reproduces today's message set. Subset 67/67 asserts-clean; two seeded tripwires (level-1 size, parent size) each aborted at the first rebuild. Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…lies, with the deadlock-freedom argument Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…th reviewers; amended to parent-position-split sends plus four bindings Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…it per owned box (gather-batching step 2) Both families through one chunked plan-then-execute (amr_regrid_gather_batching.md, amended form): phase A pre-posts every owned box's recvs for the chunk from the plan (level-1 contributor slices + split level>=2 parent patches, tag = slot, box-order contiguous request runs); phase B issues the plan-driven sends, deferring a level>=2 send to the child's consume position when the parent shares the chunk (its store is unbuilt until its own consume - the D1 defect both reviewers found, now also guarded by an always-on parents-first ASSERT); phase C consumes boxes in order with one WAITALL per owned box. The parent-field gathers take the child block explicitly (cblk) instead of reading amr_cur, since phase B runs before the consume geometry. Validated: AMR subset 67/67 goldens green; XA exchange report line-for-line identical to the per-box path on the S0-style np=4 tripwire (F1 312 msgs / 468087984 words, F2 1448 / 1400815296, F4-F7 untouched). Claude-Session: https://claude.ai/code/session_01N8xV1fowU5LmyfxCivNLDH
…lement; the alternatives reproduce a map already rejected
…m 2.24x to 1.14x per doubling
…r stage over the plan's flat transfer list (amr_device_pack)
…se-grid halo is 39% of growth, batched advance saves 0.60 s/step, our geo-mean equals the bar rather than beating it
…the batched advance worth 0.38; the uniform denominator re-measured at 240 steps
…gather is 0.224 s/step of non-wait work -- 25% wait, not 12%, after review folded pgather into its own phase bracket
… GB of output byte-identical, the cost removed is per-map not per-launch, and three measurement failures including a retraction of ledger 74's invented bracket-inflation caveat
…receives per box-peer run) and flag the exactly-tiled pool assumption at the fused copyout
…ff amr_device_pack flag (-0.14 s/step, 8.1x fewer device maps, 12 GB of field output byte-identical)
…chemistry-AMR tests it does not cover
… np8 spread was one outlier, settled by the 2-node follow-on; k004-004 and k004-005 are sick
…overs 7 percent of what the fused packs recover, so the cost is the map machinery not the malloc, and ledger 75's hedge is corrected
…istry-AMR tests needed the bench fftw on LD_LIBRARY_PATH, not a code fix
… session -- it explains the drift, the unusable step column and probably the np16 hang, and the wait-free gather column was immune
…was too weak to see the degradation it exists to catch
…ing has ever set, whose dead brackets owned instrument rows that could never record (-56 LOC, all 70 AMR tests green)
Lines of Code
|
…er-dispatch not per-byte, a quarter to a third of the gap -- two drafts withdrawn on review, the clustering knobs are inert, the cap-32 work saving belonged to a mesh nobody stepped
… gather consume is bit-identical and saves nothing; the census shows the pooled kernels are not slower, the per-block cost in the consume is non-kernel time; parked on task12/batched-gather
…pk/fill) around the per-block gather consume in both waves, same clock as [mpiwait]
…_OMP_PRESENT_ALLOCATABLE); m_amr.fpp opts in Without it amdflang maps every allocatable array of derived type a kernel touches on every launch, walking and re-attaching each component (~0.3 ms per launch for a 10-component array, linear in components; a per-element mapper generated for the type taxes every kernel in that compilation unit). m_amr opts in after an audit of its 95 kernels (bare module allocatables named: amr_cg, amr_cons_br, both allocated before first use; conditional ones are GPU_DECLARE'd). Not global: a kernel naming an unallocated allocatable variable or component aborts under present (m_variables_conversion's q_T_sf%sf). Step-loop wall -8% at cap 64, -18% at cap 32; restart files byte-identical; 70 AMR goldens; np=2 oracle. Ledger 82. Gated as 00a7c56/dede9e33 on origin/task13/host-profile (squashed here: the intermediate global form does not build).
…f allocatable derived-type arrays; per-file defaultmap(present:allocatable) opt-in for m_amr, -8%/-18% wall, bit-identical, gated
… in every form, null components do not; m_amr re-audited under the corrected rule (comments updated)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An opt-in, default-off family of performance features and the measurement infrastructure they rest on. With all flags at their defaults the only touched production path is
s_mpi_decompose_computational_domain, refactored through the newm_boxmodule (byte-identical; covered by the existing suite).m_box(partition arithmetic),m_load_weight/load_weight_wrt(per-cell load-weight field + imbalance metric),m_sfc_partition/sfc_partition_wrt(Morton-SFC predicted-imbalance diagnostic),m_load_balance/load_balance(weighted static decomposition at init; AMR-fine-work-aware),m_rank_timing/rank_time_wrt(per-rank compute-time diagnostic).m_active_box/active_box: restricts reconstruction/Riemann/RK windows to a light-cone-grown box around non-ambient flow; strict-subset golden-tested.hybrid_wenoandhybrid_riemann(+hybrid_weno_eps,hybrid_smooth_flux): linear-optimal weights / central-or-Rusanov flux in smooth cells, full WENO/HLLC at flagged discontinuities (Jameson sensor, stencil-dilated, per-level under AMR).m_amr+m_amr_registers: two-level 2:1 refined block hierarchy; conservative restriction and conservative-linear prolongation with physics-specific closures; per-stage flux registers with Berger–Colella refluxing; Berger–Rigoutsos multi-block dynamic regrid; optional dt/2 subcycling; multi-rank (single-owner blocks assigned by Morton-SFC work balancing at each regrid, with migration; blocks may span rank seams via P2P coarse↔fine gather/scatter; same-level seam halo; distributed registers); restart (both IO modes, regridded-layout persistence); AMR-aware post-processing (fine blocks visualizable as Silo overlay domains); GPU-resident fine level on both OpenACC and OpenMP offload.Full algorithm and user documentation:
docs/documentation/amr.md(support matrix enforced at runtime by the checker — unsupported combinations abort with named messages, never silently).AMR physics support matrix (abridged; authoritative table in amr.md)
Supported and golden-tested: single- and multi-fluid (5-eq,
mpp_lim) · 6-eq with per-block pressure relaxation · viscous (refluxed) · phase change (relax) · chemistry incl. species diffusion · Euler–Euler bubbles (polytropic/non-polytropic, mono/polydisperse, QBMM incl. non-polytropic with per-blockpb/mvside-state; dynamic regrid + subcycle) · acoustic sources (coarse-grid support with regrid exclusion) · immersed boundaries (multi-body, static or prescribed-motion, incl. dynamic regrid with body-containment expansion and per-substage guards) · 2D axisymmetric (per-block WENO-coefficient recompute) · stretched grids (exact parent-bisection ghost coordinates + per-swap coefficient recompute) · hybrid WENO/Riemann sensors (per-level) · Lagrangian bubbles (cloud excluded from blocks; two-way coupling on the coarse grid; regrid clips around the moving cloud) ·active_box(blocks contained in the growing window; agrees with plain AMR to ~1e-14) · IGR (restriction-only coupling: fine sigma solve seeded/Dirichlet-bounded by the coarse solve; documented truncation-order seam, exact free-stream) · 1D MHD/RMHD (div(B)=0 by construction in 1D; HLL and HLLD, incl. relativistic).Gated with named aborts (documented rationale): surface tension (seam force imbalance is structural — three fixes attempted and diagnosed in amr.md) · 2D/3D MHD (attempted and measured: the c/f seam is a continuous O(1) div(B) source GLM cleaning cannot remove — needs constrained-transport-class B prolongation/reflux) · hyperelasticity · 3D cylindrical (global azimuthal filter) · force-driven IB (
moving_ibm=2) · STL bodies · Riemann-extrapolation BCs (bc=-4) ·amr_subcycleunder IGR · stretched grids with Lagrangian/IB-regrid (uniform-spacing index conversions).Validation evidence
Known issues (all non-gating or in progress)
continue-on-error): an intermittent post-detected NaN on the two Lagrangian+AMR goldens. Exhaustively unreproducible off GitHub's runners — the exact failing stack (NVHPC 24.3 SDK,-tp=px -Kieee, HPC-X MPI, and the CI docker image itself under apptainer) passes elsewhere, as do native/zen2 builds; 24.5+ green. Documented at the golden definitions.Review guide
The commit history is arc-ordered (active-box → load-weight → SFC → weighted decomposition → rank timing → hybrid → m_box → AMR rungs → physics envelope → CI/GPU hardening); reviewing by arc is much easier than by file. The AMR arc builds stepwise: static hierarchy → restriction/prolongation → fine advance → refluxing → regrid → subcycling → multi-rank → GPU → each physics rung with its own validation. Commit messages carry the validation evidence for their change (measured defects, golden UUIDs, repro details for CI fixes).
All parameters ship default-off with
case_validatorentries, runtime checker gates, andcase.md/amr.mddocumentation.