DEBUG: diagnose the stochastic appctx loss in test_baseform_coarsening

[pbrubeck]

Description

AI-assisted (Claude Code)

Draft, for debugging. This branch exists to make one stochastic CI failure diagnosable. The
parallel test steps are switched off while it runs, so it must not be merged as it stands.

The failure

tests/firedrake/multigrid/test_poisson_gmg.py::test_baseform_coarsening[scalar-mg] fails
occasionally at nprocs = 1 (one instance). CI reports it as a bare AssertionError at
dmhooks.py:250, which is not where anything went wrong.

The real error is in solving_utils.form_jacobian:

    ctx = dmhooks.get_appctx(dm)
>   problem = ctx._problem
E   AttributeError: 'NoneType' object has no attribute '_problem'

get_appctx(snes.getDM()) returned None, that surfaced as PETSc.Error(101) out of
SNESSolve, and while ExitStack unwound it, add_hooks.__exit__ asserted on a hook stack that
was empty as well. The assertion is the failure pytest reports, and it discards the traceback of
the error that caused it.

The window is narrow. In SNESSolve_KSPONLY, SNESComputeFunction is line 27 and
SNESComputeJacobian is line 41; form_function performs the identical get_appctx lookup and
succeeded. Between the two, PETSc runs SNESGetNormSchedule, a monitor block that is skipped
(snes->numbermonitors is 0 here) and PetscTryTypeMethod(snes, update), for which Firedrake
registers nothing. So the application context went missing from a live DM during
ctx._assemble_residual, and by unwind time the setup hooks had gone with it.

I could not reproduce this, in isolation or otherwise. Ruled out so far: petsc4py holds setAttr
data on the C object, where it survives collection, temporary-wrapper collection and reference
cycles routed through the shared dict; the DM cannot be silently rebuilt, since DataSet.dm is a
cached_property on an ObjectCached DataSet held in mesh._shared_data_cache, a plain
defaultdict(dict) with no eviction; and the push/pop of __appctx__ balances exactly over all
four right-hand sides. The scalar-mg case survived 89 iterations of all eight parametrizations in
one process with gc.set_threshold(1, 1, 1) and a forced gc.collect() at the end of every
form_function. CI runs Python 3.14.4 against 3.12.13 locally, which is the obvious remaining
difference.

What's in it

• dmhooks.add_hooks.__exit__ no longer masks an in-flight exception. A broken hook stack now
  raises a RuntimeError naming the DM and the solver when nothing else is propagating, and warns
  instead of raising when something is, so the original error reaches the report intact. This is
  the change that makes the next occurrence readable.
• solving_utils._callback_appctx replaces the five open-coded get_appctx lookups in the
  PETSc callbacks. A missing context is now a RuntimeError naming the DM and the options prefix,
  rather than an AttributeError on None several lines later.
• A bounded trace of DM attribute stack operations in dmhooks, printed by both of the above.
  A DM's stacks are touched a handful of times per solve, so recording the last 512 pushes and
  pops costs nothing and shows which DM lost what, and when.
• Parallel test steps disabled in the Linux default job of core.yml (nprocs 2 to 8, and the
  parallel tests/pyop2 runs). The failure is serial, so the parallel runs are pure cost while
  this branch is being re-run. The linux_extra jobs are untouched; they only run behind a label.

What to do with it

Re-run the nprocs = 1 step until it trips. The report should then carry the RuntimeError from
_callback_appctx together with the stack history, which says whether the appctx was popped, by
what, and off which DM.

Notes for review

Two related weaknesses turned up while reading this code. Neither is addressed here, and neither
explains the failure, but both let a missing application context pass unnoticed:

• mg/ufl_utils.py:447 pushes an appctx onto the coarse DM and registers the matching teardown
  only if parentdm.getAttr("__setup_hooks__"). When that is falsy the push leaks, and the
  if get_appctx(newdm) is None guard then makes later solves reuse a stale context.
• dmhooks.get_transfer_manager treats a missing appctx as "not in a solve" and builds a fresh
  TransferManager. test_baseform_coarsening[mixed-*] trips this today: reconstruct_function
  calls get_transfer_manager(V.dm) for an IndexedProxyFunctionSpace coming off a
  DirichletBC, and that sub-DM never receives an appctx. It warns and carries on; the same
  condition in form_jacobian is fatal.