Stop a broker restart from silently killing a service's JMS consumer

[jcschaff]

Fixes #2031.

A broker restart could kill a service's JMS consumer thread permanently while the process stayed
up and the pod stayed Ready. Dev's submit service ran for 6h50m on 2026-08-24 in exactly
that state, consuming nothing.

What actually happened

The dev logs give the whole sequence, and it turned out to be two defects rather than one:

19:51:15.914Z  WARN   JmsFailoverWatchdog  JMS transport interrupted, failover reconnecting   (x3)
19:59:16.954Z  ERROR  ConsumerContextJms   Connection refused
19:59:16.957Z  FATAL  JmsFailoverWatchdog  JMS transport unrecoverable, invoking terminal handler
19:59:16.959Z  ERROR  ConsumerContextJms   Connection refused
19:59:16.960Z  INFO   ConsumerContextJms   ConsumerContextJms@6796aa72 consumer thread exiting.

481 seconds between interrupt and giving up, matching maxReconnectAttempts=20 with the 1s→30s
backoff. Then nothing at all for six hours.

1. The consumer misread a dead session as shutdown. ConsumerContextJms treated every
javax.jms.IllegalStateException out of receive() as deliberate shutdown and broke out of the
poll loop. Half of that is right — close() unblocks a thread parked in receive(), and looping
on a closed consumer would spin. But the provider raises the same exception when the session dies
underneath a consumer we are still meant to be polling, which is the case that most needs
reporting.

2. The terminal condition was detected and then ignored. d58cd1292e bounded the failover
reconnect budget on the reasoning that in K8s a pod restart is the right response to a sustained
broker outage, added jvmExitOnTerminal() as the escape hatch, and left every service on
logOnly() for a future caller to take up. git log -S setFailoverWatchdog returns exactly that
one commit — nobody ever did. So the FATAL above invoked a no-op.

Either defect alone is enough to produce the zombie: fixing only (1) leaves the process alive
around a connection it can never use again, and fixing only (2) leaves the consumer thread dying
silently wherever the watchdog is not attached — attach() installs nothing for a
non-ActiveMQConnection, which matters if the Artemis work ever moves off the OpenWire client.

The change

• JmsFailoverWatchdog.onTerminalFailure(what, cause) — a route to the terminal action for a
  failure the caller detected itself. The existing transport path folds through it; passing
  "transport" reproduces today's log line exactly.
• ConsumerContextJms now tests bProcessing alone to recognise shutdown (both closeAll() and
  stopAndClose() clear it before close(), so it is sufficient), and escalates a session lost
  while still processing instead of exiting quietly.
• submit, sched, data and db build their messaging service with
  createForLongLivedConsumerService(), taking the escape hatch. Short-lived batch processes
  (SolverPreprocessor, SolverPostprocessor, JavaSimulationExecutable, OptimizationBatchServer)
  and the API server keep the log-only default — they outlive neither the broker outage nor their
  own task.
• docs/MESSAGING.md §8 said the watchdog exits the JVM in production. That was the intent, not
  the behaviour; it now says who opts in.

A restarted pod does not crash-loop while the broker is still down: startupMaxReconnectAttempts=-1
leaves the initial connect unbounded, so the process waits at boot and picks up when the broker
returns.

Tests

ConsumerContextJmsTest — one regression case and two controls.

The failure is injected at the consumer rather than by stopping the broker on purpose. Stopping
the broker also drives the failover transport to the same terminal handler, so that version of
the test passes on the broken code for the wrong reason; here the transport stays healthy and the
consumer's own escalation is the only thing that can fire the latch.

Verified against the pre-fix bytecode (javap-checked, not just assumed):

	before	after
aSessionLostWhileStillPollingIsEscalated	fails	passes
anOrdinaryShutdownIsNotEscalated	passes	passes
aTransientPollFailureIsNotEscalated	passes	passes

vcell-server Fast group is otherwise unchanged.

Not covered here

Nothing outside the process would still have noticed. #2032 tracks the missing livenessProbe on
these four services — worth doing regardless, since this fix covers one way a consumer can die and
a probe covers all of them.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01PFAraxXXN2KvRqgo9GPmv5