Queue-depth-only escalation throttles an idle database: Jul 20 flood ran 2h23m with threads_running ≤ 5

[mrtwebdesign]

Summary

During the Jul 20, 2026 queue flood on bloomzhemp production, the Action Scheduler throttle (HYPERCART_QUERY_GUARD_THROTTLE_MODE = enforce) ran a 2h23m intervention triggered entirely by queue depth while every database signal showed an idle server. The throttle's response — batch size 1 and mass deferral — slows backlog drain and increases PHP-worker cost per action, so on a site where PHP workers (not MySQL) are the constrained resource, it plausibly extended the event rather than absorbing it.

Analysis by Claude Code, from the Jul 11–20 bloomzhemp production log review (nginx, Apache, WP debug, HyperCart query_guard logs, plus WP Engine's pt-query-digest), cross-referenced against this repo's source.

Evidence from the Jul 20 event

Trigger: a 74.9 s wp-cron.php run at 07:09:29 PT. Queue depth went 5–15 → 112 → peak 847; the guard logged normal → elevated at 07:09:55, first critical at 07:17:39, seven separate critical transitions, and the last elevated → normal at 09:32:16 (10.0 min critical, 51.9 min elevated).

Database state during the same window:

• threads_running never exceeded 5 during the flood (peak 6 across the entire ten-day period; value was 2 on 86% of all samples)
• the guard's own DB probe answered in 1 ms median, 4 ms max, zero errors
• WP Engine's pt-query-digest covering the entire flood window found zero slow queries from this production site

So the max(threads_level, queue_level) in HCQG_Load_Monitor::evaluate_level() was driven by queue_depth alone for the full 2h23m. Queue depth crossed queue_depth_elevated (100) and queue_depth_critical (500) while threads_running never came near threads_running_elevated (5).

Cost of the intervention:

• 9,689 deferrals; 953 actions hit the 5-defer cap and executed anyway — for those, the throttle bought pure delay with no shedding
• 3,486 wp_mail_smtp_send_enqueued_email actions (customer emails, potentially order confirmations) released in a burst 1–2 hours late
• 2,441 wc-admin_import_orders and 2,010 woocommerce_deliver_webhook_async actions delayed

Why this is a design problem, not a tuning problem

1. Queue depth measures backlog volume, not server distress. A deep queue with an idle database means "drain faster." The throttle's response is "drain slower." When threads_running corroborates, throttling is defensible; when it doesn't, the queue-depth signal alone cannot distinguish "MySQL is drowning" from "a cron hiccup batched up work for an idle server."

2. The deferral loop re-triggers its own signal. Wave B clones an action to now + delay (300–900 s for normal/deferrable tiers). Deferred clones come back due in 5–15 minute waves, re-inflating the due-queue count that is the only active trigger. The seven distinct critical transitions in the log are the signature of deferral waves re-entering the queue and re-escalating the level. The event self-sustains until enough actions hit max_defers and force-execute.

3. batch_size: 1 at critical inverts the protection on plugin-heavy sites. Each queue-runner call is a full WordPress bootstrap (~180 plugins on this site). At the default AS batch size, that bootstrap amortizes over up to 25 actions; at batch_size: 1 it is paid per action. The observed effect: runner calls that took 5–17 s before onset took 20–44 s each from 07:10, and did not recover until 09:32. Total PHP-worker seconds per unit of completed work went up under throttling — and PHP worker time, not MySQL, is the scarce resource on this site (threads_running data above; meanwhile the site spends 6.55 h/day of worker time on admin-ajax.php alone).

The one genuinely protective element was concurrent_batches: 1, which capped AS to roughly one PHP worker and left the rest for shoppers during the flood. That is the right instinct gated on the wrong sensor — the throttle has no PHP-worker-availability signal at all; queue depth just loosely correlates with one.

4. Counterfactual. 847 pending actions at the site's ~7.6 s per-runner-call average is roughly 1.8 h of sequential work. Drained at default batch size against a database answering probes in 1 ms, the backlog plausibly clears in well under an hour instead of degrading background processing for 2h23m and delaying customer email.

For balance: on ordinary days the throttle is inert (queue depth 5–15 vs. an entry threshold of 100) and its overhead is one ~1 ms probe per AS request. This issue is strictly about flood-time behavior under a queue-depth-only trigger.

Proposed changes

1. Require DB corroboration for queue-depth escalation in HCQG_Load_Monitor::evaluate_level(): queue depth alone can raise the level to at most elevated (or not at all) unless threads_running is also above its entry threshold. This preserves current behavior in real DB-load incidents and removes the failure mode above. (Alternative interim mitigation, no code change: neuter the queue-depth thresholds on affected sites via the hypercart_query_guard_load_thresholds filter.)
2. Reconsider batch_size: 1 in the critical policy (THROTTLE_POLICY, hypercart-query-guard.php:100-111). concurrent_batches: 1 already achieves the worker-protection effect; batch size 1 only multiplies per-action bootstrap overhead. A critical-level batch size of 10–25 with concurrency 1 protects foreground workers without inflating total work.
3. Exempt or fast-lane email actions. wp_mail_smtp_send_enqueued_email resolves to the default normal tier (300 s at elevated, 900 s at critical) and was the single most-deferred hook. Order-confirmation email is arguably critical tier — customer-facing latency with no meaningful DB cost per action.
4. Until (1) lands, recommend observe for HYPERCART_QUERY_GUARD_THROTTLE_MODE on sites with no history of DB pressure. The production record here shows zero slow queries and threads_running ≤ 6 over ten days — enforce mode currently has nothing to protect on this site and a demonstrated cost.

Related

• Default threads_running_elevated threshold (5) may be too low for busy stores #10 — threads_running_elevated default of 5 may be too low; note that on Jul 20 even 5 was never crossed, so raising it would not have changed this event
• Add filter for no-signal default throttle level (currently hardcoded to elevated) #17 — no-signal default level is hardcoded to elevated; same family of "escalation without evidence of DB load"
• Coverage gap: 87% of slow queries invisible to observation subsystem #43 / Feature: consequence-aware query thresholds #33 — separate detector blind spot, listed here only because both erode confidence that the guard's picture of "DB load" matches reality

[mrtwebdesign]

Update: bloomzhemp production was switched to HYPERCART_QUERY_GUARD_THROTTLE_MODE = 'observe' on 2026-07-20 (proposal 4), via wp-config.php line 161. The query-timeout subsystem remains in enforce. Config backup saved on the server as wp-config.php.bak-2026-07-20-qg-observe. Observe-mode decision logs from the next flood event can serve as the counterfactual data for proposals 1–3.