Skip to content

fix(scheduler): contain panics so a bad cycle skips instead of wedging background refresh - #428

Merged
runyourempire merged 1 commit into
mainfrom
worktree-agent-adb1fafb1e978405c
Aug 14, 2026
Merged

fix(scheduler): contain panics so a bad cycle skips instead of wedging background refresh#428
runyourempire merged 1 commit into
mainfrom
worktree-agent-adb1fafb1e978405c

Conversation

@runyourempire

Copy link
Copy Markdown
Collaborator

The bug

A panic anywhere in the scheduled-analysis path permanently disabled background refresh for the rest of the process lifetime. No error, no UI signal — the feed just quietly stopped updating.

Verified mechanism:

  1. monitoring.rs:1183 / :1335 claims the gate: is_checking.swap(true, SeqCst)before scheduled-analysis is emitted.
  2. app_setup.rs:785 ran run_scheduled_analysis(handle) inside a bare tauri::async_runtime::spawn. The JoinHandle is dropped, so nothing ever observes a JoinError.
  3. That awaits fill_cache_background(&handle).await unguarded (app_setup.rs:1985).
  4. Every site that clears the gate — app_setup.rs:2011, :2109, monitoring.rs:1348, monitoring_notifications.rs:69 — sits after the work.

So one unwind skipped all four clear sites and latched is_checking at true forever. Every subsequent tick saw is_checking == true and silently skipped itself.

This was already a known shape with no recovery path: void_engine/heartbeat.rs:158 carries the comment "check if monitoring is_checking stuck (simple heuristic)". headless.rs:258 had the twin failure — an unwind out of a cycle propagated through the daemon loop and killed fourda-engine outright.

The fix

Contain the panic at the boundary so a bad cycle becomes a skipped cycle, not a permanent wedge.

  • src-tauri/src/task_guard.rs (new) — contain(), an AssertUnwindSafe(..).catch_unwind() wrapper that logs the panic payload and returns None. Mirrors the pattern already proven in analysis_status.rs:59-68.

  • app_setup.rs:1983-2025run_scheduled_cycle_contained() wraps the cycle and releases the gate in the recovery arm only.

    Deliberately not an unconditional clear: clearing after a normal completion could race a tick that has already claimed the gate for the next cycle, trading a wedge for two concurrent scheduled analyses against shared state. The normal paths already release it; only the unwind path was unreachable.

  • headless.rs:583-599 — the daemon loop survives a panicking cycle and retries on the next tick instead of dying.

  • headless.rs:133-144--once exits with its documented code 1 instead of an unwind/abort, so an OS scheduler records a clean failed run.

Proving the regression test is real

A test that passes both before and after proves nothing, so this was falsified explicitly: the containment in run_scheduled_cycle_contained was temporarily replaced with a bare cycle.await (the exact unfixed shape) and the suite re-run.

cycle_panicking_before_first_await_releases_the_gate ......... FAILED
panicking_scheduled_cycle_releases_the_in_flight_gate ........ FAILED
normal_completion_leaves_the_gate_exactly_as_the_cycle_left_it  ok
normal_completion_preserves_the_cycles_own_gate_release ...... ok

Both panic tests fail against unfixed code (the unwind escapes and kills the test); both success-path tests still pass, confirming they are not just passing by accident. The containment was then restored and all four pass.

The tests use a local MonitoringState rather than the process-wide global, so they are hermetic under parallel execution.

clippy::string_slice — measured, then scoped

The intent was a crate-wide string_slice = "deny" on the theory that residual hits would be few after #422. Measured against this tree it fires 280 times, so that premise did not hold:

Area Hits Area Hits
sources/ 55 monitoring_briefing.rs 13
ace/ 38 preemption.rs 10
scoring/ 36 diagnostics.rs 7
content_personalization/ 25 (~90 others)

The lint is the right one: the only two hits in source_fetching/ are #422's own floor_char_boundary slices, which confirms it would have flagged all 23 original panic sites. But bulk-#[allow]ing 278 pre-existing sites — 55 of them in sources/ adapters that parse ingested text — would risk cementing a live panic behind an allow that claims safety without an audit. That is the same "large churn" bar this PR was told to apply to indexing_slicing.

So it is adopted per-module, starting where the panic class actually lives:

  • #![deny(clippy::string_slice)] on src/source_fetching/mod.rs — the module that caps scraped bodies, and where two of fix(utf8): eliminate 23 string byte-slicing panic vectors on ingested text #422's 23 panics were.
  • The two flagged sites collapse into one cap_on_char_boundary() helper carrying the single char-boundary proof, removing the duplication.
  • fetcher_tests.rs previously re-implemented the capping logic inline (content[..CONTENT_CAP]), so it asserted against a copy of the logic and could never have caught the original bug. It now calls the real function, plus new multi-byte (3-byte across every cap 1..=8) and emoji (4-byte) boundary cases.
  • Cargo.toml carries the measurement and the promotion path for whoever audits the backlog.

Verification

Check Result
cargo clippy -- -D warnings (default) clean
cargo clippy --features experimental -- -D warnings clean
cargo fmt --check clean
cargo test --lib 4446 passed, 1 pre-existing flake

The one failure is db::migrations::recovery_tests::locked_db_returns_recovery_failed_without_quarantine, a timing assertion ("locked DB recovery must return quickly, elapsed 32.7s"). It passes in 2.03s in isolation — starved by parallel load on a busy machine, unrelated to this diff, which touches no DB-migration code.

cargo clippy --all-features is pre-existing broken on main (~56 errors in team_sync/webhooks, chacha20poly1305::generate_nonce gone from the locked aead 0.6.1) and was not touched. This diff cannot affect it: source_fetching contains zero cfg(feature) code, so the scoped deny behaves identically under every feature combination.

Follow-ups deliberately left out

  • clippy::indexing_slicing — not added, as instructed. Not separately measured; expected to exceed the 280 string_slice count since it covers slices and index ops on all sequence types.
  • The 278-site string_slice backlog — needs a real per-site audit, best done module-by-module (sources/ 55 first — highest ingested-text exposure). Some of those may be live panics of exactly the fix(utf8): eliminate 23 string byte-slicing panic vectors on ingested text #422 kind.
  • The other ~54 unguarded spawn sites — scoped out per instructions. contain() is now available for them. No spawn_guarded wrapper was added: both call sites here need custom recovery (a gate release / an exit code), so a generic spawner would have had zero users and tripped the dead-code doctrine.
  • crash_guard interaction (pre-existing, worth knowing)crash_guard.rs installs a panic hook that zeroizes API keys on any unwind, including ones caught by catch_unwind. That already applied to the existing guards in analysis_status.rs and extractors/pdf.rs; this PR does not change it. But it means a survived scheduled panic leaves the process running with zeroized LLM keys. std's hook API cannot distinguish caught from fatal panics, so fixing it is a separate design question.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2

…g refresh

A panic anywhere in the scheduled-analysis path permanently disabled
background refresh for the rest of the process lifetime.

Mechanism: monitoring.rs claims the gate with `is_checking.swap(true,
SeqCst)` BEFORE emitting `scheduled-analysis`; app_setup then ran
`run_scheduled_analysis` inside a bare `tauri::async_runtime::spawn`
whose JoinHandle is dropped, awaiting `fill_cache_background` unguarded.
Every site that clears the gate — `complete_scheduled_check`, the
scoring-error arm, both foreground-collision guards — sits AFTER the
work. So one unwind skipped all of them and latched the gate at `true`:
every later tick saw `is_checking == true` and silently skipped itself.
No error, no UI signal — the feed just stopped updating.
`void_engine::heartbeat` already carried a "check if monitoring
is_checking stuck" probe, i.e. the wedge was a known shape with no
recovery path. `headless.rs` had the twin failure: an unwind out of a
cycle propagated through the daemon loop and killed `fourda-engine`.

- new `task_guard::contain` — catch_unwind wrapper mirroring the existing
  pattern in analysis_status.rs, logging the panic payload
- `run_scheduled_cycle_contained` releases the gate in the recovery arm
  only. NOT unconditional: clearing after a normal completion could race
  a tick that already claimed the gate for the next cycle, trading a
  wedge for a double-run
- headless daemon loop survives a panicking cycle and retries next tick;
  `--once` exits with its documented code 1 instead of an unwind

Regression cover proven by falsification: with the containment replaced
by a bare `.await`, both panic tests fail (the unwind escapes) while the
two success-path tests still pass.

Also adopts `clippy::string_slice` for `source_fetching`, the module
where two of the 23 byte-slice panics #422 fixed lived. Measured against
this tree the lint fires 280 times crate-wide (sources/ 55, ace/ 38,
scoring/ 36, content_personalization/ 25), so it is NOT enabled globally
— bulk-allowing that backlog would cement any live panic hiding in it.
Adopting per-module keeps the guarantee real where ingested text is cut.
The two flagged sites collapse into one proven `cap_on_char_boundary`
helper; fetcher_tests previously re-implemented the capping inline and
so could not have caught the original bug — they now call the real
function, plus new multi-byte/emoji boundary cases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2
@runyourempire
runyourempire enabled auto-merge (squash) August 14, 2026 13:49
@runyourempire
runyourempire merged commit 4e8e197 into main Aug 14, 2026
9 checks passed
@runyourempire
runyourempire deleted the worktree-agent-adb1fafb1e978405c branch August 14, 2026 13:53
runyourempire added a commit that referenced this pull request Aug 14, 2026
…mmit gate (fleet cannot commit) (#430)

## Why this is urgent

`src-tauri/src/analysis_rerank.rs` landed at **1032 lines** in #423,
over the **1000-line hard error threshold** in
`scripts/check-file-sizes.cjs`. That script exits `1`, and
`.husky/pre-commit:38-41` treats a non-zero exit as blocking:

```sh
node scripts/check-file-sizes.cjs || {
    echo "File size check failed. Split large files or add justified exceptions."
    exit 1
}
```

**Every developer on this repo currently cannot commit anything
locally.** This PR restores that.

### How it got in

#423 was Rust-only, so CI's `Frontend` job — the only place the
file-size check runs — was skipped by the path filter and the gate never
fired. The CI path-filter fix is a **separate change owned by another
agent**; this PR deliberately does not touch `.github/workflows/`.

## What this does

Option A (split), not an exception-list entry. The file had a
self-contained 169-line inline `#[cfg(test)] mod rerank_breaker_tests`
block at the bottom, which is the cleanest possible seam — it touches
**zero production logic**.

Moved it into `src-tauri/src/analysis_rerank_tests.rs`, declared with:

```rust
#[cfg(test)]
#[path = "analysis_rerank_tests.rs"]
mod rerank_breaker_tests;
```

This is the established pattern in this crate — **45 other modules**
already use `#[cfg(test)] #[path = "*_tests.rs"] mod ...;` with the test
file holding the module body directly (e.g. `adversarial.rs:495-497`).
Test files get 2x the size limit, so the extracted file sits well inside
it.

| File | Before | After | Status |
|---|---|---|---|
| `analysis_rerank.rs` | 1032 | **866** | warn only (non-blocking) |
| `analysis_rerank_tests.rs` | — | 166 | test file, exempt from warnings
|

The module name is unchanged, so test paths stay
`analysis_rerank::rerank_breaker_tests::*` — no test-name churn.

## Verification

Gate, before and after:

```
# before
1 file(s) exceed error threshold. Split large files or add justified exceptions.
EXIT=1

# after
43 file(s) approaching size limits (warnings only).
EXIT=0
```

| Check | Result |
|---|---|
| `node scripts/check-file-sizes.cjs` | **exit 0** (was exit 1) |
| `cargo fmt --check` | clean |
| `cargo clippy -- -D warnings` (default) | clean |
| `cargo clippy --features experimental -- -D warnings` | clean |
| `cargo test --lib` | **4290 passed, 0 failed, 10 ignored** |

The test count is **identical** to the pre-change baseline measured on
the same tree. Beyond the count, a full test-**name** set diff between
the two runs shows **4300 unique names on both sides, 0 added, 0
removed** — proving nothing was orphaned by the module move (a missed
`#[path]` declaration would silently drop tests, which a passing run
alone would not catch).

The production diff is a single hunk starting at line 862; everything
above it is byte-for-byte untouched. The extracted body was verified as
a byte-exact dedent of the original, with the only difference being
rustfmt's re-wrap of the `use super::{...}` block at the reduced indent
level.

## Notes for the reviewer

- Rebased onto latest `main` (`4e8e197b`, #428). No overlap — #428 does
not touch `analysis_rerank.rs`.
- Local pre-commit and pre-push hooks both ran in full and passed. No
`--no-verify` was used anywhere. (Hooks were invoked via `git -c
core.hooksPath=.husky ...` so the worktree ran **its own tracked hooks**
rather than the shared tree's `core.hooksPath` absolute path — that is
the correct branch's gate, not a bypass.)
- `cargo audit` reports a pre-existing `RUSTSEC-2026-0221`
(`event-listener` unsound) on main; unrelated to this change and
warn-only in the hook.
- The exception list in `scripts/check-file-sizes.cjs` was **not**
touched. Worth a separate cleanup pass: it reportedly carries 5 stale
entries for deleted files and 3 with false justifications.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant