Supervisor remains alive with missing worker process; due scheduled jobs stop being dispatched until restart

[sapandiwakar]

Description

We hit a production incident where Solid Queue stopped processing due scheduled jobs even though Mission Control still showed active workers with recent heartbeats.

Restarting the Solid Queue jobs process immediately fixed the issue.

This appears to be a supervisor/dispatcher/process-liveness issue rather than an application job failure: the process table was missing one expected worker before restart, the remaining workers continued heartbeating, scheduled jobs accumulated, and the supervisor did not appear to replace the missing worker or prune it as dead.

Environment

• solid_queue: 1.4.0
• Rails: 8.1.3
• Ruby: 3.4.9
• Queue DB: SQLite
• App DB: PostgreSQL
• Mode: fork
• Started via bin/jobs
• Deployment: Kamal container
• Worker config:

dispatchers:
  - polling_interval: 1
    batch_size: 500

workers:
  - queues: [high_priority, medium_priority, default]
    threads: 3
    processes: 1
    polling_interval: 0.1

  - queues: [high_priority_transport, medium_priority_transport, default_transport]
    threads: 6
    processes: 3
    polling_interval: 0.1

Effective production env:

JOB_CONCURRENCY=1
JOB_THREADS=3
TRANSPORT_JOB_CONCURRENCY=3
TRANSPORT_JOB_THREADS=6
DB_POOL_SIZE=40

Expected behavior

The supervisor should keep the configured process set alive:

• 1 Dispatcher
• 1 Scheduler
• 1 Supervisor(fork)
• 4 Workers

If a worker exits, the supervisor should replace it or mark/prune it clearly.

Due scheduled jobs should be moved from solid_queue_scheduled_executions to solid_queue_ready_executions and then picked up by workers.

Actual behavior

Mission Control showed only 3 workers before restart:

worker 416 PID 229965
worker 417 PID 236467
worker 418 PID 236473

All three had recent heartbeats.

At the same time:

• Scheduled jobs accumulated: 264 scheduled jobs, many delayed by around 20+ hours
• In progress jobs: 0
• Blocked jobs: 0
• Some transport jobs had recently finished, but old delayed scheduled jobs remained stuck
• Restarting the jobs container restored the expected process shape:

{
  "Dispatcher" => 1,
  "Scheduler" => 1,
  "Supervisor(fork)" => 1,
  "Worker" => 4
}

After restart, job processing resumed.

Log evidence

The old worker log had continuous heartbeats and pruning from the supervisor, but no obvious process replacement or dead-process pruning:

SolidQueue-1.4.0 Prune dead processes (...) size: 0

This repeated every 5 minutes.

The log also showed that work stopped moving after approximately 2026-06-24 11:06:28 Europe/Zurich. After that, the process kept heartbeating/pruning but did not perform jobs.

Aggregated log counts by minute near the incident:

2026-06-24T11:05 enq=41 ready=12 performing=22 performed=19 hb=5 prune=0 err=18
2026-06-24T11:06 enq=81 ready=19 performing=32 performed=34 hb=5 prune=0 err=33
2026-06-24T11:07 enq=0  ready=0  performing=0  performed=0  hb=5 prune=0 err=0
...
2026-06-24T11:29 enq=0  ready=0  performing=0  performed=0  hb=6 prune=1 err=0

I also searched the log for replacement/termination signals and did not find any replace_fork, process-exit, shutdown-timeout, or prune signal explaining the missing worker.

Why this looks like a Solid Queue liveness issue

From the README, workers process jobs from solid_queue_ready_executions, while dispatchers move due scheduled jobs from solid_queue_scheduled_executions to ready executions.

In this incident:

• due scheduled jobs were present and delayed
• workers/supervisor were still heartbeating
• one configured worker process was missing
• no replacement/prune log was emitted
• restart restored the missing process and resumed processing

So the failure mode appears to be: the supervisor remained alive, but the configured process set was incomplete and/or dispatching stopped, without recovery or visible error.

This looks similar in shape to #204, but we are seeing it on Solid Queue 1.4.0.

[sapandiwakar]

I dug further into the Solid Queue 1.4.0 code and the old logs, and I think the more precise failure mode is dispatcher liveness rather than worker liveness.

Delayed jobs live in solid_queue_scheduled_executions; workers only consume from solid_queue_ready_executions. In 1.4.0, the path that moves due scheduled jobs to ready jobs is SolidQueue::Dispatcher#poll:

def poll
  batch = dispatch_next_batch
  batch.zero? ? polling_interval : 0.seconds
end

def dispatch_next_batch
  with_polling_volume do
    ScheduledExecution.dispatch_next_batch(batch_size)
  end
end

and ScheduledExecution.dispatch_next_batch does the scheduled-to-ready transition:

transaction do
  job_ids = next_batch(batch_size).non_blocking_lock.pluck(:job_id)
  if job_ids.empty? then 0
  else
    SolidQueue.instrument(:dispatch_scheduled, batch_size: batch_size) do |payload|
      payload[:size] = dispatch_jobs(job_ids)
    end
  end
end

Looking back at the pre-restart log, I can identify these active PIDs:

• #1: supervisor, pruning dead processes every 5 minutes
• #39: scheduler; it logs the recurring task at 11:00 and heartbeats process id 392
• #229965, #236467, #236473: workers

After restart, the healthy process counts were:

{
  "Dispatcher" => 1,
  "Scheduler" => 1,
  "Supervisor(fork)" => 1,
  "Worker" => 4
}

So before restart we effectively had 5 live process records/PIDs visible in logs instead of the expected 7: supervisor + scheduler + 3 workers. The dispatcher was absent or not making progress, and one worker was also missing.

This also explains why there were recent finished jobs: immediate jobs can be enqueued directly into solid_queue_ready_executions, so workers can still finish some jobs while delayed/scheduled jobs accumulate because the dispatcher is not moving them to ready.

The recovery gap I see is in the fork supervisor. ForkSupervisor#check_and_replace_terminated_processes only replaces a child when Process.waitpid2(-1, Process::WNOHANG) reports that child exited:

def check_and_replace_terminated_processes
  loop do
    pid, status = ::Process.waitpid2(-1, ::Process::WNOHANG)
    break unless pid

    replace_fork(pid, status)
  end
end

I do not see a reconciliation path that checks whether the configured process set still matches the registered/active process set, nor a progress watchdog such as “dispatcher has not dispatched anything for N minutes while due scheduled jobs exist.”

So the current theory is:

1. The dispatcher fork became missing, wedged, unregistered, or otherwise stopped making progress.
2. Because the supervisor only replaces observed exited child PIDs, it did not recover this state.
3. The supervisor/scheduler/workers continued heartbeating, so Mission Control looked partially healthy.
4. Due scheduled jobs accumulated until the jobs container was restarted.
5. Restart rebuilt the configured process set and processing resumed.

I cannot prove the original trigger from the logs alone. It could be SQLite lock contention, child-process state drift, or a blocked dispatcher poll loop. But the missing recovery path seems concrete: heartbeat/process presence does not prove dispatcher progress, and the supervisor does not appear to reconcile configured process counts/kinds against actual registered process state.