ruby / ci-queue: Recover queue initialization when the master exits

[ckwarteng]

There’s a long-standing ci-queue bug that causes CI::Queue::Redis::LostMaster. If the initializer dies (example during spot terminations) before publishing the queue, its setup marker remains indefinitely, preventing surviving workers and retried jobs from taking over. This change simplifies the flow and makes initialization recoverable.

• A worker claims initialization by creating a setup marker only if one doesn’t already exist.
• The marker expires after half the initialization timeout, leaving time for another worker to take over.
• The first initializer to successfully publish atomically writes the queue, test count, and ready status.
• Once the queue is ready, workers begin running tests normally.

We also add some new queue initialization unit tests. The downside is that we make more redis requests than before (about 20 extra requests per second per waiting worker). This will well within how much we can support so I think the simplicity is worth it.

Written by AI

When the elected initializer dies before publishing the test queue, its master-status=setup marker previously remained indefinitely. Surviving workers and replacement jobs could not initialize the queue and failed with CI::Queue::Redis::LostMaster. In build 8569214, a spot termination during initialization left both surviving workers and the initializer's retry waiting on the abandoned marker.

The setup marker now expires after half of queue_init_timeout, and waiting workers attempt initialization again. A WATCH-guarded transaction publishes the queue, total, and ready status together, so the first initializer to publish wins. A slow initializer can overlap its replacement; each attempt gives its chunks unique IDs, keeping their metadata and dynamic timeouts separate.

Chunk metadata is staged in batches of five, using at most 22 commands per transaction. Final publication uses six commands. This respects Sinatra's 25-command transaction limit. Metadata retains the existing Redis TTL, including unused chunks from an interrupted or superseded attempt.

Validation:

• 93 Redis tests, 410 assertions, no failures or errors with both Redis clients 4.2.5 (Sinatra's version) and 6.0.0 on Ruby 3.4.7.
• Regression cases cover initializer death, retry and takeover, setup exceptions, stale publication and metadata, partial metadata staging, and the 25-command transaction limit.
• All 100 Sinatra test workers passed in both baseline build 8600764 and fixed build 8600872, using the same Figma base revision and full main test group (83,367 tests). Initialization took 22.622 seconds before and 21.054 seconds after; group elapsed time was 19.0 versus 15.7 minutes. This single comparison shows no observed slowdown; cache and agent placement differences prevent attributing the improvement to this patch.
• A controlled local experiment with 100 workers and a delayed initializer exercised takeover: all 1,000 tests executed exactly once and the original initializer lost its master role. The healthy Sinatra experiment had one initializer and no polling initialization retries, so it did not measure production takeover polling load.
• Datadog during the fixed run showed queue Redis engine CPU at most 2.533%, timing Redis at most 6.333%, and zero queue throttling or rejected connections. At 100 workers, 99 waiters would add approximately 1,980 Redis commands/second while initialization is pending. This is an estimate; simultaneous waiting builds multiply it.
• The first patched experiment exposed the transaction limit in build 8600767; it was canceled after diagnosis, and this revision restores bounded batches.
• Both overall experiment builds failed afterward in sinatra-collate-coverage with No Sinatra test coverage results were found: forced job selection included default-branch coverage reporting on feature branches, where coverage collection is disabled. All 100 Sinatra test workers passed in each build; these are successful test groups, not fully green builds.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 8b3da9d. Configure here.}