Skip to content

fix(queue): bound pending depth and make dedup constant time - #594

Open
walsykerem wants to merge 2 commits into
kernel:mainfrom
walsykerem:queue-pending-unbounded-problem
Open

walsykerem wants to merge 2 commits into
kernel:mainfrom
walsykerem:queue-pending-unbounded-problem

Conversation

@walsykerem

@walsykerem walsykerem commented Nov 21, 2026 •

Copy link
Copy Markdown

The in-memory job queue behind image pulls and pushes had no limit on waiting jobs. A caller could hand it 20,000 distinct keys; each one appended a pending entry, and duplicate detection walked the whole pending slice from the start. Memory and enqueue time therefore grew with the backlog: 20,001 pending entries, about 1.7 MB of heap, 20.6 s to fill the first batch and 56.3 s for a second batch at that depth, all under the race detector.

The queue now stops accepting work at 512 pending entries and reports a refused job with position -1. Keys live in a map, so duplicate detection stays constant time however deep the queue gets. The pull and push managers check that answer: a refused push drops its record and reservation and returns an error; a refused recovered job is marked failed with an explanation instead of waiting forever. No call blocks, and no job that was accepted is dropped.

After the change: 513 tracked jobs, and 0.36 s and 0.38 s for the two 20,000 call batches.


Note

Medium Risk
Touches core queueing, recovery, and borrowed-credential bookkeeping for image pulls and pushes; behavior changes when backlog hits 512, but refusals are explicit and terminal rather than silent growth.

Overview
Caps the shared in-memory job queue at 512 pending entries: Enqueue now returns -1 with queue.ErrFull, and duplicate detection uses a pendingIndex map instead of scanning the pending slice.

Image pull and push managers handle refusals instead of growing unbounded backlogs. Refused CreatePush calls drop the inflight reservation and persisted record so retries start clean; recovery paths mark jobs failed with an explanatory error when the queue cannot accept them. Image creates that are rejected before enqueue restore the previous inflight pull (so an already-running build keeps borrowed credentials and expiry), fail the new pending record, and return ErrFull without blocking other operations.

Adds tests for the pending limit, full-queue create/recovery behavior, and successor refusal restoring prior inflight state.

Reviewed by Cursor Bugbot for commit a93ff27. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 18af2f9. Configure here.

Comment thread lib/images/manager.go Resolved
Comment thread lib/imagepush/manager.go Resolved
Comment thread lib/images/manager.go Resolved
A full pending queue used to leave the image and push managers in a bad
state. Refusing a pull deadlocked the image manager, a refused recovery
leaked a push reservation, and a refused successor left the running pull
without a handle to build with.

- images: split releaseInflightPull and updateStatusByDigest into
  ...Locked bodies plus locking wrappers, and use the locked variants in
  the rejection path where createMu is already held. The record still
  ends failed with the queue error, credentials and timer cleared.
- images: registerInflightPull now returns the handle it replaced, and a
  refused successor restores it with its credential expiry re-armed, so
  the pull already running keeps going and no expired handle stays
  current.
- imagepush: a recovery refused by the full queue drops its inflight
  entry before failing the record, so a later push for the same digest
  and target can run and garbage collection stops treating it as live.

Tests fill the queue through lib/queue directly and cover the refused
create, the refused successor and the refused recovery.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant