Skip to content

fix: recover multiscan locks with missing or malformed owner metadata - #170

Open
alarcritty wants to merge 1 commit into
openai:mainfrom
alarcritty:fix/recover-incomplete-multiscan-lock
Open

fix: recover multiscan locks with missing or malformed owner metadata#170
alarcritty wants to merge 1 commit into
openai:mainfrom
alarcritty:fix/recover-incomplete-multiscan-lock

Conversation

@alarcritty

Copy link
Copy Markdown
Contributor

Summary

Fixes #102.

Bulk scan creates its lock in two steps: create <output>/.lock, then write <output>/.lock/owner.json. A process kill, machine restart, or write failure between those steps leaves a lock that every later run tries to read unconditionally, so acquisition fails forever with:

codex-security: ENOENT: no such file or directory, open '.../results/.lock/owner.json'

Truncated or malformed owner.json contents fail the same way through JSON.parse. The only escape was deleting .lock by hand. The existing recovery path handled a valid owner file with a dead PID, but not missing or invalid owner metadata.

Changes

acquireLock in src/multiscan.ts now routes contention through a recoverStaleLock helper that mirrors the credential-home lock recovery already shipped in src/runtime.ts (recoverStaleCredentialHomeLock):

  • Missing owner.json, malformed JSON, or owner metadata without a numeric pid no longer crash acquisition. The lock is treated as incomplete and reclaimed through the existing rename-aside path.
  • A lock whose directory was modified within the last 30 seconds is still reported as an active supervisor when its owner metadata is absent or invalid. This protects a concurrent supervisor that has created the directory but not yet published owner.json, a window the previous code left unguarded.
  • EPERM from the process.kill(pid, 0) liveness probe now reports an active supervisor instead of crashing, matching the runtime lock. ESRCH still means the owner is dead and the lock is reclaimed.
  • A lock that disappears mid-recovery (ENOENT on rename) retries acquisition instead of failing, and stale quarantine directories are removed with force: true so concurrent recovery attempts do not race each other into an error.
  • The recursive retry became an acquisition loop; behavior for the existing cases (live owner blocks, dead owner reclaims) is unchanged.

The owner file format is unchanged, so locks written by older versions are still recognized.

Relationship to #98

#98 proposes a broader redesign of this lock (atomic hard-link publication with ownership tokens). This change instead applies the recovery pattern the codebase already uses for the credential-home lock, keeping the directory-based format and the existing acquisition semantics. If maintainers prefer the redesign, this can serve as the minimal interim fix or be superseded.

Tests

Two new tests in tests-ts/multiscan.test.ts:

  • recovers locks with missing or malformed owner metadata: iterates the three shapes from the issue (empty .lock directory, truncated '{"pid":', and '{"pid":"invalid"}'), ages the lock past the publication grace window, and asserts each run completes, removes the lock, and leaves no .lock.stale- residue.
  • treats a fresh lock without owner metadata as an active supervisor: a just-created empty .lock is reported as an active supervisor instead of crashing with ENOENT.

Both tests fail against the previous implementation with the exact ENOENT error from the issue, and pass with this change. The existing exclusive-lock and dead-owner recovery tests continue to pass.

Verification:

  • pnpm run types
  • pnpm run test (708 pass, 0 fail)
  • pnpm run format
  • pnpm run build

Bulk scan creates its lock in two steps: create the .lock directory,
then write owner.json inside it. A crash between those steps left a
lock that every later run tried to read unconditionally, so recovery
failed forever with ENOENT until the directory was deleted by hand.
Truncated or structurally invalid owner files failed the same way.

Treat missing, malformed, or invalid owner metadata as a stale lock
and reclaim it through the existing rename-aside path, mirroring the
credential-home lock recovery in runtime.ts. A lock modified within
the last 30 seconds is still treated as an active supervisor so a
concurrent process that is mid-publication is not stolen from, and
EPERM from the liveness probe now reports an active supervisor
instead of crashing.

Fixes openai#102
@github-actions github-actions Bot added the bug Something isn't working label Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incomplete .lock directory permanently blocks bulk-scan recovery

1 participant