fix(worker): Fix issues with gracefully shutting down

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
    participant Worker
    participant JobQueue
    participant DB as Prisma/Redis
    participant Handler

    rect rgb(200, 220, 255)
    note over Worker: Job Lifecycle with Graceful Timeout
    end

    Worker->>Worker: runJob called
    Worker->>DB: Fetch current job status
    DB-->>Worker: Job status
    
    alt Status is PENDING or IN_PROGRESS
        Worker->>Worker: Execute job
        alt Normal Completion
            Worker->>Handler: Job success
        else Graceful Timeout
            Handler->>Worker: onJobGracefulTimeout
            Worker->>DB: Mark job as FAILED
            Worker->>Handler: Update metrics & log
        end
    else Status is other
        Worker->>Handler: Throw error (skip invalid state)
    end

Loading

sequenceDiagram
    participant System
    participant Index as index.ts
    participant Managers as ConnectionManager<br/>RepoIndexManager
    participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
    participant Signals

    rect rgb(255, 220, 200)
    note over System: Graceful Shutdown Flow
    end

    Signals->>Index: SIGTERM/SIGINT/etc.
    Index->>Index: listenToShutdownSignals
    
    Index->>Managers: dispose() each manager
    
    par Manager Cleanup
        Managers->>Managers: Worker.gracefulStop(timeout)
        Managers->>Services: Delete Redis lock keys
        Managers->>Managers: Log lock release
    end
    
    Index->>Services: Disconnect each service
    Services-->>Index: Disconnected
    
    Index->>Index: Exit gracefully
    Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts