Skip to content

[Bug]: Blocks not released when jobs stop concurrently #3956

@jvstme

Description

@jvstme

Steps to reproduce

  1. Create this fleet:
    type: fleet
    name: test-fleet
    nodes: 1
    blocks: 2
    resources:
      cpu: 2
    backends: [aws]
  2. Run this service:
    type: service
    name: test-service
    image: nginx
    port: 80
    replicas: 2
    gateway: false
    fleets: [test-fleet]
    resources:
      cpu: 1..
      memory: 1GB..
  3. Wait until both replicas are running.
  4. Stop the service.

Actual behaviour

One of the blocks is still identified as busy, although no jobs are running on the instance.

$ dstack event --within-fleet test-fleet
[2026-06-11 13:25:22] [👤admin] [fleet test-fleet] Fleet created. Status: ACTIVE
[2026-06-11 13:25:22] [instance test-fleet-0] Instance created on fleet submission. Status: PENDING
[2026-06-11 13:25:41] [instance test-fleet-0] Instance status changed PENDING -> PROVISIONING
[2026-06-11 13:26:12] [instance test-fleet-0] Instance status changed PROVISIONING -> IDLE
[2026-06-11 13:28:30] [instance test-fleet-0] Instance status changed IDLE -> BUSY
[2026-06-11 13:28:30] [job test-service-0-0, instance test-fleet-0] Job assigned to instance. Instance blocks: 1/2 busy
[2026-06-11 13:28:44] [job test-service-0-1, instance test-fleet-0] Job assigned to instance. Instance blocks: 2/2 busy
[2026-06-11 13:31:26] [job test-service-0-0, instance test-fleet-0] Job unassigned from instance. Instance blocks: 1/2 busy
[2026-06-11 13:31:26] [job test-service-0-1, instance test-fleet-0] Job unassigned from instance. Instance blocks: 1/2 busy

Expected behaviour

All blocks are idle.

dstack version

0.20.23

Server logs

[13:31:12] DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item a53688cc-6807-45d7-bb11-af8757805f15
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 53edc4c0-3f15-48af-846f-65666db52a4f
           DEBUG    dstack._internal.server.services.jobs:355 job(a53688)test-service-0-0: stopping runner
           DEBUG    dstack._internal.server.services.jobs:355 job(53edc4)test-service-0-1: stopping runner
[13:31:13] ERROR    dstack._internal.server.background.pipeline_tasks.jobs_terminating:525 Failed to update related instance 5a674418-03f3-41a3-a400-962d8a3bd845 for
                    terminating job 53edc4c0-3f15-48af-846f-65666db52a4f.
           INFO     dstack._internal.server.services.events:205 Emitting event: Graceful job stop requested. Event targets: job(53edc4)test-service-0-1. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 53edc4c0-3f15-48af-846f-65666db52a4f in 0.982
           INFO     dstack._internal.server.services.events:205 Emitting event: Graceful job stop requested. Event targets: job(a53688)test-service-0-0. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item a53688cc-6807-45d7-bb11-af8757805f15 in 0.999
[13:31:16] DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4 in 0.028
[13:31:24] DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4 in 0.027
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 53edc4c0-3f15-48af-846f-65666db52a4f
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item a53688cc-6807-45d7-bb11-af8757805f15
           DEBUG    dstack._internal.server.background.pipeline_tasks.jobs_terminating:663 job(53edc4)test-service-0-1: stopping container
           DEBUG    dstack._internal.server.background.pipeline_tasks.jobs_terminating:663 job(a53688)test-service-0-0: stopping container
[13:31:26] INFO     dstack._internal.server.services.events:205 Emitting event: Job status changed TERMINATING -> TERMINATED. Event targets: job(a53688)test-service-0-0.
                    Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Job unassigned from instance. Instance blocks: 1/2 busy. Event targets:
                    job(a53688)test-service-0-0, instance(5a6744)test-fleet-0. Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Service replica unregistered from receiving requests. Event targets:
                    job(a53688)test-service-0-0. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item a53688cc-6807-45d7-bb11-af8757805f15 in 1.654
           ERROR    dstack._internal.server.background.pipeline_tasks.jobs_terminating:525 Failed to update related instance 5a674418-03f3-41a3-a400-962d8a3bd845 for
                    terminating job 53edc4c0-3f15-48af-846f-65666db52a4f.
           INFO     dstack._internal.server.services.events:205 Emitting event: Job status changed TERMINATING -> TERMINATED. Event targets: job(53edc4)test-service-0-1.
                    Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Job unassigned from instance. Instance blocks: 1/2 busy. Event targets:
                    job(53edc4)test-service-0-1, instance(5a6744)test-fleet-0. Actor: system
           INFO     dstack._internal.server.services.events:205 Emitting event: Service replica unregistered from receiving requests. Event targets:
                    job(53edc4)test-service-0-1. Actor: system
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 53edc4c0-3f15-48af-846f-65666db52a4f in 1.687
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing fleets item d90f5d4a-5594-4373-8012-7340605bdc4a
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:364 Processed fleets item d90f5d4a-5594-4373-8012-7340605bdc4a in 0.051
           DEBUG    dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
           INFO     dstack._internal.server.services.events:205 Emitting event: Run status changed TERMINATING -> TERMINATED. Event targets: run(9e74d3)test-service. Actor:
                    system

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions