Steps to reproduce
- Create this fleet:
type: fleet
name: test-fleet
nodes: 1
blocks: 2
resources:
cpu: 2
backends: [aws]
- Run this service:
type: service
name: test-service
image: nginx
port: 80
replicas: 2
gateway: false
fleets: [test-fleet]
resources:
cpu: 1..
memory: 1GB..
- Wait until both replicas are running.
- Stop the service.
Actual behaviour
One of the blocks is still identified as busy, although no jobs are running on the instance.
$ dstack event --within-fleet test-fleet
[2026-06-11 13:25:22] [👤admin] [fleet test-fleet] Fleet created. Status: ACTIVE
[2026-06-11 13:25:22] [instance test-fleet-0] Instance created on fleet submission. Status: PENDING
[2026-06-11 13:25:41] [instance test-fleet-0] Instance status changed PENDING -> PROVISIONING
[2026-06-11 13:26:12] [instance test-fleet-0] Instance status changed PROVISIONING -> IDLE
[2026-06-11 13:28:30] [instance test-fleet-0] Instance status changed IDLE -> BUSY
[2026-06-11 13:28:30] [job test-service-0-0, instance test-fleet-0] Job assigned to instance. Instance blocks: 1/2 busy
[2026-06-11 13:28:44] [job test-service-0-1, instance test-fleet-0] Job assigned to instance. Instance blocks: 2/2 busy
[2026-06-11 13:31:26] [job test-service-0-0, instance test-fleet-0] Job unassigned from instance. Instance blocks: 1/2 busy
[2026-06-11 13:31:26] [job test-service-0-1, instance test-fleet-0] Job unassigned from instance. Instance blocks: 1/2 busy
Expected behaviour
All blocks are idle.
dstack version
0.20.23
Server logs
[13:31:12] DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item a53688cc-6807-45d7-bb11-af8757805f15
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 53edc4c0-3f15-48af-846f-65666db52a4f
DEBUG dstack._internal.server.services.jobs:355 job(a53688)test-service-0-0: stopping runner
DEBUG dstack._internal.server.services.jobs:355 job(53edc4)test-service-0-1: stopping runner
[13:31:13] ERROR dstack._internal.server.background.pipeline_tasks.jobs_terminating:525 Failed to update related instance 5a674418-03f3-41a3-a400-962d8a3bd845 for
terminating job 53edc4c0-3f15-48af-846f-65666db52a4f.
INFO dstack._internal.server.services.events:205 Emitting event: Graceful job stop requested. Event targets: job(53edc4)test-service-0-1. Actor: system
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 53edc4c0-3f15-48af-846f-65666db52a4f in 0.982
INFO dstack._internal.server.services.events:205 Emitting event: Graceful job stop requested. Event targets: job(a53688)test-service-0-0. Actor: system
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item a53688cc-6807-45d7-bb11-af8757805f15 in 0.999
[13:31:16] DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4 in 0.028
[13:31:24] DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4 in 0.027
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item 53edc4c0-3f15-48af-846f-65666db52a4f
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing jobs item a53688cc-6807-45d7-bb11-af8757805f15
DEBUG dstack._internal.server.background.pipeline_tasks.jobs_terminating:663 job(53edc4)test-service-0-1: stopping container
DEBUG dstack._internal.server.background.pipeline_tasks.jobs_terminating:663 job(a53688)test-service-0-0: stopping container
[13:31:26] INFO dstack._internal.server.services.events:205 Emitting event: Job status changed TERMINATING -> TERMINATED. Event targets: job(a53688)test-service-0-0.
Actor: system
INFO dstack._internal.server.services.events:205 Emitting event: Job unassigned from instance. Instance blocks: 1/2 busy. Event targets:
job(a53688)test-service-0-0, instance(5a6744)test-fleet-0. Actor: system
INFO dstack._internal.server.services.events:205 Emitting event: Service replica unregistered from receiving requests. Event targets:
job(a53688)test-service-0-0. Actor: system
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item a53688cc-6807-45d7-bb11-af8757805f15 in 1.654
ERROR dstack._internal.server.background.pipeline_tasks.jobs_terminating:525 Failed to update related instance 5a674418-03f3-41a3-a400-962d8a3bd845 for
terminating job 53edc4c0-3f15-48af-846f-65666db52a4f.
INFO dstack._internal.server.services.events:205 Emitting event: Job status changed TERMINATING -> TERMINATED. Event targets: job(53edc4)test-service-0-1.
Actor: system
INFO dstack._internal.server.services.events:205 Emitting event: Job unassigned from instance. Instance blocks: 1/2 busy. Event targets:
job(53edc4)test-service-0-1, instance(5a6744)test-fleet-0. Actor: system
INFO dstack._internal.server.services.events:205 Emitting event: Service replica unregistered from receiving requests. Event targets:
job(53edc4)test-service-0-1. Actor: system
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed jobs item 53edc4c0-3f15-48af-846f-65666db52a4f in 1.687
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing fleets item d90f5d4a-5594-4373-8012-7340605bdc4a
DEBUG dstack._internal.server.background.pipeline_tasks.base:364 Processed fleets item d90f5d4a-5594-4373-8012-7340605bdc4a in 0.051
DEBUG dstack._internal.server.background.pipeline_tasks.base:357 Processing runs item 9e74d3bc-076a-4a63-aeb6-c52aee9b51e4
INFO dstack._internal.server.services.events:205 Emitting event: Run status changed TERMINATING -> TERMINATED. Event targets: run(9e74d3)test-service. Actor:
system
Additional information
No response
Steps to reproduce
Actual behaviour
One of the blocks is still identified as busy, although no jobs are running on the instance.
Expected behaviour
All blocks are idle.
dstack version
0.20.23
Server logs
Additional information
No response