Skip to content

perf(sched): wake one worker per queued global EDF task - #789

Closed
atsushi421 wants to merge 3 commits into
mainfrom
feat/edf-budget-global-wakes
Closed

atsushi421 wants to merge 3 commits into
mainfrom
feat/edf-budget-global-wakes

Conversation

@atsushi421

@atsushi421 atsushi421 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR wakes one sleeping worker per queued EDF task whose set holds every worker core. It is the third PR of a stack on top of #787.

Before this PR, the EDF scheduler kept every task in one affinity queue, and wake_workers() woke every core in the affinity mask of that queue. One queued task whose set holds every worker core put every worker in the mask, so wake_workers() woke every sleeping worker. The extra workers took GLOBAL_WAKE_GET_MUTEX, found no task, and slept again. GEDF woke one worker per task, so #787 added this cost for every former GEDF task.

Changes:

  • The EDF scheduler keeps two queues. The global queue holds the tasks whose set holds every worker core, and the restricted queue holds the other tasks. get_next() pops the earliest task of either queue that can run on the calling core.
  • The queue key becomes (absolute_deadline, wake_time, seq). seq grows with each push, so tasks with equal deadlines and wake times still dequeue in push order across both queues.
  • Scheduler::queued_cpu_mask() becomes queued_workers(), which returns the mask of the restricted queue and the number of global tasks. wake_workers() wakes every core in the mask and one sleeping worker per global task, as it does for the tasks of the other schedulers.
  • The mdbook describes the two queues.

Related links

How was this PR tested?

  • scripts/ci.sh passes on my machine.
  • New unit tests in scheduler/edf.rs check that pop_for_cpu takes the earliest task of either queue, that equal keys are FIFO across both queues, that a restricted task stays off other cores, and that a global task stays out of the restricted mask.
  • On QEMU (aarch64_virt, 4 CPUs), test_edf logs no [FAIL], every node of test_dag runs, and every task of test_multi_prioritized_scheduler starts and finishes, with no panic.

Notes for reviewers

With temporary counters during 30 seconds of test_dag on QEMU (aarch64_virt, 4 CPUs), wake_workers() woke 13 workers on #792 (3 for the affinity mask and 10 for the budget) and 11 workers with this PR (0 and 11). test_dag rarely queues a global EDF task while workers sleep, so the saving is small there. It grows with the number of sleeping workers and of global EDF tasks that wait in the queue.

I also ran a benchmark that wakes global EDF tasks while workers sleep. Eight EDF(10 ms, CpuSet::all()) tasks run 200 us every 10 ms, with releases spread 1.25 ms apart, for 10 seconds (8000 jobs). The table shows two runs per configuration on QEMU (aarch64_virt).

CPUs Branch Wakes by wake_workers() Average latency (us) Workers that slept without work
4 #792 520, 188 762, 652 71373, 76397
4 this PR 199, 108 712, 646 71078, 75556
8 #792 2201, 2259 1530, 1090 110830, 121856
8 this PR 855, 880 1297, 833 117749, 130802

This PR removes about 60% of the wakes by wake_workers() with 8 CPUs. But on QEMU, these wakes are about 2% of the times that a worker woke and slept again without work. Most of those wakes come from another source, which I did not identify. The latency differences are within the run-to-run noise.

Merge #792 first. This PR targets its branch.

@atsushi421
atsushi421 added this pull request to stack #791 November 13, 2026 05:21
@atsushi421
atsushi421 force-pushed the feat/edf-budget-global-wakes branch from 9f7658e to 183d771 Compare November 13, 2026 05:50
@atsushi421
atsushi421 removed this pull request from stack #791 November 13, 2026 05:51
@atsushi421
atsushi421 changed the base branch from feat/edf-idle-core-preemption to fix/sched-num-task-in-queue November 13, 2026 05:51
@atsushi421
atsushi421 added this pull request to stack #793 November 13, 2026 05:53
Before this change, the EDF scheduler did not preempt if one core in the
set of the woken task was idle. An idle core takes only one task, and it
stays idle until the primary CPU wakes it. If several tasks woke in that
window, all of them were enqueued, and only one of them got the idle
core. The others waited behind running tasks with later deadlines.

Now the scheduler skips preemption only if an idle core takes the task.
An idle core takes the earliest queued task runnable on it. So it takes
the woken task if the task is earlier than every queued task runnable on
that core, or if the queue holds fewer tasks than the set has idle cores.
Otherwise, the scheduler chooses a victim among the busy cores of the set.

Signed-off-by: atsushi421 <atsushi.yano.2@tier4.jp>
NUM_TASK_IN_QUEUE underflowed to u32::MAX. ArcWake::wake incremented
it, and get_next_task() decremented it when a non-clustered scheduler
returned a task. Other paths did not match:

- The re-wakes in do_preemption() and run_main(), and the re-wake of a
  preempted task, call wake_task() without ArcWake::wake, so their
  tasks were decremented but never incremented.
- A task that invoke_preemption() pushed to a preemption-pending heap
  was incremented but runs without get_next_task().
- get_next() skips terminated and panicked tasks without a decrement.

With the counter at u32::MAX, wake_workers() woke every sleeping worker
on every loop of the primary CPU.

Now the PrioritizedFIFO, PrioritizedRR, and Panicked schedulers
increment the counter when they push a task to their queue and
decrement it when they pop one. So the counter equals the number of
queued non-clustered tasks on every path. These schedulers also wake
the primary CPU after they enqueue a task, as the EDF scheduler does,
because the re-wakes do not go through ArcWake::wake. The two
target-specific definitions of the counter become one.

Signed-off-by: atsushi421 <atsushi.yano.2@tier4.jp>
The EDF scheduler kept every task in one affinity queue, and
wake_workers() woke every core in the affinity mask of that queue. One
queued task whose set holds every worker core put every worker in the
mask, so wake_workers() woke every sleeping worker. The extra workers
found no task and slept again.

Now the EDF scheduler keeps two queues. The global queue holds the tasks
whose set holds every worker core, and the restricted queue holds the
other tasks. wake_workers() wakes every core in the mask of the
restricted queue, and one sleeping worker per task of the global queue,
as it does for the tasks of the other schedulers. Scheduler::queued_cpu_mask()
becomes queued_workers(), which also returns the number of global tasks.

The queue key gets a sequence number, so tasks with equal deadlines and
wake times still dequeue in push order across both queues.

Signed-off-by: atsushi421 <atsushi.yano.2@tier4.jp>
@romancardenas
romancardenas force-pushed the feat/edf-budget-global-wakes branch from 183d771 to 302181f Compare November 16, 2026 00:20
@romancardenas
romancardenas force-pushed the fix/sched-num-task-in-queue branch from d417ea8 to 47b6f5d Compare November 16, 2026 05:11
Base automatically changed from fix/sched-num-task-in-queue to main November 17, 2026 05:57
@atsushi421

Copy link
Copy Markdown
Contributor Author

Closing this PR. I measured it again on current main, which includes #788 and #792. On QEMU (aarch64_virt), the change removes about 20% of the wakes by wake_workers() with 4 CPUs and about 43% with 8 CPUs. But the average latency does not change. The removed wakes are about 1.2% of the times that a worker woke and slept again without work. This gain does not justify a second EDF queue.

@atsushi421 atsushi421 closed this Nov 17, 2026
@atsushi421
atsushi421 removed this pull request from stack #793 November 17, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant