diff --git a/awkernel_async_lib/src/scheduler/edf.rs b/awkernel_async_lib/src/scheduler/edf.rs index 065a880e8..6539edc35 100644 --- a/awkernel_async_lib/src/scheduler/edf.rs +++ b/awkernel_async_lib/src/scheduler/edf.rs @@ -169,7 +169,7 @@ impl EDFScheduler { let cpu_set = task.cpu_set.expect("Task has no CPU set"); // Find the CPU whose target (running or pending-preemption) task has - // the lowest priority among the set. + // the lowest priority among the busy CPUs of the set. let mut victim: Option<(usize, Arc)> = None; for cpu_id in cpu_set.iter() { @@ -180,19 +180,27 @@ impl EDFScheduler { return false; } - // An idle CPU in the set will pick the task up; no preemption needed. + // An idle CPU does not stop the preemption. A task queued before + // or after this one can take the idle CPU first, and a sleeping + // CPU runs a queued task only after the primary CPU wakes it. So + // this task can preempt the lowest-priority busy CPU even if the + // set has an idle CPU. The cost is that the preemption can be + // unneeded, and the victim waits for the primary CPU to wake a + // sleeping CPU. If the set of the victim holds no idle CPU, the + // victim waits for its own CPU while the idle CPU stays idle. if task_running.task_id == 0 { - return false; + continue; } - let task_running = get_task(task_running.task_id); - let task_pending = peek_preemption_pending(cpu_id); + // A running task that is missing from the task table finished + // after the read above, so this CPU counts as idle too. + let Some(task_running) = get_task(task_running.task_id) else { + continue; + }; - let target_task = match (task_running, task_pending) { - (Some(running), Some(pending)) => running.max(pending), - (Some(running), None) => running, - (None, Some(pending)) => pending, - (None, None) => return false, + let target_task = match peek_preemption_pending(cpu_id) { + Some(pending) => task_running.max(pending), + None => task_running, }; match victim { @@ -215,13 +223,14 @@ impl EDFScheduler { // heap and is not in the affinity queue, so it is invisible to the // other CPUs in its `cpu_set` (neither `pop_for_cpu`, the clustered // task counter, nor `queued_cpu_mask` sees it). If another CPU in - // the set becomes idle while the IPI is in flight, it cannot pick + // the set is or becomes idle while the IPI is in flight, it cannot pick // the task up and may go to sleep — a latency window bounded by the // IPI delivery plus the victim's interrupt-disabled sections. This // is not a lost wakeup: the task is eventually run either by the // victim's preemption handler (`do_preemption`), or re-woken by the // victim's `run_main` if its running task finishes first, in which - // case the idle check above routes it to the queue. Future work: + // case this function decides again. `run_main` checks the pending + // heap again before the victim sleeps. Future work: // an enqueue-instead-of-pin strategy (push to the affinity queue // and treat the IPI as a hint) would close this window, but it // requires redesigning the duplicate-IPI suppression that the diff --git a/awkernel_async_lib/src/task.rs b/awkernel_async_lib/src/task.rs index 346c0c890..eee6a3a3f 100644 --- a/awkernel_async_lib/src/task.rs +++ b/awkernel_async_lib/src/task.rs @@ -1411,6 +1411,16 @@ pub fn run_main() { } } } else { + // A waker that read this CPU as busy can pin a task here after the + // re-wake at the top of the loop. While this CPU runs no task, the + // preemption IPI can leave that task pinned, so re-wake it instead + // of sleeping. `get_next_task` took `GLOBAL_WAKE_GET_MUTEX`, so this + // check sees the pin of every such waker. + if scheduler::PREEMPTION_ENABLED && scheduler::peek_preemption_pending(cpu_id).is_some() + { + continue; + } + #[cfg(feature = "perf")] perf::start_idle(); diff --git a/mdbook/src/internal/scheduler.md b/mdbook/src/internal/scheduler.md index c15853e4e..9ce66e97d 100644 --- a/mdbook/src/internal/scheduler.md +++ b/mdbook/src/internal/scheduler.md @@ -75,7 +75,9 @@ The Earliest Deadline First (EDF) scheduler is implemented in [edf.rs](https://g The scheduler holds a single affinity-aware priority queue, [`AffinityBTreeQueue`](https://crates.io/crates/affinity_btree_queue), backed by an augmented B-tree. Each entry carries `(priority, affinity, task)`, where the priority is the pair `(absolute_deadline, wake_time)`; smaller values dequeue first, so tasks are ordered by earliest deadline with wake time as a tie-breaker. Every B-tree node stores the OR of the affinities in its subtree, which lets `pop_for_cpu(cpu)` find the earliest-deadline task runnable on `cpu` in logarithmic time while skipping subtrees with no eligible entry. -When a task is enqueued via `wake_task()`, the scheduler reads the `SchedulerType::EDF(relative_deadline, cpu_set)` attached to the task and calculates the absolute deadline as `uptime + relative_deadline`. If the task is part of a DAG, `calculate_and_update_dag_deadline()` is used instead to propagate deadlines through the DAG. The task priority (`PriorityInfo`) becomes `MAX_TASK_PRIORITY - absolute_deadline`, so an earlier deadline gives a higher priority. The task is then pushed into the queue with its `cpu_set` as the affinity mask. Preemption is handled via `invoke_preemption()`: if no core in the set is idle and the task is not already running, the core running the lowest-priority task among the set is chosen, and an IPI is sent to it when the newly enqueued task has an earlier deadline than the task currently running (or pending preemption) on that core. +When a task is enqueued via `wake_task()`, the scheduler reads the `SchedulerType::EDF(relative_deadline, cpu_set)` attached to the task and calculates the absolute deadline as `uptime + relative_deadline`. If the task is part of a DAG, `calculate_and_update_dag_deadline()` is used instead to propagate deadlines through the DAG. The task priority (`PriorityInfo`) becomes `MAX_TASK_PRIORITY - absolute_deadline`, so an earlier deadline gives a higher priority. `invoke_preemption()` handles preemption. If the task is already running, the scheduler does not preempt. Otherwise, the scheduler chooses the core running the lowest-priority task among the busy cores of the set. If the new task has a higher priority than the task running (or pending preemption) on that core, the scheduler sends an IPI to it. In that case, the scheduler pins the new task to the preemption-pending heap of that core and does not push it into the queue. If the scheduler sends no IPI, it pushes the task into the queue with its `cpu_set` as the affinity mask. One such case is a set with no busy core. Against another EDF task, a higher priority means an earlier deadline. An EDF task preempts a task of a lower-priority scheduler whatever its deadline. + +An idle core in the set does not stop the preemption. A task queued before or after the new task can take the idle core first. Also, a sleeping core runs a queued task only after the primary CPU wakes it. The cost is that the preemption can be unneeded, and the preempted task waits for the primary CPU to wake a sleeping core. If the set of the preempted task holds no idle core, the preempted task waits for its own core while the idle core stays idle. `get_next()` pops the earliest-deadline task whose `cpu_set` contains the calling CPU (`pop_for_cpu`), so a task is only ever dequeued by a core within its set. This guarantees CPU affinity. CPU 0 (the primary core) is always excluded from `cpu_set` when a task is spawned. The `cpu_set` is normalized at spawn time by removing CPU 0 and out-of-range bits; if this leaves the set empty (for example an empty set, or one naming only CPU 0 or out-of-range cores), the task is not rejected but falls back to all worker cores (`1..num_cpu()`) with a warning.