MultiThreadedExecutor can permanently drop a MutuallyExclusive callback group from the wait set when the rebuild trigger coincides with an rmw_wait timeout (rmw_fastrtps)

[atsushi421]

Generated by Generative AI

The investigation and this report were done with the help of Claude Code (Claude Fable 5). Every code reference below was checked by hand against the linked sources.

Operating System:

Linux 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
(host; the tests ran inside an Ubuntu 24.04.1 container using the ros-jazzy binary packages)

ROS version or commit hash:

jazzy (rclcpp 28.1.18, rcl 9.2.9). The same code is present on rolling, see the permalinks below.

RMW implementation (if applicable):

rmw_fastrtps_cpp (rmw_fastrtps_shared_cpp 8.4.3)

RMW Configuration (if applicable):

Default, no XML profile.

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

Not collected (the container only has the packages needed to build and run the tests).

Steps to reproduce issue

The failure is a race, so it needs a setup where a thread can be preempted for a while at an unlucky moment (a hosted CI runner with a few vCPUs, or a local perturbation as described below).

1. Create a node with a wall timer in the node's default (MutuallyExclusive) callback group whose callback takes a few milliseconds. Do not create any other MutuallyExclusive callback group in the executor (put subscriptions in a Reentrant group, or have none).
2. Spin it with rclcpp::executors::MultiThreadedExecutor (2 or more threads) constructed with a finite next_exec_timeout (we used 25 to 400 ms). See "Preconditions" below for why the default -1 does not hit this.
3. Run the process repeatedly on a 4 CPU cpuset while a bursty SCHED_FIFO hog steals one random CPU for 30 to 300 ms at a time (timeout 0.<n>s taskset -c <cpu> chrt -f 50 yes >/dev/null, then sleep 100 to 500 ms, in a loop). This mimics vCPU steal on hosted runners. Without the hog we could not reproduce it in 6,000+ runs; with it, 3 out of 800 runs failed.

Expected behavior

A timer (or any entity) in a MutuallyExclusive callback group keeps being serviced by MultiThreadedExecutor for as long as the node lives.

Actual behavior

Occasionally the timer fires a few times and then never again for the rest of the process lifetime. The other entities of the same callback group stop being serviced as well.

Mechanism (thread A is inside wait_for_work(), thread B executes a callback of the MutuallyExclusive group G):

1. build_entities_collection() skips callback groups whose can_be_taken_from is false (rolling:

   rclcpp/rclcpp/src/rclcpp/executors/executor_entities_collection.cpp

   Line 79 in 90a1e56

   if (group_ptr->can_be_taken_from().load()) {

   , jazzy:

   rclcpp/rclcpp/src/rclcpp/executors/executor_entities_collection.cpp

   Line 79 in e4c23a5

   if (group_ptr->can_be_taken_from().load()) {

   ). While thread B executes a callback of G, can_be_taken_from of G is false, so a collect_entities() that thread A runs at that moment builds a wait set without any entity of G (for the node's default group that is every timer of the node).
2. Those entities return only with the next rebuild. MultiThreadedExecutor::run() requests it by calling interrupt_guard_condition_->trigger() after each MutuallyExclusive callback (rolling:

   rclcpp/rclcpp/src/rclcpp/executors/multi_threaded_executor.cpp

   Lines 105 to 117 in 90a1e56

   	execute_any_executable(any_exec);
   	if (any_exec.callback_group &&
   	any_exec.callback_group->type() == CallbackGroupType::MutuallyExclusive)
   	{
   	try {
   	interrupt_guard_condition_->trigger();
   	} catch (const rclcpp::exceptions::RCLError & ex) {
   	throw std::runtime_error(
   	std::string(
   	"Failed to trigger guard condition on callback group change: ") + ex.what());
   	}
   	}

   , jazzy:

   rclcpp/rclcpp/src/rclcpp/executors/multi_threaded_executor.cpp

   Lines 100 to 112 in e4c23a5

   	execute_any_executable(any_exec);
   	if (any_exec.callback_group &&
   	any_exec.callback_group->type() == CallbackGroupType::MutuallyExclusive)
   	{
   	try {
   	interrupt_guard_condition_->trigger();
   	} catch (const rclcpp::exceptions::RCLError & ex) {
   	throw std::runtime_error(
   	std::string(
   	"Failed to trigger guard condition on callback group change: ") + ex.what());
   	}
   	}

   ). The trigger sets entities_need_rebuild_ only indirectly: it has to be reported as ready by rmw_wait() and then executed through ExecutorNotifyWaitable.
3. rmw_wait() in rmw_fastrtps_shared_cpp calls set_trigger_value(false) on every guard condition after the DDS wait returns, even when it returns RMW_RET_TIMEOUT (rolling: https://github.com/ros2/rmw_fastrtps/blob/e5cc0e6422b5311691228953cea9ace87fc109e3/rmw_fastrtps_shared_cpp/src/rmw_wait.cpp#L328-L339, jazzy: https://github.com/ros2/rmw_fastrtps/blob/54d2240637f18e3ec710bb73c51c79abaf91ece4/rmw_fastrtps_shared_cpp/src/rmw_wait.cpp#L306-L317). A guard condition that was triggered keeps its slot in guard_conditions->guard_conditions[] (it is reported as ready), but the return value stays RMW_RET_TIMEOUT because wait_result is false.
4. rcl_wait() turns RMW_RET_TIMEOUT into RCL_RET_TIMEOUT unless a timer is ready (rolling: https://github.com/ros2/rcl/blob/db3d1c8ea6eb22235494cfc03249155e35c47511/rcl/src/rcl/wait.c#L777-L779, jazzy: https://github.com/ros2/rcl/blob/22c0b957140035c675f0a061daaab5fd6b2080e7/rcl/src/rcl/wait.c#L744-L746), and rclcpp::WaitSet::wait() maps that to WaitResultKind::Timeout.
5. Executor::wait_for_work() evaluates the notify waitable only when the result kind is Ready (rolling:

   rclcpp/rclcpp/src/rclcpp/executor.cpp

   Lines 804 to 809 in 90a1e56

   	if (this->wait_result_->kind() == WaitResultKind::Ready && current_notify_waitable_) {
   	auto & rcl_wait_set = this->wait_result_->get_wait_set().get_rcl_wait_set();
   	if (current_notify_waitable_->is_ready(rcl_wait_set)) {
   	current_notify_waitable_->execute(current_notify_waitable_->take_data());
   	}
   	}

   , jazzy:

   rclcpp/rclcpp/src/rclcpp/executor.cpp

   Lines 781 to 786 in e4c23a5

   	if (this->wait_result_->kind() == WaitResultKind::Ready && current_notify_waitable_) {
   	auto & rcl_wait_set = this->wait_result_->get_wait_set().get_rcl_wait_set();
   	if (current_notify_waitable_->is_ready(rcl_wait_set)) {
   	current_notify_waitable_->execute(current_notify_waitable_->take_data());
   	}
   	}

   ). On Timeout it never calls is_ready(), so the ready slot left by rcl_wait() is ignored and entities_need_rebuild_ stays false.

So if thread B's trigger() lands in the window between the DDS wait of thread A returning with a timeout and the guard condition loop at the end of rmw_wait(), the trigger value is cleared, the result is Timeout, and nothing records that a rebuild was requested. The next rmw_wait() does not see it either (has_triggered_condition() reads the already cleared value). The window is normally microseconds, but preemption of thread A inside it (vCPU steal, an RT task) stretches it to milliseconds. The state is also self reinforcing: once the timers of G are out of the wait set, rcl_wait() no longer shortens the timeout for them, so every subsequent wait ends in a genuine timeout, which is exactly the path on which the trigger can be lost.

Preconditions:

• A finite next_exec_timeout. With the default -1, rcl_wait() derives its timeout only from the timers in the wait set, so whenever rmw_wait() returns RMW_RET_TIMEOUT a timer is due, rcl_wait() returns RCL_RET_OK, the result is Ready, and the trigger is picked up from the ready slot. Once the timers of G are gone the wait blocks indefinitely and the trigger wakes it. WaitResultKind::Timeout therefore does not occur in normal operation with -1 (remaining corner cases: a timer cancelled or reset by another thread during the wait, ROS time jumps). With a finite timeout, Timeout results are routine.
• No other MutuallyExclusive callback group in the executor. Otherwise its callbacks keep producing triggers, one of them eventually causes a rebuild, and G recovers after a delay. With all other groups Reentrant (or with a single group), the loss is permanent.

Humble is not affected because its executor rebuilds the wait set on every wait_for_work().

Additional information

• The behavior appeared with the wait set based executor rewrite (Utilize rclcpp::WaitSet as part of the executors #2142), which introduced both the incremental rebuild and the guard condition trigger after MutuallyExclusive callbacks.
• Setting entities_need_rebuild_ explicitly in MultiThreadedExecutor::run() fixes it; the trigger is still needed to wake the waiting thread:

  if (any_exec.callback_group &&
    any_exec.callback_group->type() == CallbackGroupType::MutuallyExclusive)
  {
    entities_need_rebuild_.store(true);
    interrupt_guard_condition_->trigger();
  }

  With this change the failure did not reappear in 2,500+ runs under the same perturbation (0 failures, versus 3 out of 800 before). Alternatives would be to have wait_for_work() check current_notify_waitable_->is_ready() regardless of the result kind (rcl already leaves the ready slot in the wait set), or to have rmw_fastrtps not clear guard conditions on a timeout, but the first option keeps the rebuild request out of the rmw layer entirely.
• The failure was observed and the fix verified in an executor built on rclcpp::Executor whose spin loop is the same as MultiThreadedExecutor::run(), including the trigger after MutuallyExclusive callbacks (fix(agnocastlib): request a wait set rebuild explicitly after MutuallyExclusive callbacks on Jazzy autowarefoundation/agnocast#1591). I have not yet written a standalone rclcpp-only reproducer; the mechanism above involves only rclcpp::Executor, MultiThreadedExecutor::run(), rcl_wait() and rmw_fastrtps.
• Only rmw_fastrtps was checked. Whether other RMW implementations clear guard conditions on a timeout was not verified.
• Related: Starvation in MultiThreadedExecutor #2645 and Multithreaded executor behavior #2360 describe starvation in MultiThreadedExecutor; this issue is a different failure (entities permanently missing from the wait set, not delayed), but PR Multi-threaded Executor starvation fix #2702 touches the same trigger.