Bug report
Steps to reproduce
- Build with
-DSANITIZER=tsan (RelWithDebInfo) and run the integration tests.
test_triggers_data intermittently fails: the gateway process exits with code 66 (TSan detected a data race). It is nondeterministic and has been seen on main as well.
Expected behavior
No data race; the gateway shuts down cleanly.
Actual behavior
TSan reports a data race during shutdown. A trigger topic subscription callback runs on a main-executor thread (deserializing a message via JsonSerializer::deserialize -> TypeCache::parse_type_string) while the same subscription is being torn down on the main thread, destroying the callback lambda's captured strings and unloading type-support libraries.
Two racing stacks (abridged):
Write (main thread): ~lambda trigger_topic_subscriber.cpp:92 (destroy captured strings)
Read (executor): trigger_topic_subscriber.cpp:111 -> JsonSerializer::deserialize -> TypeCache::parse_type_string
Root cause: TriggerTopicSubscriber assumes rclcpp::GenericSubscription destruction drains in-flight callbacks (see the comments in unsubscribe() and shutdown()). That assumption is false: destroying a GenericSubscription does not wait for a callback already running on an executor thread. Trigger subscriptions run on the shared main MultiThreadedExecutor, so at shutdown a callback can still be running when the subscription (and the subscriber's serializer and captured strings) is destroyed.
Environment
- ros2_medkit version: current main
- ROS 2 distro: Jazzy (TSan sanitizer job)
- OS: Ubuntu 24.04
Additional information
Impact: shutdown-only race. The process is already exiting, so operational impact is low, but TSan fails the sanitizer job whenever it triggers.
Proposed fix: route trigger subscriptions through the isolated subscription executor (the same one /data uses), so their callbacks dispatch on a dedicated worker thread that is drained and joined before teardown, instead of the shared main executor. Two things to get right:
- Do not call the executor's synchronous run-on-worker helper while holding the subscriber mutex that a callback also takes, or shutdown can deadlock.
- Order teardown so trigger subscriptions are released before the subscription executor is reset.
A narrow TSan suppression for this race is in tsan_suppressions.txt (referencing this issue) so it does not block unrelated work. Remove it when the fix lands.
Bug report
Steps to reproduce
-DSANITIZER=tsan(RelWithDebInfo) and run the integration tests.test_triggers_dataintermittently fails: the gateway process exits with code 66 (TSan detected a data race). It is nondeterministic and has been seen onmainas well.Expected behavior
No data race; the gateway shuts down cleanly.
Actual behavior
TSan reports a data race during shutdown. A trigger topic subscription callback runs on a main-executor thread (deserializing a message via
JsonSerializer::deserialize->TypeCache::parse_type_string) while the same subscription is being torn down on the main thread, destroying the callback lambda's captured strings and unloading type-support libraries.Two racing stacks (abridged):
Root cause:
TriggerTopicSubscriberassumesrclcpp::GenericSubscriptiondestruction drains in-flight callbacks (see the comments inunsubscribe()andshutdown()). That assumption is false: destroying aGenericSubscriptiondoes not wait for a callback already running on an executor thread. Trigger subscriptions run on the shared mainMultiThreadedExecutor, so at shutdown a callback can still be running when the subscription (and the subscriber's serializer and captured strings) is destroyed.Environment
Additional information
Impact: shutdown-only race. The process is already exiting, so operational impact is low, but TSan fails the sanitizer job whenever it triggers.
Proposed fix: route trigger subscriptions through the isolated subscription executor (the same one
/datauses), so their callbacks dispatch on a dedicated worker thread that is drained and joined before teardown, instead of the shared main executor. Two things to get right:A narrow TSan suppression for this race is in
tsan_suppressions.txt(referencing this issue) so it does not block unrelated work. Remove it when the fix lands.