Skip to content

[BUG] TSan data race: trigger topic subscription callback races shutdown teardown #548

Description

@bburda

Bug report

Steps to reproduce

  1. Build with -DSANITIZER=tsan (RelWithDebInfo) and run the integration tests.
  2. test_triggers_data intermittently fails: the gateway process exits with code 66 (TSan detected a data race). It is nondeterministic and has been seen on main as well.

Expected behavior

No data race; the gateway shuts down cleanly.

Actual behavior

TSan reports a data race during shutdown. A trigger topic subscription callback runs on a main-executor thread (deserializing a message via JsonSerializer::deserialize -> TypeCache::parse_type_string) while the same subscription is being torn down on the main thread, destroying the callback lambda's captured strings and unloading type-support libraries.

Two racing stacks (abridged):

Write (main thread): ~lambda  trigger_topic_subscriber.cpp:92   (destroy captured strings)
Read  (executor):    trigger_topic_subscriber.cpp:111 -> JsonSerializer::deserialize -> TypeCache::parse_type_string

Root cause: TriggerTopicSubscriber assumes rclcpp::GenericSubscription destruction drains in-flight callbacks (see the comments in unsubscribe() and shutdown()). That assumption is false: destroying a GenericSubscription does not wait for a callback already running on an executor thread. Trigger subscriptions run on the shared main MultiThreadedExecutor, so at shutdown a callback can still be running when the subscription (and the subscriber's serializer and captured strings) is destroyed.

Environment

  • ros2_medkit version: current main
  • ROS 2 distro: Jazzy (TSan sanitizer job)
  • OS: Ubuntu 24.04

Additional information

Impact: shutdown-only race. The process is already exiting, so operational impact is low, but TSan fails the sanitizer job whenever it triggers.

Proposed fix: route trigger subscriptions through the isolated subscription executor (the same one /data uses), so their callbacks dispatch on a dedicated worker thread that is drained and joined before teardown, instead of the shared main executor. Two things to get right:

  • Do not call the executor's synchronous run-on-worker helper while holding the subscriber mutex that a callback also takes, or shutdown can deadlock.
  • Order teardown so trigger subscriptions are released before the subscription executor is reset.

A narrow TSan suppression for this race is in tsan_suppressions.txt (referencing this issue) so it does not block unrelated work. Remove it when the fix lands.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions