[feature][bug] Add tcp transport event dispatcher unsched flag & fix RDMA event dispatcher unsched flag

docs/cn/rdma.md

@@ -47,7 +47,24 @@ RDMA要求数据收发所使用的内存空间必须被注册（memory register
 
 RDMA是硬件相关的通信技术，有很多独特的概念，比如device、port、GID、LID、MaxSge等。这些参数在初始化时会从对应的网卡中读取出来，并且做出默认的选择（参见src/brpc/rdma/rdma_helper.cpp）。有时默认的选择并非用户的期望，则可以通过flag参数方式指定。
 
-RDMA支持事件驱动和轮询两种模式，默认是事件驱动模式，通过设置rdma_use_polling可以开启轮询模式。轮询模式下还可以设置轮询器数目（rdma_poller_num），以及是否主动放弃CPU（rdma_poller_yield）。轮询模式下还可以设置一个回调函数，在每次轮询时调用，可以配合io_uring/spdk等使用。在配合使用spdk等驱动的时候，因为spdk只支持轮询模式，并且只能在单线程使用（或者叫Run To Completion模式上使用）执行一个任务过程中不允许被调度到别的线程上，所以这时候需要设置（rdma_edisp_unsched）为true，使事件驱动程序一直占用一个worker线程，不能调度别的任务。
+RDMA支持事件驱动和轮询两种模式，默认是事件驱动模式，通过设置rdma_use_polling可以开启轮询模式。轮询模式下还可以设置轮询器数目（rdma_poller_num），以及是否主动放弃CPU（rdma_poller_yield）。轮询模式下还可以设置一个回调函数，在每次轮询时调用，可以配合io_uring/spdk等使用。
+
+`event_dispatcher_edisp_unsched` 是全局开关，同时影响普通模式（TCP）和RDMA模式的EventDispatcher调度行为。兼容历史配置，`rdma_edisp_unsched` 仍保留，但已标记为废弃，未来版本会移除。
+
+最终生效条件统一为：
+`event_dispatcher_edisp_unsched || rdma_edisp_unsched`
+
+启动时不会再改写用户传入的 flag，运行时严格按用户配置值生效。
+
+推荐使用方式：
+1. 新部署：只配置 `event_dispatcher_edisp_unsched`。
+2. 存量部署：`rdma_edisp_unsched` 仅作过渡兼容，逐步迁移到 `event_dispatcher_edisp_unsched`。
+3. 避免脚本中给出“冲突值”；在统一 OR 语义下，只要任一 flag 为 `true`，EventDispatcher 就不可调度。
+
+行为示例：
+1. 仅设置 `-rdma_edisp_unsched=true`：`rdma_edisp_unsched=true`、`event_dispatcher_edisp_unsched=false`；TCP和RDMA均不可调度。
+2. 仅设置 `-event_dispatcher_edisp_unsched=true`：两个flag同为`true`；TCP和RDMA均不可调度。
+3. 同时设置 `-rdma_edisp_unsched=true -event_dispatcher_edisp_unsched=false`：`rdma_edisp_unsched=true`、`event_dispatcher_edisp_unsched=false`；TCP和RDMA均不可调度。
 
 # 参数
 
@@ -73,5 +90,6 @@ RDMA支持事件驱动和轮询两种模式，默认是事件驱动模式，通
 * rdma_use_polling: 是否使用RDMA的轮询模式，默认false。
 * rdma_poller_num: 轮询模式下的poller数目，默认1。
 * rdma_poller_yield: 轮询模式下的poller是否主动放弃CPU，默认是false。
-* rdma_edisp_unsched: 让事件驱动器不可以被调度，默认是false。
+* event_dispatcher_edisp_unsched: 全局开关，控制EventDispatcher是否不可被调度（true时不可调度），默认是false。
+* rdma_edisp_unsched: 废弃兼容参数（未来版本计划移除）。当前仍参与统一生效判断，默认是false。
 * rdma_disable_bthread: 禁用bthread，默认是false。

docs/en/rdma.md

@@ -47,6 +47,23 @@ The application can manage memory by itself and send data with IOBuf::append_use
 
 RDMA is hardware-related. It has some different concepts such as device, port, GID, LID, MaxSge and so on. These parameters can be read from NICs at initialization, and brpc will make the default choice (see src/brpc/rdma/rdma_helper.cpp). Sometimes the default choice is not the expectation, then it can be changed in the flag way.
 
+`event_dispatcher_edisp_unsched` is a global flag and affects EventDispatcher scheduling in both normal mode (TCP) and RDMA mode. For backward compatibility, `rdma_edisp_unsched` is still kept, but it is deprecated and will be removed in a future release.
+
+The effective unsched condition is unified as:
+`event_dispatcher_edisp_unsched || rdma_edisp_unsched`
+
+No startup synchronization rewrites user flags. Runtime behavior is determined directly from user-provided values.
+
+Recommended usage:
+1. New deployment: set only `event_dispatcher_edisp_unsched`.
+2. Existing deployment: keep `rdma_edisp_unsched` temporarily, but migrate to `event_dispatcher_edisp_unsched`.
+3. Avoid conflicting values in scripts; with unified OR semantics, either flag being `true` makes EventDispatcher unschedulable.
+
+Examples:
+1. Only `-rdma_edisp_unsched=true`: `rdma_edisp_unsched=true`, `event_dispatcher_edisp_unsched=false`; both TCP and RDMA are unschedulable.
+2. Only `-event_dispatcher_edisp_unsched=true`: both flags are `true`; both TCP and RDMA are unschedulable.
+3. Both `-rdma_edisp_unsched=true -event_dispatcher_edisp_unsched=false`: `rdma_edisp_unsched=true`, `event_dispatcher_edisp_unsched=false`; both TCP and RDMA are unschedulable.
+
 # Parameters
 
 Configurable parameters:
@@ -71,5 +88,6 @@ Configurable parameters:
 * rdma_use_polling: Whether to use RDMA polling mode, default is false.
 * rdma_poller_num: The number of pollers in polling mode, default is 1.
 * rdma_poller_yield: Whether pollers in polling mode voluntarily relinquish the CPU, default is false.
-* rdma_edisp_unsched`: Prevents the event driver from being scheduled, default is false.
+* event_dispatcher_edisp_unsched: Global switch for EventDispatcher scheduling (true means unschedulable), default is false.
+* rdma_edisp_unsched: Deprecated compatibility flag (planned removal in a future release). It still participates in unified unsched condition, default is false.
 * rdma_disable_bthread: Disables bthread, default is false.

src/brpc/event_dispatcher.cpp

@@ -30,6 +30,16 @@ DECLARE_int32(task_group_ntags);
 namespace brpc {
 
 DEFINE_int32(event_dispatcher_num, 1, "Number of event dispatcher");
+DEFINE_bool(event_dispatcher_edisp_unsched, false,
+            "Disable event dispatcher schedule");
+
+#if BRPC_WITH_RDMA
+namespace rdma {
+DEFINE_bool(rdma_edisp_unsched, false,
+            "Deprecated and will be removed in a future release, "
+            "use event_dispatcher_edisp_unsched instead");
+}  // namespace rdma
+#endif
 
 DEFINE_bool(usercode_in_pthread, false, 
             "Call user's callback in pthreads, use bthreads otherwise");
@@ -41,6 +51,15 @@ static bvar::LatencyRecorder* g_edisp_read_lantency = NULL;
 static bvar::LatencyRecorder* g_edisp_write_lantency = NULL;
 static pthread_once_t g_edisp_once = PTHREAD_ONCE_INIT;
 
+bool EventDispatcherUnsched() {
+#if BRPC_WITH_RDMA
+    return FLAGS_event_dispatcher_edisp_unsched ||
+           rdma::FLAGS_rdma_edisp_unsched;
+#else
+    return FLAGS_event_dispatcher_edisp_unsched;
+#endif
+}
+
 static void StopAndJoinGlobalDispatchers() {
     for (int i = 0; i < FLAGS_task_group_ntags; ++i) {
         for (int j = 0; j < FLAGS_event_dispatcher_num; ++j) {

src/brpc/event_dispatcher.h

@@ -19,13 +19,16 @@
 #ifndef BRPC_EVENT_DISPATCHER_H
 #define BRPC_EVENT_DISPATCHER_H
 
+#include <gflags/gflags_declare.h>          // DECLARE_bool
 #include "butil/macros.h"                     // DISALLOW_COPY_AND_ASSIGN
 #include "bthread/types.h"                   // bthread_t, bthread_attr_t
 #include "brpc/versioned_ref_with_id.h"
 
 
 namespace brpc {
 
+DECLARE_bool(event_dispatcher_edisp_unsched);
+
 // Unique identifier of a IOEventData.
 // Users shall store EventDataId instead of EventData and call EventData::Address()
 // to convert the identifier to an unique_ptr at each access. Whenever a
@@ -188,6 +191,11 @@ template <typename T> friend class IOEvent;
 
 EventDispatcher& GetGlobalEventDispatcher(int fd, bthread_tag_t tag);
 
+// Unified unsched switch for transport layer.
+// false -> background start (allowing schedule away),
+// true  -> urgent start (foreground scheduling before caller continues).
+bool EventDispatcherUnsched();
+
 // IOEvent class manages the IO events of a file descriptor conveniently.
 template <typename T>
 class IOEvent {

src/brpc/rdma/rdma_endpoint.cpp

@@ -63,7 +63,6 @@ BRPC_VALIDATE_GFLAG(rdma_trace_verbose, brpc::PassValidate);
 DEFINE_bool(rdma_use_polling, false, "Use polling mode for RDMA.");
 DEFINE_int32(rdma_poller_num, 1, "Poller number in RDMA polling mode.");
 DEFINE_bool(rdma_poller_yield, false, "Yield thread in RDMA polling mode.");
-DEFINE_bool(rdma_edisp_unsched, false, "Disable event dispatcher schedule");
 DEFINE_bool(rdma_disable_bthread, false, "Disable bthread in RDMA");
 
 static const size_t IOBUF_BLOCK_HEADER_LEN = 32; // implementation-dependent

src/brpc/rdma/rdma_endpoint.h

@@ -38,7 +38,6 @@ namespace rdma {
 
 DECLARE_bool(rdma_use_polling);
 DECLARE_int32(rdma_poller_num);
-DECLARE_bool(rdma_edisp_unsched);
 DECLARE_bool(rdma_disable_bthread);
 
 class RdmaConnect : public AppConnect {

src/brpc/rdma_transport.cpp

@@ -18,6 +18,7 @@
 #if BRPC_WITH_RDMA
 
 #include "brpc/rdma_transport.h"
+#include "brpc/event_dispatcher.h"
 #include "brpc/tcp_transport.h"
 #include "brpc/rdma/rdma_endpoint.h"
 #include "brpc/rdma/rdma_helper.h"
@@ -127,13 +128,13 @@ void RdmaTransport::ProcessEvent(bthread_attr_t attr) {
     bthread_t tid;
     if (FLAGS_usercode_in_coroutine) {
         OnEdge(_socket);
-    } else if (rdma::FLAGS_rdma_edisp_unsched == false) {
-        auto rc = bthread_start_background(&tid, &attr, OnEdge, _socket);
+    } else if (!EventDispatcherUnsched()) {
+        auto rc = bthread_start_urgent(&tid, &attr, OnEdge, _socket);
         if (rc != 0) {
             LOG(FATAL) << "Fail to start ProcessEvent";
             OnEdge(_socket);
         }
-    } else if (bthread_start_urgent(&tid, &attr, OnEdge, _socket) != 0) {
+    } else if (bthread_start_background(&tid, &attr, OnEdge, _socket) != 0) {
         LOG(FATAL) << "Fail to start ProcessEvent";
         OnEdge(_socket);
     }
@@ -235,4 +236,4 @@ bool RdmaTransport::OptionsAvailableOverRdma(const ServerOptions* opt) {
     return true;
 }
 } // namespace brpc
-#endif
+#endif

src/brpc/tcp_transport.cpp

@@ -16,6 +16,7 @@
 // under the License.
 
 #include "brpc/tcp_transport.h"
+#include "brpc/event_dispatcher.h"
 
 namespace brpc {
 DECLARE_bool(usercode_in_coroutine);
@@ -68,7 +69,13 @@ void TcpTransport::ProcessEvent(bthread_attr_t attr) {
     bthread_t tid;
     if (FLAGS_usercode_in_coroutine) {
         OnEdge(_socket);
-    } else if (bthread_start_urgent(&tid, &attr, OnEdge, _socket) != 0) {
+    } else if (!EventDispatcherUnsched()) {
+        auto rc = bthread_start_urgent(&tid, &attr, OnEdge, _socket);
+        if (rc != 0) {
+            LOG(FATAL) << "Fail to start ProcessEvent";
+            OnEdge(_socket);
+        }
+    } else if (bthread_start_background(&tid, &attr, OnEdge, _socket) != 0) {
         LOG(FATAL) << "Fail to start ProcessEvent";
         OnEdge(_socket);
     }
@@ -96,4 +103,4 @@ void TcpTransport::QueueMessage(InputMessageClosure& input_msg,
     }
 }
 
-} // namespace brpc
+} // namespace brpc