Skip to content

fix(connectors): hand off SDK worker during blocking send callbacks - #3797

Open
mlevkov wants to merge 1 commit into
apache:masterfrom
mlevkov:sdk-callback-worker-handoff
Open

fix(connectors): hand off SDK worker during blocking send callbacks#3797
mlevkov wants to merge 1 commit into
apache:masterfrom
mlevkov:sdk-callback-worker-handoff

Conversation

@mlevkov

@mlevkov mlevkov commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #3796. The send callback runs synchronously inside the SDK's polling task (handle_messages in core/connectors/sdk/src/source.rs), on the tokio runtime shared by every connector instance loaded from the same plugin library (static RUNTIME: OnceLock<Runtime> in lib.rs). A callback that blocks pins one worker for the duration. With #3795's bounded forwarding channel, a callback legitimately blocks for backpressure while the channel is full, so enough saturated instances of one library can occupy all workers, and iggy_source_close for a sibling instance then waits behind them: the close blocks on the sibling's polling task getting scheduled to observe its shutdown signal.

Change

One call site: wrap the callback invocation in tokio::task::block_in_place, so the worker is handed off before the callback runs and the runtime keeps scheduling sibling tasks regardless of how long the callback blocks.

Notes from the analysis in #3796:

  • The wrap has to live in the SDK, not the runtime: block_in_place consults the calling thread's tokio context, which is only set on the plugin runtime's own worker threads.
  • The SDK runtime is Runtime::new(), i.e. multi-threaded, which block_in_place requires.
  • No FFI or ABI change. Existing plugin binaries keep working and pick the fix up when rebuilt against the updated SDK.

This composes with #3795 (bounded channel + shutdown signaling) but does not depend on it: any long-blocking callback benefits.

Test plan

  • cargo clippy -p iggy_connector_sdk --all-targets --all-features -- -D warnings clean
  • cargo test -p iggy_connector_sdk --all-features passes
  • cargo build -p iggy_connector_stdout_sink -p iggy_connector_random_source -p iggy-connectors (macro consumers + runtime rebuild cleanly)
  • Behavior under saturation is exercised end to end by the integration suite in CI

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 2, 2026
@codecov

codecov Bot commented Aug 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.19%. Comparing base (be5012e) to head (50ef34d).

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3797       +/-   ##
=============================================
- Coverage     82.99%   68.19%   -14.80%     
  Complexity     1339     1339               
=============================================
  Files          1218     1218               
  Lines        165627   139885    -25742     
  Branches     133940   108198    -25742     
=============================================
- Hits         137459    95398    -42061     
- Misses        24509    40788    +16279     
- Partials       3659     3699       +40     
Components Coverage Δ
Rust Core 64.19% <100.00%> (-19.32%) ⬇️
Java SDK 66.55% <ø> (ø)
C# SDK 76.43% <ø> (ø)
Python SDK 90.00% <ø> (ø)
PHP SDK 84.48% <ø> (ø)
Node SDK 95.68% <ø> (ø)
Go SDK 69.04% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sdk/src/source.rs 82.55% <100.00%> (-1.13%) ⬇️

... and 336 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

/request-review: could not request mlevkov - a handle is not a repo collaborator, or the team is unknown.

@mlevkov

mlevkov commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @hubcio

@github-actions
github-actions Bot requested a review from hubcio August 2, 2026 03:00
@mlevkov
mlevkov force-pushed the sdk-callback-worker-handoff branch 2 times, most recently from 2772550 to cc9107f Compare August 9, 2026 01:28
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added the S-stale Inactive issue or pull request label Aug 17, 2026
The send callback runs synchronously inside the SDK's polling task, on
a tokio runtime shared by every connector instance loaded from the same
plugin library. A callback that blocks for backpressure pins one worker
for the duration, and with enough saturated instances iggy_source_close
for a sibling waits behind them, since the close blocks on the sibling's
polling task getting scheduled to observe its shutdown signal.

Wrap the callback in tokio::task::block_in_place so the worker is
handed off before the callback runs. The SDK runtime is multi-threaded,
which block_in_place requires. No FFI or ABI change; plugins pick this
up when rebuilt against the updated SDK.

Fixes apache#3796.

Co-authored-by: Claude <noreply@anthropic.com>
@mlevkov
mlevkov force-pushed the sdk-callback-worker-handoff branch from cc9107f to 50ef34d Compare August 17, 2026 02:10
@github-actions github-actions Bot removed the S-stale Inactive issue or pull request label Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SDK: hand off the plugin-runtime worker during blocking source send callbacks

1 participant