Skip to content

fix(pipeline): stop idle keepalive polls, reduce idle requests#1322

Merged
therealaleph merged 2 commits into
therealaleph:mainfrom
yyoyoian-pixel:fix/idle-keepalive-backoff
May 20, 2026
Merged

fix(pipeline): stop idle keepalive polls, reduce idle requests#1322
therealaleph merged 2 commits into
therealaleph:mainfrom
yyoyoian-pixel:fix/idle-keepalive-backoff

Conversation

@yyoyoian-pixel

@yyoyoian-pixel yyoyoian-pixel commented May 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Stops keepalive polling on truly idle sessions and adds gradual backoff with separate idle tier tracking. Reduces idle Apps Script requests from ~1200/5min to ~35/5min for truly idle sessions.

How it works

  1. Early backoff (first 3 empty replies): 20ms → 80ms → 4s → 10s
  2. Stop polling after idle_tier > 3 (~16s of empties): session stops sending keepalive polls entirely, just waits for client data
  3. Apps maintain their own heartbeats (MQTT PINGREQ, FCM keepalive, Telegram ping) which trigger client writes — those act as natural polls that pick up server-pushed data

Separate counters

  • consecutive_empty: controls pipeline depth management (resets to 0 on data — sessions can ramp up immediately)
  • idle_tier: controls keepalive delay (halves on server-pushed data instead of full reset — prevents push notifications from yo-yoing the backoff)

Measured results

Minute Requests Notes
0:00-1:00 23 Initial ramp-down
1:00-2:00 63 Sessions settling
2:00+ 7-50 Steady state (app heartbeats only)
Version Idle requests/5min
v1.9.13 (serial) ~300
v1.9.30 (pipelined, 2s cap) ~1200
v1.9.31 (pipelined, 2s cap) ~1000
This PR (truly idle) ~35
This PR (background apps active) ~250-350 (real traffic, not wasted polls)

Note: sessions receiving periodic server pushes (Instagram MQTT, Telegram, etc.) never reach idle_tier > 3 because they keep getting data. Those requests are real traffic, not wasted keepalive polls. True idle sessions (no server pushes) drop to zero polls.

No latency impact

  • tokio::select! races delay timer against client socket reads
  • When app writes data (heartbeat, user action), fires immediately
  • Pipeline depth uses consecutive_empty (resets to 0) — not affected by idle_tier

Test plan

  • Verified idle requests drop from ~1200 to ~35/5min for truly idle sessions
  • Verified background app traffic (Telegram, Instagram) still works
  • Verify active browsing/video works without added latency
  • Verify push notifications still arrive promptly
  • Test with customer's setup (15 deployments, Chrome + Instagram only)

🤖 Generated with Claude Code

@github-actions github-actions Bot added the type: fix fix: PR — auto-applied by release-drafter label May 20, 2026
@yyoyoian-pixel yyoyoian-pixel force-pushed the fix/idle-keepalive-backoff branch 2 times, most recently from c605acb to d943fef Compare May 20, 2026 12:20
@yyoyoian-pixel yyoyoian-pixel changed the title fix(pipeline): escalate idle keepalive backoff — reduces idle requests 6x fix(pipeline): stop idle keepalive polls — 1200/5min → ~35/5min May 20, 2026
@yyoyoian-pixel yyoyoian-pixel changed the title fix(pipeline): stop idle keepalive polls — 1200/5min → ~35/5min fix(pipeline): stop idle keepalive polls, reduce idle requests ~97% May 20, 2026
@yyoyoian-pixel yyoyoian-pixel changed the title fix(pipeline): stop idle keepalive polls, reduce idle requests ~97% fix(pipeline): stop idle keepalive polls, reduce idle requests May 20, 2026
@yyoyoian-pixel yyoyoian-pixel force-pushed the fix/idle-keepalive-backoff branch from d943fef to e870fb0 Compare May 20, 2026 12:39
Previous cap of 2s caused ~1200 requests/5min idle with 15 deployments.
New escalation: 20ms→80ms→200ms→500ms→2s→5s→10s→20s.
After 15+ consecutive empties, sessions poll every 20s.

Estimated idle reduction: ~1200/5min → ~200/5min.
Zero latency impact on active traffic — select! races timer against
client reads, so real data fires immediately.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@yyoyoian-pixel yyoyoian-pixel force-pushed the fix/idle-keepalive-backoff branch from e870fb0 to 259be25 Compare May 20, 2026 12:54

@therealaleph therealaleph left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally after adding the mixed-deployment guard: all-legacy idle sessions can stop polling, but mixed fleets keep emitting empty polls so round-robin can still reach a long-poll-capable peer.

Tests:

  • cargo test --lib tunnel_client::tests::tunnel_loop_keeps_polling_when_only_some_deployments_legacy -- --nocapture
  • cargo test --lib
  • cargo build --release

Answered via LLM, Supervised @therealaleph

@therealaleph therealaleph merged commit 02f2765 into therealaleph:main May 20, 2026
1 check passed
therealaleph added a commit that referenced this pull request May 20, 2026
Ship PR #1322 by @yyoyoian-pixel: Full Tunnel idle sessions now reduce empty keepalive request load, while mixed deployments keep polling so long-poll-capable peers can continue delivering remote-to-client data.

Local verification:
- cargo test --lib
- cargo build --release
- cargo build --bin mhrv-rs-ui --release --features ui
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix fix: PR — auto-applied by release-drafter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants