Skip to content

improvement(db): reduce connection saturation and egress hotspots#4594

Merged
waleedlatif1 merged 3 commits into
stagingfrom
waleedlatif1/pgbouncer-error-debug
May 14, 2026
Merged

improvement(db): reduce connection saturation and egress hotspots#4594
waleedlatif1 merged 3 commits into
stagingfrom
waleedlatif1/pgbouncer-error-debug

Conversation

@waleedlatif1
Copy link
Copy Markdown
Collaborator

Summary

  • Reduce postgres-js pool max from 30 → 15 in @sim/db and realtime app (validated against 24h PlanetScale Insights: 3.5 avg fleet concurrency, 25× headroom on steady state, 2.5× under 10× burst)
  • Scope workflow_blocks agent query to workspace in MCP refresh + stored routes (eliminates ~88 GB/day egress from cross-tenant scan)
  • Project copilot_chats.messages SQL-side in workspace VFS materializer (eliminates ~58 GB/day egress; semantically identical to prior JS-side filter)

Type of Change

  • Bug fix

Testing

Tested manually. Validated:

  • bun run check:api-validation passes
  • bun run type-check passes (no errors)
  • SQL projection produces exactly the shape serializeTaskChat consumes
  • MCP route filter migrated from JS to SQL is logically equivalent; inArray guarded by existing workflowIds.length === 0 early return

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel
Copy link
Copy Markdown

vercel Bot commented May 14, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped May 14, 2026 6:31am

Request Review

@cursor
Copy link
Copy Markdown

cursor Bot commented May 14, 2026

PR Summary

Medium Risk
Moderate risk because it changes database connection pooling limits and rewrites query projections/filters (including JSONB SQL), which could impact performance or edge-case data shapes if assumptions differ in production.

Overview
Reduces postgres client pool max from 30 to 15 in both packages/db/db.ts and realtime’s operations.ts to lower connection saturation.

Optimizes MCP endpoints (/api/mcp/servers/[id]/refresh and /api/mcp/tools/stored) by scoping workflow_blocks agent-block reads to the current workspace’s workflow IDs via inArray, removing the prior cross-workspace scan + in-process filtering.

Updates Copilot workspace VFS task materialization to compute messageCount and project a filtered/normalized copilot_chats.messages array in SQL (only user/assistant roles and text contentBlocks), reducing JSON egress while keeping the serialized output shape consistent.

Reviewed by Cursor Bugbot for commit 631e60f. Configure here.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 14, 2026

Greptile Summary

This PR addresses connection saturation and egress hotspots through three targeted changes: halving the postgres-js pool max from 30 to 15 in both @sim/db and the realtime app, and pushing two JS-side filters into SQL to eliminate ~146 GB/day of cross-tenant data transfer.

  • Pool reduction (packages/db/db.ts, apps/realtime/src/database/operations.ts): max lowered from 30 → 15, validated against 24 h PlanetScale metrics showing ample headroom.
  • MCP workspace scoping (refresh/route.ts, stored/route.ts): workflowBlocks query now includes inArray(workflowBlocks.workflowId, workflowIds), eliminating the full-table agent-block scan; both call-sites have an existing workflowIds.length === 0 early return that guards against empty-array SQL issues.
  • Copilot VFS SQL projection (workspace-vfs.ts): copilot_chats.messages is now filtered and shaped server-side via an inline jsonb_agg subquery rather than shipping the full column and filtering in JS.

Confidence Score: 5/5

Safe to merge — all three changes are well-scoped optimizations with no behaviour regressions.

Pool reduction is purely a config tuning backed by production metrics. Both MCP workspace-scoping changes move an existing JS filter into SQL with the empty-array edge case correctly handled by pre-existing early returns. The copilot VFS SQL projection preserves pre-PR semantics: role/content operators (->> / ->) are used correctly, and COALESCE guards cover NULL and empty-array column states.

No files require special attention.

Important Files Changed

Filename Overview
packages/db/db.ts Pool max reduced from 30 to 15; no logic changes, straightforward config tuning.
apps/realtime/src/database/operations.ts Pool max reduced from 30 to 15; mirrors the packages/db change, no other modifications.
apps/sim/app/api/mcp/servers/[id]/refresh/route.ts Workspace filter moved from post-query JS to SQL inArray; empty-array case correctly guarded by existing early return at line 71.
apps/sim/app/api/mcp/tools/stored/route.ts Same workspace-scoping improvement as the refresh route; workflowIds.length === 0 guard at line 29 prevents empty-array inArray issues.
apps/sim/lib/copilot/vfs/workspace-vfs.ts Messages column projected SQL-side; role/content extraction uses correct ->> / -> operators; COALESCE guards handle NULL/empty columns safely.

Reviews (3): Last reviewed commit: "fix(vfs): guard jsonb_array_elements aga..." | Re-trigger Greptile

Comment thread apps/sim/lib/copilot/vfs/workspace-vfs.ts
Comment thread apps/sim/lib/copilot/vfs/workspace-vfs.ts
@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread apps/sim/lib/copilot/vfs/workspace-vfs.ts
@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@cursor review

@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@greptile

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 631e60f. Configure here.

@waleedlatif1 waleedlatif1 merged commit b5dba82 into staging May 14, 2026
14 checks passed
@waleedlatif1 waleedlatif1 deleted the waleedlatif1/pgbouncer-error-debug branch May 14, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant