Measured against the live cluster on 2026-08-15 while evaluating whether to delete default-pool for cost. Conclusion was: don't, for now. Filing so the arithmetic isn't re-derived, and because one of these is a live latent bug worth fixing on its own.
Burn is ~$300/mo total (operator figure). Compute is 3 × n2-standard-2, all in us-central1-a.
1. schedulerService is not a singleton — this one is a real bug
server.ts:299 calls schedulerService.start() unconditionally. No leader election, no gate.
The deployment is one kubectl scale away from running every cron twice — including the daily digest, which sends real email to real users. Also agent heartbeats (double LLM spend), summarizer, pod curation, auto-join, ensemble, skills refresh.
This is latent today only because replicas: 1. It fires on: a manual scale during an incident, any autoscaling, or a stuck rollout leaving two pods up briefly.
Cheap fix: a short-TTL Redis lease; only the holder runs jobs. Redis is already connected and already used by the socket adapter.
(Note: onboardingSilenceService and stalledConnectService are the accidental exceptions — their unique indexes make a duplicate pass a no-op by design.)
2. Two places ADR-015 and the cluster disagree
ADR-015 says backend + frontend + redis schedule on spot-pool, and that dev-pool carries taint pool=dev:NoSchedule. Neither is true:
| claim |
reality |
| backend/redis on spot-pool |
both have nodeSelector: pool=dev — pinned to the on-demand pool |
dev-pool tainted pool=dev:NoSchedule |
dev-pool has no taint; only spot-pool is tainted (workload-tier) |
Result: spot-pool (the cheap node) runs one frontend pod, and dev-pool (on-demand) carries backend, redis, commonly-bot, litellm, clawdbot-gateway, nginx-ingress and both cloudflared pods.
Either the doc is stale or the cluster drifted. Worth resolving either way — a doc claiming work nobody did is the drift class that has already cost this repo a fix commit.
Caveat if anyone acts on it: ADR-015 also warns that anything holding session state must stay off spot (30s reclaim notice). Backend holds Socket.io connections, so the pinning may be a deliberate correction rather than drift. Find out which before moving anything.
3. Why default-pool cannot simply be deleted
It looks nearly idle — but only if you filter out kube-system, which is the mistake I made first. It actually carries a kube-dns replica, both konnectivity agents, metrics-server, external-secrets ×3, kube-state-metrics, l7-default-backend and gmp-operator.
Measured:
must reschedule off default-pool 492m (a further 276m is DaemonSet/static — dies with the node)
dev-pool free 599m (1930m allocatable, 1331m requested = 68%)
→ after absorbing 107m free
backend is maxUnavailable=0, maxSurge=1, requests 200m, and is pinned nodeSelector: pool=dev. Every rolling deploy needs a second 200m pod on dev-pool before the old one terminates. 107m < 200m, so the next Deploy Dev would stall with the surge pod Pending. It cannot spill to spot-pool (tainted, and no pool label).
It would not fail at deletion time. It would fail at the next deploy.
Cleared as not blockers, for the record: kube-dns already has a second replica on dev-pool; all three nodes are in one zone so no PVC affinity problem; none of default-pool's workloads own a PVC.
4. Zero-downtime deploys are real and worth protecting
Verified end to end, and better configured than assumed:
maxSurge=1, maxUnavailable=0- readiness probe
GET /api/health/ready that 503s until Mongo is connected and runs a real SELECT 1 against PG — with a deliberate 200-plus-degraded:['postgresql'] branch so a PG blip doesn't pull the pod from rotation
preStop: sleep 10 — drains before SIGTERM, which is the detail that actually prevents dropped requeststerminationGracePeriodSeconds: 30
Two honest limits: open Socket.io connections still drop when the old pod terminates (clients auto-reconnect; invisible at current concurrency, real at scale), and at replicas: 1 this covers deploys but not crashes.
If someone picks this up later
Order matters:
- Scheduler singleton — independently valuable, do first.
- Resolve the ADR-015 contradiction as a decision (is backend allowed on spot given Socket.io?).
- Only then consider the node. The likely-safe shape is replacing
default-pool with an untainted spot pool so the 492m lands there rather than on dev-pool — the workloads on it are stateless and replicated, which is exactly the spot profile.
- Verify by running a real
Deploy Dev and watching the backend rollout complete. Not by checking that pods moved.
replicas: 2 is not the win it appears to be until (1) is done: both replicas would land on the same node anyway (nodeSelector: pool=dev, one dev-pool node), so it buys crash resilience but not node-failure resilience.
Dollar figures are list prices; I could not see the bill.
Measured against the live cluster on 2026-08-15 while evaluating whether to delete
default-poolfor cost. Conclusion was: don't, for now. Filing so the arithmetic isn't re-derived, and because one of these is a live latent bug worth fixing on its own.Burn is ~$300/mo total (operator figure). Compute is 3 ×
n2-standard-2, all inus-central1-a.1.
schedulerServiceis not a singleton — this one is a real bugserver.ts:299callsschedulerService.start()unconditionally. No leader election, no gate.The deployment is one
kubectl scaleaway from running every cron twice — including the daily digest, which sends real email to real users. Also agent heartbeats (double LLM spend), summarizer, pod curation, auto-join, ensemble, skills refresh.This is latent today only because
replicas: 1. It fires on: a manual scale during an incident, any autoscaling, or a stuck rollout leaving two pods up briefly.Cheap fix: a short-TTL Redis lease; only the holder runs jobs. Redis is already connected and already used by the socket adapter.
(Note:
onboardingSilenceServiceandstalledConnectServiceare the accidental exceptions — their unique indexes make a duplicate pass a no-op by design.)2. Two places ADR-015 and the cluster disagree
ADR-015 says
backend+frontend+redisschedule onspot-pool, and thatdev-poolcarries taintpool=dev:NoSchedule. Neither is true:nodeSelector: pool=dev— pinned to the on-demand pooldev-pooltaintedpool=dev:NoScheduledev-poolhas no taint; onlyspot-poolis tainted (workload-tier)Result:
spot-pool(the cheap node) runs one frontend pod, anddev-pool(on-demand) carries backend, redis, commonly-bot, litellm, clawdbot-gateway, nginx-ingress and both cloudflared pods.Either the doc is stale or the cluster drifted. Worth resolving either way — a doc claiming work nobody did is the drift class that has already cost this repo a fix commit.
Caveat if anyone acts on it: ADR-015 also warns that anything holding session state must stay off spot (30s reclaim notice). Backend holds Socket.io connections, so the pinning may be a deliberate correction rather than drift. Find out which before moving anything.
3. Why
default-poolcannot simply be deletedIt looks nearly idle — but only if you filter out
kube-system, which is the mistake I made first. It actually carries a kube-dns replica, both konnectivity agents, metrics-server, external-secrets ×3, kube-state-metrics, l7-default-backend and gmp-operator.Measured:
backendismaxUnavailable=0, maxSurge=1, requests 200m, and is pinnednodeSelector: pool=dev. Every rolling deploy needs a second 200m pod on dev-pool before the old one terminates. 107m < 200m, so the nextDeploy Devwould stall with the surge podPending. It cannot spill tospot-pool(tainted, and nopoollabel).It would not fail at deletion time. It would fail at the next deploy.
Cleared as not blockers, for the record: kube-dns already has a second replica on dev-pool; all three nodes are in one zone so no PVC affinity problem; none of default-pool's workloads own a PVC.
4. Zero-downtime deploys are real and worth protecting
Verified end to end, and better configured than assumed:
maxSurge=1, maxUnavailable=0GET /api/health/readythat 503s until Mongo is connected and runs a realSELECT 1against PG — with a deliberate 200-plus-degraded:['postgresql']branch so a PG blip doesn't pull the pod from rotationpreStop: sleep 10— drains before SIGTERM, which is the detail that actually prevents dropped requeststerminationGracePeriodSeconds: 30Two honest limits: open Socket.io connections still drop when the old pod terminates (clients auto-reconnect; invisible at current concurrency, real at scale), and at
replicas: 1this covers deploys but not crashes.If someone picks this up later
Order matters:
default-poolwith an untainted spot pool so the 492m lands there rather than on dev-pool — the workloads on it are stateless and replicated, which is exactly the spot profile.Deploy Devand watching the backend rollout complete. Not by checking that pods moved.replicas: 2is not the win it appears to be until (1) is done: both replicas would land on the same node anyway (nodeSelector: pool=dev, one dev-pool node), so it buys crash resilience but not node-failure resilience.Dollar figures are list prices; I could not see the bill.