fix(warehouse): repair capability probing and the service-map rollup

[Makisuo]

Investigated production query performance via tinybird.pipe_stats_rt and Maple's own traces. Aggregate p99 is healthy (0.198s over 174k queries/day) — but two things turned out to be completely broken, silently.

1. Capability probing never succeeded — not once

inspectCapabilities runs four system.* probes. In production:

probe	result
system.columns	403 ×2,523/day (Tinybird SDK), or 200 at p50 2.196s via the gateway
system.data_skipping_indices	403 ×2,523/day
system.projections	403 ×691/day, zero successes
system.settings, version()	fine

Tinybird's /v0/sql does not expose system.* to workspace tokens; the gateway serves system.columns in 2.2s, over the 2s budget. Maple's own spans confirm it — inspectCapabilities p99 was exactly 2000ms, clamped at the timeout, with ~370 "fell back to conservative plan" log lines/day.

Because Effect.all failed the whole set on one denied probe, every query got baselineWarehouseCapabilities() → attributeIndexMode: "none", logBodySearchMode: "scan". With indexMode === "none", buildAttrFilterCondition emits only the exact Map lookup and skips the has(mapKeys(...)) candidate prefilter — so idx_span_attr_keys, idx_log_attr_keys and idx_lower_body are all deployed and were never used by a single production query. On top of that, misses (10% of logs/traces queries) added p50 286ms / p99 4.5s of overhead before the real query started.

Fix: we deploy the managed schema, so its indices are a compile-time fact. BackendDialect.managedSchema (true for tinybird + tinybird-gateway) short-circuits resolveCapabilities to managedWarehouseCapabilities(), derived by parsing latestSnapshotStatements — not hand-listed, so adding an index to datasources.ts enables the matching feature with no change here. BYO ClickHouse still probes live, now with per-probe degradation (degradeToEmpty) and a 1h cache TTL.

2. The service-map edge rollup has been dead since 2026-05-15

Chasing a 73k-queries/day fan-out led to the actual bug: service_map_edges_hourly holds 32 rows, newest 2026-05-15.

HTTP 400 Bad Request: Data Source needs to have JSONPaths defined

Both rollup targets declare jsonPaths: false but are written directly via POST /v0/events by ServiceMapRollupService, not by a materialized view. Every write was rejected. Since an hour seals only once its edge rows land (if (rows.length > 0)), nothing ever sealed — so all six LOOKBACK_HOURS candidates re-ran on every tick for ~260 orgs, forever: ~73,000 queries and ~175 GB/day. The fan-out was the symptom, not the bug.

Fixes:

• Removed jsonPaths: false from serviceMapEdgesHourly and serviceAddressResolutionsHourly (alertChecks is the exemplar — the other directly-ingested datasource). Manifest regenerated.
• Gated the tick on CH.activeOrgsByTracesQuery(), the same pattern the error-issue and anomaly ticks already use. Only 14 orgs had spans in the last 7h, vs ~260 being processed. Fails open (unlike the anomaly tick) because a skipped rollup hour is a permanent gap once it ages out, whereas a skipped anomaly evaluation is just a late alert.
• Bounded the resolutions repair pass with a new serviceMapResolutionsExistingHoursSQL probe. It previously re-ran the raw-traces self-join — the most expensive query in the tick — for every sealed hour on every tick, unconditionally.

3. spanSearchQuery two-staging

Without a traceId it reads raw traces, whose sort key (OrgId, ServiceName, SpanName, toDateTime(Timestamp)) cannot serve ORDER BY Timestamp DESC. Single-stage, it materialized both attribute Maps for every matching row in range before LIMIT discarded all but N — 9–13 GB per call, occasionally timing out. Now two-staged exactly like its sibling tracesListQuery. The trace_detail_spans path is unchanged (sort-key prefix, one trace, no cutoff needed).

Reviewer notes

• ⚠️ The datasource change needs a Tinybird deploy before edge writes resume.
• spanSearchQuery's ORDER BY still uses the toString(Timestamp) alias — left alone deliberately to keep results identical; the win is the cutoff, and stage 2's row set is now small.
• The plan also called for pushing a time predicate into traceSummariesQuery's TraceId IN (SELECT ...) subquery. Dropped — it already has one; I verified by compiling the query. tracesBaseWhereConditions adds the Timestamp bounds unconditionally.
• Out of scope: the 66 timeouts/7d, all of which are ad-hoc raw-SQL dashboard widgets at max_execution_time=10 (28 of them the internal ingest-volume dashboard).

Verification

• 988 query-engine tests pass; domain tinybird tests pass.
• tsc --noEmit clean in packages/query-engine, packages/domain, apps/api.
• SQL baseline snapshot regenerated — the diff contains only the new builder and the spanSearchQuery cutoff, nothing else.
• New regression guard: managed-capabilities.test.ts asserts the managed schema yields logs.attributes.bloom, traces.attributes.bloom and logs.body.tokenbf. The rewritten gateway test asserts no system.* query is issued at all.

After deploy, re-check: system.columns request count in pipe_stats_rt should go to ~zero (and the 403s with it); maple.query.capabilities.metadata_available should be true; the "conservative plan" log line should stop; service_map_edges_hourly should start filling for active orgs; read_bytes on attribute-filtered log searches should drop.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[github-actions]

🍁 Maple PR preview

Note

Preview resources were removed when this pull request closed.

Final commit 2b3dd58 · View workflow run