Repository navigation
feat(ci): materialize CI runs and jobs off github-actions traces - #1718
Confidence 10/10 · No issues found
🟢 Confidence 10/10 · safe to merge
No findings; the only unverified piece is the backfill's (TraceId, SpanId) NOT IN (...) subquery, the repo's first tuple subquery, proven only by the new ClickHouse e2e.
no findings · tests covered · risk medium · 1/1 new units observable
Adds migration 0047: ci_runs and ci_jobs, two one-row-per-run/job projections of github-actions traces with their views and a deduplicating backfill, plus the Tinybird datasources, local schema v38 and catalog notes. Nothing reads them yet; safe to merge. The Warehouse*Error issues whose frames land in packages/query-engine/src/ch/tables.ts are unrelated: the diff only adds two table exports and no query path, so it neither causes nor fixes them.
- Migration 0047 creates
ci_runs,ci_jobsand their views, then backfills retainedtracesskipping rows already stored ciRuns/ciJobsTinybird datasources and MVs extract thecicd.*attributes into columns, 365-day TTL- Local schema bumps to v38 via an additive, forward-filled step; scoped delete gets
unkeyedplans
Before merge
- Migration · Check it is safe on production data and with the code still deployed ·
packages/domain/src/clickhouse/migrations/0047_ci_runs_jobs.ts - Warehouse · Deploy the schema before the code that reads it ·
packages/domain/src/tinybird/datasources.ts
Production impact
Open errors in the changed files
| Issue | Service | Occurrences | File |
|---|---|---|---|
| @maple/http/errors/WarehouseUpstreamError | maple-api | 271 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseQueryError | maple-api | 264 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseUpstreamError | maple-api | 154 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseUpstreamError | maple-api | 112 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseUpstreamError | maple-api | 20 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseAuthError | maple-api | 18 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseUpstreamError | maple-vcs-sync | 15 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseUpstreamError | maple-api | 14 | packages/query-engine/src/ch/tables.ts |
| @maple/http/errors/WarehouseAuthError | maple-api | 13 | packages/query-engine/src/ch/tables.ts |
| @maple/backend/lib/DatabaseError | maple-vcs-sync | 13 | packages/query-engine/src/ch/tables.ts |
After this merges, Maple checks whether they stop.
What was checked
- Runner waits and steps stay out of
ci_jobs: the wait span carries nocicd.pipeline.task.name(CiTraceServiceotlp.ts:282-287) and steps carrymaple.cicd.step.number - The projections carry every field the current raw reads select (
ch/queries/ci-runs.ts:57-97and:122-141), including theskiprows those reads filter - The backfill windows on source bounds (
apply-plan.ts:86-113), so a fresh-target migration still chunks overtracesrather than issuing one Worker-sized statement
Observability coverage: 1 of 1 changes observable
| Change | Kind | Observable | Evidence |
|---|---|---|---|
Migration 0047 backfill of ci_runs / ci_jobs |
background job | yes | expands into durable schema-apply Workflow steps (packages/domain/src/clickhouse/apply-plan.ts:106-113), the shared path every backfilling migration uses |
4e74953 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.