[Makisuo]

What

Two filtered projections of the github-actions CI traces, one row per workflow run (ci_runs) and one per job (ci_jobs), with the cicd.* attributes as columns and a 365-day TTL. Migration 0047 creates them and their views, then backfills the retained raw traces. Nothing reads them yet; the next PR moves the CI analytics over.

Why

The CI page's queries read raw traces and pull every field out of the Map columns. Measured in prod on Maple's own org: a 6-day job query reads 569k rows / 205 MB to count 22k jobs, and the same kind of scan over 30 days took 16 s (cold) against 181 ms (warm). In the app that showed up as p95 3 to 4.4 s and spikes to 15.7 s per query, with 11 queries per page load in three waves.

Step spans are 80% of CI spans and no CI query reads them; runner-wait spans duplicate the queue time the job span already carries. The projections keep runs and jobs only.

Layout, chosen by benchmark

Real CI span shapes, synthetic volume: a Maple-sized org (790 runs/day) and a 10x org over 365 days, plus 9M non-CI spans in raw traces. ClickHouse 25.8, 2 threads, bun run bench:queries, 7 runs.

candidate	verdict
two tables, (OrgId, Timestamp, TraceId), monthly partitions	chosen
same, daily partitions	slower everywhere (365d runs list 180 ms vs 11 ms: 495+ tiny parts)
(OrgId, Owner, Repository, Pipeline, Timestamp)	wins only the per-workflow streak read (3 to 24 ms either way), loses the runs list and PR lookup
hourly aggregates	not built: the runs list, red streaks and cursors need per-run rows, and the volume is small

Rows: 88 B per run and 55 B per job compressed on synthetic ids (real ids compress better).

Safety

• The backfill skips every (TraceId, SpanId) its target already holds, so the spans the views write while it runs are never doubled, and a replay keeps the rows older than raw retention. Nothing is truncated: ci_* outlive their source by 335 days. The convergence gate in migrations/index.test.ts now accepts that guard.
• requiredForIngest: false; ingest readiness is unaffected.
• Tinybird fills the new datasources by replaying the views over traces on deploy. The views filter ServiceName = 'github-actions', the second sort-key column, so each org's replay reads only its CI granules (Maple's own org: ~450 MB over 30 days).
• Local mode: schema v38 installs the tables filled forward (CI traces never reach a local store).

Verification

• New ci-runs.clickhouse.e2e.test.ts on ClickHouse 25.8 and 26.2: CiTraceService-shaped spans land exactly (steps, runner waits and other services excluded; skipped runs kept), the backfill rebuilds the same rows without doubling the views' rows, and a full replay keeps a run older than raw retention. A mutation (putting a TRUNCATE back) fails it.
• CI's migration replay step (109 tests) and the SQL analyzer sweep (529 cases) pass on 26.2.
• clickhouse:schema:check, tinybird:manifest:check, domain (169), CLI (626) tests, scoped typecheck and effect lint.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

Stack: #1718 (tables) → #1719 (reads) → #1720 (one-year range)

Stack: #1718 (tables) → #1719 (reads) → #1720 (one-year range)

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added dedicated workflow run and job data, including pipeline, result, duration, queue time, runner, and failure details.
  • Workflow runs and jobs can be queried separately, with job records linked to their runs.
• Documentation
  • Added guidance on querying workflow data, interpreting durations and results, and understanding data retention.

[maple-review-bot]

Maple review

🟢 Confidence 10/10 · safe to merge
No findings; the only unverified piece is the backfill's (TraceId, SpanId) NOT IN (...) subquery, the repo's first tuple subquery, proven only by the new ClickHouse e2e.
_{no findings · tests covered · risk medium · 1/1 new units observable}

Adds migration 0047: ci_runs and ci_jobs, two one-row-per-run/job projections of github-actions traces with their views and a deduplicating backfill, plus the Tinybird datasources, local schema v38 and catalog notes. Nothing reads them yet; safe to merge. The Warehouse*Error issues whose frames land in packages/query-engine/src/ch/tables.ts are unrelated: the diff only adds two table exports and no query path, so it neither causes nor fixes them.

• Migration 0047 creates ci_runs, ci_jobs and their views, then backfills retained traces skipping rows already stored
• ciRuns/ciJobs Tinybird datasources and MVs extract the cicd.* attributes into columns, 365-day TTL
• Local schema bumps to v38 via an additive, forward-filled step; scoped delete gets unkeyed plans

Before merge

• Migration · Check it is safe on production data and with the code still deployed · packages/domain/src/clickhouse/migrations/0047_ci_runs_jobs.ts
• Warehouse · Deploy the schema before the code that reads it · packages/domain/src/tinybird/datasources.ts

Production impact

Open errors in the changed files

Issue	Service	Occurrences	File
@maple/http/errors/WarehouseUpstreamError	maple-api	271	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseQueryError	maple-api	264	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseUpstreamError	maple-api	154	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseUpstreamError	maple-api	112	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseUpstreamError	maple-api	20	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseAuthError	maple-api	18	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseUpstreamError	maple-vcs-sync	15	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseUpstreamError	maple-api	14	packages/query-engine/src/ch/tables.ts
@maple/http/errors/WarehouseAuthError	maple-api	13	packages/query-engine/src/ch/tables.ts
@maple/backend/lib/DatabaseError	maple-vcs-sync	13	packages/query-engine/src/ch/tables.ts

After this merges, Maple checks whether they stop.

What was checked

• Runner waits and steps stay out of ci_jobs: the wait span carries no cicd.pipeline.task.name (CiTraceService otlp.ts:282-287) and steps carry maple.cicd.step.number
• The projections carry every field the current raw reads select (ch/queries/ci-runs.ts:57-97 and :122-141), including the skip rows those reads filter
• The backfill windows on source bounds (apply-plan.ts:86-113), so a fresh-target migration still chunks over traces rather than issuing one Worker-sized statement

Observability coverage: 1 of 1 changes observable

Change	Kind	Observable	Evidence
Migration 0047 backfill of ci_runs / ci_jobs	background job	yes	expands into durable schema-apply Workflow steps (packages/domain/src/clickhouse/apply-plan.ts:106-113), the shared path every backfilling migration uses

_{4e74953 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 50 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: cd3b78ac-273f-411e-8d84-4b8751a96584

📥 Commits

Reviewing files that changed from the base of the PR and between 137e60c and 4e74953.

⛔ Files ignored due to path filters: 2

• packages/domain/src/generated/clickhouse-schema.ts is excluded by !**/generated/**
• packages/domain/src/generated/tinybird-project-manifest.ts is excluded by !**/generated/**

📒 Files selected for processing: 24

• .github/workflows/ci.yml
• apps/cli/src/server/local-schema-history.ts
• apps/cli/src/server/local-schema-version.ts
• apps/cli/src/server/local-store-migrations/steps.ts
• apps/cli/src/server/schema-identity.ts
• apps/cli/src/server/schema/local-inserts.json
• apps/cli/src/server/schema/local-schema-v38.sql
• apps/cli/src/server/schema/local-schema.sql
• apps/cli/src/server/scoped-delete.ts
• apps/cli/test/local-store-migrations.test.ts
• apps/cli/test/native-local-store-migration.sh
• apps/ingest/src/clickhouse_insert_mappings.rs
• docs/self-hosted-clickhouse.md
• packages/backend/src/services/warehouse/ci-runs-materialization.clickhouse.e2e.test.ts
• packages/backend/src/services/warehouse/warehouse-catalog.ts
• packages/domain/src/clickhouse/migrations/0047_ci_runs_jobs.ts
• packages/domain/src/clickhouse/migrations/index.test.ts
• packages/domain/src/clickhouse/migrations/index.ts
• packages/domain/src/tinybird/datasources.ts
• packages/domain/src/tinybird/materializations.ts
• packages/domain/src/tinybird/materialized-projection-order.test.ts
• packages/domain/src/tinybird/retention-matrix.test.ts
• packages/query-engine/src/ch/tables.ts
• scripts/purge-org-traces.ts

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[maple-review-bot]

Merged with 2 steps from "Before merge" still open. If they are done, tick them on the review comment; if not, now is the time.

• Migration · Check it is safe on production data and with the code still deployed
• Warehouse · Deploy the schema before the code that reads it

[maple-review-bot]

Maple: after this shipped

✅ Shipped clean. Nothing this change touches got worse in the hour after it deployed.

Deployed to maple-api (pr-1727) at 2026-10-23 18:40 UTC as 3d9881b, the first version after the merge (the merge commit itself never reported). Compared the 60 minutes before and after.

Errors in the changed files:

• ➖ @maple/http/errors/WarehouseUpstreamError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseQueryError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseUpstreamError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseUpstreamError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseUpstreamError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseAuthError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseUpstreamError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseUpstreamError: 0/h → 0/h
• ➖ @maple/http/errors/WarehouseAuthError: 0/h → 0/h
• ➖ @maple/backend/lib/DatabaseError: 0/h → 0/h

_{Maple looks once, an hour after the first deploy that carries the merge.}