Ledger engine: crash-safe, resumable drill_events migration — no Redis - #7
Open
ar2rsawseen wants to merge 69 commits into
Open
Ledger engine: crash-safe, resumable drill_events migration — no Redis#7ar2rsawseen wants to merge 69 commits into
ar2rsawseen wants to merge 69 commits into
Conversation
…A/B vs classic) Adds a second migration engine behind MIGRATION_ENGINE=ledger for A/B testing against the current architecture (classic remains the default and is untouched). Ledger engine design: - Progress state = one MongoDB ledger row per cd-bounded chunk (pending → in_progress → written → attaching → done). No Redis anywhere; MongoDB + ClickHouse are the only dependencies. - Each chunk copies into its own staging table (clone of the live DDL), verified by read-tally vs exact ClickHouse count(), then promoted via verify-then-ATTACH PARTITION per partition (INSERT SELECT fallback), then dropped. The live table only ever receives whole verified chunks. - Crash recovery never trusts the ledger: in_progress chunks are dropped and redone; written chunks are recounted; attaching chunks verify each partition against the live table before attaching (no double-attach). - Synchronous inserts (errors surface; dedup token effective) + startup dedup canary that measures whether the token works on the target engine. - Error classifier: permanent ClickHouse data errors fail immediately instead of burning the 8x retry backoff; transient errors keep retrying. - Pipelined reads (prefetch) + bounded concurrent insert window. - Fixes the page-boundary double-read (inclusive min() re-returns the previous page's last doc) — the classic engine exhibits this on main: measured 25 duplicate rows + 1 lost boundary doc per 250k clean run. A/B harness in bench/: seed script, kill-drill (random SIGKILL until convergence, verifies zero loss + zero duplicates), instructions. Measured on the same 250k-doc dataset, same machine: - classic: 44s, 250,024 rows / 249,999 unique (dups + loss on a clean run) - ledger: 12s, 250,000 / 250,000 exact; 4x SIGKILL drill converges exact Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the Redis-fed viz for the ledger engine: data comes from the chunk ledger (MongoDB) + in-process engine counters, polled every 2s. Brand tokens sampled from countly.com (#21B566 green, #24292E ink, Plus Jakarta Sans + Inter). Shows live counters, per-collection progress, a chunk map colored by ledger status (newest-first), dedup-canary and engine badges, and failed chunks with their errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ure, monitor, dry-run, report Closes every gap the initial POC deliberately skipped, so the branch is a complete solution rather than a proof of concept: - Bisection → doc-level DLQ (mig_dlq_docs): permanent insert errors are halved-and-retried down to the exact offending documents, which are stored WITH their full raw source doc. Every unmigratable doc (invalid ts, missing fields, transform errors) is likewise captured — accounted for and replayable, never silently dropped. - DLQ replay (POST /control/replay-dlq): re-transforms stored raw docs under the current TRANSFORM_VERSION and inserts into the live table; still-broken docs stay pending with updated errors. Never re-reads the source collection. - Circuit breaker: pauses the engine when >LEDGER_BREAKER_PCT% of a chunk's docs fail (systematic bug) or after N consecutive failed chunks. Resume via POST /control/resume. - ClickHouse backpressure: TTL-cached sampler (never 3 system queries per batch); waits out parts pressure between pages. - Streaming reads (C3): one long-lived cursor per chunk instead of a fresh find() per page, reopened from the last position on cursor death; kills the per-page boundary re-read class entirely. - Multi-pod lease reclaim tick: expired claims are recovered during the work loop, not only at collection start. - Invariant monitor: background spot checks of done chunks against live-table counts; violation → pause + chunk flagged. - Dry-run mode (DRY_RUN=1): ≤5% stratified sample against a Null-engine clone — full parse/type validation, nothing stored; DLQ + coercions become the pre-run report. - Coercion policy (two-tier): Countly-owned c clamped to UInt32; customer sg/custom/cmp/up values that can't survive the numeric path stringified losslessly (zero-copy when clean). Every coercion counted with samples. - GET /report: chunk status, skips by reason, coercions per key, DLQ summary. - Tests: 12 new (classifier, coercions, ledger claims/leases/transitions, end-to-end pipeline with poisoned docs → DLQ with raw docs, replay). Validated: 250k-doc run exact (250,000/250,000) with the full feature set at identical speed to the POC; SIGKILL drill converges exact; typecheck clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rely The team compares branches (main vs this one), so this branch carries only the new implementation. Removed: BatchRunner, collection/range orchestration, Redis hot state, collection locks, global progress, async batch writer, manifest batch store, coverage math, legacy ClickHouse writer, legacy HTTP routes, GC controller, process metrics, their tests and helpers, the engine switch, all classic-only config (Redis, rerun modes, range-parallel, GC, async-write, lock tuning), and the ioredis dependency. Dependencies are now MongoDB + ClickHouse, full stop. Added the one capability only the legacy engine had: a null-cd sweep — a dedicated chunk (sentinel bounds) pages by _id over documents without a cd value, with id-based verify-then-attach (no cd window exists for them) and a monitor mode that stays sound when null-cd rows land inside regular chunks' cd windows. Validated: 12/12 tests (incl. null-cd end-to-end), 250k straight run exact in 8s, SIGKILL drill (2 kills) converges exact — zero loss, zero duplicates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Completion bug: the work loop treated "no pending chunks" as collection complete, silently skipping chunks still leased by a dead pod (SIGKILL orphan under an unexpired lease) — a run could report completed with a hole. Complete now means NO non-terminal chunks: single-pod recovers orphans immediately; multi-pod waits and reclaims on lease expiry (reclaim tick capped at 30s cadence). - POST /control/retry-failed: resets failed chunks to pending and resumes; chunks that were already promoted (e.g. flagged by the invariant monitor) get their live cd window purged first so redo is clean. - Circuit-breaker path now drops its staging table. - bench/seed-failures.ts: seeds breaker-burst / scattered-DLQ / coercion scenarios for failure drills. Drill verified end-to-end on 100k docs: SIGKILL → breaker trip (805 docs DLQ'd with raw docs) → deliberate live-table corruption caught by the invariant monitor in seconds → retry-failed purge+redo → orphan recovery → final 100,003/100,003 exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… harness (D4, #6) Vendors the differential harness from countly-platform (corpus.json 71 fixtures, goldens.json generated from the LIVE ingestion normalization, decode/canonicalize, hash-tied sync contract) into tests/differential/, adds the repo's first CI workflow (typecheck + harness, service-free), and adopts the shared normalization spec in src/transform: - normalize.ts/validators.ts rewritten to the spec (platform branch claude/jovial-shannon-b3dd29 is the source of truth): existing non-blank doc.n wins over sg-derived names (dedup identity with live rows), clampUInt32/clampDateTime64 for Countly-owned fields, sanitizeJsonValue for customer bags — stringify ONLY what JSON cannot carry (NaN/±Infinity, bigint, BSON Decimal128/Long). Notably this DROPS the earlier >2^53-stringify rule: live ingestion keeps finite large doubles numeric, and matching live is the whole point — the harness caught that divergence. - CoercionCounter re-threaded as pure accounting (optional param, zero behavior change): clamp + stringify events counted per (rule, bag) with samples for the /report endpoint. 86/86 tests green (71 differential fixtures + engine suite); 100k end-to-end run exact after the transform change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the 3-pod drill: concurrent pods probing a shared canary table name race on CREATE/DROP and false-flag dedup as inert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A document that kills the process on every touch (OOM-class, not a clean insert rejection) previously crash-looped until the whole multi-million-doc chunk was quarantined. Now: after 3 crash-retries a splittable chunk is bisected into 4 sub-chunks instead of retried — repeated splitting converges on a <=1-minute window around the poison doc, quarantined as a tiny failed chunk while everything else migrates. Originals become 'superseded' (terminal); the null-cd sentinel and <=1-min windows quarantine directly. Includes a gated chaos hook (LEDGER_TEST_CRASH_ID) and bench/poison-drill.ts. Drill result: 20k docs + 1 poison -> converged in 25 restarts / 7 split generations to a 0.5-min 2-doc window, 19,999/20,001 migrated with the poison active, exact 20,001/20,001 after the operator fix + retry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /control/waive-dlq (optionally {ids}): explicitly accept that pending
DLQ docs will not migrate. Waived is terminal but reversible; raw docs stay
in the DLQ permanently as the record of what was excluded. Sign-off requires
pending = 0 — every entry must end resolved (fixed+replayed) or waived.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dashboard now covers the complete operator workflow, not just state:
- Action buttons wired to the /control endpoints (pause, resume, retry
failed chunks, replay DLQ, waive pending DLQ) with confirmation prompts
on the destructive ones and toast receipts that distinguish success from
HTTP errors (an error response no longer masquerades as success).
- Dead-letter queue panel: pending/resolved/waived pills with the sign-off
gate spelled out (pending must reach 0), top errors table, and expandable
per-doc samples showing the stored raw source document.
- Coercions panel: per-(rule, field) counts with before→after samples.
- New GET /api/dlq feeding the panel.
Fixes from driving it in a real browser: POST fetches now send '{}' with the
JSON content-type (Fastify 400s an empty JSON body — the button clicks were
silently failing), and toasts report non-2xx responses as failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, verification
Turns the dashboard into a console a self-hosted customer can migrate with,
not just watch:
- Migration Guide tab: the runbook as a guided checklist. Automated phases
report their own status (index coverage, dry-run state, live progress);
manual phases (Prepare, Cutover) are persistent checkboxes (localStorage).
Sign-off is three explicit gates: all chunks done, DLQ pending = 0, full
verification passed.
- Preflight (GET /api/preflight + button): MongoDB reachability, per-
collection {cd,_id} index coverage, doc estimates, ClickHouse target
existence, dedup-canary verdict, dry-run status — read-only, run anytime.
- One-click verification (GET /api/verify + button): every completed chunk
recounted against the live table, plus table totals and duplicate check.
Exact; feeds the sign-off gate.
- Help & Recovery tab: the runbook's incident scenarios as expandable
entries with the relevant action buttons inline (incl. a cross-tab jump
to verification).
- Two-step confirmation replaces native confirm() dialogs: first click arms
the button (auto-disarms after 4s), second click fires. Testable,
consistent, no browser dialogs.
Every element driven and verified in a real browser: tabs, preflight,
checkbox persistence across reload, verify (6 chunks, 0 duplicates, gates
updating truthfully mid-incident), DLQ raw-doc expander, arm/disarm/fire on
waive (receipt {"waived":505}, gate flip to ready-for-sign-off), replay,
pause, resume receipts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-service & scaling:
- Pods panel (GET /api/pods): per-pod chunks done/active/last-seen with
alive/gone pills; README gains a scaling guide (pods scale across
machines — one pod is CPU-bound; find the ceiling by adding pods and
watching per-pod docs/s).
- Preflight became actionable: POST /control/build-indexes builds missing
{cd,_id} indexes server-side with live progress (GET /api/index-progress,
incl. $currentOp build percentage); POST /control/dry-run runs the sampled
rehearsal in-process with its own reader (guarded while migrating);
both wired into the Guide tab.
- New preflight checks: replica-set detection with a secondaryPreferred
suggestion (source is frozen after cutover — secondary reads are exact),
MongoDB and ClickHouse disk headroom (the #1 preventable incident).
Chaos-verified (scratch containers + TCP chaos proxy; shared dev services
untouched), all with exact final counts:
- mongod hard-killed 8s mid-run (OOM/crash surface): self-healed, driver
retry layer alone absorbed it
- mongod CPU-starved to 0.15 cores mid-run: slowed, completed exact
- ClickHouse unreachable 8s mid-run: insert retries rode it out
Disk-full stance documented in the classifier: capacity errors are
transient → retries → attempts → breaker pause → operator frees space →
retry-failed.
Found & fixed by the chaos run: estimatedDocumentCount resets after an
unclean mongod shutdown, which collapsed chunk sizing to one mega-chunk.
Chunk count now also floors by time span (LEDGER_MAX_CHUNK_DAYS, default 7)
so a bad estimate can never produce a whole-collection chunk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ge, state location GET /api/config surfaces the sizing knobs (chunk target, max chunk days, page size, insert window, lease, breaker, read preference) with current vs default values and guidance, plus where progress state physically lives (mig_ranges / mig_dlq_docs in MANIFEST_DB) and the recovery stance. Rendered as a Guide-tab card with 'changed' pills on non-default values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…by review) Walking the state machine for 'can any crash point produce duplicates or missing data' surfaced a real gap, no crash required: the null-cd sweep chunk had the highest idx, so newest-first claiming ran it FIRST — and its rows carry cd derived from ts, landing inside regular chunks' cd windows. A regular chunk attaching afterwards saw rows in its window during verify-then-attach and skipped attaching a never-attached partition: silently missing data. (The invariant monitor would flag it, but the retry purge would then also delete sweep rows in that window.) Fixes: - The sweep is now gated: claimNext excludes the sentinel until every regular chunk of the collection is terminal (multi-pod safe — the gate counts in-flight chunks too). - retry-failed on a regular chunk of a collection whose sweep already ran also resets the sweep, purging its remaining rows precisely by id (it has no cd window of its own). - Orphaned staging tables (crash between done and drop) are swept at collection completion. - Regression test: a null-cd doc whose derived cd lands inside regular windows — exact totals now; would silently lose data before this fix. Residual (documented): on dedup-inert targets only, an ack-lost crash during DLQ REPLAY can duplicate one replay batch; the canary identifies such targets and /api/verify's uniqExact check detects it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t, null-cd preflight count - docker-compose.yml: the migrator only, connecting to YOUR Mongo/ClickHouse (previous file still shipped Redis and a bundled MongoDB from the legacy architecture — misleading for setup). - .env.example: current variables only (required trio up top, common, scaling, sizing, rehearsal), replica-set read-preference note included. - README 'Setup & run': prerequisites, two start paths, then hand over to the dashboard. Explicit split: README = everything BEFORE the dashboard exists (install, env, start, automation reference); the UI = everything after (guide, actions, troubleshooting, verification); RUNBOOK.md = the cross-system cutover procedure and incident tables. - Preflight now counts null-cd outliers per collection (pass when zero, which is the expected case) and labels the doc-count estimate as span-guarded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No reason an operator should know a path fragment; the dashboard is the product's front door. Docs updated to plain http://host:port. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found while smoke-testing the new root route: pointing MONGO_DB at a
database without drill_events collections (or any orchestrator startup
error) hit process.exit(1) — taking the dashboard down with it, so a
fresh operator with a config typo saw a dead process instead of the UI.
Now the crash marks the run failed and the console stays up:
- red 'Engine stopped' banner on the dashboard with the actual error and
'fix env + restart, state is untouched' guidance
- /healthz returns {status:'error', error} for orchestration/probes
- /stats carries fatalError; status badge shows FAILED
Verified live: bad MONGO_DB → console at / renders the banner, healthz
reports the error, run resumes normally once config is fixed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d-window queries
Two things, the second found while designing the first.
1) SCOPING FIX (latent multi-collection bug). Every test and drill so far
used ONE source collection; production has many drill_events{hash}
collections all overlapping in wall-clock time, landing in one CH table
partitioned by month only. Four live-table cd-window queries were
collection-agnostic, so on real deployments:
- retry-failed's window purge DELETED sibling collections' rows
- crash-during-attach recovery could see sibling rows in the same
partition+window and skip a never-attached partition (silent loss)
- the invariant monitor and verify would false-alarm (live > expected)
Chunks now persist their ClickHouse row identity at creation
(scope_a/scope_e/scope_n — custom events map to e='[CLY]_custom' with
the name in n, so their scope is (a,e,n); internal events (a,e)):
- countLiveInCdRange / deleteLiveCdRange take the scope
- attach-recovery now checks staged row ids universally (precise for
the chunk regardless of siblings; window-count check removed)
- unresolvable collections in multi-collection runs purge by Mongo ids
and are skipped by per-window equality checks (global totals still
verify); single-collection runs keep exact unscoped semantics
Regression suite: two hashed sibling collections over the same time
range — exact migration, scoped verify clean, retry-failed leaves the
sibling untouched.
2) LEDGER REBUILD (operator request): regenerate mig_ranges from the data
itself when progress state is lost. Frozen source ⇒ chunk grid is
re-derivable; per window, exact Mongo count vs scoped live CH count:
equal→done, zero→pending, partial→failed (redo purges first). Post-
cutover live ingestion is untouchable by construction (newer cd than
every window); the tool's own null-cd sweep rows are attributed by id
and subtracted per window. Guarded: not while copying, not with other
pods active, force required to replace an existing ledger.
- POST /control/rebuild-ledger {force} + GET /api/rebuild
- Help & Recovery: 'Migration progress state lost' scenario with
two-step Rebuild button, force-overwrite path, live progress and
per-collection summary table (verified in-browser end to end)
- tests: rebuild after wipe (all done, live rows ignored), partial
window → failed → retry+resume heals exactly
Suite: 91 tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The engine was already Kubernetes-shaped — pod identity defaults to the hostname (= pod name), coordination is Mongo chunk leases with no shared service, /healthz exists, and abrupt kills are the designed recovery path — but the repo shipped no manifests, so only Docker had a concrete artifact. - k8s/migration.yaml: ConfigMap + Secret + Deployment + Service. Pods stay up after completion so the dashboard remains available for verification and sign-off; any pod shows the whole run (state is in MongoDB). - k8s/job.yaml: batch Job variant with EXIT_ON_COMPLETE=true (pods exit 0 when every chunk is terminal), generous backoffLimit since crash-redo is normal operation. - README: Kubernetes subsection in Setup & run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decision context: the countly-platform eventTransformer rewrite (claude/jovial-shannon-b3dd29) will not merge for now. Audited every divergence between this tool's transform and platform MAIN: - The tool has no code dependency on platform — it writes ClickHouse directly. Ledger, DLQ, rebuild, verify, UI: all unaffected. - Cross-query semantics already agree with main's live pipeline: custom events e='[CLY]_custom' + name in n (confirmed), uid_canon left to identity machinery on both sides, cd = history vs receive-time. - Everything else in the spec (NaN/Decimal128/Long stringification, ts heuristics, clamps, skip rules) concerns BSON-only shapes that JSON SDK ingestion can never produce — divergence is unobservable. - The rebuild's non-overlap assumption is GUARANTEED by main's behavior (cd always re-stamped to now for live rows). One real hazard documented as a guardrail instead of a code change: replaying historical drill docs through platform ingestion on main re-stamps cd to insert time → history duplicated at today's date. Added to RUNBOOK incident table and the DLQ Help scenario: replay only via the tool's Replay DLQ. normalize.ts header + differential README no longer claim a two-repo CI lock; the goldens are this repo's frozen spec, the platform PR is optional platform-side hardening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cd fix was split out of the shelved transformer-spec branch into its own minimal PR — it affects LIVE rows (Kafka offset replay and connector redelivery re-date events), not just doc replay. Guardrail wording fixed: platform-side replay of already-migrated docs is off-limits regardless of that fix, since the live table does not dedup by _id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live ingestion on platform main is at-least-once (connector exactlyOnce=false); ordinary redelivery leaves a handful of duplicate _ids until the platform's nightly EventDeduplicationJob cleans them. Verify's global count-vs-uniqExact check surfaced those as bare 'duplicates: N', indistinguishable from a migration defect at sign-off. Verify now samples duplicate groups with their cd spread and classifies each against the EXACT migrated-data boundary (max chunk upper_cd from the ledger): groups entirely above it are live at-least-once artifacts (reported, do NOT fail verification — the nightly job cleans them); any group reaching below it involves migrated data and fails verification for investigation. UI verify panel shows the attribution. Also the written record of the compatibility audit against platform main-as-deployed: ingestor owns the [CLY]_custom/n mapping and cd stamping, EventDeduplicationJob's 26h/7d cd window can never scan or delete historical-cd migrated rows, and cross-cutover SDK-retry dups resolve to the older (migrated) copy when the job sees both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Arturs' question 'can new incoming data mix with migrated data in these checks?' had a real yes: cd-window checks can't mix (live cd is always newer than every migrated window), but ID-BASED checks could. An SDK retry straddling cutover lands the same _id in both stacks, in the SAME ts-month partition — so attach-recovery's staged-ids sample could see the live retry copy, conclude 'partition already attached', and skip a never-attached partition (silent loss). Duplicate attribution by cd boundary was likewise heuristic at the edge. Now provenance is a column, not an inference: - connect() adds 'migrated Bool DEFAULT false' to the live table (metadata-only ALTER, instant at any size; live inserts default false) - the INSERT layer stamps migrated=true on every row (staging + DLQ replay); the transform/goldens stay unaware — it's transport metadata - every migration-side query filters on it: staged-ids attach recovery, window counts (verify/monitor/rebuild), window purges, by-id purges, null-cd sweep attribution - verify's duplicate attribution is now exact with three verdicts: 0 migrated copies = live at-least-once artifact (nightly platform job cleans), 1 = cross-cutover retry (benign, reported), 2+ = migration defect (fails sign-off) - guard: resuming a run whose completed chunks predate the flag fails fast with the backfill recipe (checks would otherwise see zero rows) New precision test pins the loss vector: staged-ids check returns 0 when only a live retry copy of a staged _id exists, 1 once the migrated copy is live. 93 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the migrated column Team direction: no new column on the production drill_events table. The column is gone entirely (no ALTER, nothing for future rows to inherit) and the 100% migrated/live distinction is preserved by construction: cd IS the provenance marker. Migrated rows carry historical cd from the source; live rows are stamped at post-cutover insert time. A cross- cutover SDK retry shares _id with its migrated twin but can never share cd — so where an id alone is ambiguous, checks match (_id, cd) pairs: - attach-recovery (the loss vector): staged (_id, cd) pairs vs live — the retry copy is invisible to it, the chunk's own promoted rows are matched exactly - purges: deleteLiveByPairs (parallel arrays zipped server-side — Array(Tuple) params don't parse over HTTP); the null-cd sweep purge reconstructs its ts-derived cd values so even it is pair-exact - verify's duplicate attribution: classified against the ledger's end-of-migrated-data boundary (max chunk upper_cd) — same three verdicts (live artifact / cross-cutover retry / migration defect) - window counts need no pairs at all: the historical cd range excludes live rows and is minmax-index-accelerated (cheaper than any flag) New preflight check guards the one assumption this rests on: 'Source frozen & clocks sane' fails if the newest source cd is within 60s of ClickHouse server time (source still ingesting, or skewed clocks would blur the boundary). Cost note: pair matching runs only on rare recovery/purge paths where the partition scan dominates either way; the hot checks use the cd minmax index. Nothing is stamped on future live rows, ever. 93 tests green (attach-recovery precision test now pins pair semantics). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e 26.4) CI previously ran only typecheck + the pure differential harness; the 93-test suite (engine e2e, multi-collection scoping, rebuild, duplicate attribution, pair-based recovery) now runs against service containers pinned to the production ClickHouse version (26.4, per countly-platform deploy/compose/images.standard.env). 26.4 requires a password for non-localhost clients, so the tests accept TEST_CLICKHOUSE_URL/ TEST_CLICKHOUSE_PASSWORD (defaults unchanged for local runs). Verified on GKE the same day: image runs on a dedicated cluster against CH 26.4 — 120,200 docs exact, all chunks promoted via real ATTACH, 2-pod run with a SIGKILL mid-flight converged exact (zero loss, zero duplicates). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live-review findings from Arturs walking the dashboard: - docs/second kept 'declining' after completion — the elapsed clock never froze, so the card showed the run average decaying while the finished engine idled. finishedAt now freezes the clock (completed/stopped/ fatal); the card shows the true run average with an 'avg' suffix, or a dash when this process copied nothing. - 'Docs migrated' now prefers the durable ledger sum (done chunks' rows_expected) over process-local counters, so a restarted engine shows 80,001 — not 0 — for a completed run. - DLQ panel now answers 'where do I run the update?': names the fix location (<manifestDb>.mig_dlq_docs — Replay re-transforms the STORED raw_doc, never the source) and each entry carries its source collection plus a copy-pasteable updateOne targeting its dlq _id. - expanded DLQ entries survive the 2s re-render (open-state preserved by dlq _id). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings from Arturs (what happens at a billion DLQ docs; why is the tool ASKING the operator to set read preference when it can decide): Scale: - /api/chunks now ships an O(collections) aggregation summary (status counts, docs done, remaining estimates per collection); full chunk details only under 2,000 chunks, else just pending/active/failed capped at 500 — a 10TB run no longer streams tens of thousands of chunk docs to the browser every 2s. Cards, bars, ETA and gates all compute from the summary; the chunk map notes when done-cells are summarized away. - DLQ panel paginates (8 per page, stable _id order, Prev/Next with 'x–y of N pending'); counts stay index-served aggregates. - CoercionCounter caps distinct (rule, field) keys at 10k with an overflow bucket — totals stay exact under pathological field-name cardinality. Self-driving checks: - MONGO_READ_PREFERENCE defaults to 'auto': the engine probes hello at startup and picks secondaryPreferred on replica sets itself (frozen source ⇒ secondary reads exact). Explicit env still wins; preflight and the config card show '(auto-selected)'. - New preflight check 'Old ingestion stopped (source frozen)': double probe of newest cd + estimated counts 4s apart — any advance fails the check and names the still-growing collections. - New preflight check 'New ingestion flowing into ClickHouse': rows with cd in the last 15 min (pass with count, warn when zero — traffic may legitimately be zero). Both checks are topology-agnostic: they read only the source handle and the target handle, so new-cluster and same-cluster migrations behave identically. 96 tests (coercion cap, frozen-probe detection, DLQ pagination added). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, full DLQ drain
Second pass of the billion-document audit, this time below the UI:
- verifyMigration recounted every window SEQUENTIALLY inside one HTTP
request — hours and a guaranteed timeout at tens of thousands of
chunks. It now runs as a background task (POST /control/verify +
GET /api/verify with {status, progress, result}), counts windows with
bounded concurrency (8), and reports live progress; the UI button
polls and shows 'checked X/Y · phase'.
- The global uniqExact(_id) + whole-table GROUP BY duplicate check can
exhaust ClickHouse memory at billions of distinct ids. Replaced with
duplicateStats(): partition-by-partition scans (external group-by
enabled) — exact for every duplicate class we act on, because copies
of the same document share their ts month (a retry RESENDS the same
event ⇒ same ts ⇒ same partition), and memory-bounded per month.
Dead countAndUniq/duplicateSample removed.
- replayDlq silently processed only the first 10,000 pending entries
(listPending's default limit) — one click on a large DLQ reported
success while replaying a fraction. Now a keyset drain (pages of 500
by _id) processes the entire queue; still-failing entries stay
pending but sort behind the advancing cursor, so it terminates.
Batch dedup tokens are keyed by the page's first _id (stable across
retries, unlike positional counters over a shifting list).
- Test fixture correction that validates the partition assumption: a
cross-cutover retry shares the original event's ts (a retry resends
the same event) — the earlier fixture gave the copy a fresh ts, which
no real duplicate has.
96 tests green; async verify exercised live end-to-end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two field crashes from devops' first dry run, same root pair: 1. socketTimeoutMS (maxTimeMs+30s ≈ 90s) killed the createIndex await on any real-sized collection. Removed: every read op already carries its own maxTimeMS, so the client-wide socket timeout protected nothing and murdered legitimately long awaits (index builds, cold window counts). Index builds now run as long as they need. 2. The crash left a half-built index, and on restart hasRequiredIndex counted it as present (listIndexes lists in-progress builds), so the engine skipped building and hinted an unfinished index: 'hint provided does not correspond to an existing index'. startIndexCreation is replaced by ensureIndex with join semantics — ALWAYS called per collection: instant no-op when ready, JOINS an in-progress identical build and waits for readiness, builds when missing. hasRequiredIndex remains for display/log purposes only; correctness no longer depends on it. Corrects the claim I gave devops earlier: 'no timeout on createIndex' was true of the command but false of the client socket config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ow window The 4 docs in devops' dry-run DLQ, made migratable. Mechanics: JS serializes integer-valued doubles below 1e21 in FIXED notation (5.26e19 → '52601586211929000000'), so ClickHouse's JSON parser infers an integer type and overflows UInt64 — while values ≥1e21 serialize in exponent notation and land as Float64 (the 9.2e25 golden). The unmigratable band is exactly [2^64, 1e21) and (-1e21, -2^63). Values in that band are now stringified losslessly (as they would have serialized) and counted under 'stringify_int64_overflow' — the same spec philosophy as NaN/Infinity/bigint/Decimal128/Long: values the target cannot carry numerically become strings, never nulls, never dropped rows. No live-parity divergence: live ingestion ERRORS on such payloads (same CH rejection), so there is no live row whose identity a migrated row must match. TRANSFORM_VERSION default bumped to v2 — devops can simply pull and press Replay DLQ: the 4 entries re-transform under v2, insert, and resolve with resolved_by_version=v2. No golden fixture sits in the window (differential suite unchanged and green). Boundary unit tests + e2e proof (overflow value lands in real CH as the expected string). Integration validated on CI — local Docker Desktop is currently down. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
countly-platform's WhereClauseConverter accesses sg fields numerically via toFloat64OrNull(CAST(field, 'String')) — string-cast first, precisely because JSON paths hold mixed Dynamic types. Through that pattern a stringified overflow value behaves identically to the number, so product filters/aggregations are unaffected by the coercion. Pinned with an executable assertion against real ClickHouse. (Also the answer to 'is mixed-type under one path a problem': the platform's standard access pattern was built for exactly that; only hand-written typed-subcolumn SQL differs, and the fix there is the same cast the platform uses.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Field bug (devops): pressing Replay DLQ during a dry run inserted the replayed rows into the REAL drill_events table — the replay path had no dry-run awareness. Consequence beyond surprise: the actual run then migrates those same docs from source, guaranteeing duplicate (_id, cd) pairs that verify flags as migration defects. Replay now honors dry-run mode: rows rehearse against the Null-engine table (full parse/type validation — the dry-run contract), the already-live pair check is skipped (meaningless against a Null target), and the Null table is ensured non-destructively so replay can run beside the main dry loop. Regression test pins: dry replay reports replayed=1, live table stays untouched, dry DLQ entry resolves. Operational note for the field: rows a PRE-FIX dry-run replay already wrote to live must be removed before the real run (TRUNCATE if live is otherwise empty pre-cutover, else delete by (_id, cd) pairs). Also in this push: LedgerStore gains claimNextGlobal / claimById / listPendingSentinels and optional-collection filters (groundwork for cross-collection pod scheduling, orchestrator switch lands separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Field shape from the first customer dry run: ~2,400 mostly-single-chunk collections at 12k docs/s on one pod — and the old scheduler could not use more pods on it: every pod walked the same collection list and waited for the current collection to finish before advancing (convoy). Arturs' design, implemented: map EVERY collection's chunk grid upfront (index ensure + bounds + initChunks per collection, idempotent — resume unchanged), then claim globally: next pending chunk anywhere, sorted (collection asc, idx desc), so pods drain a collection together and spill into the next one the instant nothing is claimable. Reservation stays a single atomic findOneAndUpdate with a lease, exactly as before. The one per-collection ordering constraint survives as its own phase: each null-cd sentinel runs only after ITS collection's regulars are terminal (gate re-checked per sentinel; guarded claimById so racing pods can't double-run a sweep). Completion = zero non-terminal chunks anywhere; recovery/lease-reclaim are now run-scoped. New in-process two-pod test: 8 single-chunk collections, both pods run concurrently — exact counts, zero duplicates, both pods complete ≥2 collections each (the convoy would have let one pod monopolize). All 104 tests green, including the sweep-ordering regression and every resume/heal choreography on the new control flow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Arturs' question on the map-upfront design: what about data still coming into OLD ingestion? Old ingestion assigns cd at write time, so post-map docs land strictly BEYOND every mapped window — existing chunks and all window checks stay valid; the data just needs chunks of its own. The run is now map → drain → re-map → ... until a pass finds nothing new: re-mapping re-discovers collections (new event types included) and appends delta chunks after each collection's high-water mark (idx continues, so delta = newest data claims first). Frozen source costs one extra cheap pass; a live source is legitimately chased, which makes bulk-before-cutover + final-drain a supported flow (runbook row added): migrate history while the old stack still serves, freeze, and the last pass drains minutes of delta instead of hours of backfill. Backward compatible in both directions: mapping is idempotent, ledger schema is unchanged (collection was always a first-class field), and an upgraded pod resumes an old ledger exactly as the soak's mid-run image swap already proved. Dry runs skip top-up (sampled rehearsal). Test: run to completion, insert late-cd docs into two collections (simulated old-ingestion tail), resume — delta chunks appended with continuing idx, drained exactly, zero duplicates. 105 tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… misclassified Mixed shape = a data-bearing base drill_events (new format, a/e/n in docs) ALONGSIDE hashed collections. verifyMigration already skips per-window equality for the unscopable base collection in multi-collection runs — but rebuild/audit lacked the same guard: an unscoped window count includes sibling collections' rows, so rebuild would have misclassified base windows (and the source audit would have false-flagged them). Now: audit reports nothing for them; rebuild marks them pending — redo is idempotent (promotion pair-checks staged rows before every attach), so recopying an already-migrated window skips attaches instead of duplicating. Live two-pod mixed-shape run (docker compose scale=2, 20 hashed collections + base with new-format docs, 530k docs): cross-collection parallelism observed mid-run (both pods in different collections in 6 sampled moments; 17 distinct collections each), exit 0 both, 530,000 rows / 530,000 unique, chunks split 21/21, base mapping exact (checkout_new → [CLY]_custom/n, [CLY]_view kept, explicit doc.n preserved), verify ok with exactly the 2 base windows unscoped-skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l_events (new format) The transform was always written against the UNION of both shapes (doc.a/e/n win, collection-hash defaults fill absence, ts unit heuristic, old-only helper fields dropped like live's whitelist) and the mixed-shape live run proved it empirically — but the golden corpus turned out to cover it barely (1 of 71 fixtures without embedded a, none exercising collection defaults). Six unit cases now pin the schema-difference matrix explicitly: defaults identity for old-shape custom + internal events, embedded-wins for new shape, seconds-unit ts, skip-not-garbage when identity is absent everywhere, and helper-field dropping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
@codex review |
Member
Author
Review findingsI found two blocking multi-pod correctness issues at head [P1] Reclaimed leases are not fenced from the previous worker
Please add a claim-generation/fencing token to every worker-owned ledger mutation (including attachment bookkeeping), include it in the staging-table identity, and abort processing whenever heartbeat or a guarded transition no longer matches the claim. Relevant code: [P1] Top-up completion has no distributed barrier before null-cd sweepsEach pod independently decides that Please persist a shared mapping/top-up generation and prevent sentinel claims until every active mapper has completed the same stable generation. Relevant code: Related
|
… barrier-lite Codex review on PR #7 found two P1s and a race at head 7b1de12; all three addressed, each with a pinning test. FENCING (P1): a worker stalled past its lease could resume after reclamation sharing the new owner's deterministic staging table and moving its claim (transitions guarded only by id+status). attempts — already atomically incremented per claim — is now the fencing token: - staging tables are generation-suffixed (…_g<attempts>): a stale worker writes only its own table, orphan sweep collects it - every worker-path transition and recordAttached carries an optimistic (pod_id, attempts) fence; recovery passes the CURRENT doc snapshot so legitimate takeover still works, while a stale ex-owner's outdated snapshot is rejected - heartbeat reports ownership; loss aborts the chunk at the next page (ClaimLostError → abandon quietly, no transitions, drop own staging) APPEND RESERVATION: racing top-up pods that observed different source maxima could interleave two grids into OVERLAPPING windows (docs migrated twice) — unordered insertMany swallowed the duplicate keys. The first delta chunk is now the reservation: exactly one insertOne wins startIdx, the loser returns 0 and re-probes. The new test reproduced the original bug (a=2 b=3 both 'succeeded') before the fix. SWEEP BARRIER-LITE (P1): a pod entering the sweep phase re-probes the sentinel's collection for delta right before claiming — a racing mapper's append sends it back to regular draining. Note the invariant has softened since pair-matched promotion: a lost race degrades to idempotent redo (attach pair-checks staged rows), not data loss — the probe is ordering defense-in-depth, so no distributed barrier needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Member
Author
|
All three findings addressed in [P1] Lease fencing — fixed as suggested.
appendChunks race — confirmed and fixed. The new regression test reproduced your exact scenario pre-fix (both racing appends "succeeded", mixed grid). The first delta chunk is now the reservation: exactly one [P1] Sweep/top-up barrier — mitigated with a lighter mechanism, and honestly re-scoped. The strict sweep-after-regulars invariant dates from when attach-recovery was window-count-based; since promotion became 🤖 Generated with Claude Code |
… sweep Concurrent INITIAL mapping had the same interleaving bug the review round fixed for top-up appends: two pods that probed a live source at different instants compute different grids, and unordered insertMany with swallowed duplicate keys interleaved them into overlapping/gapping windows. The first document (idx 0, same _id for every racer) is now the reservation — exactly one grid stands; the loser returns 0 and heals any genuine delta through the top-up path. Remainder inserts are ordered so a crash mid-init leaves a contiguous prefix the top-up pass can extend. New deterministic race tests (the divergent-grids one reproduces the original bug when run against the unfixed store): - initChunks divergent grids: exactly one coherent grid stands - initChunks identical grids (frozen source): collapse to one - sweep sentinel double-claim: one winner - retryFailed double-fire: each failed chunk retried exactly once - zombie owner mid-chunk: processChunk abandon path drops only its own staging generation, leaves the reclaimed chunk untouched Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…chaos harness The new chaos harness (real worker processes SIGKILLed at random moments, plus a forced kill between ATTACH and recordAttached) caught a 400-row duplication on its first run: recovery of a lease-expired 'attaching' chunk was not atomic, so TWO surviving pods could both pick it up from findRecoverable and both run check-then-ATTACH on the same staging partition — double-attaching it. No zombie required; pod death mid-attach plus two recoverers is an ordinary multi-pod production scenario. Fixes: - LedgerStore.reclaim: atomic single-winner takeover of a recoverable chunk (status + expired-lease filtered findOneAndUpdate, attempts bump = new claim generation). recoverOne reclaims FIRST for all three states; losers walk away, zombies are fenced out of every subsequent ledger mutation. - finishAttaching: exact pair accounting per partition (live-matching count vs staged count) replaces the sampled any>0 check — equal means already attached, zero means attach, anything else self-heals (fenced-touch, then delete matched pairs + attach fresh), including a post-attach re-count for the concurrent window. Partial promotions now converge immediately instead of silently under-recording. - Heartbeat interval is capped at lease/2 so small leases keep the heartbeat-outlives-lease contract (the 10s floor broke it below 30s). - staging: countPartitionRows, countLiveMatchingStaged (exact), and provenance-exact deleteLiveMatchingStaged. New tests: pod-chaos harness (phase A forced torn commit, phase B random 2-pod kill cycles, phase C undisturbed drain + quarantine heal, end-state exactness + verify + both audits + zero debris), reclaim single-winner race, and the double-attach heal pinned deterministically. Full suite 121 tests; chaos run green 5x consecutively. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion under random pod kills The chaos harness now runs the complete cutover-first situation: a Mongo writer keeps appending to the old source through the kill phases (top-up mapping under chaos), and a ClickHouse writer pours live rows into the target the whole time — half into the same (a,e,n) scope being migrated — through verification. End-state asserts use dynamic source counts, exclude live rows from migration exactness, and require every live-ingested row to survive untouched. CHAOS_PODS/CHAOS_CYCLES knobs; campaign green at 2, 3, and 4 pods. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eported
Devops running 4 pods saw the dashboard report ~10k docs/s while the real
aggregate was ~39k: docsPerSecond was computed from this pod's in-memory
counters only, so with N pods the stat undercounted ~N-fold (same flaw the
previous migration tool had).
The rate now comes from the shared ledger: LedgerStore.clusterRate sums
docs_read of chunks that finished in the last 120s across ALL pods. /stats
carries it as {docsPerSecond, pods, windowSec}; the dashboard
card shows the cluster rate with a pod count ('21,750 · 2 pods') while
running, and the ETA uses the same effective rate. Single-pod display and
the completed-run average are unchanged.
Verified live: real engine + a simulated second pod's ledger entries —
/stats cluster math exact, card renders the pod-count label mid-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two operator-safety flaws around multi-pod runs (the setup devops is on right now): - The rebuild guard counted a CRASHED pod's leftover claims as "active", refusing rebuild after a crash-storm — the exact scenario rebuild exists for. It now checks live leases only (activeClaims: non-terminal chunks with unexpired lease_until): dead pods' claims expire and stop blocking; genuinely working pods still block. - The two audit buttons had NO cross-pod guard at all — clicking "Audit vs source" on any pod's dashboard while other pods migrate would report a wall of false mismatches. Both audits now refuse while other pods hold live leases, with a message saying why. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ld failure) Chunks are cut on cd (server insert time) but drill_events partitions by toYYYYMM(ts) (device event time). Historical data carries garbage device clocks — 1970s epochs, far-future dates — so one cd-week of documents can span more than 100 distinct ts-months, and ClickHouse rejects such an INSERT block at the default max_partitions_per_insert_block=100. Hit in the field on two chunks of a real migration (both parked as failed after 3 attempts, exactly as designed — no data damage). Unlimited is correct for a one-off bulk load: the partitions exist in the data regardless of how the insert batches are shaped, and the setting is client-level so the staging copy path, the DLQ replay path, and the INSERT SELECT fallback are all covered. Regression test inserts 120 rows across 120 ts-months within one cd window — reproduces the exact field error on the unfixed client settings. Operator path for the parked chunks: pull this image, click Retry failed chunks — the windows redo cleanly (nothing was attached, so the purge is a no-op). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…able ceiling CI typecheck: the client types UInt64 settings as strings. And instead of unlimited (0), use 4800 — every month of the DateTime64 range (1900..2299) the transform clamps ts into. Functionally identical for legitimate data (no block can exceed the theoretical month count), but keeps the tripwire: the default 100 exists to catch misconfigured partition keys, not to cap table partitions — those come from the data regardless of batch shape. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…unk #110) The insert regression covered writing >100 ts-months into staging; this extends it through the full promote path — recovery of an 'attaching' chunk with 120 partitions: per-partition pair-checked ATTACH, per-partition recordAttached bookkeeping, chunk done, exactly one live copy per doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New outage-chaos suite (CHAOS_OUTAGE=1, dedicated throwaway containers — never the shared dev stack) SIGKILLs nothing: it restarts CLICKHOUSE mid-copy (connections reset) and freezes MONGODB for 6s (connections hang) under two live workers. First run surfaced a real liveness gap: chunks failing during the outage tripped the consecutive-failure circuit breaker, which paused BOTH pods — and a paused headless pod never resumes. Workers sat alive-but-idle with 24 pending chunks for 90s+ (data was never at risk; the end state healed via kill+retry, but a 3am network blip would have parked a 20h migration until an operator woke up). Fix — classify the pause instead of weakening the breaker: - pause(reason): 'operator' | 'breaker-transient' | 'breaker-data'. Streaks containing any KNOWN-permanent failure (classifier verdict at the failure site, source-count mismatch, fail-rate breaker, invariant trip, mass-DLQ guard) pause as breaker-data: operator-owned, exactly as before. Streaks of transient/unknown failures pause as breaker-transient. - breaker-transient arms a 15s backend probe (ClickHouse + Mongo); two consecutive healthy probes re-queue the failed chunks (retryFailed) and resume. Poison data that slipped through re-fails via the classified path and re-pauses as breaker-data — converges to operator-owned. - resume() resets the failure streak (one stray failure after resume must not instantly re-trip), stats expose pauseReason, and the dashboard status shows 'auto-resume armed' vs 'needs operator (data)'. - chaos worker now uses the config-driven retry policy (prod-like backoff absorbs multi-second blips; pod-chaos pins fast retries via env). Verified: outage suite green 4x consecutively with workers finishing on their own (exit 0, wedge watchdog never fires — asserted); deterministic test pins the probe state machine (2-probe resume + failed-chunk re-queue, and that breaker-data/operator pauses never auto-resume). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er loop live Second real finding from the outage chaos round: an ECONNREFUSED thrown by a recovery-path ClickHouse call (recoverOne dropping/promoting a reclaimed chunk during a target outage) propagated up through reclaimExpiredLeases into run() and CRASHED the pod (observed live: worker exit 1). In k8s that means pods crash-looping for the length of any CH outage. recoverOne now absorbs non-fence errors: the chunk stays claimed, the lease expires, and a later tick retries recovery — the pod never dies for a failed attempt. The outage suite now forces and proves the full self-healing loop LIVE (previously only unit-verified): tight retry budget + 90s ClickHouse stop deterministically parks chunks and trips the breaker on both pods; ledger telemetry during the outage shows them correctly frozen while probes fail; strict asserts demand 'Circuit breaker' AND 'auto-resumed' in the worker logs, exit 0, wedge watchdog silent, exact end state. Measured on the way: with PROD backoff the engine absorbed a 150s hard stop with zero failed chunks — the patience is real, which is why the test needs tight knobs to reach the breaker at all. Also: worker staging client timeout is config-driven now, and the harness captures stdout AND stderr (pino logs to stdout — the breaker evidence was invisible in stderr-only capture). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-chunk scale Field round 4. All source-count-mismatch failures sat in windows exactly one retention period old (2025-08-23..30, one year before run date): the Mongo TTL reaper deletes at the retention horizon while the migration passes it, so read > source by the reaper's progress during the chunk. Failing those chunks flaps forever (retry chases a moving target). - sourceCountGuard is now directional: source < read (shrinkage: TTL, GDPR) logs + counts (stats.sourceShrankChunks) and the chunk stays done — the migration is a snapshot at read time. source > read (UNDER-read, the true data-loss signal) still fails the chunk. Deterministic test pins both directions. - Source audit classifies live > source windows as deletionDriftWindows (reported separately), never as mismatches; live < source remains a defect. Test: delete a migrated source doc -> drift 1 / mismatches 0. Dashboard fixes (field screenshots, 73k chunks): - Per-collection bars were built from the TRUNCATED chunk list (500 active rows once total > 2000): a collection whose only listed chunk was failed rendered as 0/1 = 100% failed while its done chunks were invisible. Bars now come from the server-side summary — accurate at any scale, with an explicit 'N failed' note instead of a red takeover. - Chunk map states what it shows when truncated (active/failed window; done chunks omitted) instead of presenting a sea of grey as stalled. - ETA used a GLOBAL avg-docs-per-done-chunk; the done set is dominated by big collections while the pending tail is thousands of tiny ones — field showed 8630 min for a ~1-day remainder. Now per-collection averages with global fallback. - Failed-chunks stat card reads the ledger (cluster truth), not this pod's in-memory counter. - Long error text wraps inside its table cell; coercion samples quote strings so stringify coercions no longer display as no-ops. All verified live against a 2,668-chunk truncated-mode run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bounded migration Field requirement: customers who need approval before switching keep the OLD architecture authoritative, the new one receives a live copy, and history is backfilled up to the moment mirroring began. Mirror mode (MIRROR_MODE=true, its own pod, same image): - ONE database-level change stream filtered to the drill prefix; inserts are transformed with the SAME transform v2 as the bulk migration (cd preserved), pair-checked against live (_id, cd), and batch-inserted. Identical transform on both halves of the timeline means verify and the source audit cover migrated + mirrored data as one consistent set. - Checkpoint T0 is captured AFTER the stream opens and persisted once ($setOnInsert): overlap around the boundary is delivered by BOTH sides and converges via the pair-check — loss is structurally impossible, restarts never move the boundary. - Crash-safe: resume token saved per flushed batch; redelivery converges (pinned by a deliberate token-REWIND test). A ClickHouse outage makes the mirror lag, never drop (retry-until-landed). Oplog-rolled tokens are a loud fatal with gap-recovery instructions, not a silent gap. - Collections created after startup trigger a resolver refresh; non-insert ops (uid merges, GDPR, TTL) are counted and surfaced — the runbook's reconciliation step owns them (v1 mirrors inserts). - Idle heartbeat marks the stream at-head so dashboards distinguish "caught up" from "mirror died" during quiet hours. LEDGER_CD_UPPER_BOUND (the other half): - Mapper clamps windows to the bound; collections born after it are skipped; top-up is disabled (post-checkpoint data belongs to the mirror); clamped spans re-estimate doc counts with an exact indexed count. Preflight becomes bound-aware: "source keeps growing" is the expected state, and the sanity check moves to the bound itself. UI: Mirror card on every pod's dashboard (live/caught-up/lag, docs mirrored, non-insert ops, checkpoint with the exact LEDGER_CD_UPPER_BOUND value to copy, and a mismatch warning if a pod runs with a different or missing bound); "bounded · cd < ..." header badge. Verified live. CI: MongoDB service replaced with a step-launched single-node replica set (change streams). Tests: mirror identity/restart/rewind suite + THE SCENARIO — mirror running, source still growing, bulk migration bounded at the checkpoint → exact whole-timeline convergence with verify + audit green over the combined table. RUNBOOK: mirror-first playbook section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e bound Arturs' review of the mirror-first design surfaced two decisive flaws in the change-stream mirror: (1) not all deployments run replica-set MongoDB, and (2) copying drill DOCUMENTS is the wrong layer — the new architecture re-ingests with different logic, so a doc-level mirror fills drill_events while sessions/aggregations/profiles stay empty, making side-by-side validation meaningless. The real mirror is the nginx-level REQUEST tee that already exists: same SDK traffic re-ingested natively by both stacks. Removed: MirrorEngine, MirrorStore, MIRROR_MODE config, /api/mirror, dashboard Mirror card, mirror tests, and the CI replica-set plumbing (the tool is back to running on any MongoDB). Kept and now load-bearing: LEDGER_CD_UPPER_BOUND. With a request tee the same event exists in both systems under DIFFERENT identities (new _id, new cd), so (_id, cd) convergence cannot deduplicate across the seam — any post-flip doc the bulk migration copies is an UNDETECTABLE duplicate. The bound is the only protection: mapper never crosses it, top-up is disabled, born-after collections are skipped, preflight expects a growing source, and the header badge shows the bound on every pod. New test (no change streams needed): history + post-flip slice + a live writer growing the source THROUGH the run → exactly the pre-bound docs land (zero leaked, max cd < bound), the born-after collection is never mapped, a second full run appends nothing, verify + source audit stay green against the growing source. RUNBOOK: tee-mirror playbook — flip the tee inside a ~60s old-ingestion pause (SDKs retry; the pause creates a sharp loss-free/dup-free boundary), record the pause timestamp as the bound; fallback margin rule when a pause is impossible; GDPR/merge re-apply and retention-drift notes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"Can we auto-detect the cut-off from incoming data in both stores?" — yes, and without per-event identity (teed events carry different _id/cd on each side, and device ts is server-stamped independently when requests omit it, so matching keys are out — Arturs' catch). Rate SHAPES only: - Anchor: min(cd) in ClickHouse — the store that starts cold in BOTH tee directions (old→new and new→old), so one detector covers both. Refused when the run already mapped chunks (migrated rows poison the anchor) or when CH already holds historical data. - Per-minute count curves from both stores around the anchor. The recommended flip-inside-an-ingestion-pause leaves a zero-traffic minute on BOTH curves: the detector finds the gap, suggests a bound from inside it, and the seam is provably exact. Without a gap it suggests the anchor and QUANTIFIES the stake — how many old-side docs sit within +-2 min — so the dup/loss trade is a number the operator decides on, never hidden. - Sync parity (valid before AND after migration): hourly count parity between the stores from the boundary onward. nginx mirror is fire-and-forget — a secondary outage silently drops mirrored requests and only count parity reveals the hole; each flagged hour is a bounded backfill window. Sub-100-doc hours are below the noise floor; the still-filling current hour is never flagged. Runs as a background task (the Mongo scan spans thousands of collections); dashboard card shows the suggestion with the exact LEDGER_CD_UPPER_BOUND line to copy, the minute table around the seam (gap highlighted), and the parity verdict with flagged hours. Tests: pause-gap detection lands inside the simulated pause; no-gap mode quantifies ambiguity; mapped-run refusal (sync still reports); the simulated tee-outage hour is flagged precisely. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Without LEDGER_CD_UPPER_BOUND the migration deliberately chases newly arriving old-cluster data via top-up (correct when nothing mirrors traffic to ClickHouse — CH's existing rows play no role in mapping). The bound is only for tee setups where post-flip old-cluster data is the tee's copy. The detector suggests, never applies — stated on the card and as a mode table in the runbook. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The detected boundary can now be adopted from the dashboard: the Apply
button (double-click confirm) stores the bound in the run's shared config
(mig_run_config) and prunes the shared ledger — pending chunks fully past
the bound are deleted, a straddler is clamped to end AT the bound. Because
both the store and the grid are shared, ONE click covers every pod:
nothing beyond the bound is claimable the moment it lands, and each pod
adopts the bound itself at its next map pass (top-up disables) — no
ConfigMap edit, no rollout restart.
Safety rails:
- env stays king: with LEDGER_CD_UPPER_BOUND set, the route refuses and
points at the deployment config (one source of truth). A pod booted with
a CONFLICTING env value fails loudly ("bound conflict"), never guesses.
- refuses when any EXECUTED chunk reaches past the bound (that data may
already be live — purge/retry first, then apply).
- adoption re-prunes: a pod whose unbounded map pass raced the click may
append post-bound chunks after the apply-time prune; every adopting pod
prunes again, so stragglers die within one pass cycle.
- bound must be >60s in the past; dry runs refuse.
Tests: prune semantics (delete/clamp/refuse-on-executed), a fresh pod
adopting the stored bound end-to-end (migrates only below it, stats show
adoption), and the env-conflict fatal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The four supported scenarios, now selectable on the Guide tab (persisted) with per-scenario checklists, and as a decision matrix in the RUNBOOK: 1. Two clusters, no mirroring (plain switch) -> bound UNSET; top-up chases new old-cluster data until the final drain. 2. Two clusters, mirror old->new (old primary) -> bound SET at the tee flip; detect + one-click apply; badge on every pod; sync parity during validation; GDPR/merge re-apply before sign-off. 3. Two clusters, mirror new->old (rollback net) -> same machinery as 2; the detector is direction-agnostic (ClickHouse starts cold either way). 4. Single cluster, in-place upgrade -> bound UNSET; the upgrade froze mongo drill collections; live-parallel path with backpressure protecting the production ClickHouse. The picker states the one decision that changes configuration — tee or no tee — because a wrong bound setting is the one mistake the tool cannot detect afterwards (unset with a tee = undetectable duplicates; set without one = orphaned new data). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l collapse Field report at 1.2B rows live: docs/s collapsed to 20-200 in the tail. The tail is the many-single-chunk-collections regime (41% of chunks held the last 20% of docs), where per-chunk FIXED costs dominate — and the dominant fixed cost was the pair check: countLiveMatchingStaged filtered (_id, cd) IN (staged) against the live PARTITION with no pruning — a partition column scan over billions of rows, paid twice per chunk (pre-attach check + post-attach double-attach guard), identical cost for a 300-doc chunk and a 50k-doc one. Fix: the chunk's (a, e, n) scope plus the staged rows' ts min/max are exactly the live table's primary key (a, e, n, ts) — the pair check and the heal DELETE now carry those predicates, turning the partition scan into a keyed range read. Scopeless chunks (unresolvable collections) fall back to the old plan — correct, just slower. Also: the per-chunk statusCounts refresh aggregated over ALL chunk docs after EVERY chunk — O(n^2) across a 73k-chunk run; now throttled to once per 2s (the UI reads a snapshot; 2s staleness is free). Correctness unchanged — every attach-path test (heal, torn-commit, 120-partition, chaos, scenario) runs through the pruned queries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A ground-up replacement of the migration engine, built for the three problems that block large (10TB-class) drill_events migrations today: instability under bad data, throughput, and the operational risk of long ingestion downtime.
This is the complete solution, not a slimmed-down PoC — every improvement from the redesign is included (chunk ledger, DLQ with raw docs, verify-then-attach, poison-pill quarantine, circuit breaker, error classifier + bisection, sampled dry run, preflight incl. clock-skew check, index builds from the UI, multi-pod scaling, ledger rebuild, duplicate attribution, operator dashboard with embedded runbook, k8s manifests, CI). What the branch does NOT carry is the legacy engine: no Redis code paths, no dual-engine switches, no dead config. Reviewing against
mainis therefore a clean either/or comparison of two complete engines rather than a diff tangled through both.Architecture in one paragraph
Work is cut into cd-bounded chunks tracked in a MongoDB ledger (
mig_ranges) — ~50-100 tiny documents, claimed atomically with leases, safe for N pods with zero coordination infrastructure (Redis is gone entirely). Each chunk copies into its own ClickHouse staging table (sync inserts), is count-verified there, then promoted into the live table via verify-then-ATTACH per partition (INSERT SELECT fallback). Recovery never trusts the ledger: in-flight chunks are redone, promoted chunks are recounted, half-attached chunks are checked by staged(_id, cd)pairs. Unmigratable documents land in a DLQ with their full raw source (replay / waive from the UI); repeated crashers are auto-bisected down to the poison document; a circuit breaker pauses on systematic failure. A built-in dashboard (/) carries the whole runbook: preflight, index builds, sampled dry-run, live progress, verification gates, incident recovery — self-hosted customers can run this without us.Key correctness properties (each has a pinning test)
uniqExactcheck; continuous invariant monitor(a, e, n)identity — overlapping hashed collections cannot corrupt each other (purge/verify/recovery)cdconstruction +(_id, cd)pair matching; cross-cutover SDK retries are harmless; nothing added to the production tableRebuild ledger from dataregeneratesmig_rangesby recounting windows (Mongo vs scoped ClickHouse)Evidence
What reviewers should look at first
src/runtime/chunk-orchestrator.ts— the chunk lifecycle + every recovery pathsrc/state/ledger-store.ts— claim/lease/transition semanticssrc/target/staging-manager.ts— staging lifecycle, verify-then-attach, scoped queriesdocs/RUNBOOK.md— the cutover choreography and incident tabletests/integration/— the correctness contract, in executable formNot in this PR
Platform-side items tracked separately: countly-platform#1105 (cd passthrough, draft — interacts with EventDeduplicationJob), #722 (dedup job replay resilience). This tool depends on neither.
🤖 Generated with Claude Code