feat(core): percentage-based canary rollouts + calibration experiment

docs/contributing/canary-rollouts.mdx

@@ -0,0 +1,310 @@
+---
+title: "Canary rollouts"
+description: "Ship a change to a percentage of installs instead of all-or-nothing."
+---
+
+Most flags in this repo are binary: a feature is off (and therefore never
+exercised on real traffic) or on for everyone (and therefore a fleet-wide
+bet). A canary is the rung in between — the same change, enabled for a stable
+slice of installs, ramped as the signal holds.
+
+<Note>
+  **Canaries are part of telemetry, not a separate system.** A canary is a
+  *measured* rollout: a slice is enrolled specifically so it can be compared
+  against everyone else, and the comparison is what makes ramping safe. An
+  install that reports nothing cannot be compared, so **opting out of telemetry
+  opts you out of canaries** — via `hyperframes telemetry disable`,
+  `HYPERFRAMES_NO_TELEMETRY=1`, `DO_NOT_TRACK=1`, a dev build, or Studio's
+  `hyperframes-studio:telemetryDisabled`. See the
+  [`telemetry` command](/packages/cli#telemetry) for what is collected and how
+  to turn it off; everything on this page sits behind that switch.
+</Note>
+
+## Add one
+
+**1. Register it at 0%.** In `packages/core/src/canaryRegistry.ts`:
+
+```ts
+{
+  name: "my-feature",
+  percentage: 0,
+  description: "One line on what turning this on actually changes.",
+  owner: "your-handle",
+  sunsetAfter: "2026-12-01",
+}
+```
+
+At `0` it is inert, so this lands safely on its own.
+
+**2. Read it at the decision point.**
+
+```ts
+import { isCanaryEnabled } from "../telemetry/canary.js";
+
+if (isCanaryEnabled("my-feature")) {
+  // new path
+} else {
+  // existing path
+}
+```
+
+That is the whole API. The percentage lives in the registry, never at the
+call site.
+
+**3. Ramp it** by editing `percentage` in a patch release: `0 → 5 → 25 → 100`.
+
+**4. Delete it** once it is at 100 and holding — both the registry entry and
+the branch it guarded. `sunsetAfter` exists to force this: a test fails once
+the date passes.
+
+## Overriding
+
+```bash
+HF_CANARY_MY_FEATURE=on   # or off / true / false / 1 / 0 / yes / no
+```
+
+Upper-snake-case the name. An override always wins over the percentage, in
+both directions — use it for support escalations, dogfooding, bisects, or a
+panic-off.
+
+## Measuring
+
+Every telemetry event carries the assignment as a flag-shaped property:
+
+```
+$feature/canary-my-feature: "true" | "false"
+```
+
+`$feature/<key>` is the de-facto flag-property convention in analytics
+tooling, so whoever operates the telemetry backend can split any metric by
+cohort with **nothing configured server-side** — while the decision itself
+still happens locally and offline, which the render path requires.
+(Assignments ride the same anonymous, opt-out telemetry pipeline as every
+other event — and disabling telemetry disables the **enrolment**, not just the
+reporting: an opted-out install is never bucketed at all. See the note at the
+top of this page.)
+
+Two details worth knowing:
+
+- **Both arms are emitted.** A non-enrolled install reports `"false"`, not a
+  missing property. Absent means *this build predates the canary*, which is a
+  different fact from *this install is control* — collapsing them makes a ramp
+  unreadable.
+- **Keys are namespaced with `canary-`.** The `$feature/` namespace is shared
+  with real product feature flags elsewhere in the analytics pipeline. The
+  infix guarantees a canary can never alias one and fight it for the same
+  property.
+
+## Cumulative exposure — the number that actually bounds blast radius
+
+The instantaneous share holds at the target forever: enrolment is a pure
+function of `(feature, installId, percentage)` and fresh ids are uniformly
+random, so ~10% of active installs and ~10% of renders are enrolled on any
+given day. That part does not drift.
+
+**The cumulative set does grow.** Install ids churn — measured on the desktop
+render population, there are ~25× more distinct ids over 30 days than in any
+single day. Ten percent of a pool that keeps turning over is a steadily larger
+group of installs that have been enrolled *at some point*. If a single person
+cycles through N ids during a rollout, their chance of having been exposed is
+`1 − (1 − p)^N`, so at 10%: 19% after two ids, 41% after five.
+
+So the number that bounds blast radius is *distinct installs ever enrolled
+during the window*, not the instantaneous rate. Two practical consequences:
+
+- **Keep canaries short.** Drift compounds with time; a 5-day window at 10% is
+  far tighter than a 60-day one.
+- **The percentage is not the safety mechanism.** It bounds *initial* exposure
+  and decays from there. Per-render verification and per-install circuit
+  breakers are what actually bound harm. A breaker's tripped state lives in
+  `install-state.json`, separate from `config.json` and merged back in on every
+  read, so a `config.json` re-mint cannot re-enrol an install into a path that
+  already failed it. Deleting `~/.hyperframes` clears both, deliberately —
+  that is the user's reset.
+
+For *measurement* — "is this feature better?" — churn is harmless: re-bucketing
+is random, so it adds noise, not bias. It is specifically the blast-radius
+guarantee that degrades.
+
+## Calibration: validating the mechanism in the wild
+
+Two inert canaries — `calibration-10` (10%) and `calibration-50` (50%) — gate
+nothing and exist only to prove the mechanism behaves as designed on real
+traffic. Their percentages stay FIXED for the whole window, which is what
+makes check 3 below meaningful: while a percentage is constant, a cohort flip
+is a bug, whereas during a real ramp a `false → true` flip is correct and
+expected.
+
+Four checks, written down before the data arrives so the read is not post-hoc.
+(The maintainers monitor these on an internal dashboard; the definitions live
+here so the experiment's terms are public and fixed.)
+
+**1. Accuracy — does 10% mean 10%?** Measured install-weighted AND
+event-weighted separately: render volume is heavily skewed toward a few heavy
+installs, so the two can differ even when bucketing is perfect. Install share
+is the one that must land on target.
+
+**2. Drift — does cumulative exposure climb?** Measured over widening windows
+(1/7/14/30 days). The instantaneous share should stay flat at 10%; the
+cumulative enrolled-install count should grow with id churn.
+
+**3. Stability — does any install ever change cohort?** MUST be zero. A single
+id reporting both `true` and `false` at a fixed percentage means something is
+broken: memoization, a registry edit mid-window, or two code paths
+disagreeing. One innocent explanation to rule out first on a small-nonzero
+read: events carry the assignment but not the decision *reason*, so a dev
+toggling `HF_CANARY_CALIBRATION_10=on/off` mid-window is indistinguishable
+from a real flip. Emitting a reason property is deliberately skipped — add it
+only if this check comes back dirty.
+
+**4. Cross-surface agreement — do CLI and Studio agree for the same install?**
+A CLI-launched Studio adopts the CLI's id, so the same install must report the
+same cohort on both. Disagreement means the two bindings have diverged. This
+compares the *value reported per surface* — any disagreement here is also a
+check-3 flip, so this check's job is attribution: it isolates the flips that
+are binding divergence rather than within-surface instability.
+
+**Independence bonus:** overlap between `calibration-10` and `calibration-50`
+should be ~5% of installs (p1 x p2), not ~10% (which would mean the slices are
+correlated and every canary hits the same unlucky cohort).
+
+### What passing looks like, and what cannot be fixed
+
+Checks 1, 3 and 4 should pass outright — they are properties of the design,
+and failing any of them is a bug to fix before shipping a real rollout.
+
+Check 2 will NOT come back flat, and that is expected rather than a defect.
+Per-install cohorts never flip, but a *person* who wipes their config gets a
+new id and a fresh roll of the dice. Preventing that entirely needs a stable
+identity across resets, and both candidates were rejected: hardware
+fingerprinting correlates the cohort with hardware (fatal for a rendering
+experiment, and it survives uninstall), and account identity covers only
+~3.6% of local rendering installs. So the drift itself stands as a measured,
+accepted limit — but its worst consequence is mitigated, and its size is now
+directly measurable rather than inferred:
+
+- **Cohorts survive a config re-mint.** Canaries bucket on a dedicated
+  `bucketSeed` — not the telemetry id — held in `install-state.json` beside
+  `config.json` and write-once (the first install's seed wins forever). This
+  matters because `config.json` is rewritten on every command and every
+  render, and any parse/permission/IO failure makes the CLI mint a fresh
+  identity. The seed is never emitted, so it does not link the old id to the
+  new one.
+- **Deleting `~/.hyperframes` clears the cohort too, deliberately.** The seed
+  does not live outside the config directory, so the user's reset is a real
+  reset. A seed that outlived it would be a persistent identifier defeating
+  the only lever they have — see [the removal path](#removing-your-canary-state).
+- **A re-minted install cannot re-enter a path that already failed on that
+  machine.** The circuit breaker's tripped state is mirrored to the same
+  state file.
+- **`install_predecessor_found`** on every event says whether this install's
+  mint found a previous install's state marker — i.e. `config.json` was lost
+  while `install-state.json` survived. Its true-share is the re-mint rate
+  directly; drift from a deliberate reset, a fresh machine, a container, or a
+  new user is not linkable by any local mechanism and is not counted.
+
+With the seed carryover, check 2's residual drift comes from the
+unrecoverable buckets only — deliberate resets, fresh machines, containers,
+and genuinely new users — and canary window lengths get picked from that
+residual, not the raw turnover.
+
+## Behaviour worth knowing
+
+**Ramping is inclusive.** Widening `10 → 25` keeps everyone who was already
+in the 10. Cohorts never reshuffle, so a before/after comparison stays valid
+across a ramp.
+
+**Slices are independent per feature.** The bucket hashes `feature:seed`,
+not the seed alone — two canaries at 10% select two different 10%s. If
+they shared a slice, one unlucky cohort would receive every experiment at
+once and no two rollouts could be read apart.
+
+**Cohorts are keyed to the install directory, not to `config.json`.** The
+bucketing unit is a dedicated seed in `install-state.json`, inherited across
+`config.json` re-mints (see the calibration section) — distinct from the
+telemetry id and never emitted. It does not outlive `~/.hyperframes`.
+
+**A CLI-launched Studio adopts the CLI's decisions rather than re-deriving
+them.** The CLI publishes `window.__HF_CLI_CANARY_DECISIONS` — a
+`{ name: { enabled, forced } }` map. Re-deriving cannot agree in the cases
+that matter: telemetry off (the CLI resolves `telemetry_opt_out`, but Studio's
+opt-out is a separate localStorage flag it cannot see), an `HF_CANARY_*`
+override (env vars never cross into the browser), or no seed injected (Studio
+falls back to a different unit, so a different bucket). One render spanning
+both surfaces must not run half-enrolled.
+
+`forced` carries the provenance, and the precedence follows from it — highest
+first:
+
+1. A **forced** CLI decision (`HF_CANARY_*`). Wins outright, including over
+   this profile's opt-out, exactly as a local URL override does.
+2. A local `?hf_canary_*=` override, same reasoning.
+3. **This profile's telemetry opt-out.** Checked before any percentage
+   decision: the two surfaces have independent opt-outs, and CLI telemetry
+   being on says nothing about whether this browser profile agreed to be
+   measured. A cohort roll must never enrol an opted-out profile.
+4. The CLI's percentage decision.
+5. Local evaluation (standalone Studio, or a canary the CLI did not publish).
+
+Identity is treated differently from decisions. `__HF_CLI_DISTINCT_ID` and
+`__HF_CLI_BUCKET_SEED` are injected only for a loopback `Host` — a page that
+rebinds its hostname to `127.0.0.1` would otherwise read them as same-origin.
+The decisions map is not identifying, so it is published regardless, which
+keeps a LAN preview (`HYPERFRAMES_PREVIEW_HOST=0.0.0.0`) in agreement with the
+CLI instead of silently re-deriving.
+
+The decisions map is published even when telemetry is off — that is the case
+it exists for. It is safe to expose where the seed is not: booleans about
+features, not the value cohorts are derived from. Studio still evaluates
+locally when opened standalone, or for any canary the CLI did not publish.
+
+## Removing your canary state
+
+Both `config.json` and `install-state.json` live in `~/.hyperframes`, so:
+
+```bash
+rm -rf ~/.hyperframes    # clears telemetry id, canary cohorts, and breaker state
+```
+
+`hyperframes telemetry status` prints both paths if you want to inspect or
+delete them individually. Nothing canary-related is stored anywhere else.
+
+**Opting out of telemetry opts you out of canaries.** A canary is a *measured*
+rollout — we enrol a slice precisely so it can be compared against everyone
+else. An install that sends nothing can't be compared, so enrolling it buys no
+signal and only changes that user's code path, on an experimental feature,
+without their knowledge. Every opt-out route counts: the persisted preference
+(`hyperframes telemetry disable`), the runtime env vars
+(`HYPERFRAMES_NO_TELEMETRY`, `DO_NOT_TRACK`), dev/telemetry-disabled builds,
+and Studio's `hyperframes-studio:telemetryDisabled`. The decision resolves to
+`telemetry_opt_out` *before* bucketing, so no cohort is assigned at all.
+
+An explicit `HF_CANARY_<FEATURE>` (or `?hf_canary_<feature>=` in Studio)
+override still wins — that is a deliberate local choice, and it stays the way
+to exercise a canary with telemetry off.
+
+**It fails closed.** No install id, an unregistered name, or a CI machine all
+resolve to *not enrolled*. A canary exists to bound blast radius, so "we
+don't know who this is" must never mean "enrol everyone". CI is excluded
+because its config is regenerated per run, so its ids are ephemeral and would
+hop cohorts between runs — an explicit override still reaches it, which is
+how you test a canary in CI.
+
+**Decisions are stable and memoized.** The same install always resolves the
+same way, and the answer is fixed for the life of a process — a render that
+starts enrolled finishes enrolled, and its telemetry agrees with what ran.
+
+**Do not rename a live canary.** The name is part of the hash, so renaming
+reshuffles the cohort mid-rollout and invalidates the comparison.
+
+## Where it lives
+
+| File | Role |
+| --- | --- |
+| `packages/core/src/canary.ts` | Pure evaluator — no fs, no network, browser-safe |
+| `packages/core/src/canaryRegistry.ts` | Every rollout, with owner and sunset date |
+| `packages/cli/src/telemetry/canary.ts` | CLI binding: install id, env override, CI detection |
+
+The evaluator is deliberately dependency-free so studio and the embeddable
+player can use it too; those surfaces need their own thin binding to supply an
+id, since only the CLI has `anonymousId`.

docs/docs.json

@@ -472,6 +472,7 @@
               "contributing/release-channels",
               "contributing/changelog-process",
               "contributing/testing-local-changes",
+              "contributing/canary-rollouts",
               "contributing/studio-manual-dom-editing"
             ]
           },

docs/guides/feedback.mdx

@@ -199,7 +199,7 @@ Studio stores the equivalent state in `localStorage` under the `hyperframes-stud
 
 ### CLI — disable telemetry entirely
 
-Disabling telemetry suppresses the CLI feedback prompt and all other CLI usage tracking:
+Disabling telemetry suppresses the CLI feedback prompt, all other CLI usage tracking, and enrolment in any [canary rollout](/contributing/canary-rollouts) — an install that reports nothing is never picked for a staged rollout:
 
 ```bash
 # Via CLI command (persisted to ~/.hyperframes/config.json)

docs/packages/cli.mdx

@@ -992,6 +992,13 @@ Word-level transcripts (whisper output) are grouped into readable caption cues o
 
     Telemetry collects command names, render performance, render checkpoint/error names, aggregate browser diagnostic counts, browser initialization duration and tween count, aggregate video extraction workload counts (such as extracted frame count and VFR preflight count), example choices, and system info — including a coarse environment fingerprint (OS, kernel string, CPU/memory shape, sandbox runtime such as gVisor or Docker, and the *name* of a coding agent driving the CLI when one is detected, e.g. `claude_code` / `codex` / `cursor`). The agent name is derived from the existence of well-known environment variables; their values are never read. Telemetry redacts local paths and URL query strings from render error/checkpoint messages and does **not** collect project names, video content, or environment variable values. It collects no personally identifiable information until you sign in: when you authenticate with `hyperframes auth login`, your HeyGen account email (or your username, if your account has no email) is linked to your usage so CLI activity can be associated with your account (and your prior anonymous usage is stitched to it). Nothing else personal is collected, and this only happens after you choose to sign in. Disable all telemetry with `HYPERFRAMES_NO_TELEMETRY=1` or the command above.
 
+    Telemetry state also controls **canary enrolment**: staged rollouts pick a
+    stable slice of installs to enable a change for, and an install that reports
+    nothing cannot be compared against anyone, so opting out of telemetry opts you
+    out of canaries too. Every route counts — `hyperframes telemetry disable`,
+    `HYPERFRAMES_NO_TELEMETRY=1`, `DO_NOT_TRACK=1`, and dev builds. See
+    [Canary rollouts](/contributing/canary-rollouts).
+
     See [Feedback Collection](/guides/feedback) for how the periodic post-render prompt and Studio feedback bar work, what data they collect, and how to opt out.
 
     ### `skills`