fix(producer): retry transient deterministic font fetches

[jrusso1020]

What

Retry transient deterministic Google Fonts fetch failures during distributed render planning and preserve a retryable FONT_FETCH_UNAVAILABLE error across the AWS and GCP adapters.

Why

The render-error spike was caused by transient Google Fonts CSS/woff2 502 responses. The existing path made one request, collapsed the failure into FONT_FETCH_FAILED, and caused orchestration to treat temporary dependency unavailability as terminal.

The affected fonts are reachable now. This change does not modify font aliases, URLs, or authored font data, and intentionally does not include or depend on #2768 or #2778.

How

• Retry 408/425/429/5xx, network errors, timeouts, and response-body transport failures.
• Bound each URL to 2 total attempts, 8 seconds per attempt, and one 20-second shared compile budget.
• Apply full-jitter backoff and respect Retry-After only when it fits the shared budget.
• Propagate caller cancellation through fetch attempts and retry waits without wrapping or retrying it.
• Keep deterministic 4xx/unresolved-font failures as FONT_FETCH_FAILED.
• Surface exhausted transient failures as FONT_FETCH_UNAVAILABLE.
• Normalize both codes at AWS Lambda and GCP Cloud Run boundaries while keeping only FONT_FETCH_FAILED terminal.
• Preserve the existing fail-open, single-attempt behavior for non-distributed callers.

Test plan

• 100 focused producer, compile-stage, AWS, and GCP tests pass (315 assertions).
• Producer, AWS, and GCP TypeScript typechecks pass.
• Producer, AWS, and GCP production builds passed during validation.
• Repository pre-commit lint, format, fallow, typecheck, and artifact checks pass.
• Manual production testing after package release and internal dependency bump.
• Documentation updated (not applicable; internal resilience behavior only).

Rollout

1. Merge and publish the OSS packages.
2. Merge the internal sidecar mapping PR.
3. Bump the internal sidecar from Hyperframes 0.7.78 to the newly published version and deploy.
4. Merge/deploy the Experiment Framework retry-classification PR (safe before the package bump).

Companion PRs:

• heygen-com/hyperframes-internal#533
• heygen-com/experiment-framework#43672

[james-russo-rames-d-jusso]

Reviewed at 24bc8a8.

Clean, tightly-scoped fix. The typed-error hierarchy (FontFetchError base + FontFetchUnavailableError subclass carrying code=FONT_FETCH_UNAVAILABLE), the bounded retry mechanism (2 attempts × 8s attempt-timeout × 20s shared budget, honors Retry-After only when it fits the budget, full-jitter backoff), determinism preservation (4xx not retried, still classifies as FONT_FETCH_FAILED), and the AWS/GCP boundary normalizers all read as a considered production fix rather than a shotgun retry-everywhere.

Verified:

• Determinism preserved. 4xx statuses (excluding 408/425/429) short-circuit out of the retry loop and reach fetchGoogleFont's deterministic return [] (missing-font) branch. Only retryable statuses (5xx/408/425/429) + transport errors + timeouts enter the loop; only exhaustion produces FONT_FETCH_UNAVAILABLE.
• CodeQL fix at this head is real. resolveFontCacheRoot now uses mkdtempSync(join(tmpdir(), "hyperframes-fonts-")) for the Lambda path, memoized in lambdaFontCacheRoot per warm process. Owner-only 0o700 from mkdtempSync, unguessable subdirectory. Combined with the pre-existing {flag: "wx", mode: 0o644} on the actual file write, the symlink-race surface is closed on Lambda.
• Cancellation propagation. abortSignal threads through runCompileStage → compileForRender → injectDeterministicFontFaces → fetchFontResource → AbortSignal.any merge with the per-attempt timeout signal. throwIfCallerAborted after every fetch attempt and inside waitForFontFetchRetry. The compileStage.test.ts:242-278 test does await expect(result).rejects.toBe(cancellation) — reference-equality, proving the caller's Error is propagated as-is through the whole chain.
• Test coverage. 15 retry-focused specs in the new deterministicFonts-retries.test.ts cover: retry-per-status (408/425/429/500/502/503), CSS retry recovery, woff2 retry recovery, exhaustion → UNAVAILABLE, stalled body-cancel path, response-body transport failure, per-attempt timeout, long Retry-After exceeding budget, shared elapsed budget across multiple font resources, 4xx no-retry (still FONT_FETCH_FAILED), caller cancellation without retry/wrap, fail-open one-attempt-no-typed-failure. Plus updated deterministicFonts-failClosed.test.ts for the new error types. Plus AWS/GCP boundary normalization tests. Plus the snapshot-test assertion that FONT_FETCH_UNAVAILABLE is NOT in the terminal-code set.

Concerns are all nits — inline for the specific lines.

Scope check

PR body explicitly excludes #2768 / #2778. Files touched are all deterministicFonts.ts/tests, adapter boundaries, and compileStage/plan abort-signal threading. No changes to font aliases, URLs, or authored font data. Scope check passes.

Rollout ordering

Rollout order (Merge OSS → publish → merge internal mapping → bump internal → EF classification) is safe: hf-internal#533's mapping is inert against 0.7.78 (only fires when OSS emits the new code), so #533 is mergeable independently and only activates after the OSS package publishes and the internal bumps.

— Review by Rames D Jusso

[james-russo-rames-d-jusso]

🟢 Full-jitter can produce 0-ms retries. Math.floor(Math.random() * (ceiling + 1)) where ceiling = baseDelayMs * 2 ** attempt. At baseDelayMs=250 on attempt 0, this returns [0, 250]. Roughly ~0.4% of retries will land at 0-ms delay.

For a real Google Fonts 502 storm with a large HF install base retrying concurrently, full-jitter's whole point is to spread the retry wave across the delay window. A 0-ms delay is a legitimate outcome of the full-jitter formula (that's what "full" means), but it means every ~250th retry hammers Google immediately. Whether this matters depends on your concurrent-install scale during a 502 incident.

If the intent is "spread but never zero-delay", the fix is Math.floor(Math.random() * ceiling) + minDelayMs with a small floor (say 50ms). If full-jitter-with-zero is intentional (matching AWS's canonical full-jitter recipe), leave the code and add a short comment justifying it. Non-blocking. — Rames D Jusso

[james-russo-rames-d-jusso]

🟢 Error message says "deterministic" for both codes. FontFetchError and FontFetchUnavailableError share this factory, so both variants get the trailing sentence "Distributed renders require deterministic fonts; system-font fallback would produce non-byte-identical output."

That's accurate for FONT_FETCH_FAILED (deterministic — will always fail, needs terminal handling). For FONT_FETCH_UNAVAILABLE it's misleading: the retry is expected to succeed and produce byte-identical output. A support engineer reading a log line for an UNAVAILABLE case would reasonably conclude the render is permanently broken when it's actually retryable.

Cosmetic — worth splitting the trailing sentence into two variants (retryable → "Distributed adapters will retry; check upstream health if this persists."; terminal → the current text). — Rames D Jusso

[james-russo-rames-d-jusso]

🟢 Only the Lambda path gets mkdtempSync protection. The CodeQL flag was Lambda-specific (shared /tmp), and the fix here closes it correctly. But HYPERFRAMES_FONT_CACHE_DIR (explicit override) and the non-Lambda homedir()/.cache/hyperframes/fonts path retain the pre-fix posture — predictable path, world-visible on multi-user machines, no mkdtempSync wrapping.

For local dev this is fine — single-user machines, public font content, and the {flag: "wx"} O_EXCL flag on the actual file write prevents symlink races. For any future multi-tenant surface that plumbs HYPERFRAMES_FONT_CACHE_DIR to a shared dir, the CodeQL concern re-emerges. Worth a comment above resolveFontCacheRoot locking the invariant ("the HYPERFRAMES_FONT_CACHE_DIR override MUST be a private per-process directory"). Non-blocking. — Rames D Jusso

[miguel-heygen]

Exact-head review at 24bc8a8221a632471f2d917d4600f30a5080643e: approved. The transient/deterministic error split, bounded per-URL retries, shared deadline/abort propagation, adapter normalization, and regression coverage are coherent; exact-head CI is fully green. Rollout remains gated on publishing a newly versioned OSS producer/adapters package before the internal dependency bump activates this contract.