Merged
Conversation
- Fix outgoing links bypassing shouldSkipTracking when disabled/bot - All plugins (interactions, scroll-depth, errors) now return cleanup functions, wired into destroy() via cleanupFns - Fix HttpClient double-read of response body (use text+parse instead) - Fix flush race condition: check isFlushing before clearing timer - destroy() flushes all queues via sendBeacon before clearing - handlePageUnload uses sendBeacon with fetch fallback for all queues - databuddyOptIn reinitializes tracker without requiring page reload - Cache timezone at init instead of creating Intl.DateTimeFormat per event - Add regression tests for all fixed bugs
Remove unused R2 storage and Logtail env vars.
Upgrade to schema v2, switch to strict envMode with explicit globalEnv, simplify task configs by removing redundant fields.
Rename typecheck/type-check to check-types across packages, use turbo for test runner, remove unused root dependencies (opentelemetry, maxmind).
Replace adapter class pattern with plain mapUmamiRow function. Add createImport helper that handles session exit detection. Remove old test script, csv-parse/zod/drizzle deps, and utils-map-events.
Align with bun test glob pattern (tests/*.test.ts).
Enable affectedUsingTaskInputs, watchUsingTaskInputs, and filterUsingTasks to prepare for Turbo 3.0.
- ci.yml: split into 3 parallel jobs (lint, check-types, test), add concurrency group, path-ignore for docs, pin bun to 1.3.4, add postgres service, remove redundant full build step - health-check.yml: add concurrency group, restrict triggers to Dockerfile and app source changes only - docker-publish.yml: switch to Blacksmith Docker tools (setup-docker-builder, build-push-action, stickydisk), use native arm64 runners instead of QEMU emulation, add concurrency group, downsize manifest runners to 2vcpu - codeql.yml: use Blacksmith runner, add staging branch, add concurrency group, remove boilerplate - dependency-review.yml: add staging branch, use Blacksmith runner
- Root check-types now delegates to turbo (packages already use tsc) - Lint set to continue-on-error until 166 pre-existing errors are fixed
check-types needs package dist outputs to resolve cross-package types
…board turbo build --filter=@databuddy/api... was resolving to 21 packages (including dashboard) due to ^build dependency traversal. turbo prune correctly scopes to only the 14 actual API dependencies.
* chore(notifications): add test scripts to package.json * test(notifications): add BaseProvider unit tests * test(notifications): add uptime template tests * test(notifications): add anomaly template tests
…ion (#375) * feat(docker): add self-hosting support with docker-compose configuration * feat(docker): update docker-compose for production readiness and security enhancements * feat(docker): enhance docker-compose with required APP_URL and GEOIP_DB_URL configurations
- basket: loop HTML tag strip in sanitizeString to defeat stacked-tag bypass (js/incomplete-multi-character-sanitization, alert #43) - tracker: replace Math.random() fallback in generateUUIDv4 with crypto.getRandomValues() so downstream IDs are cryptographically random (js/insecure-randomness, alerts #59 and #60)
…oute The outgoing-links plugin has been POSTing to `/outgoing` since 2025-12-27. basket has no such route — it serves outgoing-link events at `POST /` with `type: "outgoing_link"` in the body. Every external link click was 404'ing silently in production. Confirmed via direct ClickHouse query: zero rows in `analytics.outgoing_links` for any site that has `trackOutgoingLinks: true` enabled. Switch the plugin to: - POST `/` with `type: "outgoing_link"` (the route basket actually serves) - Send `client_id` as a query param so beacon transport works (sendBeacon strips custom headers, including `databuddy-client-id`) - Include `anonymousId`, `sessionId`, `timestamp` so clicks are attributed to a session — basket's insertOutgoingLink reads these but the old payload never sent them, so all click rows would have been anonymous even if the route had worked Verified end-to-end against prod basket: probe POST returned 200 success and the row landed in `analytics.outgoing_links` with the correct href/text/session_id (the first row ever for a real customer site).
Why this change: The /outgoing tracker bug fixed in 918d196 hid in green E2E tests for 3+ months because every spec used the same blanket mock pattern: await page.route("**/basket.databuddy.cc/*", async (route) => { await route.fulfill({ status: 200, ... }); }); A catch-all that returns 200 for any path. Assertions only checked tracker-side behaviour (`req.url().includes("/outgoing")`), so the contract between tracker and basket was never tested. What this adds: - BASKET_ROUTES allowlist in tests/test-utils.ts mirroring the routes basket actually serves (apps/basket/src/routes/{basket,track,llm}.ts) - A custom Playwright `test` exported from test-utils with an `auto: true` fixture that: 1. Tracks every basket request via `page.on("request", ...)` so the observer fires before any test mock can intercept and shadow it 2. Default-fulfills known routes 200, unknown routes 404 3. Throws at teardown if any unknown route was hit, with a clear message pointing at BASKET_ROUTES so the next dev knows where to update if a new route is genuinely added Migrating the tests: - All 16 spec files now `import { test } from "./test-utils"` instead of `@playwright/test` — auto-fixtures run for every test, no opt-in - Deleted ~15 redundant `**/basket.databuddy.cc/*` catch-all beforeEach blocks (the fixture handles them) - Refactored one test in audit-bugs.spec.ts that was using `page.route` as a request observer — now uses `page.on("request")` so it doesn't shadow the fixture's tracking - Updated outgoing-links spec predicates from `req.url().includes("/outgoing")` to `e.type === "outgoing_link"` (matches the new payload from the plugin fix and is the body-shape pattern the hardened mocks expect) - Marked edge-cases pixel-mode test as `test.fixme()` — the hardening immediately uncovered a separate real bug in pixel.ts (it routes `/batch`/`/track`/`/vitals`/`/errors` as GET image loads to paths basket only serves as POST). Tracked separately, fix is out of scope here. Verification: - 126 passed, 6 skipped, 0 failed on chromium - Temporarily reverted the outgoing-links fix and re-ran the spec: 6 of 13 outgoing-links tests fail with the exact "Tracker hit unknown basket route(s): POST /outgoing" message — proving the regression guard catches the bug class it's meant to catch
basket only serves pixel transport at GET /px.jpg — there is no GET /batch, /track, /vitals, /errors. The pixel plugin only translated the `/` endpoint to `/px.jpg` and left every other endpoint alone, so batched screen views, custom track events, vitals and errors all fired GET image loads to dead routes in pixel mode (`usePixel: true`). The new strict basket route allowlist in tests/test-utils.ts caught this the moment the hardening landed. basket's parsePixelQuery (apps/basket/src/utils/pixel.ts) dispatches on a `type` query param and only handles `track` / `outgoing_link`. So: - Always route to `/px.jpg`, never the original endpoint - Add a `pixelEventTypeFor()` mapping that translates `/`, `/batch`, `/track` → `track` and `/outgoing` → `outgoing_link` - /vitals and /errors return null and silently no-op — they have no pixel equivalent and the pre-existing GET /vitals, GET /errors behaviour was already broken in production - Set `type` as a query param so basket's parsePixelQuery dispatches correctly (overridable by an explicit `type` field on the data) - Split batched arrays into one pixel call per event since /px.jpg only accepts a single event per GET. Pre-existing behaviour silently flattened the array indices into garbage params like `0[name]=...` Updated the pixel test to assert the request actually lands at /px.jpg (not just "any GET"), which would have caught this bug if it had been written that way originally. Verified: 127 passed, 5 skipped (WebKit only), 0 failed. The pixel test that was test.fixme'd in the previous commit now passes.
Net -112 LOC across 3 files (101 added, 213 removed).
test-utils.ts (259 → 170):
- Inline setupBasketMock into the auto-fixture; nothing imports it
externally so the indirection added zero value
- Inline isKnownBasketRoute; replace the regex array with a Set<string>
keyed on `${method} ${path}` — the routes are exact matches, regex
was overkill
- Drop UnknownBasketRouteHit and BasketMock interfaces; use inline
type literals (also unused externally)
- Hoist CORS_HEADERS to module scope, it's static
- Drop the try/catch around `new URL(req.url())` — Playwright requests
always have valid URLs
- Collapse the duplicate 200/404 fulfill blocks into a single call
- Strip JSDoc walls that just restated function signatures
pixel.ts (143 → 121):
- Replace pixelEventTypeFor() with a PIXEL_TYPE_BY_ENDPOINT lookup
table — three branches into a four-line const
- Hoist flatten() out of the closure into a module-level
flattenIntoParams(); doesn't depend on any closure state
- Delete the dead `prefix === "" && key === "properties"` branch — it
called the same code path as the else branch (when prefix is "",
newKey === key, so both branches were identical)
- Strip the wall comment explaining the endpoint mapping; the const
table speaks for itself
edge-cases.spec.ts:
- Drop the wall comment on the pixel test assertion
Verified: 127 passed, 5 skipped, 0 failed. tsc + ultracite clean.
Pulls in tokenlens@1.3.1 — a small registry of LLM model metadata (context windows + per-token pricing) for the Vercel AI Gateway catalog. Used by the upcoming agent route telemetry to compute USD cost per chat turn from result.totalUsage.
Add a small summarizeAgentUsage helper that reads result.totalUsage
from the agent stream and emits raw token counts (input, output,
cache read/write, reasoning) plus best-effort USD cost via the
tokenlens Vercel AI Gateway catalog. Falls back to anthropic/claude-4-sonnet
when the exact gateway model id isn't in the catalog yet — directionally
correct, with cost_fallback flagged so analytics can correct estimated
rows later.
The agent route fires the telemetry as a parallel side effect after
result.consumeStream() — the totalUsage promise resolves once the stream
finishes, so awaiting it never blocks the response. Output goes to:
- mergeWideEvent (evlog wide-event coverage rule)
- trackAgentEvent("agent_activity", { action: "chat_usage", ... })
Failures are captured via captureError and never break the chat flow.
Also export modelNames from ai/config/models so the telemetry helper
can resolve the canonical id without re-declaring it.
- usePendingQueue / useChatLoading: remove JSDoc that just restated the function signature. - useChatLoading JSDoc referenced "post-stream metadata sync" which was removed alongside the followup suggestions earlier this session — drop it. - Trim the verbose 4-line "Token + cost telemetry. Fire-and-forget side effect..." comment in agent.ts to a single sentence; the helper name and Promise.resolve pattern carry the meaning already.
…ated features Adds full feature coverage to autumn so every billable resource is declared in one place. Server-side enforcement is wired up in follow-up tickets; this commit is config-only. New features: - monitors — already declared, now used in main plans (Free 1 / Hobby 5 / Pro 25 / Scale 50) plus expanded Pulse counts (Pulse Hobby 25 → was 10, Pulse Pro 150 → was 50). Cost driver is checks not monitors, so frequency is gated separately. - uptime_minute_checks (boolean) — gates sub-5-minute granularity. Pro+ on main plans, all Pulse plans. Free/Hobby capped at 10-min granularity. - status_pages (metered) — count cap. Hobby 1 / Pro 3 / Scale 5 / Pulse Hobby 3 / Pulse Pro 10. - status_page_custom_branding (boolean) — paid feature, was free for all. - status_page_custom_domain (boolean) — Pulse Pro only. - alarms (metered) — count cap. Hobby 5 / Pro 50 / Scale unlimited / Pulse Hobby 25 / Pulse Pro unlimited. - webhook_alert_destinations (boolean) — Pro+ and all Pulse plans. Free and Hobby get email-only. - funnels, goals, feature_flags, target_groups (metered) — moved from shared/features.ts hardcoded limits into autumn so billing is the single source of truth. - retention_analytics, error_tracking (boolean) — Hobby+, was previously only enforced in shared/features.ts. Existing features wired up properly: - seats — was declared but never attached to any plan. Now Free 2 / Hobby 5 / Pro 25 / Scale unlimited. - rbac — was declared but never attached. Now Pro+ and Scale. - sso — sso_plan addon now actually has the boolean item. Plan structure preserved: - Free $0, Hobby $9.99, Pro $49.99, Scale $99.99 (phasing out — no expansion), Pulse Hobby $14.99, Pulse Pro $49.99, SSO addon $100. - All existing event tier overage pricing kept identical. - Agent credits + rollover on Pro unchanged from prior commit.
Wire the agent route to autumn for credit enforcement now that the config is pushed. - Resolve billing customer id via getBillingCustomerId once user auth succeeds. Skipped for API-key flows (no clear billing owner). - Pre-stream check on agent_credits returns 402 OUT_OF_CREDITS if the customer has no remaining balance and no overage allowance. - Post-stream telemetry side effect now also fires autumn.track for the 4 metered token features (input/output/cache_read/cache_write). Autumn auto-deducts credits via the credit_system creditSchema mapping. - Track failures captured via captureError, never block the chat flow (Promise.allSettled around the 4 tracks). - Zero-value tracks filtered out so we don't pollute autumn with no-op events.
Drops the seats metered feature entirely from autumn.config.ts (was Free 2 / Hobby 5 / Pro 25 / Scale unlimited) and removes all seats items from every plan. Also sets PLAN_FEATURE_LIMITS.TEAM_ROLES to "unlimited" across all tiers in packages/shared/src/types/features.ts so the dashboard UI stops gating team size. Upgrade message updated to "Team members are unlimited on all plans". Seat-based pricing is off the table for Databuddy.
Add an AgentCreditBalance pill next to the chat history and new-chat buttons in the agent header. Uses the existing useUsageFeature hook from billing-provider (which proxies autumn's useCustomer). Shows: - "234 / 5,000" format with tabular-nums - Amber warning state at <20% remaining - Destructive state at 0 remaining - ∞ when the plan grants unlimited (Scale, comped orgs) - Click → /billing - Skeleton during first load - Auto-refetches 1.5s after stream finish to pick up the post-turn autumn.track decrement
Latest day appeared empty on pulse and status pages for users west of UTC because the client re-parsed backend UTC date strings through localDayjs, shifting them back a day. Uptime time-series queries also bucketed in server tz instead of user tz.
Compose `coerceMcpInput` into the schema once with `z.preprocess` instead of calling it inline at parse time. Drop redundant explanatory comments now that the wiring is the obvious shape.
Pull in `atmn` so we can `atmn push` autumn config from this repo. The companion `@useautumn-sdk.d.ts` file is regenerated on every `atmn pull` and would otherwise drift constantly — gitignore it.
Old schema: 1 credit ≈ \$0.005 of LLM compute. Free tier of 100 credits covered ~13 first-turn chats — felt stingy. New schema: 1 credit ≈ \$0.001, free 500, hobby 2.5k, pro 25k (with rollover + rescaled overage tiers). Same USD margin per plan, ~5x more perceived value. Also fold the comment blocks into /* */ syntax — atmn pull was shredding consecutive // lines into single-line paragraphs separated by blank lines, and biome was happy to keep that mess.
Telemetry: bill fresh input tokens, not the full input count. Cache
read/write were tracked as their own metered features, so the prior
code was charging cached tokens twice (once at the input rate, once
at the cache rate). UsageTelemetry now exposes fresh_input_tokens
from inputTokenDetails.noCacheTokens and the route bills that.
Tools: delete redundant tools that get_data already covers
(execute_query_builder, get_top_pages, get_link, search_links,
get_funnel/goal/annotation_by_id) and trim every remaining tool's
description + parameter describe() calls. The 151-builder list dump
moves out of the schema and into a discoverable error on unknown type.
Prompts: drop CLICKHOUSE_SCHEMA_DOCS from the always-on system prompt
and inline a minimal table hint into execute_sql_query (the only tool
that needs schema knowledge). Condense analytics rules, examples, and
chart guide to the essentials.
Dead code: triage and reflection agents were never invoked from
production — only createAgentConfig("analytics", ...) is called.
Delete the agent files, their prompts, the tool files only they
imported, and collapse createAgentConfig to a passthrough.
Result on a "hi" baseline turn: cache_write 17,322 → 7,683 tokens
(-56%), credits 13.27 → 6.38 (-52%) under the old schema. Stacked
with the credit rescale, free tier delivers ~200 turns vs ~13.
Thinking: user-selectable extended thinking via a new compact control
in the agent input footer. AgentThinking = "off" | "low" | "medium"
| "high" flows from a jotai atom (atomWithStorage so it persists)
through the chat transport into the route body, into AgentContext,
and into Anthropic's thinking.budgetTokens via providerOptions. The
route now drops temperature when thinking is enabled because Anthropic
rejects the combination. createToolLoopAgent actually threads
providerOptions through now — it was dead code before.
Layout fix: agent input footer was shifting on every submit because
KeyboardHints returned null while loading and the Stop button got
inserted between the Thinking control and Send. Hints now swap to
"Generating…" in the same slot, and Send/Stop share one slot that
toggles by state. Footer is pixel-stable across both states.
Probe harness: new apps/api/scripts/agent-cost-probe.ts that runs the
real agent pipeline (same model, same tools, same providerOptions),
supports multi-turn chats and --thinking=off|low|medium|high, and
prints token + credit breakdowns under both schemas.
Cleanup: drop the double cast on apiKey.organizationId, drop the as
LanguageModel cast on models.analytics, drop unused textareaRef in
agent-input, fold get-data's truncation into the mapping pass, remove
a leftover formatLinkForDisplay helper that only existed for the
deleted search_links output.
Back to 1 credit ≈ \$0.005 of compute (input 0.000_6, output 0.003, cache read 0.000_06, cache write 0.000_75). Plan budgets stay at the bigger 500/2500/25k numbers — users still get meaningful headroom, we just no longer subsidize the per-token math on top of that. Probe now prints runway against the three real plan sizes (free, hobby, pro) instead of comparing against a theoretical proposed schema.
Items and sections inside the sidebar nav memo were filtered by getFlag() directly, with no isHydrated guard. The FlagsProvider reads from localStorage synchronously on the first client render, so the flag store was populated on the client but empty on the server. Any flag-gated nav item (e.g. Home > Insights) was rendered on the client and absent on the server, producing React error #418 hydration mismatches on every page sharing the (main) layout: /websites, /onboarding, /demo/*, /status/*, and nested website routes. Mirror the existing filterCategoriesByFlags pattern: treat all flags as off until isHydrated is true. Server and first client paint now agree, and flag-gated items appear on the next render once hydration completes.
Thinking picker moves from Popover + button grid to DropdownMenu + DropdownMenuRadioGroup so keyboard nav + selected state come from the primitive. Header drops the inline favicon/domain and adds a thin separator before the right-side action cluster.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dashboard sidebar refactor
SidebarNavigationProviderandNavigationRendererfor cleaner sidebar architecturePerformance optimizations
searchParamsAuth hardening
customStoragewith per-route custom rules (sign-up, sign-in, forget-password, magic-link, email-otp)rateLimit()guards to all email-sending callbacks (reset password, verify email, OTP, magic link, invitation) with evloglog.warncoverage so throttled events are visible in tracessecondaryStoragefrom Better-Auth config to fix 401s on staging; Redis is now only used for rate limiting, sessions stay in PostgreSQLBasket (ingestion service)
Docker & self-hosting
SDK & tracker
apiKeyinstead ofclientIdto the SDK; updated flag method names to match current SDK surface (SDK itself was not renamed, dashboard was brought in line)CI/CD & tooling
check-types, made lint non-blockingCode quality & misc
Review fixes
Review caught and fixed in this PR:
log.warnwithauth_rate_limitedwide-event fields.chore/gstack-routingbranch for follow-up.clientIdwas renamed toapiKey. Corrected to reflect that only the dashboard callsite changed.Test plan
Verified
stagingHEAD (enforced by lint-staged pre-commit)packages/authtypechecks after rate-limit + evlog changesTo verify before merging to main
docker-compose -f docker-compose.selfhost.yml upsecondaryStorageremoval)secondaryStorageremoval lands on main