feat: Add file-based dashboard provisioner

[ZeynelKoca]

Summary

Add a provision-dashboards task that reads .json files from a directory and upserts dashboards into MongoDB, following the existing task system pattern (same as check-alerts).

Provisioned dashboards are flagged with provisioned: true so they never overwrite user-created dashboards with the same name. Files are validated against DashboardWithoutIdSchema. Removing a file does not delete the dashboard (safe by default, same as Grafana). The task is deployment-agnostic: it reads from a directory, regardless of how files get there.

When DASHBOARD_PROVISIONER_DIR is set, entry.prod.sh automatically starts the task as an additional process alongside the API, App, and check-alerts.

Note: Users can currently edit provisioned dashboards through the UI, but changes will be overwritten on the next sync cycle. Grafana handles this by blocking saves on provisioned dashboards. Adding a similar guard would be a good follow-up to improve UX.

Variable	Required	Default	Description
DASHBOARD_PROVISIONER_DIR	Yes		Directory to read .json files from
DASHBOARD_PROVISIONER_TEAM_ID	No*		Scope to a specific team ID
DASHBOARD_PROVISIONER_ALL_TEAMS	No*	false	Set to true to provision to all teams

*One of DASHBOARD_PROVISIONER_TEAM_ID or DASHBOARD_PROVISIONER_ALL_TEAMS=true is required.

How to test locally or on Vercel

1. Create a directory with a dashboard JSON file:

   mkdir /tmp/dashboards
   echo '{"name":"Test Dashboard","tiles":[],"tags":[]}' > /tmp/dashboards/test.json

2. Run the task:
   DASHBOARD_PROVISIONER_DIR=/tmp/dashboards DASHBOARD_PROVISIONER_ALL_TEAMS=true
   ./packages/api/bin/hyperdx task provision-dashboards
3. Verify the dashboard appears in the UI
4. Modify the JSON file, run again, verify it updates
5. Delete the JSON file, run again, verify the dashboard persists

References

• Related PRs: HyperDX Dashboard Provisioner ClickHouse/ClickStack-helm-charts#190

[changeset-bot]

🦋 Changeset detected

Latest commit: 65f5509

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@ZeynelKoca is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

PR Review

The implementation is well-structured and follows existing task patterns closely. A few items worth addressing:

• ⚠️ asyncDispose() closes the mongoose connection after every cron tick → In built-in scheduler mode (RUN_SCHEDULED_TASKS_EXTERNALLY=false), asyncDispose() is called in the finally block after each tick, closing the connection, then connectDB() re-opens it on the next tick. Verify connectDB() handles reconnection after explicit close (pre-existing pattern for other tasks, but worth confirming it works here).

• ⚠️ No API-level guard for provisioned dashboards → Users can freely edit or delete provisioned dashboards via the API. The provisioner will overwrite their edits on the next sync. Consider either blocking edits to provisioned: true dashboards in the update/delete API, or at minimum surfacing the provisioned flag in the UI so users understand the behavior.

• ⚠️ Partial unique index added to an existing collection → The new { name, team } unique partial index on documents with provisioned: true will be built against existing data on first deployment. Since no existing documents have provisioned: true, this is safe, but worth verifying the index creation is non-blocking for large deployments (consider { background: true } if the schema DSL supports it).

• ✅ Shell script changes look safe — DASHBOARD_PROVISIONER_DIR is only used in a conditional check; no injection risk.

• ✅ asyncDispose() matches the HdxTask interface contract correctly.

• ✅ Test coverage is comprehensive and uses real DB (consistent with project pattern of no DB mocks).

[dhable]

Thanks for the well thought out contribution. Overall this looks good but we don't typically have long running processes running as tasks inside the api process. We have put background processing like this as a task.

This approach provides a flexible deployment. Using our full stack image, we start the tasks as separate processes and then have a CronJob library manage scheduling of execution. For more advanced deployments, these scheduled tasks can run outside the main process, like using a k8s CronJob or other scheduling system.

I think this implementation could be easily adapted to that design since the startDashboardProvisioner() function is almost a direct fit for the task system. The check-alert task also needs to access Mongo so you should be able to connect in the same way.

[ZeynelKoca]

PR Review

The implementation is well-structured and follows existing task patterns closely. A few items worth addressing:

* ⚠️ **`asyncDispose()` closes the mongoose connection after every cron tick** → In built-in scheduler mode (`RUN_SCHEDULED_TASKS_EXTERNALLY=false`), `asyncDispose()` is called in the `finally` block after each tick, closing the connection, then `connectDB()` re-opens it on the next tick. Verify `connectDB()` handles reconnection after explicit close (pre-existing pattern for other tasks, but worth confirming it works here).

* ⚠️ **No API-level guard for provisioned dashboards** → Users can freely edit or delete provisioned dashboards via the API. The provisioner will overwrite their edits on the next sync. Consider either blocking edits to `provisioned: true` dashboards in the update/delete API, or at minimum surfacing the `provisioned` flag in the UI so users understand the behavior.

* ⚠️ **Partial unique index added to an existing collection** → The new `{ name, team }` unique partial index on documents with `provisioned: true` will be built against existing data on first deployment. Since no existing documents have `provisioned: true`, this is safe, but worth verifying the index creation is non-blocking for large deployments (consider `{ background: true }` if the schema DSL supports it).

* ✅ Shell script changes look safe — `DASHBOARD_PROVISIONER_DIR` is only used in a conditional check; no injection risk.

* ✅ `asyncDispose()` matches the `HdxTask` interface contract correctly.

* ✅ Test coverage is comprehensive and uses real DB (consistent with project pattern of no DB mocks).

1. asyncDispose reconnection: tested and works as intended. Same pattern as check-alerts, which also closes and reconnects each tick via its provider
2. API guard: was already mentioned in the PR description. Recommend a follow-up PR to not overbloat this PR with code changes (since it also involves frontend work)
3. Index creation: as mentioned, no existing documents have provisioned: true, so index builds instantly

[ZeynelKoca]

Thanks for the well thought out contribution. Overall this looks good but we don't typically have long running processes running as tasks inside the api process. We have put background processing like this as a task.

This approach provides a flexible deployment. Using our full stack image, we start the tasks as separate processes and then have a CronJob library manage scheduling of execution. For more advanced deployments, these scheduled tasks can run outside the main process, like using a k8s CronJob or other scheduling system.

I think this implementation could be easily adapted to that design since the startDashboardProvisioner() function is almost a direct fit for the task system. The check-alert task also needs to access Mongo so you should be able to connect in the same way.

Reimplemented with the existing concurrent task system. Relevant ClickStack helm PR is also updated