SDK design improvements: fold operator surface into @benchsdk/runner and add ergonomics/observability

[dtice25]

Summary

This PR implements the SDK-design improvements we discussed: @benchsdk/runner is now the canonical operator package, @benchsdk/client is a compatibility re-export shim, and a set of ergonomics/observability/validation fixes round out the surface.

Breaking

• TaskStepOptions.concurrency is renamed to parallelInvocations to avoid ambiguity with config.concurrency (max in-flight tasks). The old field is still accepted and logs a one-time deprecation warning.
• runWorker is now a free function: runWorker(client, options). The legacy createBenchmarkClient().runWorker(options) spelling is preserved in @benchsdk/client.

Added

• runBenchmarkWorker(options) one-shot operator helper in @benchsdk/runner — run a single participant's worker without a *.bench.ts file.
• bench check <file.bench.ts> CLI command validates env vars, API connectivity, participant availability, and scoring weights before a run.
• validateBenchmarkConfig(config) returns structured { field, message }[]; defineBenchmarkConfig throws BenchmarkConfigError with the same shape.
• defineOnComplete(handler) helper for typed onComplete callbacks.
• RunWorkerOptions.processKey now defaults to os.hostname().
• RunWorkerOptions.onTelemetryError / BenchmarkReporterConfig.onTelemetryError callbacks expose heartbeat/log-upload/artifact telemetry failures.
• TaskError now includes step, timeoutMs, and participant context for step timeouts.

Fixed

• Worker telemetry failures now emit console.warn by default instead of failing silently.
• Step timeout diagnostics now include the step name, configured timeout, and participant name.
• @benchsdk/client public API contract test now derives the expected type/value surface from @benchsdk/runner rather than a hand-maintained list.

Package boundaries

• @benchsdk/client now depends only on @benchsdk/runner and re-exports its entire surface. createBenchmarkClient is a thin wrapper that attaches the legacy runWorker method.
• @benchsdk/runner re-exports the low-level worker/API primitives (runWorker, BenchmarkReporter, createBenchmarkClient, etc.) so operators can import from one package.

Verification

• pnpm typecheck passes.
• pnpm --filter @benchsdk/runner test passes (96 tests).
• pnpm --filter @benchsdk/client test passes (122 tests).
• pnpm exec bench run examples/*.bench.ts --dry-run and pnpm exec bench check examples/*.bench.ts --dry-run pass for all examples.

Link to Devin session: https://app.devin.ai/sessions/f67b6ba526fe43619684b90f0d19e045
Open in Devin Desktop: https://app.devin.ai/desktop/session/f67b6ba526fe43619684b90f0d19e045?variant=devin
Requested by: @dtice25

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 1 new potential issue.

🐛 1 issue in files not directly in the diff

🐛 Scale image build loses dependencies

pnpm install --frozen-lockfile cannot link the omitted runner workspace required by @benchsdk/client. The runner also requires the omitted CLI workspace, so scale image builds stop before bundling.

9 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)