Skip to content

feat: trace Portkey chat completions - #2601

Draft
KarthikS (karthiksenv) wants to merge 1 commit into
braintrustdata:mainfrom
karthiksenv:codex/portkey-instrumentation
Draft

KarthikS (karthiksenv) wants to merge 1 commit into
braintrustdata:mainfrom
karthiksenv:codex/portkey-instrumentation

Conversation

@karthiksenv

Copy link
Copy Markdown

Portkey's chat.completions.create(...) calls currently produce no Braintrust generation spans. This adds the normal Orchestrion configuration, a Portkey channel/plugin, and wrapPortkey(client) for portkey-ai v3 chat completions.

Both non-streaming and stream: true calls record messages, choices, model metadata, token usage, errors, and Portkey provenance. Streams aggregate text and tool-call arguments and record time to first token. Gateway configuration and HTTP options remain separate from logged metadata. Combining the wrapper with the automatic hook emits one span, and portkey / portkey-ai disable aliases are supported.

Refs #2482. This change covers the primary chat-generation API; prompt management, legacy text completions, and other Portkey APIs are outside this PR's scope.

Validation on Windows with Node 24.18.0, pnpm 10.33.0, and the official portkey-ai@3.1.0 package:

  • Replaced the registration with the original main version: all four original automatic-tracing regression cases failed because no Portkey spans were emitted. Restored the fix afterward.
  • 156 focused tests pass, including 21 Portkey cases across manual, automatic, and combined instrumentation. These use the real installed SDK with stubbed HTTP and exercise non-streaming output, SSE aggregation, tool calls, request-option forwarding, parentage, errors, early stream exit, stream failure, disable aliases, and wrapper idempotence.
  • Source and test TypeScript checks pass; the SDK build (including declaration generation) passes.
  • Packaged Node --import .../hook.mjs smoke test emits one Portkey span; disabling portkey-ai emits none.
  • Repository formatter and git diff --check pass. ESLint reports no errors (two type-assertion warnings in the wrapper; test/config files follow the existing lint exclusions).

Draft / remaining validation: No Portkey credentials were available to record real provider responses. The repository requires real-provider e2e cassettes, pinned/latest scenario variants, paired span-tree snapshots, and CI summary entries for new instrumentation. That provider-backed scenario coverage is still outstanding; the synthetic unit-test responses above are not provider recordings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant