Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 54 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,19 +83,20 @@ Run a Codex turn, then open your Langfuse project to see the trace.

## Environment variables

| Variable | Required | Default | Description |
| ------------------------------------------------------------- | -------- | ---------------------------- | -------------------------------------------------------- |
| `TRACE_TO_LANGFUSE` | Yes | `false` | Set to `"true"` to enable tracing |
| `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_CODEX_PUBLIC_KEY` | Yes | — | Langfuse public key (`pk-lf-...`) |
| `LANGFUSE_SECRET_KEY` / `LANGFUSE_CODEX_SECRET_KEY` | Yes | — | Langfuse secret key (`sk-lf-...`) |
| `LANGFUSE_BASE_URL` / `LANGFUSE_CODEX_BASE_URL` | No | `https://cloud.langfuse.com` | Langfuse host / data region |
| `LANGFUSE_TRACING_ENVIRONMENT` / `LANGFUSE_CODEX_ENVIRONMENT` | No | — | Environment label for the traces (e.g. `production`) |
| `LANGFUSE_CODEX_USER_ID` | No | Codex auth email, if found | Attach a user id to all traces |
| `LANGFUSE_CODEX_TAGS` | No | — | Tags for all traces (JSON array or comma-separated) |
| `LANGFUSE_CODEX_METADATA` | No | — | JSON object of metadata to attach to all traces |
| `LANGFUSE_CODEX_MAX_CHARS` | No | `20000` | Truncate inputs/outputs longer than this many characters |
| `LANGFUSE_CODEX_DEBUG` | No | `false` | Set to `"true"` for verbose logging to stderr |
| `LANGFUSE_CODEX_FAIL_ON_ERROR` | No | `false` | Set to `"true"` to make hook upload errors fail the hook |
| Variable | Required | Default | Description |
| ------------------------------------------------------------- | -------- | ---------------------------- | -------------------------------------------------------------------- |
| `TRACE_TO_LANGFUSE` | Yes | `false` | Set to `"true"` to enable tracing |
| `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_CODEX_PUBLIC_KEY` | Yes | — | Langfuse public key (`pk-lf-...`) |
| `LANGFUSE_SECRET_KEY` / `LANGFUSE_CODEX_SECRET_KEY` | Yes | — | Langfuse secret key (`sk-lf-...`) |
| `LANGFUSE_BASE_URL` / `LANGFUSE_CODEX_BASE_URL` | No | `https://cloud.langfuse.com` | Langfuse host / data region |
| `LANGFUSE_TRACING_ENVIRONMENT` / `LANGFUSE_CODEX_ENVIRONMENT` | No | — | Environment label for the traces (e.g. `production`) |
| `LANGFUSE_CODEX_USER_ID` | No | Codex auth email, if found | Attach a user id to all traces |
| `LANGFUSE_CODEX_TAGS` | No | — | Tags for all traces (JSON array or comma-separated) |
| `LANGFUSE_CODEX_METADATA` | No | — | JSON object of metadata to attach to all traces |
| `LANGFUSE_CODEX_TRACE_SEED` | No | — | Derive deterministic trace ids ([details](#deterministic-trace-ids)) |
| `LANGFUSE_CODEX_MAX_CHARS` | No | `20000` | Truncate inputs/outputs longer than this many characters |
| `LANGFUSE_CODEX_DEBUG` | No | `false` | Set to `"true"` for verbose logging to stderr |
| `LANGFUSE_CODEX_FAIL_ON_ERROR` | No | `false` | Set to `"true"` to make hook upload errors fail the hook |

### Data regions

Expand All @@ -106,6 +107,45 @@ Run a Codex turn, then open your Langfuse project to see the trace.
| 🇯🇵 Japan | `https://jp.cloud.langfuse.com` |
| ⚕️ HIPAA | `https://hipaa.cloud.langfuse.com` |

## Deterministic trace ids

By default, trace ids are auto-generated, and an external system (a CI harness, benchmark runner, or dataset-experiment service) that runs `codex exec` headlessly has to poll the Langfuse API to discover the trace a run produced. Set `LANGFUSE_CODEX_TRACE_SEED` (or `trace_seed` in `langfuse.json`) to make trace ids predictable instead:

- **Turn N of the main thread** (1-based, in rollout order) gets the trace id `hex(sha256("<seed>:<N>")).slice(0, 32)`.
- **Turn N of a subagent thread** gets `hex(sha256("<seed>:<childThreadId>:<N>")).slice(0, 32)`, scoped by the subagent's thread id so it cannot collide with main-thread ids. (Subagent turns spawned _within_ a main-thread turn are nested inside that turn's trace as usual and don't get their own trace id.)

The main-thread formula deliberately excludes the Codex thread id, so you can compute the trace id **before** the run starts — no thread id, no polling. The derivation matches the Langfuse SDKs' `createTraceId(seed)` helper and always yields a valid W3C trace id.

**Use a unique seed per session** (e.g. a UUID or your job/run id). Reusing a seed across sessions produces colliding trace ids, and the second upload would merge into (and overwrite parts of) the first trace.

If derivation ever fails, the hook falls back to auto-generated ids and still uploads — it never blocks the session (set `LANGFUSE_CODEX_FAIL_ON_ERROR=true` while testing to surface such errors).

### Example: link a Codex run to a dataset run item

A harness can compute the trace id up front and register it with a [dataset run](https://langfuse.com/docs/evaluation/dataset-runs/native-run) — without ever fetching traces:

```bash
SEED="$(uuidgen)" # unique per codex exec invocation

# Trace id of the first main-thread turn: hex(sha256("<seed>:1")).slice(0, 32)
TRACE_ID=$(printf '%s:1' "$SEED" | shasum -a 256 | cut -c1-32)

# Link the precomputed trace id to a dataset run item before (or after) the run.
curl -s -X POST "$LANGFUSE_BASE_URL/api/public/dataset-run-items" \
-u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
-H "Content-Type: application/json" \
-d "{
\"runName\": \"codex-benchmark-2026-07-13\",
\"datasetItemId\": \"$DATASET_ITEM_ID\",
\"traceId\": \"$TRACE_ID\"
}"

# Run Codex; the Stop hook uploads the turn with exactly $TRACE_ID.
LANGFUSE_CODEX_TRACE_SEED="$SEED" codex exec "your prompt"
```

The same works from JavaScript with the Langfuse SDK: `await createTraceId(`${seed}:1`)` (from `@langfuse/tracing`) returns the identical id.

## JSON config reference

| Config key | Environment variable | Default | Description |
Expand All @@ -118,6 +158,7 @@ Run a Codex turn, then open your Langfuse project to see the trace.
| `user_id` | `LANGFUSE_CODEX_USER_ID` | Codex auth email, if found | User id for all traces |
| `tags` | `LANGFUSE_CODEX_TAGS` | — | Tags for all traces |
| `metadata` | `LANGFUSE_CODEX_METADATA` | — | Metadata object for all traces |
| `trace_seed` | `LANGFUSE_CODEX_TRACE_SEED` | — | Deterministic trace-id seed |
| `max_chars` | `LANGFUSE_CODEX_MAX_CHARS` | `20000` | Input/output truncation threshold |
| `debug` | `LANGFUSE_CODEX_DEBUG` | `false` | Verbose logging |
| `fail_on_error` | `LANGFUSE_CODEX_FAIL_ON_ERROR` | `false` | Fail the hook on upload errors |
Expand Down
59 changes: 55 additions & 4 deletions plugins/tracing/dist/index.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -4274,6 +4274,7 @@ const ConfigSchema = object({
user_id: string().optional(),
tags: array(string()).optional(),
metadata: record(string(), string()).optional(),
trace_seed: string().optional(),
max_chars: number().int().positive(),
debug: boolean(),
fail_on_error: boolean()
Expand Down Expand Up @@ -4391,6 +4392,7 @@ function readEnvConfig(env) {
user_id: env.LANGFUSE_CODEX_USER_ID,
tags: parseTags(env.LANGFUSE_CODEX_TAGS),
metadata: parseMetadata(env.LANGFUSE_CODEX_METADATA),
trace_seed: env.LANGFUSE_CODEX_TRACE_SEED,
max_chars: parseInteger(env.LANGFUSE_CODEX_MAX_CHARS),
debug: parseBoolean(env.LANGFUSE_CODEX_DEBUG),
fail_on_error: parseBoolean(env.LANGFUSE_CODEX_FAIL_ON_ERROR)
Expand Down Expand Up @@ -46540,6 +46542,17 @@ function startObservation(name, attributes, options) {
});
}
}
async function createTraceId(seed) {
if (seed) {
const data = new TextEncoder().encode(seed);
const hashBuffer = await crypto.subtle.digest("SHA-256", data);
return uint8ArrayToHex(new Uint8Array(hashBuffer)).slice(0, 32);
}
return uint8ArrayToHex(crypto.getRandomValues(new Uint8Array(16)));
}
function uint8ArrayToHex(array$1) {
return Array.from(array$1).map((b) => b.toString(16).padStart(2, "0")).join("");
}

//#endregion
//#region src/utils.ts
Expand Down Expand Up @@ -46704,7 +46717,8 @@ function parseSession(lines) {
sessionId: typeof p.id === "string" ? p.id : sessionMeta.sessionId,
cliVersion: p.cli_version,
modelProvider: p.model_provider ?? void 0,
baseInstructions: p.base_instructions?.text
baseInstructions: p.base_instructions?.text,
isSubagentThread: typeof p.parent_thread_id === "string" || p.thread_source === "subagent"
};
continue;
}
Expand Down Expand Up @@ -46843,6 +46857,7 @@ async function markTurnUploaded(rolloutFile, turnId) {

//#endregion
//#region src/trace.ts
init_esm$2();
async function loadSession(file) {
const data = await fs.readFile(file, "utf-8");
const lines = [];
Expand Down Expand Up @@ -46882,6 +46897,39 @@ async function findSubagentRollout(parentFile, threadId) {
}
return walk(root);
}
/**
* Placeholder parent span id used to pin a deterministic trace id on a root
* span (the pattern the Langfuse SDK documents for custom trace ids). The id
* never exists as a real span, so Langfuse still renders the turn as the
* trace root.
*/
const SEED_PARENT_SPAN_ID = "0123456789abcdef";
/**
* Derive the deterministic trace id for a turn from `config.trace_seed`.
*
* Main-thread turn N (1-based, rollout order): createTraceId(`${seed}:${N}`)
* Subagent-thread turn N: createTraceId(`${seed}:${threadId}:${N}`)
*
* The main-thread form deliberately excludes the thread id so external systems
* can precompute trace ids (hex(sha256(seed)).slice(0, 32)) before the Codex
* thread exists. Returns `undefined` (auto-generated ids) when no seed is set
* or derivation fails — the hook must never block an upload.
*/
async function seededTraceParent(config$1, sessionMeta, turnNumber) {
if (!config$1.trace_seed) return void 0;
try {
return {
traceId: await createTraceId(sessionMeta.isSubagentThread ? `${config$1.trace_seed}:${sessionMeta.sessionId}:${turnNumber}` : `${config$1.trace_seed}:${turnNumber}`),
spanId: SEED_PARENT_SPAN_ID,
traceFlags: TraceFlags.SAMPLED,
isRemote: true
};
} catch (error) {
debugLog("failed to derive seeded trace id; falling back to auto-generated:", error);
if (config$1.fail_on_error) throw error;
return;
}
}
function toUsageDetails(usage) {
if (!usage) return void 0;
const details = {};
Expand Down Expand Up @@ -46932,7 +46980,7 @@ async function emitTurn(turn, sessionMeta, ctx) {
}, {
asType: "agent",
startTime: new Date(turn.startTime),
parentSpanContext: ctx.parentObservation?.otelSpan.spanContext()
parentSpanContext: ctx.parentObservation?.otelSpan.spanContext() ?? ctx.seededParent
});
let previousToolResults = void 0;
for (let i = 0; i < turn.steps.length; i++) {
Expand Down Expand Up @@ -47001,8 +47049,10 @@ async function convertRollout(rolloutFile, options) {
return;
}
const uploaded = await loadUploadedTurnIds(rolloutFile);
for (const turn of turns) {
for (let turnIndex = 0; turnIndex < turns.length; turnIndex++) {
const turn = turns[turnIndex];
if (turn.completed && turn.turnId && uploaded.has(turn.turnId)) continue;
const seededParent = await seededTraceParent(options.config, sessionMeta, turnIndex + 1);
await propagateAttributes({
sessionId: sessionMeta.sessionId,
traceName: "Codex Turn",
Expand All @@ -47012,7 +47062,8 @@ async function convertRollout(rolloutFile, options) {
}, async () => {
await emitTurn(turn, sessionMeta, {
config: options.config,
rolloutFile
rolloutFile,
seededParent
});
});
if (turn.completed && turn.turnId) {
Expand Down
3 changes: 3 additions & 0 deletions plugins/tracing/src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ export const ConfigSchema = z.object({
tags: z.array(z.string()).optional(),
// LANGFUSE_CODEX_METADATA (JSON object; values coerced to strings)
metadata: z.record(z.string(), z.string()).optional(),
// LANGFUSE_CODEX_TRACE_SEED — deterministic trace ids derived from this seed
trace_seed: z.string().optional(),
// LANGFUSE_CODEX_MAX_CHARS — truncate large inputs/outputs
max_chars: z.number().int().positive(),
// LANGFUSE_CODEX_DEBUG
Expand Down Expand Up @@ -184,6 +186,7 @@ function readEnvConfig(env: Record<string, string | undefined>): Partial<Config>
user_id: env.LANGFUSE_CODEX_USER_ID,
tags: parseTags(env.LANGFUSE_CODEX_TAGS),
metadata: parseMetadata(env.LANGFUSE_CODEX_METADATA),
trace_seed: env.LANGFUSE_CODEX_TRACE_SEED,
max_chars: parseInteger(env.LANGFUSE_CODEX_MAX_CHARS),
debug: parseBoolean(env.LANGFUSE_CODEX_DEBUG),
fail_on_error: parseBoolean(env.LANGFUSE_CODEX_FAIL_ON_ERROR),
Expand Down
3 changes: 3 additions & 0 deletions plugins/tracing/src/parse.ts
Original file line number Diff line number Diff line change
Expand Up @@ -158,12 +158,15 @@ export function parseSession(lines: RolloutLine[]): {
cli_version?: string;
model_provider?: string | null;
base_instructions?: { text?: string } | null;
parent_thread_id?: string | null;
thread_source?: string | null;
};
sessionMeta = {
sessionId: typeof p.id === "string" ? p.id : sessionMeta.sessionId,
cliVersion: p.cli_version,
modelProvider: p.model_provider ?? undefined,
baseInstructions: p.base_instructions?.text,
isSubagentThread: typeof p.parent_thread_id === "string" || p.thread_source === "subagent",
};
continue;
}
Expand Down
62 changes: 59 additions & 3 deletions plugins/tracing/src/trace.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,13 @@ import type { Dirent } from "node:fs";
import * as fs from "node:fs/promises";
import * as path from "node:path";

import { propagateAttributes, startObservation, type LangfuseObservation } from "@langfuse/tracing";
import {
createTraceId,
propagateAttributes,
startObservation,
type LangfuseObservation,
} from "@langfuse/tracing";
import { TraceFlags, type SpanContext } from "@opentelemetry/api";

import type { Config } from "./config.js";
import { parseSession } from "./parse.js";
Expand Down Expand Up @@ -61,6 +67,48 @@ async function findSubagentRollout(
return walk(root);
}

/**
* Placeholder parent span id used to pin a deterministic trace id on a root
* span (the pattern the Langfuse SDK documents for custom trace ids). The id
* never exists as a real span, so Langfuse still renders the turn as the
* trace root.
*/
const SEED_PARENT_SPAN_ID = "0123456789abcdef";

/**
* Derive the deterministic trace id for a turn from `config.trace_seed`.
*
* Main-thread turn N (1-based, rollout order): createTraceId(`${seed}:${N}`)
* Subagent-thread turn N: createTraceId(`${seed}:${threadId}:${N}`)
*
* The main-thread form deliberately excludes the thread id so external systems
* can precompute trace ids (hex(sha256(seed)).slice(0, 32)) before the Codex
* thread exists. Returns `undefined` (auto-generated ids) when no seed is set
* or derivation fails — the hook must never block an upload.
*/
async function seededTraceParent(
config: Config,
sessionMeta: SessionMeta,
turnNumber: number,
): Promise<SpanContext | undefined> {
if (!config.trace_seed) return undefined;
try {
const seed = sessionMeta.isSubagentThread
? `${config.trace_seed}:${sessionMeta.sessionId}:${turnNumber}`
: `${config.trace_seed}:${turnNumber}`;
return {
traceId: await createTraceId(seed),
spanId: SEED_PARENT_SPAN_ID,
traceFlags: TraceFlags.SAMPLED,
isRemote: true,
};
} catch (error) {
debugLog("failed to derive seeded trace id; falling back to auto-generated:", error);
if (config.fail_on_error) throw error;
return undefined;
}
}

function toUsageDetails(usage: TokenUsage | undefined): Record<string, number> | undefined {
if (!usage) return undefined;
const details: Record<string, number> = {};
Expand Down Expand Up @@ -115,6 +163,8 @@ async function emitTurn(
config: Config;
rolloutFile: string;
parentObservation?: LangfuseObservation;
/** Pre-derived trace id for top-level turns (see seededTraceParent). */
seededParent?: SpanContext;
},
): Promise<void> {
const clip = makeClip(ctx.config.max_chars);
Expand All @@ -139,7 +189,7 @@ async function emitTurn(
{
asType: "agent",
startTime: new Date(turn.startTime),
parentSpanContext: ctx.parentObservation?.otelSpan.spanContext(),
parentSpanContext: ctx.parentObservation?.otelSpan.spanContext() ?? ctx.seededParent,
},
);

Expand Down Expand Up @@ -249,11 +299,16 @@ export async function convertRollout(

const uploaded = await loadUploadedTurnIds(rolloutFile);

for (const turn of turns) {
for (let turnIndex = 0; turnIndex < turns.length; turnIndex++) {
const turn = turns[turnIndex];
if (turn.completed && turn.turnId && uploaded.has(turn.turnId)) {
continue; // already uploaded in a previous hook invocation
}

// Turn numbering stays 1-based over the full rollout (including turns
// skipped by dedup above) so the derived id is stable across hook runs.
const seededParent = await seededTraceParent(options.config, sessionMeta, turnIndex + 1);

await propagateAttributes(
{
sessionId: sessionMeta.sessionId,
Expand All @@ -266,6 +321,7 @@ export async function convertRollout(
await emitTurn(turn, sessionMeta, {
config: options.config,
rolloutFile,
seededParent,
});
},
);
Expand Down
6 changes: 6 additions & 0 deletions plugins/tracing/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,12 @@ export type SessionMeta = {
cliVersion?: string;
modelProvider?: string;
baseInstructions?: string;
/**
* Whether this rollout belongs to a subagent thread rather than the main
* session. Codex marks subagent rollouts with `parent_thread_id` and/or
* `thread_source: "subagent"` in `session_meta`.
*/
isSubagentThread?: boolean;
};

/** A single tool invocation, assembled from response items + event_msg. */
Expand Down
Loading
Loading