Skip to content

feat: tag traces with skill:<name> when a SKILL.md is read by a tool call - #67

Open
lilpacy wants to merge 1 commit into
langfuse:mainfrom
lilpacy:feat/skill-read-tags-upstream
Open

feat: tag traces with skill:<name> when a SKILL.md is read by a tool call#67
lilpacy wants to merge 1 commit into
langfuse:mainfrom
lilpacy:feat/skill-read-tags-upstream

Conversation

@lilpacy

@lilpacy lilpacy commented Sep 3, 2026

Copy link
Copy Markdown

Problem

Trace tags are currently static: whatever tags the config provides is applied to every turn. Codex sessions load skills by reading skills/<name>/SKILL.md through a tool call (e.g. bash -lc "cat ~/.codex/skills/goal-first/SKILL.md"), but nothing surfaces that on the trace — so there is no way to filter or aggregate traces by skill in Langfuse without re-parsing rollout files locally.

The companion Claude Code plugin already tags traces with skill:<name> (CC_LANGFUSE_SKILL_TAGS, on by default). Running both plugins side by side, the Claude project supports tags-dimension queries like "daily usage per skill" while the Codex project cannot answer the same question, which makes cross-agent skill analytics one-sided.

Fix

  • collectSkillTags(turn) scans each turn's toolCalls arguments for skills/<name>/SKILL.md and returns deduplicated skill:<name> tags.
  • The per-turn tags are merged after config.tags into the attributes passed to propagateAttributes, so configured tags keep working unchanged and turns without skill reads are unaffected (tags is omitted when empty, as before, unless config tags exist).
  • Semantics deliberately match the Claude Code plugin: skill: means "read in the top-level conversation". Subagent rollouts are out of scope here (the Claude plugin keeps those in a separate subagent-skill: namespace); this is documented on the helper as the extension point.

The path pattern requires a separator (or start of string) before skills/ and the exact SKILL.md filename, so reads of skills/<name>/references/*.md or unrelated paths do not tag.

Tests

Three unit tests in trace.test.ts pin the helper directly:

  • a SKILL.md read inside shell command args is tagged,
  • repeated reads dedupe and multiple skills collect in encounter order,
  • non-SKILL.md paths inside a skill directory and unrelated commands yield no tags.

vitest: 43 passed. tsc --noEmit clean. dist/index.mjs rebuilt with tsdown and included in the commit, matching the lint:dist contract.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant