feat: tag traces with skill:<name> when a SKILL.md is read by a tool call - #67
Open
lilpacy wants to merge 1 commit into
Open
feat: tag traces with skill:<name> when a SKILL.md is read by a tool call#67lilpacy wants to merge 1 commit into
lilpacy wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Trace tags are currently static: whatever
tagsthe config provides is applied to every turn. Codex sessions load skills by readingskills/<name>/SKILL.mdthrough a tool call (e.g.bash -lc "cat ~/.codex/skills/goal-first/SKILL.md"), but nothing surfaces that on the trace — so there is no way to filter or aggregate traces by skill in Langfuse without re-parsing rollout files locally.The companion Claude Code plugin already tags traces with
skill:<name>(CC_LANGFUSE_SKILL_TAGS, on by default). Running both plugins side by side, the Claude project supportstags-dimension queries like "daily usage per skill" while the Codex project cannot answer the same question, which makes cross-agent skill analytics one-sided.Fix
collectSkillTags(turn)scans each turn'stoolCallsarguments forskills/<name>/SKILL.mdand returns deduplicatedskill:<name>tags.config.tagsinto the attributes passed topropagateAttributes, so configured tags keep working unchanged and turns without skill reads are unaffected (tagsis omitted when empty, as before, unless config tags exist).skill:means "read in the top-level conversation". Subagent rollouts are out of scope here (the Claude plugin keeps those in a separatesubagent-skill:namespace); this is documented on the helper as the extension point.The path pattern requires a separator (or start of string) before
skills/and the exactSKILL.mdfilename, so reads ofskills/<name>/references/*.mdor unrelated paths do not tag.Tests
Three unit tests in
trace.test.tspin the helper directly:SKILL.mdread inside shell command args is tagged,SKILL.mdpaths inside a skill directory and unrelated commands yield no tags.vitest: 43 passed.tsc --noEmitclean.dist/index.mjsrebuilt withtsdownand included in the commit, matching thelint:distcontract.