RFC-0005 and RFC-0006: Skill Registry and Harness Integration#10
Conversation
Propose a governed, metadata-first registry for AI agent skills in MLflow with typed source pointers, lifecycle management, security scan tracking, skill groups, and federated discovery. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds RFC-0005 proposing a new governed, metadata-first Skill Registry for MLflow, including versioning, lifecycle/publish-state governance, typed source pointers (Git/OCI/ZIP), and first-class skill groups.
Changes:
- Introduces the Skill Registry conceptual model (skills, skill versions, tags, aliases) and publish-state lifecycle.
- Defines skill groups with versioned membership snapshots, plus associated tags/aliases.
- Specifies proposed DB schema (12 tables) and API surfaces (REST, Python SDK, CLI, UI).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| published skills, not the process of writing them. | ||
| - **Skill format specification.** The registry is format-agnostic. It | ||
| does not define or enforce what a skill looks like (SKILL.md, plugin | ||
| manifests, etc.). |
There was a problem hiding this comment.
Could we consider MLflow artifact storage as a first-class source type here? That would give us a natural UI upload flow and keeps the door open for more MLflow-native packaging/optimization later, even if Git and OCI stay the main paths.
There was a problem hiding this comment.
Good idea. We've added mlflow as an explicit future source type in the "Source type extensibility" section — it would allow storing skill content directly in MLflow artifact storage, providing a natural UI upload flow. It's deferred from the initial implementation to keep the registry metadata-first, but can be added as a follow-up without breaking changes. The adoption strategy's "Follow-up" section also calls this out.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| RETIRED = "retired" | ||
|
|
||
|
|
||
| class SkillSourceType(StrEnum): |
There was a problem hiding this comment.
Can we line this lifecycle up more closely with the MCP registry RFC in PR 12 (#12)? Right now this introduces draft, published, deprecated, and retired plus a separate skill-level status, which makes the cross-registry pattern feel less consistent unless there’s a strong skill-specific reason to diverge.
There was a problem hiding this comment.
Fully aligned — replaced the 4-state publish lifecycle (draft/published/deprecated/retired) with RFC-0004's 3-state model using status (active/deprecated/deleted). SkillStatus is now the only lifecycle enum. Parent entity status (Skill.status, SkillGroup.status) is derived from the latest version rather than set independently.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| does not perform scans. Scanning tools are separate. | ||
| - **Agent harness integration.** How a specific agent harness (Claude | ||
| Code, Codex, Cursor, etc.) installs or loads skills from the registry | ||
| is outside this RFC. The registry provides the metadata; harness |
There was a problem hiding this comment.
Do we want one short paragraph on external skill conventions we expect to interoperate with? Even if the registry stays format-agnostic, it would help to say whether we’re aligning with existing Claude/Cursor/GitHub-style skill packaging or deliberately avoiding standardization for now.
There was a problem hiding this comment.
Done — fully aligned with RFC-0004. Version lifecycle now uses status with three states (active, deprecated, deleted), matching MCPStatus. New versions default to active. Skill.status and SkillGroup.status are now read-only, derived from the latest version's status. Also added latest_version_alias on both Skill and SkillGroup, following RFC-0004's pattern for deterministic "latest" resolution. Store interface switched from AbstractSkillRegistryStore with @abstractmethod to SkillRegistryMixin with raise NotImplementedError, matching MCPServerRegistryMixin.
(Comment from Claude Code under the supervision of Bill Murdock.)
| #### SkillVersion | ||
|
|
||
| A versioned record containing a typed source pointer, publish state, | ||
| and tags. |
There was a problem hiding this comment.
I initially expected aliases to live on SkillVersion, but after checking MLflow model registry, the existing pattern is that aliases are owned by the top-level entity and point to a version. If the goal is consistency with MLflow and PR 12 (#12), I think this design is fine. It may just be worth making that pattern more explicit in the text.
There was a problem hiding this comment.
Aligned — set_skill_alias takes (name, alias, version) and sets the alias on the Skill entity, pointing to a specific version. Same pattern for set_skill_group_alias. This matches the MCP RFC's approach where aliases are owned by the parent entity rather than the version.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| ) | ||
| ``` | ||
|
|
||
| ## Create a skill group with a versioned membership snapshot |
There was a problem hiding this comment.
What's the justification for skill groups? Why not just using tags on skills
There was a problem hiding this comment.
Added a "Why groups instead of tags?" section to the RFC. The key reasons: (1) groups pin specific member versions for reproducibility, (2) groups have their own lifecycle and aliases independent of member status, (3) groups can reference capabilities across registries (skill registry + MCP registry), (4) groups map naturally to the "plugin" concept in agent harnesses, and (5) tags are flat key-value pairs that can't express ordered, versioned membership.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
There was a problem hiding this comment.
Quick terminology update: "skill groups" have been renamed to "skill bundles" in the latest revision, and the section is now titled "Why bundles instead of tags?" The reasoning in my earlier reply still applies, just with the updated name.
This comment was posted by Bill Murdock with assistance from Claude Code.
| trace data back to a governed skill record to understand adoption | ||
| across an organization. | ||
|
|
||
| ### Use cases |
There was a problem hiding this comment.
Could we add explicit JTBDs per persona (developer, admin etc.)? It would make it easier to evaluate which parts of the data model are required
There was a problem hiding this comment.
Done — replaced the abstract use case bullets with three end-to-end persona flows: a platform administrator (registers, scans, groups, aliases, deprecates), a developer (searches, resolves alias, pulls), and a security engineer (queries scan tags, deprecates failed versions, tracks compliance). These ground the data model in concrete user workflows.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
There was a problem hiding this comment.
The persona flows have been updated in the latest revision to reflect the new entity model. "Skill groups" are now "skill bundles," and the platform administrator scenario now references subagents explicitly (e.g., bundling a code-review skill with a security-auditor subagent and a GitHub MCP server). The three personas and their workflows are otherwise unchanged.
This comment was posted by Bill Murdock with assistance from Claude Code.
- Switch to YAML frontmatter to match repo template - Make ER diagram a mermaid code block - Rename source_url → source (MLflow consistency) - Rename content_hash → content_digest (OCI alignment) - Use SkillSourceType enum consistently for alias/membership types - Fix Skill.aliases type to list[SkillAlias] (was dict, missing source_type) - Fix search_skills example to use search_skill_versions (publish_state is on version) - Clarify alias resolve returns both version and source_type - Specify valid filter fields per search endpoint - Fix delete semantics: skill delete blocked when versions referenced by groups - Drop Databricks dogfooding anecdote - Remove "Relationship to other AI asset registries" section - Remove "Impact on existing MLflow components" table - Condense SDK/CLI sections (defer to store interface + examples) - Tighten drawbacks and trim weaker alternatives Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove redundant group_name/group_version/workspace fields from the entity — parent identity is provided by the enclosing SkillGroupVersion. The DB schema retains those columns as FKs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update: Review Feedback Round 1Pushed changes addressing the straightforward review feedback: YAML frontmatter, mermaid ER diagram, field renames ( Still openStructural:
Scope:
Content:
(Comment from Claude Code under the supervision of Bill Murdock.) |
Version uniqueness is now (name, version) instead of (name, version, source_type). source_type and source are optional fields on SkillVersion. Cascaded this change through entities, DB schema, store interface, REST API paths, examples, and CLI. Added RFC-0006 covering harness-specific installation with adapters for Claude Code, Codex CLI, Cursor, and Antigravity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Status update (2026-04-27)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Skill Registry Harness Integration) — new:
Resolved review comments (newly replied)
Still open
(Comment from Claude Code under the supervision of Bill Murdock.) |
Move marketplace.json from Alternatives into Detailed Design with full endpoint spec, response format, configuration, and limitations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jwm4
changed the title
|
Also, I've converted this to draft given all the open issues listed above. |
- Lifecycle: publish_state → status, 3 states (active/deprecated/ deleted), derived parent status. Aligns with RFC-0004. - Add latest_version_alias to Skill and SkillGroup. - Store: AbstractSkillRegistryStore → SkillRegistryMixin with NotImplementedError. Add order_by to search methods. - Cross-registry membership: rename skill_name/skill_version to member_name/member_version, add registry field (skill/mcp). - Conditional FK for MCP refs → application-layer enforcement. - Pull clarified as client-side, removed from store mixin and REST API. - Dual MCP registration: MCP registry is default, kind=mcp-server reserved for embedded configs only. - SDK namespace: mlflow.skills → mlflow.genai.skills. - Add external skill conventions paragraph. - Add skill groups justification (why not just tags). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document `mlflow` as a deferred source type in the extensibility section and adoption strategy follow-up, per review feedback. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Status update (2026-04-29)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Harness Integration):
Still open (deferred to future pass)
(Comment from Claude Code under the supervision of Bill Murdock.) |
- Replace abstract use case bullets with end-to-end persona flows
(platform admin, developer, security engineer)
- Expand security scan tracking with structured tag namespace convention
(scan.{type}.{field}) and documented fields/examples
- Add permissions section mapping operations to MLflow's READ/EDIT/MANAGE
levels, with status transitions and alias management requiring MANAGE
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Status update (2026-04-29, part 2)The four items listed as "still open" in the previous update are now all addressed:
All review comments from mprahl, serena-ruan, and Copilot now have replies. I don't have further edits planned at this time. The RFC is ready for another round of review. (Comment from Claude Code under the supervision of Bill Murdock.) |
| rfc_pr: https://github.com/mlflow/rfcs/pull/10 | ||
| --- | ||
|
|
||
| # RFC: Skill Registry |
There was a problem hiding this comment.
@Bill Murdock — Reading through the RFC and the proposed API, one thing struck me: given that the registry already supports kind=[skill, agent, mcp-server, hook], should we consider a more generic name than "skills" / "skill groups"?
The data model is already kind-agnostic — a Skill entity with kind=agent is conceptually an asset that happens to be an agent, not a skill. Calling the registry an Asset Registry (or Capability Registry) and using create_asset() / create_asset_group() instead of create_skill() / create_skill_group() would better reflect what the API actually does and avoid the naming tension where "skill" is both a specific kind and the umbrella term for all kinds.
The REST paths and CLI would follow naturally: mlflow assets create, mlflow asset-groups create-version, etc.
There was a problem hiding this comment.
+1
I'd like to also propose bundles. It satisfies the generic 'kind-agnostic' semantics of assets, while also implying plurality even in the singular term (skill-group and bundle both imply grouping of some asset).
I also agree that having an independent name (assets/bundles/capabilities) greatly improves function discoverability in the SDK
There was a problem hiding this comment.
Alternatively, if none of the names appeal, I'd suggest at least in the CLI dropping skill-groups in favor of skills with a --group flag. Something like:
mlflow skills pull --group pr-workflow --alias production
There was a problem hiding this comment.
+1 to @etirelli here, the umbrella term should not be a skill. Currently in the RFC the skill that is a kind/skill is more consistent with the industry's usage of the term, but the top level usage of the word skill would be very confusing.
My preference is asset registry or capability registry, or something equally broader.
There was a problem hiding this comment.
after further discussion last week, +1 to using the term bundle or skill bundle as per @khaledsulayman 's suggestion above.
There was a problem hiding this comment.
Resolved. Rather than renaming the umbrella term, we split the single Skill(kind=...) entity into three separate types: Skill, Subagent, and Hook. Each has full lifecycle (versions, aliases, tags, status). SkillGroup is renamed to SkillBundle with typed member lists, following the "skill bundle" suggestion from this thread. The registry is still "Skill Registry" and the SDK namespace stays mlflow.genai.skills.*. See commit d16abe0.
Bill Murdock, with assistance from Claude Code
There was a problem hiding this comment.
Adopted. SkillGroup is now SkillBundle throughout both RFCs. See commit d16abe0 for the full restructuring.
This comment was posted by Bill Murdock with assistance from Claude Code.
There was a problem hiding this comment.
Adopted a variant of this: the CLI now uses --bundle instead of --group, following the SkillGroup-to-SkillBundle rename. So mlflow skills pull --bundle pr-workflow --alias production.
This comment was posted by Bill Murdock with assistance from Claude Code.
| Add a Skill Registry to MLflow: a governed, metadata-first registry for | ||
| AI agent capabilities. The registry stores metadata and typed source | ||
| pointers (to Git repos, OCI registries, ZIP archives, etc.) rather | ||
| than artifacts directly. It provides enterprise governance on top of |
There was a problem hiding this comment.
Could we support both approaches? The user would either set pointers to external storage (Git, OCI, ZIP) or store assets directly in MLflow's internal artifact store.
The reason: source pointers are the right default for customers who already manage assets in Git or other systems. But for customers who don't have that infrastructure in place, or who already use MLflow to store models and want their agent capabilities alongside them, or who operate in airgapped or restricted environments where reaching external sources isn't practical — the ability to store assets directly in the registry would be a better fit.
There was a problem hiding this comment.
Agreed. The latest revision adds source_type="mlflow" as a first-class source type in the initial implementation. Content is stored as a directory tree of individual files in MLflow's artifact store, consistent with how model artifacts are stored. Users can choose external source pointers (Git, OCI, ZIP) or direct artifact storage depending on their environment. This covers the airgapped and "capabilities alongside models" use cases you described.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
|
|
||
| ### Out of scope | ||
|
|
||
| - **Artifact storage.** The registry stores metadata and source |
There was a problem hiding this comment.
given that mlflow already has an "artifact store" to store models and traces, I think the proposal could just embrace it and support it as an option, as per my previous comment.
There was a problem hiding this comment.
Addressed in the same change. See reply to your earlier comment.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
…, unify pull RFC-0005: - Add register_skill() SDK convenience function matching MCP RFC's register_mcp_server() pattern, with content_path for artifact upload - Remove kind=mcp-server from SkillKind (MCP servers belong in MCP registry; embedded configs are artifact content for harness adapters) - Unify pull/pull-group into single pull command with --group flag - Add shared base extraction note to adoption strategy - Update basic examples to use register_skill() RFC-0006: - Drop install POST endpoints (install is client-side) - Keep marketplace.json GET endpoint - Update SDK example to use unified install() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Status update (2026-05-17) Replied to all of Khaled's review comments. Key decisions:
Next steps: I need to review Matt's comments and the upcoming MCP Registry changes in PR #18. Once I've worked through both, I may retract some of the decisions made for this round in favor of further alignment. -- Bill Murdock (with assistance from Claude Code) |
…k file RFC-0005 changes: - Clarify summary: artifact storage is supported but metadata-first - Clarify source_type="mlflow" means MLflow-managed storage, not a specific URI scheme - Merge content integrity into one section with server-side validation for mlflow sources and client-side for external sources - Remove guidance to register separate versions for different sources - Rename SkillGroupVersionMembership to SkillGroupVersionMember - Unified UI list view showing skills and groups together - Add install count tracking to follow-up roadmap RFC-0006 changes: - Add lock file section for reproducible installs (mlflow-skills.lock) - Add Python entrypoint support for third-party harness adapters Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Status update on RFC-0005 / RFC-0006 We reviewed the new MCP Registry ADR (PR #18) and confirmed it has no impact on these RFCs. We've addressed many of Matt's latest review comments, including:
Still need to work through several of Matt's other comments. Also, we need to keep working on naming. -- Bill Murdock (with assistance from Claude Code) |
| rfc_pr: https://github.com/mlflow/rfcs/pull/10 | ||
| --- | ||
|
|
||
| # RFC: Skill Registry |
There was a problem hiding this comment.
+1 to @etirelli here, the umbrella term should not be a skill. Currently in the RFC the skill that is a kind/skill is more consistent with the industry's usage of the term, but the top level usage of the word skill would be very confusing.
My preference is asset registry or capability registry, or something equally broader.
| The registry is format-agnostic but is designed to interoperate with | ||
| the conventions gaining adoption across agent harnesses: | ||
|
|
||
| - **SKILL.md** — a markdown file with structured instructions for the |
There was a problem hiding this comment.
Since sub agent is a kind in this RFC, maybe a sentence mentioning sub agents would be useful
There was a problem hiding this comment.
Subagent is now a separate first-class entity type rather than a kind value on Skill. The entity model section has full Subagent and SubagentVersion dataclasses, dedicated DB tables, store methods, and REST endpoints. See commit d16abe0.
Bill Murdock, with assistance from Claude Code
| SkillGroupVersionMember( | ||
| member_name="github-mcp", member_version="2.0.0", | ||
| registry="mcp", | ||
| ), |
There was a problem hiding this comment.
Nit: I feel like it's a bit awkward that a user needs to mention the type of registry. Maybe we could also just allow users to pass the specific version object and the client can infer the registry based on type. So for example:
my_skill_version = mlflow.genai.skills.register_skill(
name="code-review",
version="1.0.0",
....
)
my_mcp_server_version = mlflow.genai.register_mcp_server(
server_json={
"name": "io.github.anthropic/brave-search",
"title": "Brave Search",
...
)
create_skill_group_version(
name="my_group",
version="1.0.0",
members=[
my_skill_version,
my_mcp_server_version,
],
)
)Internally this would translate to SkillGroupVersionMember(..) or users could use SkillGroupVersionMember() directly as fallback if they wanted.
There was a problem hiding this comment.
This is now addressed by the entity type split. SkillBundleVersion uses four typed member lists (skills, subagents, hooks, mcp_servers), each containing (name, version) tuples. There is no longer a generic members list or a registry field that callers need to specify.
(Comment from Bill Murdock with assistance from Claude Code)
|
|
||
| ```python | ||
| @dataclass(frozen=True) | ||
| class SkillAliasHistory: |
There was a problem hiding this comment.
We don't have an Alias history in the MCP registry, I don't see this as a requirement for the MVP, can we drop this and associated apis? I think if we are to consider this we should consider it hollistically for all assets types, mcp, skills, prompts to ensure consistency.
Same with SkillGroupAliasHistory
There was a problem hiding this comment.
Alias history was added in response to etirelli's comment, which argued that alias changes have high blast radius in enterprise settings and retrofitting audit later is harder than including it upfront. The motivating questions are things like "who promoted this to production and when?" and "what was production pointing to before the incident?"
I agree that cross-registry consistency matters. Is there a way to address your consistency concern while still covering the governance use case etirelli raised?
(Comment from Bill Murdock with assistance from Claude Code)
| class SkillGroupVersionMember: | ||
| member_name: str | ||
| member_version: str | ||
| registry: str = "skill" # "skill" or "mcp" |
There was a problem hiding this comment.
Should registry be an enum similar to SkillStatus, SkillKind, etc?
There was a problem hiding this comment.
This is now moot. The entity type split replaced the single Skill(kind=...) entity with separate Skill, Subagent, and Hook types, and SkillBundleVersion now uses typed member lists (skills, subagents, hooks, mcp_servers) instead of a generic members list with a registry string field.
(Comment from Bill Murdock with assistance from Claude Code)
| schema changes since the column stores a string value. `kind` is | ||
| immutable after creation. | ||
|
|
||
| **MCP servers.** MCP servers are registered in the MCP Server Registry |
There was a problem hiding this comment.
Have you thought about the prompt registry and where that fits in here? I feel like prompts are a real part of an agent package so some consideration to them in this RFC would be warranted. If you agree but feel it doesn't need to be part of the MVP, I think that is fine but we should list those as out of scope.
edit: Looked more into this and I see that it's just skills not prompts for most plugin bundles (is that the only intended usecase here?). Probably should add a note addressing prompts and where they fit in (or don't).
There was a problem hiding this comment.
Good question. Prompts and skills both provide instructions to a model, but they differ in structure, consumption, and audience. A prompt in MLflow's prompt registry is a template string with {{variable}} placeholders (or a chat-style message list), loaded at runtime by custom code via mlflow.genai.load_prompt(). A skill is a set of files (SKILL.md, code, configs) installed by a harness adapter that handles discovery, placement, and manifest generation.
The prompt registry serves developers writing custom agentic code who need templated instructions for specific steps in a flow. The skill registry serves harness-based agents (Claude Code, Codex CLI, Cursor) where the harness manages execution. Because both the structure and the consumption patterns differ, they belong in separate registries.
Added a note to the Out of scope section making this explicit.
(Comment from Bill Murdock with assistance from Claude Code)
| code-review/SKILL.md # Pulled from registered source | ||
| agents/ | ||
| security-auditor.md # Pulled from registered source | ||
| .mcp.json # Generated from mcp-server members |
There was a problem hiding this comment.
nit: do you feel the usage of the word install here could be a bit misleading, since mcp registry exposes canonical server_json and packages[], which naturally reads as "here is how you actually install/run this MCP server." But really we're just generating/creating files, not installing the mcp server(s). Would mlflow skills generate or something similar be better suited?
There was a problem hiding this comment.
I considered this. install covers the full operation: resolve versions from the registry (including alias resolution and status checks), pull content from sources, and generate harness-specific manifests. generate would only describe the last step. The MCP registry uses install for a different operation (installing MCP server packages), so there is some terminology tension. For now I'm keeping install since it describes the end-to-end operation, but happy to revisit if a better term emerges.
(Comment from Bill Murdock with assistance from Claude Code)
| "version": "1.0.0", | ||
| "description": "End-to-end pull request review workflow", | ||
| "author": { "name": "Generated by MLflow Skill Registry" }, | ||
| "source": "https://mlflow.example.com/ajax-api/3.0/mlflow/skill-groups/pr-workflow/install?harness=claude-code", |
There was a problem hiding this comment.
This source looks like it's implying that Claude is downloading through the MLflow server - is that what you intended?
There was a problem hiding this comment.
Good catch. In the marketplace flow, the MLflow server does serve the content. The /install endpoint resolves the bundle, pulls member content from sources, generates harness-specific manifests, and serves the assembled plugin. This works because the server can access public sources and MLflow artifact storage directly.
Bundles with private external sources (private Git repos, authenticated OCI registries) are excluded from the marketplace catalog. Those are installed via the CLI (mlflow skills install), which uses the caller's local credentials.
Added an "Install endpoint behavior" paragraph to RFC-0006 making this explicit.
(Comment from Bill Murdock with assistance from Claude Code)
| --harness claude-code --lock --update | ||
| ``` | ||
|
|
||
| The lock file records enough information to reproduce the install |
There was a problem hiding this comment.
One key benefit from having the Skills registry is the governance aspect, and the ability for admins to deprecate, pull, or roll back skills/mcp/hooks/agents, so wouldn't this by pass that governance if this doesn't contact the registry?
There was a problem hiding this comment.
Based on your point about governance, we updated the lock file section: lock file replay now checks version status against the registry, so governance actions (deprecation, deletion) take effect even for existing lock files. Install always contacts the registry for resolution and status checks, whether from a lock file or not.
(Comment from Bill Murdock with assistance from Claude Code)
| ) | ||
| ``` | ||
|
|
||
| ## Pull skills to a local directory |
There was a problem hiding this comment.
I don’t see the RFC specify how authorization works for external sources during pull, especially for private Git repos, private OCI registries, or authenticated ZIP URLs. Since pull is client-side and the registry server is not involved in content transfer, is the intent that auth is entirely delegated to the caller’s local Git/OCI/HTTP credentials? If so, it would help to state that explicitly, along with whether the registry ever validates source accessibility at registration time.
There was a problem hiding this comment.
Good catch. Added a "Source authentication" section to the pull semantics. The design is:
- The registry does not validate source accessibility at registration time and is not involved in content transfer.
- Auth is delegated to the caller's environment: Git uses standard credential resolution (SSH keys, credential helpers,
.netrc), OCI uses Docker config and registry credential helpers, MLflow uses existing API credentials. - ZIP sources require publicly accessible URLs; no auth support. For private content, use
gitorociinstead. - Pull failures due to auth errors surface the underlying error from the source system.
(Comment from Bill Murdock with assistance from Claude Code)
For OCI and ZIP source types, multiple skills may share a single artifact. The new subpath field identifies where each skill lives within the artifact. Not used for Git (tree URLs encode the path) or MLflow artifacts (path is scoped at upload time). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The MCP Registry RFC (RFC-0004) has no error handling section. Consistent with that precedent, this detail is better left to implementation. Reference copy saved in error-handling-reference.md (not checked in). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Workspace scoping now references MLflow's existing patterns instead of enumerating implementation details. Adoption strategy condensed to a three-phase summary. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Status update (2026-05-19) Addressed three more of Matt's review comments:
Also added Still open:
This comment is from Bill Murdock with assistance from Claude Code. |
| Membership is at the version level, so a group version is a | ||
| reproducible snapshot of "these specific asset versions work together." | ||
|
|
||
| **Group-level source and embedded MCP configs.** When a group version |
There was a problem hiding this comment.
Is having a group level source/source_type a requirement for MVP? To me this feels like there are unanswered questions in this + the harness integration rfc.
- if a group has a source, does install consume that artifact directly?
- is that artifact expected to already be a harness-ready plugin bundle, or just a generic package of skills/agents/hooks/MCP config?
- do adapters still generate manifests from metadata, or trust what’s inside the group artifact?
- what happens if the artifact contents disagree with the declared members?
- presumably sources like git repo would need to respect directory structures/layouts, how would you enforce that? or ensure it's adhered to?
IMO if this is not required I would take it out, and leave it for a future enhancement.
There was a problem hiding this comment.
I think bundle-level source is important for MVP because in practice, this is how skills get distributed: as bundles. Developers package their plugin as a single OCI image or Git repo containing skills, subagents, hooks, and MCP configs together. Requiring a separate artifact per member would be an unreasonable authoring burden.
To address the specific questions:
- Does install consume the artifact directly? Yes. When a bundle version has a source,
pullfetches that single artifact rather than pulling members individually. - Harness-ready or generic package? Generic. The artifact contains raw content (skill files, agent definitions, etc.). Harness adapters (RFC-0006) generate manifests from registry metadata, not from the artifact.
- Do adapters generate manifests or trust the artifact? Adapters generate manifests from registry metadata. The artifact provides content; the registry provides governance (status, membership, versions).
- What if artifact contents disagree with declared members? The registry does not validate artifact contents at registration time.
subpathon each member version points to its location within the artifact, but correctness is the publisher's responsibility. This is similar to how OCI registries don't validate image contents against external metadata. - Directory structure for Git repos? The
subpathfield handles this. Each member version specifies its path within the repo (e.g.,subpath="skills/code-review").
That said, I'm open to alternative approaches that solve the same problem (distributing multiple capabilities in a single artifact) if you can think of one. I could not find a way to avoid bundle-level source without requiring separate artifacts per member.
(Comment from Bill Murdock with assistance from Claude Code)
There was a problem hiding this comment.
Follow-up: added a paragraph to the bundle-level source section clarifying the points from my earlier reply. The artifact is a generic package that may or may not be harness-ready. Adapters generate manifests from registry metadata regardless. If artifact contents disagree with declared members, pull succeeds but install fails. Layout correctness is the publisher's responsibility; the registry does not validate artifact contents at registration time.
(Comment from Bill Murdock with assistance from Claude Code)
|
|
||
| ## Drawbacks | ||
|
|
||
| - **Source pointer validity.** The registry stores source pointers but |
There was a problem hiding this comment.
We should mention how such cases are handled. For example what happens if you try to pull a resource where the source no longer exists?
There was a problem hiding this comment.
Good point. Added a "Source availability" section to the pull semantics. If a source is unreachable or deleted, pull fails with an error surfacing the underlying failure (Git clone failure, OCI pull 404, etc.). Source availability is the publisher's responsibility. For bundle pulls, if one member's source is unavailable, the entire pull fails rather than producing a partial result.
(Comment from Bill Murdock with assistance from Claude Code)
| 2. Otherwise, pull each member individually from its own `source` to | ||
| a subdirectory of the destination, named by the member's skill name. | ||
| Members without a `source` are skipped with a warning. | ||
|
|
There was a problem hiding this comment.
Why is source not a required field for skills? This feels like it creates too much ambiguity.
There was a problem hiding this comment.
Source is optional because a skill version's content may live within a bundle-level artifact rather than at its own independent source. In that case, the bundle version has the source and the member version uses subpath to identify its location within the artifact.
Standalone skill versions (not part of a bundle) require a source for pull to work. The RFC has been updated to make this clearer: source may only be omitted when the version's content lives within a bundle-level artifact.
(Comment from Bill Murdock with assistance from Claude Code)
| code-review/SKILL.md # Pulled from registered source | ||
| agents/ | ||
| security-auditor.md # Pulled from registered source | ||
| .mcp.json # Generated from mcp-server members |
There was a problem hiding this comment.
.mcp.json contains information about what servers are available, and how the client connect to them.
MCPAccessBinding can give you information about the active endpoints. But it is worth making a brief note that considers the MCP gateway as well (see the MCP registry RFC Journey 2). The adapter would also need a selection policy (what if there is more than one MCPAccessBinding for a server?)
Also note that while we can get the user most of the way there we can't configure things like credentials, certificates, authorization headers, etc.
There was a problem hiding this comment.
Good points. Expanded the MCP server config generation section in RFC-0006 to address these:
Connection details. The adapter resolves MCPServerVersion from the MCP registry and looks for an MCPAccessBinding targeting that version or alias. If a binding exists, its endpoint_url and transport_type are used. If multiple bindings exist, the adapter uses the first one targeting the referenced version or alias. If no binding exists, the adapter falls back to server_json (e.g., remotes[]).
Embedded configs. When a bundle has a bundle-level source that already contains a .mcp.json, those embedded configs are used as-is for servers not in the mcp_servers member list. If the same server appears in both, the registry-generated entry takes precedence.
Credentials. The adapter generates connection config but does not configure credentials, certificates, or auth headers. The adapter logs a warning for servers using authenticated transport so users know to complete setup manually.
The MCP gateway (RFC-0004 Journey 2) is future work and doesn't affect the Phase 1 install flow.
(Comment from Bill Murdock with assistance from Claude Code)
| and hooks together. But there is no agent-neutral way to represent | ||
| these bundles for governance and discovery. | ||
|
|
||
| 5. **No usage analytics linkage.** MLflow traces can capture skill |
There was a problem hiding this comment.
Can you provide the mechanics of how we would link traces back to the skills/agents/hooks/mcp servers? Because at the moment this part is not clear to me. It seems majority of these are uniquely identified by (workspace, name, version), but how would mlflow know this information, specifically the workspace?
As an example for prompts would have to explicitly load or link the prompts via mlflow.genai.load_prompt or MlflowClient.link_prompt_versions_to_trace(...)
Can you provide an example of a trace call, and briefly explain how this would work? At the moment to me it seems like we do not have sufficient information to do this since the usecase described with harness integration seems to be to do an install and generate the files in the file system.
There was a problem hiding this comment.
Thanks for this question, it convinced me to make trace integration a first-class part of the design rather than deferring it.
I've added a "Trace integration" section to RFC-0005 that addresses your questions directly. The short version:
For SDK-based agents, I propose mlflow.skill_context(), a context manager that creates a span of type SKILL with {workspace, name, version} registry coordinates as span attributes. The workspace defaults to "default" and can be specified explicitly. Agent developers wrap their skill invocations:
with mlflow.skill_context(name="code-review", version="1.0.0"):
result = llm.chat(...) # autologger spans nest under the SKILL spanNesting these blocks naturally captures skill stacks (skills invoking sub-skills). Existing autologgers work without modification since child spans nest under the SKILL span automatically.
For Claude Code users, RFC-0006 now describes hook-based instrumentation: mlflow skills install writes a manifest mapping skill names to registry coordinates, and PreToolUse/PostToolUse hooks on Claude Code's built-in Skill tool create SKILL spans automatically.
This is analogous to the load_prompt / link_prompt_versions_to_trace pattern you mentioned, but uses span-level context managers rather than trace-level linking, because skills are ambient (active during inference) and can nest.
(This comment is from Bill Murdock with assistance from Claude Code.)
RFC-0005: Add skill_context() context manager that creates SKILL spans with registry coordinates, supporting nested skill stacks. Strengthen motivation item on trace-to-skill linkage. Move trace integration from Phase 3 to Phase 1 in adoption strategy. RFC-0006: Add harness trace integration via install-time manifest and Claude Code PreToolUse/PostToolUse hooks on the Skill tool for automatic SKILL span creation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…undle Split the single Skill entity (with kind field) into three separate entity types: Skill, Subagent, Hook. Rename SkillGroup to SkillBundle with typed member lists (skills, subagents, hooks, mcp_servers) replacing the SkillGroupVersionMember entity. Use "subagent" to avoid collision with a future agent registry. Updates span both RFCs: entity definitions, DB schema, store interface, REST API, SDK convenience functions, CLI, and harness adapter interfaces. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Status update (2026-05-28) Changes since last updateTwo major changes pushed since the 2026-05-19 update: 1. Trace integration (bd26ad8) Added RFC-0006 now includes harness-specific instrumentation: an install-time manifest ( 2. Entity model restructuring (d16abe0) Resolved the Open discussionsMatt (mprahl):
Humair (HumairAK): 15 comments posted 2026-05-18/19, mostly not yet replied to. Key themes:
Bill Murdock, with assistance from Claude Code |
…d ScanResult fields Replaces the tag-based security scan convention with a structured ScanResult entity on version types. ScanStatus enum (pass/fail/warning/error) enables cross-scanner filtering. Upsert semantics per (version, scan_type). Adds scan result DB tables, store methods, and REST endpoints for all version types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move local working documents (deferred-review-items, naming discussion, error handling reference) to .local/ directory. Add .gitignore to exclude .local/ and .DS_Store from version control. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Redundant with latest_version and sorting versions by creation_timestamp. Removed from all entity types (Skill, Subagent, Hook, SkillBundle) and DB schema tables. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds mutable display_name to Skill, SkillVersion, Subagent, Hook, SkillBundle, SkillBundleVersion, their DB tables, store methods, and SDK convenience functions. Consistent with MCP registry schema. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
run_id can be expressed via tags, consistent with how MLflow prompts handle experiment linkage. Removed from SkillVersion dataclass, field table, DB schema, store methods, and SDK convenience functions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clarifies that prompts (template strings loaded at runtime) and skills (file sets installed by harnesses) serve different audiences and belong in separate registries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents how each source type handles auth: Git via standard credential resolution, OCI via Docker config, ZIP requires public URLs, MLflow via existing API credentials. Registry does not validate source accessibility at registration time. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Source is required for standalone pull. It may be omitted only when the version's content lives within a bundle-level artifact. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server-side install resolves bundle, pulls content, assembles plugin. Only works for public sources and MLflow artifact storage. Private sources require the CLI path with local credentials. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Governance actions (deprecation, deletion) now take effect even for existing lock files. The registry is always contacted for status verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents pull failure behavior when sources are unreachable or deleted. Bundle pulls fail entirely rather than producing partial results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents how adapter resolves connection details from MCPAccessBinding, handles embedded .mcp.json in bundle artifacts, and notes that credentials are the user's responsibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bundle artifact may or may not be harness-ready. Adapters generate manifests from registry metadata regardless. Artifact layout correctness is the publisher's responsibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Status update (2026-05-28, part 2) All review threads now have repliesAll review comment threads across all reviewers (Matt, Humair, Khaled, etirelli, Serena, Jon, Dank, Copilot) now have replies. One thread is awaiting a follow-up response from Humair (alias history, see below). Changes since earlier todayPushed 12 commits addressing review feedback, primarily from Humair and Matt. RFC-0005 changes:
RFC-0006 changes:
Pending discussion
Bill Murdock, with assistance from Claude Code |
Summary
Related Issues
Numbering
This RFC is numbered 0005, assuming the MCP Server Registry proposal (mlflow/mlflow#22625) will be RFC-0004.
Design Highlights
mlflow.skills), and CLI (mlflow skills,mlflow skill-groups)🤖 Generated with Claude Code
This comment comes from Claude Code under the supervision of Bill Murdock.