Skip to content

Add supervisor pattern: /add-supervisor skill with best-fit selection - #30

Open
alexbaur wants to merge 2 commits into
mainfrom
add-supervisor-pattern
Open

Add supervisor pattern: /add-supervisor skill with best-fit selection#30
alexbaur wants to merge 2 commits into
mainfrom
add-supervisor-pattern

Conversation

@alexbaur

Copy link
Copy Markdown
Collaborator

What

Introduces a supervisor agent to AgentOps Stacks as a post-scaffold pattern (applied once a project has >1 agent, alongside eval gates / governance / monitoring). A supervisor routes user queries across the project's agents and other managed sub-agents.

The core contribution is a Selection Matrix that picks the best-fit supervisor pattern based on the user's needs, architecture, and required complexity — because the three options are not interchangeable. They differ in who owns the routing loop and what artifact lands in the bundle:

Pattern Loop owner In the bundle DAB-declarable Status
custom (default) your code agent App under src/agents/ yes, natively GA
supervisor_api Databricks wrapper App under src/agents/ yes, as an App Beta
agent_bricks_mas Databricks tile none — consumed endpoint no (bootstrap job) UI GA / SDK Beta

Decision matrix (first gate that fires wins)

  • D1 Lifecycle parity non-negotiable (same DAB + eval gate + promotion, GA, fully in databricks.yml) → custom
  • D2 Orchestration control (custom state, conditional routing, guardrails, HITL, Lakebase checkpointer, retries) → custom
  • D3 Managed loop: no-code/SME-iterable → agent_bricks_mas; code-first/per-request model choice → supervisor_api
  • D4 HIPAA/enhanced-security or GA-certainty → custom; >50 sub-agents or partial sub-agent access → not MAS

Why this fits AgentOps Stacks

  • Models the supervisor as "just another agent" (custom/api), so CI's detect_patterns → eval_gate picks it up with zero workflow changes.
  • Reuses the add_agent.py wiring approach; renders graph/tools/agent/deps variants per pattern.
  • Records the choice in .agentops-stacks/manifest.yml under supervisor: — the same contract CI/tooling already read.
  • No new bundle init input — the choice is made post-scaffold, when it's meaningful.
  • Honors the standing "declarative over notebook-created; document why" rule: the MAS tile can't be a DAB resource, so it's a documented, bundle-declared bootstrap job + consumed-endpoint reference.

Files

  • plugin/skills/add-supervisor/SKILL.md — skill + Selection Matrix + per-pattern security posture
  • plugin/commands/add-supervisor.md/add-supervisor command
  • plugin/skills/agentops-stacks/scripts/add_supervisor.py — engine (3 patterns)
  • .../scripts/supervisor_templates/ — graph (custom, api), agent handler, tools, MAS job + notebook
  • template/.../docs/supervisor-patterns.md.tmpl — pattern deep-dive in the rendered project
  • template/.../.agentops-stacks/manifest.yml.tmpl — supervisor contract placeholder
  • workflows/single-account-single-agent.json — machine-readable decision matrix in routing_notes
  • plugin/skills/agentops-lifecycle/SKILL.md — Step 3.5 routing note
  • plugin/README.md, install_skills.sh — register the new skill/command
  • tests/test_add_supervisor.py — 15 unit tests (no CLI needed), all passing

Testing

  • 15 unit tests for add_supervisor.py pass — build a fake project tree, assert all three patterns wire databricks.yml / manifest / layout correctly; no Databricks CLI needed.
  • End-to-end CLI smoke test of all three types: rendered databricks.yml parses (yaml.safe_load), rendered graph.py/agent.py compile, no leftover placeholders.
  • A review pass caught and fixed two runtime bugs in the rendered supervisor: (1) supervisor_api sent LangChain ai messages to the Responses API with an invalid role; (2) a supervisor cloned from a Lakebase-enabled agent inherited an agent.py importing get_async_checkpointer (undefined in the supervisor graph) — now overwritten with a dedicated stateless handler. Both are covered by tests.
  • Pre-existing test_create_project.py tests require Databricks CLI v1.1.0 (unrelated to this change; unaffected).

This pull request and its description were written by Isaac.

alexbaur added 2 commits July 26, 2026 12:42
Introduces a supervisor agent as a post-scaffold pattern (like eval gates,
governance, monitoring) applied once a project has >1 agent. A supervisor
routes user queries across the project's agents and other managed sub-agents.

The core contribution is a Selection Matrix that picks the best-fit supervisor
pattern from three options based on the user's needs, architecture, and the
complexity required — because the three are NOT interchangeable; they differ in
who owns the routing loop and what artifact lands in the bundle:

  custom          Hand-written LangGraph supervisor. GA, fully DAB-declarable
                  (an agent App under src/agents/), gated by the same CI eval
                  loop as any agent. The default.
  supervisor_api  Databricks-managed loop (Responses API) wrapped in a
                  declarable App. Beta (AI Gateway + UC OTel-traces preview).
  agent_bricks_mas  Agent Bricks Supervisor tile. NOT a DAB resource — scaffolds
                  a bundle-declared bootstrap job that provisions it imperatively
                  and a consumed-endpoint reference. UI GA / SDK Beta.

Integration is deliberately lightweight and maps to what already exists:
- Models the supervisor as "just another agent" (custom/api) so CI's
  detect_patterns -> eval_gate picks it up with zero workflow changes.
- Reuses the add_agent.py wiring approach; renders graph/tools/deps variants.
- Records the choice in .agentops-stacks/manifest.yml under `supervisor:` — the
  same contract CI/tooling already read.
- No new bundle-init input (choice made post-scaffold when it's meaningful).
- Honors the standing "declarative over notebook-created; document why" rule:
  the MAS tile can't be a DAB resource, so it's a documented bootstrap job.

Files:
- plugin/skills/add-supervisor/SKILL.md — skill + Selection Matrix + security posture
- plugin/commands/add-supervisor.md — /add-supervisor command
- plugin/skills/agentops-stacks/scripts/add_supervisor.py — engine (3 patterns)
- .../scripts/supervisor_templates/ — graph (custom, api), tools, MAS job+notebook
- template/.../docs/supervisor-patterns.md.tmpl — pattern deep-dive in rendered project
- template/.../.agentops-stacks/manifest.yml.tmpl — supervisor contract placeholder
- workflows/single-account-single-agent.json — machine-readable decision matrix
- plugin/skills/agentops-lifecycle/SKILL.md — Step 3.5 routing note
- plugin/README.md, install_skills.sh — register the new skill/command
- tests/test_add_supervisor.py — 13 unit tests (no CLI needed), all passing

Co-authored-by: Isaac
Review pass found two defects that would break the generated supervisor App:

1. supervisor_api graph: LangChain message .type "ai" was passed to the
   Responses API as role "ai" (invalid — expects "assistant"). Added an explicit
   _ROLE_MAP so human/ai/system/tool map to valid roles.

2. agent.py inheritance: the script copied the source agent's agent.py, which —
   when the base agent had Lakebase memory — imports get_async_checkpointer from
   graph. The supervisor graph.py doesn't define it, so the App crashed on
   import. Now overwrite agent.py with a dedicated stateless supervisor handler
   (agent_supervisor.py.tmpl).

Adds tests covering the agent.py overwrite, including the Lakebase-base case.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant