databricks-solutions · auschoi96 · Jul 17, 2026 · Jul 17, 2026 · Jul 17, 2026 · Jul 17, 2026
diff --git a/.claude/skills/databricks-demo-generator/SKILL.md b/.claude/skills/databricks-demo-generator/SKILL.md
@@ -29,7 +29,16 @@ The main loop lives in this file (SKILL.md) — it describes **the flow**: stage
 
 ## Paths
 
-Your system prompt defines `PROJECT`, `SKILLS`, `DEMO_SKILL_DIR`, and `DEMO_SKILL` as absolute paths. This skill refers to sibling files like `DEMO_SKILL_DIR/stages/*.md`, `DEMO_SKILL_DIR/app/app.md`, `DEMO_SKILL_DIR/references/*`.
+Your system prompt normally defines `PROJECT`, `SKILLS`, `DEMO_SKILL_DIR`, and `DEMO_SKILL` as absolute paths. This skill refers to sibling files like `DEMO_SKILL_DIR/stages/*.md`, `DEMO_SKILL_DIR/app/app.md`, `DEMO_SKILL_DIR/references/*`.
+
+When those injected aliases are absent (for example in a maintainer evaluation fixture), self-locate before reading any references:
+
+- `PROJECT` is the execution working directory.
+- `SKILLS` is `PROJECT/.claude/skills`.
+- `DEMO_SKILL_DIR` is `SKILLS/databricks-demo-generator`.
+- `DEMO_SKILL` is `DEMO_SKILL_DIR/SKILL.md`.
+
+Resolve and use absolute paths from those fallbacks. Injected aliases, when present, remain authoritative.
 
 **When spawning subagents**, substitute every placeholder (`DEMO_SKILL_DIR/…`, `PROJECT/…`, `SKILLS/…`) with its real absolute path before sending — the subagent has no system prompt defining them. The full spawn prompt is in `DEMO_SKILL_DIR/stages/03-build.md` → Step 2.
 

diff --git a/.github/workflows/skillforge-evaluation.yml b/.github/workflows/skillforge-evaluation.yml
@@ -0,0 +1,157 @@
+name: SkillForge evaluation
+
+on:
+  pull_request:
+    paths:
+      - ".claude/skills/databricks-demo-generator/**"
+      - "evaluation/**"
+      - "tests/evaluation/**"
+      - "tests/pipeline/scenarios.py"
+      - ".github/workflows/skillforge-evaluation.yml"
+  workflow_dispatch:
+    inputs:
+      scenario:
+        description: Canonical scenario id or all
+        required: true
+        default: all
+        type: string
+
+concurrency:
+  group: skillforge-${{ github.event_name == 'workflow_dispatch' && 'live' || github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  validate-and-quick-eval:
+    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    env:
+      HAS_QUICK_EVAL_SECRETS: ${{ secrets.SKILLFORGE_READ_TOKEN != '' && secrets.ANTHROPIC_API_KEY != '' }}
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+      - uses: astral-sh/setup-uv@e92bafb6253dcd438e0484186d7669ea7a8ca1cc # v6
+        with:
+          enable-cache: true
+      - name: Install maintainer environment
+        run: uv sync --group dev --locked
+      - name: Validate canonical cases
+        run: uv run sb-eval cases validate
+      - name: Run maintainer contract tests
+        run: uv run pytest -q
+      - name: Read pinned revisions
+        id: pins
+        run: |
+          uv run python - <<'PY' >> "$GITHUB_OUTPUT"
+          from evaluation.toolchain import load_lock
+          lock = load_lock()
+          print(f"skillforge_repo={lock.skillforge_repository.removesuffix('.git').removeprefix('https://github.com/')}")
+          print(f"skillforge_revision={lock.skillforge_revision}")
+          print(f"ai_dev_kit_repo={lock.ai_dev_kit_repository.removesuffix('.git').removeprefix('https://github.com/')}")
+          print(f"ai_dev_kit_revision={lock.ai_dev_kit_revision}")
+          PY
+      - name: Checkout pinned SkillForge
+        if: env.HAS_QUICK_EVAL_SECRETS == 'true'
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+        with:
+          repository: ${{ steps.pins.outputs.skillforge_repo }}
+          ref: ${{ steps.pins.outputs.skillforge_revision }}
+          token: ${{ secrets.SKILLFORGE_READ_TOKEN }}
+          path: .deps/skillforge
+      - name: Checkout pinned ai-dev-kit
+        if: env.HAS_QUICK_EVAL_SECRETS == 'true'
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+        with:
+          repository: ${{ steps.pins.outputs.ai_dev_kit_repo }}
+          ref: ${{ steps.pins.outputs.ai_dev_kit_revision }}
+          path: .deps/ai-dev-kit
+      - name: Install pinned external SkillForge
+        if: env.HAS_QUICK_EVAL_SECRETS == 'true'
+        run: uv tool install '.deps/skillforge/python[all]'
+      - name: Note unavailable fork secrets
+        if: env.HAS_QUICK_EVAL_SECRETS != 'true'
+        run: echo "External L1/L3 evaluation skipped because fork-safe maintainer secrets are unavailable."
+      - name: Informational L1/L3 evaluation
+        if: env.HAS_QUICK_EVAL_SECRETS == 'true'
+        id: quick-eval
+        continue-on-error: true
+        env:
+          SB_EVAL_AI_DEV_KIT_DIR: ${{ github.workspace }}/.deps/ai-dev-kit
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+        run: uv run sb-eval run --levels L1,L3
+      - name: Upload evaluation reports
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: skillforge-quick-${{ github.run_id }}
+          path: test-runs/skillforge/
+          if-no-files-found: warn
+
+  full-live-eval:
+    if: github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    environment: solution-builder-live-evaluation
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+      - uses: astral-sh/setup-uv@e92bafb6253dcd438e0484186d7669ea7a8ca1cc # v6
+        with:
+          enable-cache: true
+      - name: Install maintainer environment
+        run: uv sync --group dev --locked
+      - name: Validate canonical cases
+        run: uv run sb-eval cases validate
+      - name: Run maintainer contract tests
+        run: uv run pytest -q
+      - name: Read pinned revisions
+        id: pins
+        run: |
+          uv run python - <<'PY' >> "$GITHUB_OUTPUT"
+          from evaluation.toolchain import load_lock
+          lock = load_lock()
+          print(f"skillforge_repo={lock.skillforge_repository.removesuffix('.git').removeprefix('https://github.com/')}")
+          print(f"skillforge_revision={lock.skillforge_revision}")
+          print(f"ai_dev_kit_repo={lock.ai_dev_kit_repository.removesuffix('.git').removeprefix('https://github.com/')}")
+          print(f"ai_dev_kit_revision={lock.ai_dev_kit_revision}")
+          PY
+      - name: Checkout pinned SkillForge
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+        with:
+          repository: ${{ steps.pins.outputs.skillforge_repo }}
+          ref: ${{ steps.pins.outputs.skillforge_revision }}
+          token: ${{ secrets.SKILLFORGE_READ_TOKEN }}
+          path: .deps/skillforge
+      - name: Checkout pinned ai-dev-kit
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
+        with:
+          repository: ${{ steps.pins.outputs.ai_dev_kit_repo }}
+          ref: ${{ steps.pins.outputs.ai_dev_kit_revision }}
+          path: .deps/ai-dev-kit
+      - name: Install pinned external SkillForge
+        run: uv tool install '.deps/skillforge/python[all]'
+      - name: Configure dedicated non-production profile
+        env:
+          DATABRICKS_HOST: ${{ secrets.SB_EVAL_DATABRICKS_HOST }}
+          DATABRICKS_TOKEN: ${{ secrets.SB_EVAL_DATABRICKS_TOKEN }}
+        run: |
+          mkdir -p "$HOME/.databricks"
+          printf '[solution-builder-eval]\nhost = %s\ntoken = %s\n' "$DATABRICKS_HOST" "$DATABRICKS_TOKEN" > "$HOME/.databrickscfg"
+          chmod 600 "$HOME/.databrickscfg"
+      - name: Full manual L1-L5 live evaluation
+        env:
+          SB_EVAL_AI_DEV_KIT_DIR: ${{ github.workspace }}/.deps/ai-dev-kit
+          SB_EVAL_DATABRICKS_PROFILE: solution-builder-eval
+          SB_EVAL_ALLOWED_PROFILES: solution-builder-eval
+          SB_EVAL_ALLOWED_HOSTS: ${{ secrets.SB_EVAL_DATABRICKS_HOST }}
+          DATABRICKS_HOST: ${{ secrets.SB_EVAL_DATABRICKS_HOST }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          SB_EVAL_SCENARIO: ${{ inputs.scenario }}
+        run: uv run sb-eval run --levels all --live --scenario "$SB_EVAL_SCENARIO"
+      - name: Upload evaluation reports
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: skillforge-live-${{ github.run_id }}
+          path: test-runs/skillforge/
+          if-no-files-found: error
diff --git a/.gitignore b/.gitignore
@@ -39,6 +39,9 @@ screenshot-eval/
 # tests/pipeline/ harness output
 test-runs/
 
+# Maintainer evaluation dependency checkouts (never packaged or deployed)
+.deps/
+
 # Python caches
 __pycache__/
 *.pyc

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -16,6 +16,8 @@ A **system that generates Databricks demos**. Not one app — **three**, plus a
 
 Plus: **ai_dev_kit** (`app/ai_dev_kit/`) — a cloned external repo (`github.com/databricks-solutions/ai-dev-kit`) holding ~26 sub-skills for creating individual Databricks resources (pipelines, dashboards, Genie spaces, KAs, MAS, etc.). The generator's agent uses these during the Build stage.
 
+Maintainers also have an optional **evaluation harness** (`evaluation/`). It owns canonical scenarios and shells out to an exactly pinned external SkillForge executable. It is not imported by or packaged with the deployed generator app, copied into projects, or exposed through application routes.
+
 ## Mental model
 
 ```
@@ -81,6 +83,7 @@ industry-demo-prompts/
 │       ├── app.md                        # How to design+spec a demo app (read during Stage 2)
 │       └── app_template/                 # ★ The template app emitted by the generator
 ├── initial_templates/                    # Pre-built seed templates (retail/loyalty-segmentation)
+├── evaluation/                           # ★ Maintainer-only scenarios, sb-eval CLI, SkillForge adapter
 ├── tests/                                # Playwright E2E for the generator (targets :9000)
 ├── install.sh                            # End-user installer (downloads skill + ai-dev-kit)
 └── docs/                                 # Screenshots for README
@@ -199,6 +202,18 @@ cd app/test/app_template_test/app
 ./start.sh                    # Boots the LuxeBeauty test app on :8765
 ```
 
+For **maintainer evaluation** (from the repository root):
+
+```bash
+uv run sb-eval cases validate
+uv run sb-eval doctor
+uv run sb-eval run --levels L1,L3
+# Manual, guarded, non-production only:
+uv run sb-eval run --levels all --live --scenario <id|all>
+```
+
+SkillForge remains separately installed at `evaluation/skillforge.lock.yaml`'s exact revision. Reports and transient fixtures land under gitignored `test-runs/skillforge/`. Scores are advisory; live cleanup failures and leaked resources fail the evaluation command.
+
 ## Conventions
 
 - **Python**: `uv` only, never `pip`. Use `claude-agent-sdk` (NOT the deprecated `Skill` tool name — pass `skills=` to the SDK).

diff --git a/evaluation/README.md b/evaluation/README.md
@@ -0,0 +1,47 @@
+# Maintainer evaluation
+
+`evaluation/` is the durable, runner-neutral evaluation API for Solution Builder. It is outside `app/`, the shipped `databricks-demo-generator` skill, deployed app wheel artifacts, installers, and all deployed routes.
+
+## Contracts
+
+- `cases/*.yaml` contains the versioned canonical scenarios. `tests/pipeline` consumes the same files.
+- `models.py` and `schema/scenario.schema.json` define the Pydantic and JSON Schema contracts.
+- `adapter.py` converts a scenario losslessly into transient SkillForge v5 assets with `shared_cwd: true`.
+- `skillforge.lock.yaml` pins SkillForge and ai-dev-kit by full Git commit.
+- `EvalRun` is the normalized result contract. Generated fixtures, raw output, normalized JSON, HTML, and leak reports live under `test-runs/skillforge/`.
+
+SkillForge is never installed by `sb-eval`. The CLI discovers the external `stf` executable, requires `stf build-info --json`, and verifies its version, revision, and safety features against the lock.
+
+## Commands
+
+```bash
+uv run sb-eval cases validate
+uv run sb-eval doctor
+uv run sb-eval run --levels L1,L3
+uv run sb-eval run --levels all --live --scenario financial-services
+```
+
+`--scenario all` is the default. Scores and gaps are informational; the command does not apply a minimum score. External runner errors, `invalid_eval`, missing live requirements, and cleanup leaks return nonzero.
+
+## Live guardrails
+
+Live runs are manual only. Set all of:
+
+```bash
+export SB_EVAL_DATABRICKS_PROFILE=solution-builder-eval
+export SB_EVAL_ALLOWED_PROFILES=solution-builder-eval
+export SB_EVAL_ALLOWED_HOSTS=https://non-production-workspace.example.com
+```
+
+The profile name may not be `DEFAULT`, `prod`, or `production`. The resolved host must exactly match the allowlist, and `databricks current-user me` must identify the caller as a service principal. Every run/case/side receives distinct `SB_EVAL_CATALOG`, `SB_EVAL_SCHEMA`, and `SB_EVAL_RESOURCE_PREFIX` values. Setup writes `.skillforge/run-context.json`; cleanup reconciles `resources.json` with SkillForge tracking, deletes in reverse dependency order, retries, and writes `leak-report.json`. Any remainder fails the command.
+
+## Improvement loop
+
+1. Run a quick or full evaluation on a feature branch.
+2. Inspect normalized gaps, raw SkillForge output, and MLflow traces.
+3. Run `/forge-improve <skill> --run-id <id>` locally.
+4. Review the skill diff; improvement never writes directly to `main`.
+5. Re-run the identical scenarios and confirm no regressions or leaks.
+6. Submit the change through the normal reviewed PR process.
+
+Shared reports belong in MLflow and CI artifacts, not Git. No GitHub credentials, evaluation controls, or reports belong in the deployed app.
diff --git a/evaluation/__init__.py b/evaluation/__init__.py
@@ -0,0 +1,9 @@
+"""Maintainer-only evaluation tooling for Solution Builder.
+
+This package is deliberately rooted outside ``app/`` and is not included in
+the generator wheel, installer, or copied demo-generator skill.
+"""
+
+from .models import EvalRun, Scenario
+
+__all__ = ["EvalRun", "Scenario"]