Add versioned pipeline scenario contracts

.claude/skills/databricks-demo-generator/SKILL.md

@@ -29,7 +29,16 @@ The main loop lives in this file (SKILL.md) — it describes **the flow**: stage
 
 ## Paths
 
-Your system prompt defines `PROJECT`, `SKILLS`, `DEMO_SKILL_DIR`, and `DEMO_SKILL` as absolute paths. This skill refers to sibling files like `DEMO_SKILL_DIR/stages/*.md`, `DEMO_SKILL_DIR/app/app.md`, `DEMO_SKILL_DIR/references/*`.
+Your system prompt normally defines `PROJECT`, `SKILLS`, `DEMO_SKILL_DIR`, and `DEMO_SKILL` as absolute paths. This skill refers to sibling files like `DEMO_SKILL_DIR/stages/*.md`, `DEMO_SKILL_DIR/app/app.md`, `DEMO_SKILL_DIR/references/*`.
+
+When those injected aliases are absent (for example when the skill is used directly from a project checkout), self-locate before reading any references:
+
+- `PROJECT` is the execution working directory.
+- `SKILLS` is `PROJECT/.claude/skills`.
+- `DEMO_SKILL_DIR` is `SKILLS/databricks-demo-generator`.
+- `DEMO_SKILL` is `DEMO_SKILL_DIR/SKILL.md`.
+
+Resolve and use absolute paths from those fallbacks. Injected aliases, when present, remain authoritative.
 
 **When spawning subagents**, substitute every placeholder (`DEMO_SKILL_DIR/…`, `PROJECT/…`, `SKILLS/…`) with its real absolute path before sending — the subagent has no system prompt defining them. The full spawn prompt is in `DEMO_SKILL_DIR/stages/03-build.md` → Step 2.
 

CLAUDE.md

@@ -16,6 +16,10 @@ A **system that generates Databricks demos**. Not one app — **three**, plus a
 
 Plus: **ai_dev_kit** (`app/ai_dev_kit/`) — a cloned external repo (`github.com/databricks-solutions/ai-dev-kit`) holding ~26 sub-skills for creating individual Databricks resources (pipelines, dashboards, Genie spaces, KAs, MAS, etc.). The generator's agent uses these during the Build stage.
 
+Versioned **pipeline scenario contracts** live under `evaluation/`. They are
+runner-neutral test data and are not imported by or packaged with the deployed
+generator app, copied into projects, or exposed through application routes.
+
 ## Mental model
 
 ```
@@ -81,6 +85,7 @@ industry-demo-prompts/
 │       ├── app.md                        # How to design+spec a demo app (read during Stage 2)
 │       └── app_template/                 # ★ The template app emitted by the generator
 ├── initial_templates/                    # Pre-built seed templates (retail/loyalty-segmentation)
+├── evaluation/                           # Versioned pipeline scenario contracts + schema
 ├── tests/                                # Playwright E2E for the generator (targets :9000)
 ├── install.sh                            # End-user installer (downloads skill + ai-dev-kit)
 └── docs/                                 # Screenshots for README
@@ -199,6 +204,13 @@ cd app/test/app_template_test/app
 ./start.sh                    # Boots the LuxeBeauty test app on :8765
 ```
 
+To validate the shared pipeline scenarios (from the repository root):
+
+```bash
+uv run sb-eval cases validate
+uv run python -m evaluation.schema_generator --check
+```
+
 ## Conventions
 
 - **Python**: `uv` only, never `pip`. Use `claude-agent-sdk` (NOT the deprecated `Skill` tool name — pass `skills=` to the SDK).

evaluation/README.md

@@ -0,0 +1,26 @@
+# Scenario contracts
+
+`evaluation/` contains the versioned, runner-neutral scenarios used by the
+Solution Builder pipeline tests. It is outside `app/`, the shipped
+`databricks-demo-generator` skill, installers, and deployed application routes.
+
+## Contents
+
+- `cases/*.yaml` defines the canonical prompts, stages, capabilities, expected
+  artifacts, assertions, citations, and cleanup ownership.
+- `models.py` validates those files with strict Pydantic models.
+- `schema/scenario.schema.json` is the committed JSON Schema representation.
+- `tests/pipeline/scenarios.py` exposes a small compatibility view for the
+  existing pipeline harness.
+
+## Validation
+
+```bash
+uv run sb-eval cases validate
+uv run python -m evaluation.schema_generator --check
+uv run pytest tests/evaluation
+```
+
+These contracts intentionally do not select, install, or integrate an
+evaluation runner. Maintainers can consume the YAML through tooling outside
+this public repository.

evaluation/__init__.py

@@ -0,0 +1,9 @@
+"""Versioned scenario contracts for Solution Builder tests.
+
+This package is deliberately rooted outside ``app/`` and is not included in
+the generator wheel, installer, or copied demo-generator skill.
+"""
+
+from .models import Scenario
+
+__all__ = ["Scenario"]

evaluation/cases/__init__.py

@@ -0,0 +1,53 @@
+"""Load and validate canonical Solution Builder evaluation cases."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import yaml
+from pydantic import ValidationError
+
+from evaluation.models import Scenario
+
+
+CASES_DIR = Path(__file__).parent
+
+
+class CaseValidationError(ValueError):
+    pass
+
+
+def case_paths(cases_dir: Path = CASES_DIR) -> list[Path]:
+    return sorted(path for path in cases_dir.glob("*.yaml") if path.is_file())
+
+
+def load_case(path: Path) -> Scenario:
+    try:
+        raw = yaml.safe_load(path.read_text(encoding="utf-8"))
+        return Scenario.model_validate(raw)
+    except (OSError, yaml.YAMLError, ValidationError, TypeError) as exc:
+        raise CaseValidationError(f"{path}: {exc}") from exc
+
+
+def load_cases(cases_dir: Path = CASES_DIR) -> list[Scenario]:
+    paths = case_paths(cases_dir)
+    if not paths:
+        raise CaseValidationError(f"no scenario YAML files found in {cases_dir}")
+    cases = [load_case(path) for path in paths]
+    ids = [case.id for case in cases]
+    if len(ids) != len(set(ids)):
+        raise CaseValidationError("scenario ids must be unique across case files")
+    return cases
+
+
+def select_cases(selector: str | None, cases_dir: Path = CASES_DIR) -> list[Scenario]:
+    cases = load_cases(cases_dir)
+    if selector in (None, "all"):
+        return cases
+    selected = [case for case in cases if case.id == selector]
+    if not selected:
+        available = ", ".join(case.id for case in cases)
+        raise CaseValidationError(
+            f"unknown scenario {selector!r}; available: {available}"
+        )
+    return selected

evaluation/cases/financial-services.yaml

@@ -0,0 +1,94 @@
+schema_version: 1
+id: financial-services
+skill: databricks-demo-generator
+description: >-
+  Real-time fraud detection demo for a retail bank. Generate synthetic
+  credit-card transactions, surface anomalies in a Genie space, and visualize
+  trends in an AI/BI dashboard.
+capabilities: [genie, aibi-dashboards, synthetic-data-gen]
+timeout_seconds: 3600
+target_stage: BUILT
+live_resources:
+  cleanup_owner: solution-builder
+  expected_resource_kinds: [catalog, schema, table, dashboard, genie_space]
+  additional_resource_kinds: []
+  evaluation_prefix: sb_eval_
+steps:
+  - id: capture-story
+    prompt: >-
+      Build a fraud-detection demo for a retail bank. Use synthetic credit-card
+      transactions, an anomaly-detection pattern, a Genie space for ad-hoc
+      questions, and an AI/BI dashboard for the trend view.
+    expected_project_stage: SUMMARIZED
+    required_artifacts:
+      - {path: README.md, description: Approved business story}
+      - {path: resources.json, description: Capability and resource manifest}
+    semantic_assertions:
+      - The story has a named banking protagonist, a concrete fraud catalyst, and quantified business impact.
+      - Synthetic transactions, the anomaly narrative, Genie questions, and dashboard measures form one coherent story.
+    expected_facts:
+      - The proposed demo uses synthetic credit-card transactions, Genie, and an AI/BI dashboard.
+    expected_patterns: ['(?i)fraud', '(?i)(revenue|loss|risk|\$)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: README.md and resources.json exist after story capture.
+        command: "bash -lc 'test -f README.md && test -f resources.json && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects generic fraud stories whose selected products or generated data do not support the same investigation.
+    sources:
+      - &skill_source
+        type: github
+        uri: repo://.claude/skills/databricks-demo-generator/SKILL.md
+        title: Databricks Demo Generator workflow and coherence contract
+        retrieved_at: 2026-07-17T00:00:00Z
+        snippet: Story first and coherence above all; every showcased product must earn a clear beat in the walkthrough.
+  - id: design-architecture
+    prompt: Looks good — proceed to architect the pipeline and produce architecture.md.
+    expected_project_stage: ARCHITECTED
+    required_artifacts:
+      - {path: architecture.md, description: Renderable architecture diagram JSON}
+    semantic_assertions:
+      - The architecture connects transaction generation to governed data, Genie, and the dashboard without orphan components.
+    expected_facts: [The project contains architecture.md.]
+    expected_patterns: ['architecture\.md', '(?i)(genie|dashboard)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: architecture.md exists and is not empty.
+        command: "bash -lc 'test -s architecture.md && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects architecture output that omits a selected capability or breaks the end-to-end data flow.
+    sources: [*skill_source]
+  - id: write-specifications
+    prompt: Now write the specifications for each component (specifications/*.md).
+    expected_project_stage: SPECIFICATION
+    required_artifacts:
+      - {path: specifications/*.md, description: Functional component specifications}
+    semantic_assertions:
+      - Specifications define compatible transaction fields, fraud measures, dashboard visuals, and Genie questions.
+    expected_facts: [Per-component specifications are written under specifications/.]
+    expected_patterns: ['specifications/', '(?i)(transaction|anomal)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: At least one Markdown specification exists.
+        command: "bash -lc 'find specifications -name \"*.md\" -type f -print -quit | grep -q . && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects specs that are individually plausible but disagree on schemas, measures, or the investigation path.
+    sources: [*skill_source]
+  - id: build-demo
+    prompt: "Build it: write the SQL/Python and capture deployed resource IDs in resources.json."
+    expected_project_stage: BUILT
+    required_artifacts:
+      - {path: resources.json, description: Populated deployed resource manifest}
+      - {path: '**/*.{py,sql,json}', description: Buildable implementation artifacts}
+    semantic_assertions:
+      - Build artifacts implement the approved specifications and resources.json records only resources actually created.
+      - Live resource names use the evaluation namespace supplied in the environment when present.
+    expected_facts: [SQL or Python implementation artifacts and resource identifiers are produced.]
+    expected_patterns: ['resources\.json', '(?i)(sql|python|dashboard|genie)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: resources.json is valid JSON and implementation code exists.
+        command: "bash -lc 'python3 -m json.tool resources.json >/dev/null && find . -type f -name \"*.py\" -print -quit | grep -q . && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects claimed builds with placeholders, missing code, dead resource links, or resources outside the evaluation namespace.
+    sources: [*skill_source]

evaluation/cases/healthcare.yaml

@@ -0,0 +1,91 @@
+schema_version: 1
+id: healthcare
+skill: databricks-demo-generator
+description: >-
+  Patient-readmission risk demo for a hospital network. Combine clinical notes
+  (knowledge assistant and vector search) with a tabular ML model for 30-day
+  readmission risk.
+capabilities: [knowledge-assistant, vector-search, ml-training-serving]
+timeout_seconds: 3600
+target_stage: BUILT
+live_resources:
+  cleanup_owner: solution-builder
+  expected_resource_kinds: [catalog, schema, table, volume, vector_index, knowledge_assistant, model, serving_endpoint, mlflow_experiment]
+  additional_resource_kinds: []
+  evaluation_prefix: sb_eval_
+steps:
+  - id: capture-story
+    prompt: >-
+      Build a patient-readmission risk demo. Knowledge Assistant over
+      discharge-summary documents, vector search over the same corpus, and a
+      tabular ML model that scores 30-day readmission risk on synthetic patient data.
+    expected_project_stage: SUMMARIZED
+    required_artifacts:
+      - {path: README.md, description: Approved clinical story}
+      - {path: resources.json, description: Capability and resource manifest}
+    semantic_assertions:
+      - The story explains distinct, complementary roles for document retrieval and tabular risk scoring.
+      - The clinical narrative uses synthetic data and avoids claims that the demo provides medical advice.
+    expected_facts: [The demo combines discharge summaries with a 30-day readmission risk model.]
+    expected_patterns: ['(?i)readmission', '(?i)(discharge|clinical)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: README.md and resources.json exist after story capture.
+        command: "bash -lc 'test -f README.md && test -f resources.json && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects healthcare stories that collapse RAG and predictive ML into an incoherent or unsafe workflow.
+    sources:
+      - &skill_source
+        type: github
+        uri: repo://.claude/skills/databricks-demo-generator/SKILL.md
+        title: Databricks Demo Generator workflow and coherence contract
+        retrieved_at: 2026-07-17T00:00:00Z
+        snippet: Data, pipeline, dashboard, Genie, and agents must align; one broken link ruins the demo.
+  - id: design-architecture
+    prompt: "Proceed: architect the components and produce architecture.md."
+    expected_project_stage: ARCHITECTED
+    required_artifacts: [{path: architecture.md, description: Renderable architecture diagram JSON}]
+    semantic_assertions:
+      - The architecture separates document ingestion and vector retrieval from feature engineering, training, and serving while showing their user-facing convergence.
+    expected_facts: [The project contains architecture.md.]
+    expected_patterns: ['architecture\.md', '(?i)(vector|model|assistant)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: architecture.md exists and is not empty.
+        command: "bash -lc 'test -s architecture.md && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects architectures that omit the shared corpus, model-serving path, or governance boundaries.
+    sources: [*skill_source]
+  - id: write-specifications
+    prompt: Write the per-component specifications under specifications/.
+    expected_project_stage: SPECIFICATION
+    required_artifacts: [{path: specifications/*.md, description: Functional component specifications}]
+    semantic_assertions:
+      - Specs define a consistent patient identifier, temporal split, discharge corpus, vector index, model target, and serving contract.
+    expected_facts: [Per-component specifications are written under specifications/.]
+    expected_patterns: ['specifications/', '(?i)(30-day|readmission)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: At least one Markdown specification exists.
+        command: "bash -lc 'find specifications -name \"*.md\" -type f -print -quit | grep -q . && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects temporal leakage, mismatched patient keys, or a Knowledge Assistant corpus disconnected from the story.
+    sources: [*skill_source]
+  - id: build-demo
+    prompt: Build it now — generate the code, training notebook, and resources.json.
+    expected_project_stage: BUILT
+    required_artifacts:
+      - {path: resources.json, description: Populated deployed resource manifest}
+      - {path: '**/*.{py,sql,json}', description: Buildable implementation artifacts}
+    semantic_assertions:
+      - Training and retrieval artifacts implement the specifications and resources.json contains only validated IDs.
+      - Live resource names use the evaluation namespace supplied in the environment when present.
+    expected_facts: [Training code, retrieval configuration, and resource identifiers are produced.]
+    expected_patterns: ['resources\.json', '(?i)(train|vector|knowledge)']
+    tool_expectations: {required: [Read, Write], banned: []}
+    verification_commands:
+      - fact: resources.json is valid JSON and Python or SQL code exists.
+        command: "bash -lc 'python3 -m json.tool resources.json >/dev/null && find . -type f -name \"*.py\" -print -quit | grep -q . && echo {\"ok\":true}'"
+        check: .ok == true
+    regression_intent: Detects builds that claim live RAG or serving assets without code, identifiers, or cleanup-safe names.
+    sources: [*skill_source]