diff --git a/.claude/skills/agent-bricks/3-multi-agent-supervisors.md b/.claude/skills/agent-bricks/3-multi-agent-supervisors.md deleted file mode 100644 index c546d242..00000000 --- a/.claude/skills/agent-bricks/3-multi-agent-supervisors.md +++ /dev/null @@ -1,237 +0,0 @@ -# Multi-Agent Supervisors (MAS) - -Multi-Agent Supervisors orchestrate multiple specialized agents, routing user queries to the most appropriate agent based on the query content. - -## What is a Multi-Agent Supervisor? - -A MAS acts as a traffic controller for multiple AI agents. When a user asks a question: - -1. **Analyzes** the query to understand the intent -2. **Routes** to the most appropriate specialized agent -3. **Returns** the agent's response to the user - -This allows you to combine multiple specialized agents into a single unified interface. - -## When to Use - -Use a Multi-Agent Supervisor when: -- You have multiple specialized agents (billing, technical support, HR, etc.) -- Users shouldn't need to know which agent to ask -- You want to provide a unified conversational experience - -## Prerequisites - -Before creating a MAS, you need agents of one or both types: - -**Model Serving Endpoints** (`endpoint_name`): -- Knowledge Assistant (KA) endpoints (e.g., `ka-abc123-endpoint`) -- Custom agents built with LangChain, LlamaIndex, etc. -- Fine-tuned models -- RAG applications - -**Genie Spaces** (`genie_space_id`): -- Existing Genie spaces for SQL-based data exploration -- Great for analytics, metrics, and data-driven questions -- No separate endpoint deployment required - reference the space directly -- To find a Genie space by name, use `find_genie_by_name(display_name="My Genie")` -- **Note**: There is NO system table for Genie spaces - do not try to query `system.ai.genie_spaces` - -## Creating a Multi-Agent Supervisor - -Use the `create_or_update_mas` tool: - -- `name`: "Customer Support MAS" -- `agents`: - ```json - [ - { - "name": "policy_agent", - "ka_tile_id": "f32c5f73-466b-4798-b3a0-5396b5ece2a5", - "description": "Answers questions about company policies and procedures from indexed documents" - }, - { - "name": "usage_analytics", - "genie_space_id": "01abc123-def4-5678-90ab-cdef12345678", - "description": "Answers data questions about usage metrics, trends, and statistics" - }, - { - "name": "custom_agent", - "endpoint_name": "my-custom-endpoint", - "description": "Handles specialized queries via custom model endpoint" - } - ] - ``` -- `description`: "Routes customer queries to specialized support agents" -- `instructions`: "Analyze the user's question and route to the most appropriate agent. If unclear, ask for clarification." - -This example shows mixing Knowledge Assistants (policy_agent), Genie spaces (usage_analytics), and custom endpoints (custom_agent). - -## Agent Configuration - -Each agent in the `agents` list needs: - -| Field | Required | Description | -|-------|----------|-------------| -| `name` | Yes | Internal identifier for the agent | -| `description` | Yes | What this agent handles (critical for routing) | -| `ka_tile_id` | One of these | Knowledge Assistant tile ID (for document Q&A agents) | -| `genie_space_id` | One of these | Genie space ID (for SQL-based data agents) | -| `endpoint_name` | One of these | Model serving endpoint name (for custom agents) | - -**Note**: Provide exactly one of: `ka_tile_id`, `genie_space_id`, or `endpoint_name`. - -To find a KA tile_id, use `find_ka_by_name(name="Your KA Name")`. -To find a Genie space_id, use `find_genie_by_name(display_name="Your Genie Name")`. - -### Writing Good Descriptions - -The `description` field is critical for routing. Make it specific: - -**Good descriptions:** -- "Handles billing questions including invoices, payments, refunds, and subscription changes" -- "Answers technical questions about API errors, integration issues, and product bugs" -- "Provides information about HR policies, PTO, benefits, and employee handbook" - -**Bad descriptions:** -- "Billing agent" (too vague) -- "Handles stuff" (not helpful) -- "Technical" (not specific) - -## Provisioning Timeline - -After creation, the MAS endpoint needs to provision: - -| Status | Meaning | Duration | -|--------|---------|----------| -| `PROVISIONING` | Creating the supervisor | 2-5 minutes | -| `ONLINE` | Ready to route queries | - | -| `OFFLINE` | Not currently running | - | - -Use `get_mas` to check the status. - -## Adding Example Questions - -Example questions help with evaluation and can guide routing optimization: - -```json -{ - "examples": [ - { - "question": "I haven't received my invoice for this month", - "guideline": "Should be routed to billing_agent" - }, - { - "question": "The API is returning a 500 error", - "guideline": "Should be routed to technical_agent" - }, - { - "question": "How many vacation days do I have?", - "guideline": "Should be routed to hr_agent" - } - ] -} -``` - -If the MAS is not yet `ONLINE`, examples are queued and added automatically when ready. - -## Best Practices - -### Agent Design - -1. **Specialized agents**: Each agent should have a clear, distinct purpose -2. **Non-overlapping domains**: Avoid agents with similar descriptions -3. **Clear boundaries**: Define what each agent does and doesn't handle - -### Instructions - -Provide routing instructions: - -``` -You are a customer support supervisor. Your job is to route user queries to the right specialist: - -1. For billing, payments, or subscription questions → billing_agent -2. For technical issues, bugs, or API problems → technical_agent -3. For HR, benefits, or policy questions → hr_agent - -If the query is unclear or spans multiple domains, ask the user to clarify. -``` - -### Fallback Handling - -Consider adding a general-purpose agent for queries that don't fit elsewhere: - -```json -{ - "name": "general_agent", - "endpoint_name": "general-support-endpoint", - "description": "Handles general inquiries that don't fit other categories, provides navigation help" -} -``` - -## Example Workflow - -1. **Deploy specialized agents** as model serving endpoints: - - `billing-assistant-endpoint` - - `tech-support-endpoint` - - `hr-assistant-endpoint` - -2. **Create the MAS**: - - Configure agents with clear descriptions - - Add routing instructions - -3. **Wait for ONLINE status** (2-5 minutes) - -4. **Add example questions** for evaluation - -5. **Test routing** with various query types - -## Updating a Multi-Agent Supervisor - -To update an existing MAS: - -1. **Add/remove agents**: Call `create_or_update_mas` with updated `agents` list -2. **Update descriptions**: Change agent descriptions to improve routing -3. **Modify instructions**: Update routing rules - -The tool finds the existing MAS by name and updates it. - -## Troubleshooting - -### Queries routed to wrong agent - -- Review and improve agent descriptions -- Make descriptions more specific and distinct -- Add examples that demonstrate correct routing - -### Endpoint not responding - -- Verify each underlying model serving endpoint is running -- Check endpoint logs for errors -- Ensure endpoints accept the expected input format - -### Slow responses - -- Check latency of underlying endpoints -- Consider endpoint scaling settings -- Monitor for cold start issues - -## Advanced: Hierarchical Routing - -For complex scenarios, you can create multiple levels of MAS: - -``` -Top-level MAS -├── Customer Support MAS -│ ├── billing_agent -│ ├── technical_agent -│ └── general_agent -├── Sales MAS -│ ├── pricing_agent -│ ├── demo_agent -│ └── contract_agent -└── Internal MAS - ├── hr_agent - └── it_helpdesk_agent -``` - -Each sub-MAS is deployed as an endpoint and configured as an agent in the top-level MAS. diff --git a/.claude/skills/agent-bricks/SKILL.md b/.claude/skills/agent-bricks/SKILL.md deleted file mode 100644 index bd8f8488..00000000 --- a/.claude/skills/agent-bricks/SKILL.md +++ /dev/null @@ -1,151 +0,0 @@ ---- -name: agent-bricks -description: "Create and manage Databricks Agent Bricks: Knowledge Assistants (KA) for document Q&A, Genie Spaces for SQL exploration, and Multi-Agent Supervisors (MAS) for multi-agent orchestration. Use when building conversational AI applications on Databricks." ---- - -# Agent Bricks - -Create and manage Databricks Agent Bricks - pre-built AI components for building conversational applications. - -## Overview - -Agent Bricks are three types of pre-built AI tiles in Databricks: - -| Brick | Purpose | Data Source | -|-------|---------|-------------| -| **Knowledge Assistant (KA)** | Document-based Q&A using RAG | PDF/text files in Volumes | -| **Genie Space** | Natural language to SQL | Unity Catalog tables | -| **Multi-Agent Supervisor (MAS)** | Multi-agent orchestration | Model serving endpoints | - -## Prerequisites - -Before creating Agent Bricks, ensure you have the required data: - -### For Knowledge Assistants -- **Documents in a Volume**: PDF, text, or other files stored in a Unity Catalog volume -- Generate synthetic documents using the `unstructured-pdf-generation` skill if needed - -### For Genie Spaces -- **See the `databricks-genie` skill** for comprehensive Genie Space guidance -- Tables in Unity Catalog with the data to explore -- Generate raw data using the `synthetic-data-generation` skill -- Create tables using the `spark-declarative-pipelines` skill - -### For Multi-Agent Supervisors -- **Model Serving Endpoints**: Deployed agent endpoints (KA endpoints, custom agents, fine-tuned models) -- **Genie Spaces**: Existing Genie spaces can be used directly as agents for SQL-based queries -- Mix and match endpoint-based and Genie-based agents in the same MAS - -## MCP Tools - -### Knowledge Assistant Tools - -**create_or_update_ka** - Create or update a Knowledge Assistant -- `name`: Name for the KA -- `volume_path`: Path to documents (e.g., `/Volumes/catalog/schema/volume/folder`) -- `description`: (optional) What the KA does -- `instructions`: (optional) How the KA should answer -- `tile_id`: (optional) Existing tile_id to update -- `add_examples_from_volume`: (optional, default: true) Auto-add examples from JSON files - -**get_ka** - Get Knowledge Assistant details -- `tile_id`: The KA tile ID - -**find_ka_by_name** - Find a Knowledge Assistant by name -- `name`: The exact name of the KA to find -- Returns: `tile_id`, `name`, `endpoint_name`, `endpoint_status` -- Use this to look up an existing KA when you know the name but not the tile_id - -**delete_ka** - Delete a Knowledge Assistant -- `tile_id`: The KA tile ID to delete - -### Genie Space Tools - -**For comprehensive Genie guidance, use the `databricks-genie` skill.** - -Basic tools available: - -- `create_or_update_genie` - Create or update a Genie Space -- `get_genie` - Get Genie Space details -- `delete_genie` - Delete a Genie Space - -See `databricks-genie` skill for: -- Table inspection workflow -- Sample question best practices -- Curation (instructions, certified queries) - -**IMPORTANT**: There is NO system table for Genie spaces (e.g., `system.ai.genie_spaces` does not exist). To find a Genie space by name, use the `find_genie_by_name` tool. - -### Multi-Agent Supervisor Tools - -**create_or_update_mas** - Create or update a Multi-Agent Supervisor -- `name`: Name for the MAS -- `agents`: List of agent configurations, each with: - - `name`: Agent identifier (required) - - `description`: What this agent handles - critical for routing (required) - - `ka_tile_id`: Knowledge Assistant tile ID (use for document Q&A agents - recommended for KAs) - - `genie_space_id`: Genie space ID (use for SQL-based data agents) - - `endpoint_name`: Model serving endpoint name (use for custom agents) - - Note: Provide exactly one of: `ka_tile_id`, `genie_space_id`, or `endpoint_name` -- `description`: (optional) What the MAS does -- `instructions`: (optional) Routing instructions for the supervisor -- `tile_id`: (optional) Existing tile_id to update -- `examples`: (optional) List of example questions with `question` and `guideline` fields - -**get_mas** - Get Multi-Agent Supervisor details -- `tile_id`: The MAS tile ID - -**find_mas_by_name** - Find a Multi-Agent Supervisor by name -- `name`: The exact name of the MAS to find -- Returns: `tile_id`, `name`, `endpoint_status`, `agents_count` -- Use this to look up an existing MAS when you know the name but not the tile_id - -**delete_mas** - Delete a Multi-Agent Supervisor -- `tile_id`: The MAS tile ID to delete - -## Typical Workflow - -### 1. Generate Source Data - -Before creating Agent Bricks, generate the required source data: - -**For KA (document Q&A)**: -``` -1. Use `unstructured-pdf-generation` skill to generate PDFs -2. PDFs are saved to a Volume with companion JSON files (question/guideline pairs) -``` - -**For Genie (SQL exploration)**: -``` -1. Use `synthetic-data-generation` skill to create raw parquet data -2. Use `spark-declarative-pipelines` skill to create bronze/silver/gold tables -``` - -### 2. Create the Agent Brick - -Use the appropriate `create_or_update_*` tool with your data sources. - -### 3. Wait for Provisioning - -Newly created KA and MAS tiles need time to provision. The endpoint status will progress: -- `PROVISIONING` - Being created (can take 2-5 minutes) -- `ONLINE` - Ready to use -- `OFFLINE` - Not running - -### 4. Add Examples (Automatic) - -For KA, if `add_examples_from_volume=true`, examples are automatically extracted from JSON files in the volume and added once the endpoint is `ONLINE`. - -## Best Practices - -1. **Use meaningful names**: Names are sanitized automatically (spaces become underscores) -2. **Provide descriptions**: Helps users understand what the brick does -3. **Add instructions**: Guide the AI's behavior and tone -4. **Include sample questions**: Shows users how to interact with the brick -5. **Use the workflow**: Generate data first, then create the brick - -## See Also - -- `1-knowledge-assistants.md` - Detailed KA patterns and examples -- `databricks-genie` skill - Detailed Genie patterns, curation, and examples -- `3-multi-agent-supervisors.md` - Detailed MAS patterns and examples diff --git a/.claude/skills/aibi-dashboards/SKILL.md b/.claude/skills/aibi-dashboards/SKILL.md deleted file mode 100644 index bb846b05..00000000 --- a/.claude/skills/aibi-dashboards/SKILL.md +++ /dev/null @@ -1,917 +0,0 @@ ---- -name: aibi-dashboards -description: "Create AI/BI dashboards. CRITICAL: You MUST test ALL SQL queries via execute_sql BEFORE deploying. Follow guidelines strictly." ---- - -# AI/BI Dashboard Skill - -Create Databricks AI/BI dashboards (formerly Lakeview dashboards). **Follow these guidelines strictly.** - -## CRITICAL: MANDATORY VALIDATION WORKFLOW - -**You MUST follow this workflow exactly. Skipping validation causes broken dashboards.** - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ STEP 1: Get table schemas via get_table_details(catalog, schema) │ -├─────────────────────────────────────────────────────────────────────┤ -│ STEP 2: Write SQL queries for each dataset │ -├─────────────────────────────────────────────────────────────────────┤ -│ STEP 3: TEST EVERY QUERY via execute_sql() ← DO NOT SKIP! │ -│ - If query fails, FIX IT before proceeding │ -│ - Verify column names match what widgets will reference │ -│ - Verify data types are correct (dates, numbers, strings) │ -├─────────────────────────────────────────────────────────────────────┤ -│ STEP 4: Build dashboard JSON using ONLY verified queries │ -├─────────────────────────────────────────────────────────────────────┤ -│ STEP 5: Deploy via create_or_update_dashboard() │ -└─────────────────────────────────────────────────────────────────────┘ -``` - -**WARNING: If you deploy without testing queries, widgets WILL show "Invalid widget definition" errors!** - -## Available MCP Tools - -| Tool | Description | -|------|-------------| -| `get_table_details` | **STEP 1**: Get table schemas for designing queries | -| `execute_sql` | **STEP 3**: Test SQL queries - MANDATORY before deployment! | -| `get_best_warehouse` | Get available warehouse ID | -| `create_or_update_dashboard` | **STEP 5**: Deploy dashboard JSON (only after validation!) | -| `get_dashboard` | Get dashboard details by ID | -| `list_dashboards` | List dashboards in workspace | -| `trash_dashboard` | Move dashboard to trash | -| `publish_dashboard` | Publish dashboard for viewers | -| `unpublish_dashboard` | Unpublish a dashboard | - ---- - -## Implementation Guidelines - -### 1) DATASET ARCHITECTURE (STRICT) - -- **One dataset per domain** (e.g., orders, customers, products) -- **Exactly ONE valid SQL query per dataset** (no multiple queries separated by `;`) -- Always use **fully-qualified table names**: `catalog.schema.table_name` -- SELECT must include all dimensions needed by widgets and all derived columns via `AS` aliases -- Put ALL business logic (CASE/WHEN, COALESCE, ratios) into the dataset SELECT with explicit aliases -- **Contract rule**: Every widget `fieldName` must exactly match a dataset column or alias - -### 2) WIDGET FIELD EXPRESSIONS - -> **CRITICAL: Field Name Matching Rule** -> The `name` in `query.fields` MUST exactly match the `fieldName` in `encodings`. -> If they don't match, the widget shows "no selected fields to visualize" error! - -**Correct pattern for aggregations:** -```json -// In query.fields: -{"name": "sum(spend)", "expression": "SUM(`spend`)"} - -// In encodings (must match!): -{"fieldName": "sum(spend)", "displayName": "Total Spend"} -``` - -**WRONG - names don't match:** -```json -// In query.fields: -{"name": "spend", "expression": "SUM(`spend`)"} // name is "spend" - -// In encodings: -{"fieldName": "sum(spend)", ...} // ERROR: "sum(spend)" ≠ "spend" -``` - -Allowed expressions in widget queries (you CANNOT use CAST or other SQL in expressions): - -**For numbers:** -```json -{"name": "sum(revenue)", "expression": "SUM(`revenue`)"} -{"name": "avg(price)", "expression": "AVG(`price`)"} -{"name": "count(orders)", "expression": "COUNT(`order_id`)"} -{"name": "countdistinct(customers)", "expression": "COUNT(DISTINCT `customer_id`)"} -{"name": "min(date)", "expression": "MIN(`order_date`)"} -{"name": "max(date)", "expression": "MAX(`order_date`)"} -``` - -**For dates** (use daily for timeseries, weekly/monthly for grouped comparisons): -```json -{"name": "daily(date)", "expression": "DATE_TRUNC(\"DAY\", `date`)"} -{"name": "weekly(date)", "expression": "DATE_TRUNC(\"WEEK\", `date`)"} -{"name": "monthly(date)", "expression": "DATE_TRUNC(\"MONTH\", `date`)"} -``` - -**Simple field reference** (for pre-aggregated data): -```json -{"name": "category", "expression": "`category`"} -``` - -If you need conditional logic or multi-field formulas, compute a derived column in the dataset SQL first. - -### 3) SPARK SQL PATTERNS - -- Date math: `date_sub(current_date(), N)` for days, `add_months(current_date(), -N)` for months -- Date truncation: `DATE_TRUNC('DAY'|'WEEK'|'MONTH'|'QUARTER'|'YEAR', column)` -- **AVOID** `INTERVAL` syntax - use functions instead - -### 4) LAYOUT (6-Column Grid, NO GAPS) - -Each widget has a position: `{"x": 0, "y": 0, "width": 2, "height": 4}` - -**CRITICAL**: Each row must fill width=6 exactly. No gaps allowed. - -**Recommended widget sizes:** - -| Widget Type | Width | Height | Notes | -|-------------|-------|--------|-------| -| Text header | 6 | 1 | Full width; use SEPARATE widgets for title and subtitle | -| Counter/KPI | 2 | **3-4** | **NEVER height=2** - too cramped! | -| Line/Bar chart | 3 | **5-6** | Pair side-by-side to fill row | -| Pie chart | 3 | **5-6** | Needs space for legend | -| Full-width chart | 6 | 5-7 | For detailed time series | -| Table | 6 | 5-8 | Full width for readability | - -**Standard dashboard structure:** -```text -y=0: Title (w=6, h=1) - Dashboard title (use separate widget!) -y=1: Subtitle (w=6, h=1) - Description (use separate widget!) -y=2: KPIs (w=2 each, h=3) - 3 key metrics side-by-side -y=5: Section header (w=6, h=1) - "Trends" or similar -y=6: Charts (w=3 each, h=5) - Two charts side-by-side -y=11: Section header (w=6, h=1) - "Details" -y=12: Table (w=6, h=6) - Detailed data -``` - -### 5) CARDINALITY & READABILITY (CRITICAL) - -**Dashboard readability depends on limiting distinct values:** - -| Dimension Type | Max Values | Examples | -|----------------|------------|----------| -| Chart color/groups | **3-8** | 4 regions, 5 product lines, 3 tiers | -| Filters | 4-10 | 8 countries, 5 channels | -| High cardinality | **Table only** | customer_id, order_id, SKU | - -**Before creating any chart with color/grouping:** -1. Check column cardinality (use `get_table_details` to see distinct values) -2. If >10 distinct values, aggregate to higher level OR use TOP-N + "Other" bucket -3. For high-cardinality dimensions, use a table widget instead of a chart - -### 6) WIDGET SPECIFICATIONS - -**Widget Naming Convention (CRITICAL):** -- `widget.name`: alphanumeric + hyphens + underscores ONLY (no spaces, parentheses, colons) -- `frame.title`: human-readable name (any characters allowed) -- `widget.queries[0].name`: always use `"main_query"` - -**CRITICAL VERSION REQUIREMENTS:** - -| Widget Type | Version | -|-------------|---------| -| counter | 2 | -| table | 2 | -| filter-multi-select | 2 | -| filter-single-select | 2 | -| filter-date-range-picker | 2 | -| bar | 3 | -| line | 3 | -| pie | 3 | -| text | N/A (no spec block) | - ---- - -**Text (Headers/Descriptions):** -- **CRITICAL: Text widgets do NOT use a spec block!** -- Use `multilineTextboxSpec` directly on the widget -- Supports markdown: `#`, `##`, `###`, `**bold**`, `*italic*` -- **CRITICAL: Multiple items in the `lines` array are concatenated on a single line, NOT displayed as separate lines!** -- For title + subtitle, use **separate text widgets** at different y positions - -```json -// CORRECT: Separate widgets for title and subtitle -{ - "widget": { - "name": "title", - "multilineTextboxSpec": { - "lines": ["## Dashboard Title"] - } - }, - "position": {"x": 0, "y": 0, "width": 6, "height": 1} -}, -{ - "widget": { - "name": "subtitle", - "multilineTextboxSpec": { - "lines": ["Description text here"] - } - }, - "position": {"x": 0, "y": 1, "width": 6, "height": 1} -} - -// WRONG: Multiple lines concatenate into one line! -{ - "widget": { - "name": "title-widget", - "multilineTextboxSpec": { - "lines": ["## Dashboard Title", "Description text here"] // Becomes "## Dashboard TitleDescription text here" - } - }, - "position": {"x": 0, "y": 0, "width": 6, "height": 2} -} -``` - ---- - -**Counter (KPI):** -- `version`: **2** (NOT 3!) -- `widgetType`: "counter" -- **Percent values must be 0-1** in the data (not 0-100) - -**Two patterns for counters:** - -**Pattern 1: Pre-aggregated dataset (1 row, no filters)** -- Dataset returns exactly 1 row -- Use `"disaggregated": true` and simple field reference -- Field `name` matches dataset column directly - -```json -{ - "widget": { - "name": "total-revenue", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "summary_ds", - "fields": [{"name": "revenue", "expression": "`revenue`"}], - "disaggregated": true - } - }], - "spec": { - "version": 2, - "widgetType": "counter", - "encodings": { - "value": {"fieldName": "revenue", "displayName": "Total Revenue"} - }, - "frame": {"showTitle": true, "title": "Total Revenue"} - } - }, - "position": {"x": 0, "y": 0, "width": 2, "height": 3} -} -``` - -**Pattern 2: Aggregating widget (multi-row dataset, supports filters)** -- Dataset returns multiple rows (e.g., grouped by a filter dimension) -- Use `"disaggregated": false` and aggregation expression -- **CRITICAL**: Field `name` MUST match `fieldName` exactly (e.g., `"sum(spend)"`) - -```json -{ - "widget": { - "name": "total-spend", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "by_category", - "fields": [{"name": "sum(spend)", "expression": "SUM(`spend`)"}], - "disaggregated": false - } - }], - "spec": { - "version": 2, - "widgetType": "counter", - "encodings": { - "value": {"fieldName": "sum(spend)", "displayName": "Total Spend"} - }, - "frame": {"showTitle": true, "title": "Total Spend"} - } - }, - "position": {"x": 0, "y": 0, "width": 2, "height": 3} -} -``` - ---- - -**Table:** -- `version`: **2** (NOT 1 or 3!) -- `widgetType`: "table" -- **Columns only need `fieldName` and `displayName`** - no other properties! -- Use `"disaggregated": true` for raw rows - -```json -{ - "widget": { - "name": "details-table", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "details_ds", - "fields": [ - {"name": "name", "expression": "`name`"}, - {"name": "value", "expression": "`value`"} - ], - "disaggregated": true - } - }], - "spec": { - "version": 2, - "widgetType": "table", - "encodings": { - "columns": [ - {"fieldName": "name", "displayName": "Name"}, - {"fieldName": "value", "displayName": "Value"} - ] - }, - "frame": {"showTitle": true, "title": "Details"} - } - }, - "position": {"x": 0, "y": 0, "width": 6, "height": 6} -} -``` - ---- - -**Line / Bar Charts:** -- `version`: **3** -- `widgetType`: "line" or "bar" -- Use `x`, `y`, optional `color` encodings -- `scale.type`: `"temporal"` (dates), `"quantitative"` (numbers), `"categorical"` (strings) -- Use `"disaggregated": true` with pre-aggregated dataset data - -**Multiple Lines - Two Approaches:** - -1. **Multi-Y Fields** (different metrics on same chart): -```json -"y": { - "scale": {"type": "quantitative"}, - "fields": [ - {"fieldName": "sum(orders)", "displayName": "Orders"}, - {"fieldName": "sum(returns)", "displayName": "Returns"} - ] -} -``` - -2. **Color Grouping** (same metric split by dimension): -```json -"y": {"fieldName": "sum(revenue)", "scale": {"type": "quantitative"}}, -"color": {"fieldName": "region", "scale": {"type": "categorical"}, "displayName": "Region"} -``` - -**Bar Chart Modes:** -- **Stacked** (default): No `mark` field - bars stack on top of each other -- **Grouped**: Add `"mark": {"layout": "group"}` - bars side-by-side for comparison - -**Pie Chart:** -- `version`: **3** -- `widgetType`: "pie" -- `angle`: quantitative aggregate -- `color`: categorical dimension -- Limit to 3-8 categories for readability - -### 7) FILTERS (Global vs Page-Level) - -> **CRITICAL**: Filter widgets use DIFFERENT widget types than charts! -> - Valid types: `filter-multi-select`, `filter-single-select`, `filter-date-range-picker` -> - **DO NOT** use `widgetType: "filter"` - this does not exist and will cause errors -> - Filters use `spec.version: 2` -> - **ALWAYS include `frame` with `showTitle: true`** for filter widgets - -**Filter widget types:** -- `filter-date-range-picker`: for DATE/TIMESTAMP fields -- `filter-single-select`: categorical with single selection -- `filter-multi-select`: categorical with multiple selections - ---- - -#### Global Filters vs Page-Level Filters - -| Type | Placement | Scope | Use Case | -|------|-----------|-------|----------| -| **Global Filter** | Dedicated page with `"pageType": "PAGE_TYPE_GLOBAL_FILTERS"` | Affects ALL pages that have datasets with the filter field | Cross-dashboard filtering (e.g., date range, campaign) | -| **Page-Level Filter** | Regular page with `"pageType": "PAGE_TYPE_CANVAS"` | Affects ONLY widgets on that same page | Page-specific filtering (e.g., platform filter on breakdown page only) | - -**Key Insight**: A filter only affects datasets that contain the filter field. To have a filter affect only specific pages: -1. Include the filter dimension in datasets for pages that should be filtered -2. Exclude the filter dimension from datasets for pages that should NOT be filtered - ---- - -#### Filter Widget Structure - -> **CRITICAL**: Do NOT use `associative_filter_predicate_group` - it causes SQL errors! -> Use a simple field expression instead. - -```json -{ - "widget": { - "name": "filter_region", - "queries": [{ - "name": "ds_data_region", - "query": { - "datasetName": "ds_data", - "fields": [ - {"name": "region", "expression": "`region`"} - ], - "disaggregated": false - } - }], - "spec": { - "version": 2, - "widgetType": "filter-multi-select", - "encodings": { - "fields": [{ - "fieldName": "region", - "displayName": "Region", - "queryName": "ds_data_region" - }] - }, - "frame": {"showTitle": true, "title": "Region"} - } - }, - "position": {"x": 0, "y": 0, "width": 2, "height": 2} -} -``` - ---- - -#### Global Filter Example - -Place on a dedicated filter page: - -```json -{ - "name": "filters", - "displayName": "Filters", - "pageType": "PAGE_TYPE_GLOBAL_FILTERS", - "layout": [ - { - "widget": { - "name": "filter_campaign", - "queries": [{ - "name": "ds_campaign", - "query": { - "datasetName": "overview", - "fields": [{"name": "campaign_name", "expression": "`campaign_name`"}], - "disaggregated": false - } - }], - "spec": { - "version": 2, - "widgetType": "filter-multi-select", - "encodings": { - "fields": [{ - "fieldName": "campaign_name", - "displayName": "Campaign", - "queryName": "ds_campaign" - }] - }, - "frame": {"showTitle": true, "title": "Campaign"} - } - }, - "position": {"x": 0, "y": 0, "width": 2, "height": 2} - } - ] -} -``` - ---- - -#### Page-Level Filter Example - -Place directly on a canvas page (affects only that page): - -```json -{ - "name": "platform_breakdown", - "displayName": "Platform Breakdown", - "pageType": "PAGE_TYPE_CANVAS", - "layout": [ - { - "widget": { - "name": "page-title", - "multilineTextboxSpec": {"lines": ["## Platform Breakdown"]} - }, - "position": {"x": 0, "y": 0, "width": 4, "height": 1} - }, - { - "widget": { - "name": "filter_platform", - "queries": [{ - "name": "ds_platform", - "query": { - "datasetName": "platform_data", - "fields": [{"name": "platform", "expression": "`platform`"}], - "disaggregated": false - } - }], - "spec": { - "version": 2, - "widgetType": "filter-multi-select", - "encodings": { - "fields": [{ - "fieldName": "platform", - "displayName": "Platform", - "queryName": "ds_platform" - }] - }, - "frame": {"showTitle": true, "title": "Platform"} - } - }, - "position": {"x": 4, "y": 0, "width": 2, "height": 2} - } - // ... other widgets on this page - ] -} -``` - ---- - -**Filter Layout Guidelines:** -- Global filters: Position on dedicated filter page, stack vertically at `x=0` -- Page-level filters: Position in header area of page (e.g., top-right corner) -- Typical sizing: `width: 2, height: 2` - -### 8) QUALITY CHECKLIST - -Before deploying, verify: -1. All widget names use only alphanumeric + hyphens + underscores -2. All rows sum to width=6 with no gaps -3. KPIs use height 3-4, charts use height 5-6 -4. Chart dimensions have ≤8 distinct values -5. All widget fieldNames match dataset columns exactly -6. **Field `name` in query.fields matches `fieldName` in encodings exactly** (e.g., both `"sum(spend)"`) -7. Counter datasets: use `disaggregated: true` for 1-row datasets, `disaggregated: false` with aggregation for multi-row -8. Percent values are 0-1 (not 0-100) -9. SQL uses Spark syntax (date_sub, not INTERVAL) -10. **All SQL queries tested via `execute_sql` and return expected data** - ---- - -## Complete Example - -```python -import json - -# Step 1: Check table schema -table_info = get_table_details(catalog="samples", schema="nyctaxi") - -# Step 2: Test queries -execute_sql("SELECT COUNT(*) as trips, AVG(fare_amount) as avg_fare, AVG(trip_distance) as avg_distance FROM samples.nyctaxi.trips") -execute_sql(""" - SELECT pickup_zip, COUNT(*) as trip_count - FROM samples.nyctaxi.trips - GROUP BY pickup_zip - ORDER BY trip_count DESC - LIMIT 10 -""") - -# Step 3: Build dashboard JSON -dashboard = { - "datasets": [ - { - "name": "summary", - "displayName": "Summary Stats", - "queryLines": [ - "SELECT COUNT(*) as trips, AVG(fare_amount) as avg_fare, ", - "AVG(trip_distance) as avg_distance ", - "FROM samples.nyctaxi.trips " - ] - }, - { - "name": "by_zip", - "displayName": "Trips by ZIP", - "queryLines": [ - "SELECT pickup_zip, COUNT(*) as trip_count ", - "FROM samples.nyctaxi.trips ", - "GROUP BY pickup_zip ", - "ORDER BY trip_count DESC ", - "LIMIT 10 " - ] - } - ], - "pages": [{ - "name": "overview", - "displayName": "NYC Taxi Overview", - "pageType": "PAGE_TYPE_CANVAS", - "layout": [ - # Text header - NO spec block! Use SEPARATE widgets for title and subtitle! - { - "widget": { - "name": "title", - "multilineTextboxSpec": { - "lines": ["## NYC Taxi Dashboard"] - } - }, - "position": {"x": 0, "y": 0, "width": 6, "height": 1} - }, - { - "widget": { - "name": "subtitle", - "multilineTextboxSpec": { - "lines": ["Trip statistics and analysis"] - } - }, - "position": {"x": 0, "y": 1, "width": 6, "height": 1} - }, - # Counter - version 2, width 2! - { - "widget": { - "name": "total-trips", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "summary", - "fields": [{"name": "trips", "expression": "`trips`"}], - "disaggregated": True - } - }], - "spec": { - "version": 2, - "widgetType": "counter", - "encodings": { - "value": {"fieldName": "trips", "displayName": "Total Trips"} - }, - "frame": {"title": "Total Trips", "showTitle": True} - } - }, - "position": {"x": 0, "y": 2, "width": 2, "height": 3} - }, - { - "widget": { - "name": "avg-fare", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "summary", - "fields": [{"name": "avg_fare", "expression": "`avg_fare`"}], - "disaggregated": True - } - }], - "spec": { - "version": 2, - "widgetType": "counter", - "encodings": { - "value": {"fieldName": "avg_fare", "displayName": "Avg Fare"} - }, - "frame": {"title": "Average Fare", "showTitle": True} - } - }, - "position": {"x": 2, "y": 2, "width": 2, "height": 3} - }, - { - "widget": { - "name": "total-distance", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "summary", - "fields": [{"name": "avg_distance", "expression": "`avg_distance`"}], - "disaggregated": True - } - }], - "spec": { - "version": 2, - "widgetType": "counter", - "encodings": { - "value": {"fieldName": "avg_distance", "displayName": "Avg Distance"} - }, - "frame": {"title": "Average Distance", "showTitle": True} - } - }, - "position": {"x": 4, "y": 2, "width": 2, "height": 3} - }, - # Bar chart - version 3 - { - "widget": { - "name": "trips-by-zip", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "by_zip", - "fields": [ - {"name": "pickup_zip", "expression": "`pickup_zip`"}, - {"name": "trip_count", "expression": "`trip_count`"} - ], - "disaggregated": True - } - }], - "spec": { - "version": 3, - "widgetType": "bar", - "encodings": { - "x": {"fieldName": "pickup_zip", "scale": {"type": "categorical"}, "displayName": "ZIP"}, - "y": {"fieldName": "trip_count", "scale": {"type": "quantitative"}, "displayName": "Trips"} - }, - "frame": {"title": "Trips by Pickup ZIP", "showTitle": True} - } - }, - "position": {"x": 0, "y": 5, "width": 6, "height": 5} - }, - # Table - version 2, minimal column props! - { - "widget": { - "name": "zip-table", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "by_zip", - "fields": [ - {"name": "pickup_zip", "expression": "`pickup_zip`"}, - {"name": "trip_count", "expression": "`trip_count`"} - ], - "disaggregated": True - } - }], - "spec": { - "version": 2, - "widgetType": "table", - "encodings": { - "columns": [ - {"fieldName": "pickup_zip", "displayName": "ZIP Code"}, - {"fieldName": "trip_count", "displayName": "Trip Count"} - ] - }, - "frame": {"title": "Top ZIP Codes", "showTitle": True} - } - }, - "position": {"x": 0, "y": 10, "width": 6, "height": 5} - } - ] - }] -} - -# Step 4: Deploy -result = create_or_update_dashboard( - display_name="NYC Taxi Dashboard", - parent_path="/Workspace/Users/me/dashboards", - serialized_dashboard=json.dumps(dashboard), - warehouse_id=get_best_warehouse(), -) -print(result["url"]) -``` - -## Complete Example with Filters - -```python -import json - -# Dashboard with a global filter for region -dashboard_with_filters = { - "datasets": [ - { - "name": "sales", - "displayName": "Sales Data", - "queryLines": [ - "SELECT region, SUM(revenue) as total_revenue ", - "FROM catalog.schema.sales ", - "GROUP BY region" - ] - } - ], - "pages": [ - { - "name": "overview", - "displayName": "Sales Overview", - "pageType": "PAGE_TYPE_CANVAS", - "layout": [ - { - "widget": { - "name": "total-revenue", - "queries": [{ - "name": "main_query", - "query": { - "datasetName": "sales", - "fields": [{"name": "total_revenue", "expression": "`total_revenue`"}], - "disaggregated": True - } - }], - "spec": { - "version": 2, # Version 2 for counters! - "widgetType": "counter", - "encodings": { - "value": {"fieldName": "total_revenue", "displayName": "Total Revenue"} - }, - "frame": {"title": "Total Revenue", "showTitle": True} - } - }, - "position": {"x": 0, "y": 0, "width": 6, "height": 3} - } - ] - }, - { - "name": "filters", - "displayName": "Filters", - "pageType": "PAGE_TYPE_GLOBAL_FILTERS", # Required for global filter page! - "layout": [ - { - "widget": { - "name": "filter_region", - "queries": [{ - "name": "ds_sales_region", - "query": { - "datasetName": "sales", - "fields": [ - {"name": "region", "expression": "`region`"} - # DO NOT use associative_filter_predicate_group - causes SQL errors! - ], - "disaggregated": False # False for filters! - } - }], - "spec": { - "version": 2, # Version 2 for filters! - "widgetType": "filter-multi-select", # NOT "filter"! - "encodings": { - "fields": [{ - "fieldName": "region", - "displayName": "Region", - "queryName": "ds_sales_region" # Must match query name! - }] - }, - "frame": {"showTitle": True, "title": "Region"} # Always show title! - } - }, - "position": {"x": 0, "y": 0, "width": 2, "height": 2} - } - ] - } - ] -} - -# Deploy with filters -result = create_or_update_dashboard( - display_name="Sales Dashboard with Filters", - parent_path="/Workspace/Users/me/dashboards", - serialized_dashboard=json.dumps(dashboard_with_filters), - warehouse_id=get_best_warehouse(), -) -print(result["url"]) -``` - -## Troubleshooting - -### Widget shows "no selected fields to visualize" - -**This is a field name mismatch error.** The `name` in `query.fields` must exactly match the `fieldName` in `encodings`. - -**Fix:** Ensure names match exactly: -```json -// WRONG - names don't match -"fields": [{"name": "spend", "expression": "SUM(`spend`)"}] -"encodings": {"value": {"fieldName": "sum(spend)", ...}} // ERROR! - -// CORRECT - names match -"fields": [{"name": "sum(spend)", "expression": "SUM(`spend`)"}] -"encodings": {"value": {"fieldName": "sum(spend)", ...}} // OK! -``` - -### Widget shows "Invalid widget definition" - -**Check version numbers:** -- Counters: `version: 2` -- Tables: `version: 2` -- Filters: `version: 2` -- Bar/Line/Pie charts: `version: 3` - -**Text widget errors:** -- Text widgets must NOT have a `spec` block -- Use `multilineTextboxSpec` directly on the widget object -- Do NOT use `widgetType: "text"` - this is invalid - -**Table widget errors:** -- Use `version: 2` (NOT 1 or 3) -- Column objects only need `fieldName` and `displayName` -- Do NOT add `type`, `numberFormat`, or other column properties - -**Counter widget errors:** -- Use `version: 2` (NOT 3) -- Ensure dataset returns exactly 1 row - -### Dashboard shows empty widgets -- Run the dataset SQL query directly to check data exists -- Verify column aliases match widget field expressions -- Check `disaggregated` flag (should be `true` for pre-aggregated data) - -### Layout has gaps -- Ensure each row sums to width=6 -- Check that y positions don't skip values - -### Filter shows "Invalid widget definition" -- Check `widgetType` is one of: `filter-multi-select`, `filter-single-select`, `filter-date-range-picker` -- **DO NOT** use `widgetType: "filter"` - this is invalid -- Verify `spec.version` is `2` -- Ensure `queryName` in encodings matches the query `name` -- Confirm `disaggregated: false` in filter queries -- Ensure `frame` with `showTitle: true` is included - -### Filter not affecting expected pages -- **Global filters** (on `PAGE_TYPE_GLOBAL_FILTERS` page) affect all datasets containing the filter field -- **Page-level filters** (on `PAGE_TYPE_CANVAS` page) only affect widgets on that same page -- A filter only works on datasets that include the filter dimension column - -### Filter shows "UNRESOLVED_COLUMN" error for `associative_filter_predicate_group` -- **DO NOT** use `COUNT_IF(\`associative_filter_predicate_group\`)` in filter queries -- This internal expression causes SQL errors when the dashboard executes queries -- Use a simple field expression instead: `{"name": "field", "expression": "\`field\`"}` - -### Text widget shows title and description on same line -- Multiple items in the `lines` array are **concatenated**, not displayed on separate lines -- Use **separate text widgets** for title and subtitle at different y positions -- Example: title at y=0 with height=1, subtitle at y=1 with height=1 diff --git a/.claude/skills/bdd-features/SKILL.md b/.claude/skills/bdd-features/SKILL.md new file mode 100644 index 00000000..8d0bb123 --- /dev/null +++ b/.claude/skills/bdd-features/SKILL.md @@ -0,0 +1,105 @@ +--- +name: bdd-features +description: "This skill should be used when the user asks to \"write Gherkin\", \"create feature files\", \"generate BDD scenarios\", \"write acceptance tests in Gherkin\", \"create Behave features\", \"write Given When Then tests\", \"BDD test cases for my pipeline\", \"Gherkin for Unity Catalog\", or wants to translate requirements into Gherkin feature files for Databricks." +user-invocable: true +--- + +# BDD Features — Gherkin Feature File Generation + +Generate well-structured Gherkin `.feature` files for Databricks workloads. Translate requirements, user stories, or existing code into behavior specifications using Given/When/Then syntax. + +## When to use + +- Translating requirements or user stories into Gherkin acceptance criteria +- Creating feature files for Databricks pipelines, catalog permissions, jobs, or Apps +- Writing regression tests in Gherkin for existing functionality +- Generating Scenario Outlines for data-driven testing + +## Process + +### 1. Identify the test subject + +Determine what to test. Read the relevant code or ask the user: + +- A Lakeflow SDP pipeline definition → pipeline behavior tests +- Unity Catalog grants/policies → permission verification tests +- A FastAPI Databricks App → API endpoint tests +- A notebook or job → execution and output validation tests +- SQL transformations → data quality and correctness tests + +### 2. Write the feature file + +Place feature files in the appropriate subdirectory under `features/`: + +``` +features/ +├── catalog/permissions.feature +├── pipelines/events_pipeline.feature +├── apps/api_endpoints.feature +├── jobs/etl_notebook.feature +└── sql/data_quality.feature +``` + +**Structure every feature file with:** + +1. **Tags** — `@domain`, `@smoke`/`@regression`/`@integration`, optional `@slow` or `@wip` +2. **Feature header** — name + As a / I want / So that narrative +3. **Background** — shared Given steps (workspace connection, test schema) +4. **Scenarios** — one behavior per scenario, descriptive names + +Refer to `references/gherkin-patterns.md` for Databricks-specific Gherkin patterns covering: +- Pipeline lifecycle (full refresh, incremental, failure handling) +- Unity Catalog grants, column masks, row filters +- App endpoint testing with SSO headers +- Job/notebook execution and output validation +- SQL data quality assertions +- Scenario Outlines for parameterized testing + +### 3. Gherkin writing principles + +**Declarative, not imperative.** Describe *what* the system should do, not *how* to click buttons: + +```gherkin +# Good — declarative +When I grant SELECT on "catalog.schema.table" to group "readers" +Then the group "readers" should have SELECT permission + +# Bad — imperative +When I open the Catalog Explorer +And I click on the table "catalog.schema.table" +And I click "Permissions" +And I click "Grant" +And I select "SELECT" +And I type "readers" in the group field +And I click "Save" +``` + +**One behavior per scenario.** If a scenario tests two independent things, split it. + +**Use Backgrounds for shared setup.** Avoid repeating connection/schema steps across scenarios. + +**Scenario Outlines for data variations.** When the same behavior is tested with different inputs, use Examples tables instead of duplicating scenarios. + +**Tag strategically:** +- `@smoke` — fast, critical-path tests (< 30 seconds each) +- `@regression` — thorough coverage (minutes) +- `@integration` — needs live workspace (skip in unit test CI) +- `@slow` — pipeline tests, job executions (> 2 minutes) + +**CRITICAL — Curly braces break step matching.** Behave uses the `parse` library for step matching. `{anything}` in feature file text is interpreted as a capture group, not a literal. Never use `{test_schema}.table_name` in feature files — it will fail to match step definitions. Instead, use short table names (`"customers"`) and resolve the schema in step code. + +**Trailing colons matter.** When a step has an attached data table or docstring, the `:` at the end of the Gherkin line IS part of the step text. The step pattern must include it: `@given('a table "{name}" with data:')` — not `with data` (no colon). + +### 4. Validate step coverage + +After writing features, check that step definitions exist for all steps: + +```bash +uv run behave --dry-run +``` + +Any undefined steps will be reported with suggested snippets. Hand those to the `bdd-steps` skill for implementation. + +## Additional resources + +- **`references/gherkin-patterns.md`** — Complete Databricks Gherkin pattern library with examples for every domain diff --git a/.claude/skills/bdd-features/references/gherkin-patterns.md b/.claude/skills/bdd-features/references/gherkin-patterns.md new file mode 100644 index 00000000..19132522 --- /dev/null +++ b/.claude/skills/bdd-features/references/gherkin-patterns.md @@ -0,0 +1,446 @@ +# Gherkin Patterns for Databricks + +Reusable Gherkin patterns for common Databricks testing scenarios. Copy and adapt these to feature files. + +> **WARNING: Curly braces in step text break Behave's `parse` matcher.** +> +> Behave uses Python's `parse` library for step matching. Any `{...}` in step text +> is interpreted as a capture group. Writing `{test_schema}.customers` in a step line +> will **silently fail to match** your step definition. +> +> **The correct pattern:** +> - Step text uses **short table names in quotes**: `"customers"`, `"orders"` +> - SQL inside **docstrings** (triple-quoted blocks) can safely use `{schema}` because +> docstrings are accessed via `context.text`, not step matching +> - Step definitions prepend `context.test_schema + "."` internally to build the FQN +> +> ```python +> # WRONG - step text with curly braces +> @given('a table "{test_schema}.customers" exists') # BROKEN - parse eats {test_schema} +> +> # RIGHT - short name in step text, FQN built in the step body +> @given('a managed table "{table_name}" exists') +> def step_impl(context, table_name): +> fqn = f"{context.test_schema}.{table_name}" +> # ... use fqn +> ``` +> +> **Docstring SQL pattern** (safe because `context.text` is just a string): +> ```python +> @when('I execute SQL:') +> def step_impl(context): +> sql = context.text.replace("{schema}", context.test_schema) +> # ... execute sql +> ``` + +## Common Background + +Most Databricks feature files share this Background: + +```gherkin +Background: + Given a Databricks workspace connection is established + And a test schema is provisioned +``` + +--- + +## Unity Catalog + +### Table permissions + +```gherkin +@catalog @permissions +Feature: Unity Catalog table permissions + As a data engineer + I want to verify table-level permissions + So that sensitive data is properly protected + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + Scenario: Grant SELECT to a group + Given a managed table "customers" exists + When I execute SQL: + """sql + GRANT SELECT ON TABLE {schema}.customers TO `data_readers` + """ + And I execute SQL: + """sql + SHOW GRANTS ON TABLE {schema}.customers + """ + Then the result should contain a row where "ActionType" is "SELECT" and "Principal" is "data_readers" + + Scenario Outline: Verify multiple privilege types + Given a managed table "sales" exists + When I execute SQL: + """sql + GRANT ON TABLE {schema}.sales TO `` + """ + And I execute SQL: + """sql + SHOW GRANTS ON TABLE {schema}.sales + """ + Then the result should contain a row where "ActionType" is "" and "Principal" is "" + + Examples: + | privilege | group | + | SELECT | data_readers | + | MODIFY | data_writers | +``` + +### Column masks + +```gherkin +@catalog @security +Feature: Column-level security + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + Scenario: Mask PII columns for analysts + Given a managed table "customers" with columns: + | column_name | data_type | contains_pii | + | id | BIGINT | false | + | name | STRING | true | + | email | STRING | true | + | region | STRING | false | + And a column mask function "mask_pii" is applied to "name" and "email" on "customers" + When I query "customers" as group "analysts" + Then columns "name" and "email" should return masked values + But columns "id" and "region" should return actual values +``` + +### Row filters + +```gherkin +@catalog @security +Feature: Row-level security + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + Scenario: Row filter restricts by region + Given a managed table "regional_sales" with data: + | region | revenue | quarter | + | APAC | 50000 | Q1 | + | EMEA | 75000 | Q1 | + | AMER | 100000 | Q1 | + And a row filter on "regional_sales" restricts "apac_analysts" to region "APAC" + When I query "regional_sales" as group "apac_analysts" + Then I should only see rows where "region" is "APAC" + And the result should have 1 row +``` + +--- + +## Lakeflow Spark Declarative Pipelines + +### Pipeline lifecycle + +```gherkin +@pipeline @lakeflow +Feature: Events pipeline processing + As a data engineer + I want to verify the events pipeline processes data correctly + So that downstream consumers get accurate aggregations + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + @integration @slow + Scenario: Full refresh produces expected tables + Given a pipeline "events_pipeline" exists targeting the test schema + When I trigger a full refresh of the pipeline + Then the pipeline update should succeed within 600 seconds + And the streaming table "bronze_events" should exist + And the materialized view "silver_events_agg" should exist + And the table "silver_events_agg" should have more than 0 rows + + @integration + Scenario: Incremental refresh picks up new data + Given the pipeline "events_pipeline" has completed a full refresh + When I insert test records into the source + And I trigger an incremental refresh of the pipeline + Then the pipeline update should succeed within 300 seconds + And the new records should appear in "bronze_events" + + Scenario: Pipeline handles empty source gracefully + Given a pipeline "events_pipeline" exists targeting the test schema + And the source table is empty + When I trigger a full refresh of the pipeline + Then the pipeline update should succeed within 300 seconds + And the streaming table "bronze_events" should have 0 rows +``` + +### Pipeline failure handling + +```gherkin + Scenario: Pipeline surfaces schema mismatch errors + Given a pipeline "events_pipeline" exists targeting the test schema + And the source table has an unexpected column "extra_col" of type "BINARY" + When I trigger a full refresh of the pipeline + Then the pipeline update should fail + And the pipeline error should mention schema +``` + +--- + +## Jobs and Notebooks + +### Notebook execution + +```gherkin +@jobs @notebook +Feature: Customer ETL notebook + As a data engineer + I want to verify the ETL notebook produces correct output + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + @integration @slow + Scenario: Dedup notebook removes duplicates + Given a managed table "raw_customers" with data: + | customer_id | name | email | updated_at | + | 1 | Alice | alice@example.com | 2024-01-01T00:00:00 | + | 1 | Alice B. | alice@example.com | 2024-06-01T00:00:00 | + | 2 | Bob | bob@example.com | 2024-03-15T00:00:00 | + When I run the notebook "/Repos/team/etl/customer_dedup" with parameters: + | key | value | + | source_table | raw_customers | + | target_table | clean_customers| + Then the job should complete with status "SUCCESS" within 300 seconds + And the table "clean_customers" should have 2 rows + And the table "clean_customers" should contain a row where "customer_id" is "1" and "name" is "Alice B." + + Scenario: Notebook fails gracefully on missing source + When I run the notebook "/Repos/team/etl/customer_dedup" with parameters: + | key | value | + | source_table | nonexistent | + | target_table | output | + Then the job should complete with status "FAILED" within 120 seconds +``` + +--- + +## Databricks Apps (FastAPI) + +### API endpoint testing + +```gherkin +@app @fastapi +Feature: Databricks App API + As a user + I want the app endpoints to work correctly + + Background: + Given the app is running at the configured base URL + And the test user is "testuser@databricks.com" + + @smoke + Scenario: Health check + When I GET "/health" + Then the response status should be 200 + And the response JSON should contain "status" with value "healthy" + + Scenario: Authenticated user can list resources + When I GET "/api/dashboards" with auth headers + Then the response status should be 200 + And the response should be a JSON list + + Scenario: Unauthenticated request is rejected + When I GET "/api/dashboards" without auth headers + Then the response status should be 401 + + Scenario: POST creates a resource + When I POST "/api/items" with auth headers and body: + """json + {"name": "Test Item", "description": "Created by BDD test"} + """ + Then the response status should be 201 + And the response JSON should contain "name" with value "Test Item" +``` + +### App deployment testing + +```gherkin +@app @deployment @slow +Feature: App deployment lifecycle + Scenario: Deploy and verify app is running + Given a bundle project at the repository root + When I deploy using Asset Bundles with target "dev" + Then the deployment should succeed + And the app should reach "RUNNING" state within 120 seconds + And the app health endpoint should return 200 +``` + +--- + +## SQL Data Quality + +### Row counts and data validation + +```gherkin +@sql @data-quality +Feature: Data quality checks + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + @smoke + Scenario: Table is not empty + Given the table "orders" has been loaded + Then the table "orders" should have more than 0 rows + + Scenario: No duplicate primary keys + Given the table "orders" has been loaded + When I execute SQL: + """sql + SELECT order_id, COUNT(*) as cnt + FROM {schema}.orders + GROUP BY order_id + HAVING COUNT(*) > 1 + """ + Then the result should have 0 rows + + Scenario: Foreign key integrity + Given the tables "orders" and "customers" have been loaded + When I execute SQL: + """sql + SELECT o.customer_id + FROM {schema}.orders o + LEFT JOIN {schema}.customers c ON o.customer_id = c.customer_id + WHERE c.customer_id IS NULL + """ + Then the result should have 0 rows + + Scenario: No null values in required columns + When I execute SQL: + """sql + SELECT COUNT(*) as null_count + FROM {schema}.orders + WHERE order_id IS NULL OR customer_id IS NULL OR order_date IS NULL + """ + Then the first row column "null_count" should be "0" + + Scenario: Verify GRANT was applied via SQL + Given a managed table "products" exists + When I execute SQL: + """sql + GRANT SELECT ON TABLE {schema}.products TO `reporting_team` + """ + And I execute SQL: + """sql + SHOW GRANTS ON TABLE {schema}.products + """ + Then the result should contain a row where "ActionType" is "SELECT" and "Principal" is "reporting_team" +``` + +--- + +## Asset Bundles Deployment + +```gherkin +@deployment @dabs +Feature: Bundle lifecycle + @smoke + Scenario: Bundle validates successfully + When I run "databricks bundle validate" with target "dev" + Then the command should exit with code 0 + + @integration @slow + Scenario: Deploy and destroy lifecycle + When I run "databricks bundle deploy" with target "dev" + Then the command should exit with code 0 + When I run "databricks bundle destroy" with target "dev" and auto-approve + Then the command should exit with code 0 +``` + +--- + +## Scenario Outline patterns + +Use Scenario Outlines for testing multiple variations of the same behavior. + +Note: table names in the Examples table are short names (no schema prefix). The step +definition prepends `context.test_schema` to build the fully-qualified name. + +```gherkin + Scenario Outline: Verify table existence after pipeline run + Then the "" should exist + + Examples: Streaming tables + | table_type | table_name | + | streaming table | bronze_events | + | streaming table | bronze_transactions| + + Examples: Materialized views + | table_type | table_name | + | materialized view | silver_events_agg| + | materialized view | gold_summary | +``` + +--- + +## Steps with data tables and docstrings + +Steps that accept a data table or docstring **must** end with a trailing colon. The colon +is part of the step text that Behave matches against your `@given`/`@when`/`@then` decorator. + +```gherkin +# CORRECT - colon before data table +Given a managed table "customers" with data: + | id | name | region | + | 1 | Alice | APAC | + | 2 | Bob | EMEA | + +# CORRECT - colon before docstring +When I execute SQL: + """sql + SELECT * FROM {schema}.customers + """ + +# WRONG - missing colon, Behave will not match the step +Given a managed table "customers" with data + | id | name | region | +``` + +--- + +## SHOW GRANTS column names + +`SHOW GRANTS` returns PascalCase column names. Use these exact names when asserting +on grant results: + +| Column | Description | +|--------------|------------------------------------------------| +| `Principal` | The user, group, or service principal | +| `ActionType` | The privilege (SELECT, MODIFY, ALL PRIVILEGES) | +| `ObjectType` | TABLE, SCHEMA, CATALOG, etc. | +| `ObjectKey` | The fully-qualified object name | + +--- + +## Tag strategy + +| Tag | Purpose | Typical runtime | +|-----|---------|----------------| +| `@smoke` | Critical path, must always pass | < 30s per scenario | +| `@regression` | Full coverage | Minutes | +| `@integration` | Needs live workspace | Varies | +| `@slow` | Pipeline/job execution | > 2 min | +| `@wip` | Work in progress, skip by default | N/A | +| `@skip` | Explicitly disabled | N/A | +| `@catalog` | Unity Catalog tests | Varies | +| `@pipeline` | Lakeflow SDP tests | Minutes | +| `@jobs` | Job/notebook tests | Minutes | +| `@app` | Databricks Apps tests | Seconds | +| `@sql` | SQL/data quality tests | Seconds | +| `@deployment` | DABs lifecycle tests | Minutes | diff --git a/.claude/skills/bdd-run/SKILL.md b/.claude/skills/bdd-run/SKILL.md new file mode 100644 index 00000000..f8f242e1 --- /dev/null +++ b/.claude/skills/bdd-run/SKILL.md @@ -0,0 +1,145 @@ +--- +name: bdd-run +description: "This skill should be used when the user asks to \"run BDD tests\", \"execute Behave\", \"run Gherkin tests\", \"run my feature files\", \"behave test results\", \"run smoke tests\", \"BDD test report\", or needs to execute Behave test suites with specific options like tag filtering, parallel execution, or CI reporting." +user-invocable: true +--- + +# BDD Run — Execute and Report Behave Tests + +Execute Behave test suites with tag filtering, parallel execution, output formatting, and CI integration. Diagnose failures and suggest fixes. + +## When to use + +- Running the full BDD test suite or a subset by tags +- Getting JUnit/JSON reports for CI pipelines +- Re-running only failed scenarios +- Running tests in parallel for speed +- Diagnosing and triaging test failures + +## Process + +### 1. Pre-flight checks + +Before running tests, verify the environment: + +```bash +# Verify Behave is installed +uv run behave --version + +# Verify Databricks auth +uv run python -c "from databricks.sdk import WorkspaceClient; print(WorkspaceClient().current_user.me().user_name)" + +# Dry run to check step coverage +uv run behave --dry-run +``` + +If any undefined steps are found, report them and suggest using the `bdd-steps` skill. + +### 2. Execute tests + +**Run by tag (most common):** + +```bash +# Smoke tests only +uv run behave --tags="@smoke" --format=pretty + +# All except slow and WIP +uv run behave --tags="not @slow and not @wip" + +# Specific domain +uv run behave --tags="@catalog" +uv run behave --tags="@pipeline" + +# Boolean combinations +uv run behave --tags="(@catalog or @pipeline) and @smoke" +``` + +**Run specific feature file or directory:** + +```bash +uv run behave features/catalog/permissions.feature +uv run behave features/pipelines/ +``` + +**Run by scenario name:** + +```bash +uv run behave --name "Grant SELECT on a table" +``` + +**Pass runtime configuration:** + +```bash +uv run behave -D warehouse_id=abc123 -D catalog=my_catalog -D environment=dev +``` + +### 3. Output and reporting + +**For local development:** + +```bash +uv run behave --format=pretty --show-timings +``` + +**For CI pipelines (JUnit XML):** + +```bash +uv run behave --junit --junit-directory=reports/behave/ --format=progress +``` + +**JSON output for programmatic analysis:** + +```bash +uv run behave --format=json --outfile=reports/results.json --format=progress +``` + +**Multiple formatters simultaneously:** + +```bash +uv run behave --format=pretty --format=json --outfile=reports/results.json +``` + +### 4. Re-run failed tests + +Configure rerun file output, then re-run only failures: + +```bash +# First run captures failures +uv run behave --format=rerun --outfile=reports/rerun.txt --format=pretty + +# Re-run only failed scenarios +uv run behave @reports/rerun.txt +``` + +### 5. Parallel execution + +Behave has no built-in parallelism. Use `behavex` for parallel feature execution: + +```bash +uv run behavex --parallel-processes 4 --parallel-scheme feature +``` + +Each parallel worker needs its own test schema to avoid cross-contamination. The `environment.py` template from `bdd-scaffold` handles this by using timestamped schema names with worker ID suffixes. + +### 6. Failure diagnosis + +When tests fail, read the output and categorize: + +| Failure type | Symptom | Action | +|-------------|---------|--------| +| Undefined step | `NotImplementedError` or "undefined" in output | Generate step with `bdd-steps` | +| Auth failure | `PermissionDenied`, 401/403 | Check `databricks auth profiles` | +| Timeout | `TimeoutError` in polling steps | Increase timeout parameter or check resource state | +| Data mismatch | Assertion error with expected vs. actual | Check test data setup or query logic | +| Schema not found | `SCHEMA_NOT_FOUND` | Verify `before_all` created the ephemeral schema | +| Warehouse stopped | `WAREHOUSE_NOT_RUNNING` | Start warehouse or use `@fixture.sql_warehouse` tag hook | + +### 7. Makefile integration + +If a Makefile exists, prefer `make` targets: + +```bash +make bdd # Full suite +make bdd-smoke # Smoke tests +make bdd-report # JUnit for CI +``` diff --git a/.claude/skills/bdd-scaffold/SKILL.md b/.claude/skills/bdd-scaffold/SKILL.md new file mode 100644 index 00000000..d24c5ca4 --- /dev/null +++ b/.claude/skills/bdd-scaffold/SKILL.md @@ -0,0 +1,114 @@ +--- +name: bdd-scaffold +description: "This skill should be used when the user asks to \"set up BDD\", \"create a Behave project\", \"scaffold BDD tests\", \"initialize Behave\", \"add BDD to my project\", \"set up Gherkin testing\", \"create test structure for Behave\", or mentions setting up behavior-driven development testing. Generates a complete Behave project structure wired to Databricks SDK." +user-invocable: true +--- + +# BDD Scaffold — Behave + Databricks Project Setup + +Generate a complete Python Behave project structure pre-wired with Databricks SDK integration, including `environment.py` hooks, test isolation via ephemeral schemas, and `behave.ini` configuration. + +## When to use + +- Starting a new BDD test suite for a Databricks project +- Adding Behave-based acceptance tests to an existing repo +- Setting up integration testing against Unity Catalog, pipelines, jobs, or Apps + +## Process + +### 1. Detect project context + +Identify the project root and existing tooling: + +```bash +git rev-parse --show-toplevel +``` + +Check for existing test infrastructure: `pyproject.toml`, `Makefile`, `behave.ini`, `features/` directory. If a `features/` directory already exists, confirm before overwriting. + +### 2. Determine test domains + +Ask (or infer from the codebase) which Databricks domains to scaffold step files for: + +| Domain | Step file | When | +|--------|-----------|------| +| Unity Catalog | `catalog_steps.py` | Tables, schemas, grants, row filters, column masks | +| Pipelines | `pipeline_steps.py` | Lakeflow SDP, streaming tables, materialized views | +| Jobs | `job_steps.py` | Notebook runs, workflow tasks, job clusters | +| Apps | `app_steps.py` | FastAPI endpoints, SSO headers, deployment | +| SQL | `sql_steps.py` | Statement execution, warehouse queries, data validation | + +Always generate `common_steps.py` (shared workspace connection, row counting, table existence checks). + +### 3. Generate the directory structure + +``` +features/ +├── environment.py # Databricks SDK setup, ephemeral schema lifecycle +├── steps/ +│ ├── common_steps.py # Shared steps (always generated) +│ └── _steps.py # Per-domain (based on step 2) +├── catalog/ # Feature file directories (one per domain) +├── pipelines/ +├── jobs/ +├── apps/ +└── sql/ +behave.ini +Makefile # (append BDD targets if Makefile exists) +``` + +Refer to `references/environment-template.md` for the full `environment.py` template with: +- `before_all`: WorkspaceClient init, warehouse auto-discovery, ephemeral schema creation +- `after_all`: Schema cascade drop +- `before_scenario` / `after_scenario`: Per-scenario resource tracking and cleanup +- Tag-based hooks for `@wip`, `@skip`, `@slow` + +Refer to `references/behave-config.md` for `behave.ini` and `pyproject.toml` configuration. + +### 4. Add dependencies + +If `pyproject.toml` exists and uses `uv`: + +```bash +uv add --group test behave databricks-sdk httpx +``` + +If no `pyproject.toml`, create a minimal one with test dependencies. + +### 5. Add Makefile targets + +Append these targets (or create a Makefile if none exists): + +```makefile +.PHONY: bdd bdd-smoke bdd-report + +bdd: + uv run behave --format=pretty + +bdd-smoke: + uv run behave --tags="@smoke" --format=pretty + +bdd-report: + uv run behave --junit --junit-directory=reports/ --format=progress +``` + +### 6. Verify scaffold + +Run `behave --dry-run` to confirm step discovery works and there are no import errors: + +```bash +uv run behave --dry-run +``` + +Report the generated structure and next steps to the user. + +## Key design decisions + +- **Ephemeral schemas** — each test run creates a timestamped schema (`behave_test_YYYYMMDD_HHMMSS`) and drops it in `after_all`. Prevents cross-run contamination. +- **`-D` userdata** for parameterization — warehouse IDs, catalog names, and targets are passed via CLI args, never hardcoded. +- **Step files are globally scoped** in Behave — all files in `steps/` are imported regardless of which feature runs. Name step patterns carefully to avoid collisions. + +## Additional resources + +- **`references/environment-template.md`** — Full annotated environment.py template +- **`references/behave-config.md`** — behave.ini and pyproject.toml configuration reference diff --git a/.claude/skills/bdd-scaffold/references/behave-config.md b/.claude/skills/bdd-scaffold/references/behave-config.md new file mode 100644 index 00000000..d994f518 --- /dev/null +++ b/.claude/skills/bdd-scaffold/references/behave-config.md @@ -0,0 +1,134 @@ +# Behave Configuration Reference + +## behave.ini + +Standard Behave configuration file. Place at project root. + +```ini +[behave] +# Output +default_format = pretty +show_timings = true +color = true + +# Default tag filter — skip WIP and explicitly skipped tests +default_tags = not @wip and not @skip + +# Logging +logging_level = INFO +logging_format = %(asctime)s %(levelname)-8s %(name)s: %(message)s + +# Capture control +stdout_capture = true +log_capture = true + +# JUnit output (enable in CI) +junit = false +junit_directory = reports/ + +# Feature paths +paths = features/ + +[behave.userdata] +# Override with -D key=value on CLI +warehouse_id = auto +catalog = main +environment = dev +``` + +## pyproject.toml + +Alternative configuration via pyproject.toml (Behave reads `[tool.behave]`): + +**IMPORTANT:** In `pyproject.toml`, `default_tags` must be a **list**, not a string. The `behave.ini` parser accepts a plain string, but the TOML parser is stricter: + +```toml +[tool.behave] +default_format = "pretty" +show_timings = true +default_tags = ["not @wip and not @skip"] # MUST be a list in pyproject.toml +junit = false +junit_directory = "reports/" +logging_level = "INFO" + +[tool.behave.userdata] +warehouse_id = "auto" +catalog = "main" +environment = "dev" +``` + +## Dependencies + +Add to `pyproject.toml`: + +```toml +[project.optional-dependencies] +test = [ + "behave>=1.2.6", + "databricks-sdk>=0.40.0", + "httpx>=0.27.0", +] + +# Or for parallel execution +test-parallel = [ + "behave>=1.2.6", + "behavex>=3.0", + "databricks-sdk>=0.40.0", + "httpx>=0.27.0", +] +``` + +With `uv`: + +```bash +uv add --group test behave databricks-sdk httpx +``` + +## Makefile targets + +```makefile +.PHONY: bdd bdd-smoke bdd-report bdd-rerun bdd-parallel bdd-dry-run + +bdd: + uv run behave --format=pretty --show-timings + +bdd-smoke: + uv run behave --tags="@smoke" --format=pretty + +bdd-report: + uv run behave --junit --junit-directory=reports/behave/ --format=progress + +bdd-rerun: + uv run behave @reports/rerun.txt + +bdd-parallel: + uv run behavex --parallel-processes 4 --parallel-scheme feature + +bdd-dry-run: + uv run behave --dry-run +``` + +## CI integration (GitHub Actions example) + +```yaml +- name: Run BDD tests + env: + DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }} + DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }} + DATABRICKS_WAREHOUSE_ID: ${{ secrets.WAREHOUSE_ID }} + TEST_CATALOG: ci_test + run: | + uv run behave \ + --tags="not @slow" \ + --junit --junit-directory=reports/behave/ \ + --format=progress \ + -D catalog=$TEST_CATALOG \ + -D warehouse_id=$DATABRICKS_WAREHOUSE_ID + +- name: Upload test results + if: always() + uses: actions/upload-artifact@v4 + with: + name: behave-results + path: reports/behave/ +``` diff --git a/.claude/skills/bdd-scaffold/references/environment-template.md b/.claude/skills/bdd-scaffold/references/environment-template.md new file mode 100644 index 00000000..2a7dc1b7 --- /dev/null +++ b/.claude/skills/bdd-scaffold/references/environment-template.md @@ -0,0 +1,195 @@ +# environment.py Template — Databricks + Behave + +Complete annotated template for `features/environment.py`. Copy and adapt to the target project. + +## Full template + +```python +"""Behave environment hooks — Databricks SDK integration. + +Sets up workspace connection, ephemeral test schema, and per-scenario cleanup. +""" +from __future__ import annotations + +import logging +import os +from datetime import datetime + +from behave.model import Feature, Scenario, Step +from behave.runner import Context + +logger = logging.getLogger("behave.databricks") + + +# ─── Session-level hooks ──────────────────────────────────────── + +def before_all(context: Context) -> None: + """Initialize Databricks clients and create ephemeral test schema.""" + from databricks.sdk import WorkspaceClient + + context.workspace = WorkspaceClient() + + # Fix host URL — some profiles include ?o= which breaks SDK API paths. + # The CLI handles this transparently but the SDK does not. + if context.workspace.config.host and "?" in context.workspace.config.host: + clean_host = context.workspace.config.host.split("?")[0].rstrip("/") + profile = os.environ.get("DATABRICKS_CONFIG_PROFILE") + context.workspace = WorkspaceClient(profile=profile, host=clean_host) + + # Verify auth + me = context.workspace.current_user.me() + context.current_user = me.user_name + logger.info("Authenticated as: %s", context.current_user) + + # Warehouse — from -D userdata, env var, or auto-discover + userdata = context.config.userdata + context.warehouse_id = ( + userdata.get("warehouse_id") + or os.environ.get("DATABRICKS_WAREHOUSE_ID") + or _discover_warehouse(context.workspace) + ) + logger.info("Using warehouse: %s", context.warehouse_id) + + # Catalog — from -D userdata or env var + context.test_catalog = userdata.get("catalog", os.environ.get("TEST_CATALOG", "main")) + + # Create ephemeral schema (timestamped for isolation) + ts = datetime.now().strftime("%Y%m%d_%H%M%S") + worker = os.environ.get("BEHAVE_WORKER_ID", "0") + context.test_schema = f"{context.test_catalog}.behave_test_{ts}_w{worker}" + + _execute_sql(context, f"CREATE SCHEMA IF NOT EXISTS {context.test_schema}") + logger.info("Created test schema: %s", context.test_schema) + + +def after_all(context: Context) -> None: + """Drop ephemeral test schema.""" + if hasattr(context, "test_schema"): + try: + _execute_sql(context, f"DROP SCHEMA IF EXISTS {context.test_schema} CASCADE") + logger.info("Dropped test schema: %s", context.test_schema) + except Exception as e: + logger.warning("Failed to drop test schema %s: %s", context.test_schema, e) + + +# ─── Feature-level hooks ──────────────────────────────────────── + +def before_feature(context: Context, feature: Feature) -> None: + """Log feature start. Skip if tagged @skip.""" + logger.info("▶ Feature: %s", feature.name) + if "skip" in feature.tags: + feature.skip("Marked with @skip") + + +def after_feature(context: Context, feature: Feature) -> None: + logger.info("◀ Feature: %s [%s]", feature.name, feature.status) + + +# ─── Scenario-level hooks ─────────────────────────────────────── + +def before_scenario(context: Context, scenario: Scenario) -> None: + """Initialize per-scenario state. Skip @wip scenarios.""" + logger.info(" ▶ Scenario: %s", scenario.name) + if "wip" in scenario.tags: + scenario.skip("Work in progress") + return + # Track resources created during this scenario for cleanup + context.scenario_cleanup_sql = [] + + +def after_scenario(context: Context, scenario: Scenario) -> None: + """Clean up scenario-specific resources.""" + for sql in getattr(context, "scenario_cleanup_sql", []): + try: + _execute_sql(context, sql) + except Exception as e: + logger.warning("Cleanup SQL failed: %s — %s", sql, e) + if scenario.status == "failed": + logger.error(" ✗ FAILED: %s", scenario.name) + else: + logger.info(" ◀ Scenario: %s [%s]", scenario.name, scenario.status) + + +# ─── Step-level hooks ─────────────────────────────────────────── + +def before_step(context: Context, step: Step) -> None: + context._step_start = datetime.now() + + +def after_step(context: Context, step: Step) -> None: + elapsed = (datetime.now() - context._step_start).total_seconds() + if elapsed > 10: + logger.warning(" Slow step (%.1fs): %s %s", elapsed, step.keyword, step.name) + if step.status == "failed": + logger.error(" ✗ %s %s\n %s", step.keyword, step.name, step.error_message) + + +# ─── Tag-based hooks ──────────────────────────────────────────── + +def before_tag(context, tag: str) -> None: + """Ensure resources for tagged scenarios.""" + if tag == "fixture.sql_warehouse": + _ensure_warehouse_running(context) + + +# ─── Helpers ──────────────────────────────────────────────────── + +def _execute_sql(context: Context, sql: str) -> object: + """Execute a SQL statement via the Statement Execution API.""" + return context.workspace.statement_execution.execute_statement( + warehouse_id=context.warehouse_id, + statement=sql, + wait_timeout="30s", + ) + + +def _discover_warehouse(workspace) -> str: + """Find the first available SQL warehouse.""" + from databricks.sdk.service.sql import State + + warehouses = list(workspace.warehouses.list()) + # Prefer running warehouses + for wh in warehouses: + if wh.state == State.RUNNING: + return wh.id + if warehouses: + return warehouses[0].id + raise RuntimeError( + "No SQL warehouses found. Pass warehouse_id via -D warehouse_id= " + "or set DATABRICKS_WAREHOUSE_ID." + ) + + +def _ensure_warehouse_running(context: Context) -> None: + """Start warehouse if stopped. Used by @fixture.sql_warehouse tag.""" + from databricks.sdk.service.sql import State + + wh = context.workspace.warehouses.get(context.warehouse_id) + if wh.state != State.RUNNING: + logger.info("Starting warehouse %s...", context.warehouse_id) + context.workspace.warehouses.start(context.warehouse_id) + context.workspace.warehouses.wait_get_warehouse_running(context.warehouse_id) + logger.info("Warehouse %s is running.", context.warehouse_id) +``` + +## Context object layering + +Behave's `context` has scoped layers. Data set at different levels has different lifetimes: + +| Set in | Lifetime | Example | +|--------|----------|---------| +| `before_all` | Entire run | `context.workspace`, `context.test_schema` | +| `before_feature` | Current feature | `context.feature_data` | +| `before_scenario` / steps | Current scenario | `context.query_result`, `context.scenario_cleanup_sql` | + +At the end of each scenario, the scenario layer is popped — anything set during steps is gone. Root-level data persists across everything. + +## Parallel execution isolation + +When using `behavex` for parallel execution, each worker needs its own schema. The template uses `BEHAVE_WORKER_ID` from the environment. Set it in the parallel runner config or wrapper script: + +```bash +# Example wrapper for behavex +export BEHAVE_WORKER_ID=$WORKER_INDEX +behave "$@" +``` diff --git a/.claude/skills/bdd-scaffold/test-suite/.gitignore b/.claude/skills/bdd-scaffold/test-suite/.gitignore new file mode 100644 index 00000000..744fea57 --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/.gitignore @@ -0,0 +1,4 @@ +.venv/ +__pycache__/ +reports/ +*.pyc diff --git a/.claude/skills/bdd-scaffold/test-suite/behave.ini b/.claude/skills/bdd-scaffold/test-suite/behave.ini new file mode 100644 index 00000000..a3c4cb07 --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/behave.ini @@ -0,0 +1,13 @@ +[behave] +default_format = pretty +show_timings = true +color = true +default_tags = not @wip and not @skip +logging_level = INFO +stdout_capture = false +log_capture = false +paths = features/ + +[behave.userdata] +warehouse_id = auto +catalog = main diff --git a/.claude/skills/bdd-scaffold/test-suite/features/catalog/schema_operations.feature b/.claude/skills/bdd-scaffold/test-suite/features/catalog/schema_operations.feature new file mode 100644 index 00000000..45bacf92 --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/features/catalog/schema_operations.feature @@ -0,0 +1,38 @@ +@catalog @smoke +Feature: Unity Catalog schema and table operations + As a data engineer + I want to verify Unity Catalog operations work correctly + So that I can manage my data assets with confidence + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + Scenario: Ephemeral test schema was created + Then the test schema should exist in Unity Catalog + + Scenario: Create tables and list them + Given a managed table "table_alpha" exists + And a managed table "table_beta" exists + When I list tables in the test schema + Then the table list should include "table_alpha" + And the table list should include "table_beta" + + Scenario: Table with data is queryable via SQL + Given a managed table "products" with data: + | product_id | name | price | + | 1 | Widget | 9.99 | + | 2 | Gadget | 19.99 | + | 3 | Doohickey | 4.99 | + Then the managed table "products" should have 3 rows + When I execute a query on the test schema: + """ + SELECT name, price FROM {schema}.products WHERE CAST(price AS DOUBLE) > 10.0 + """ + Then the query result should have 1 rows + And the first result column "name" should be "Gadget" + + Scenario: Grant SELECT permission on a table + Given a managed table "grant_test" exists + When I grant SELECT on managed table "grant_test" to group "users" + Then the group "users" should have SELECT on managed table "grant_test" diff --git a/.claude/skills/bdd-scaffold/test-suite/features/environment.py b/.claude/skills/bdd-scaffold/test-suite/features/environment.py new file mode 100644 index 00000000..bd09e070 --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/features/environment.py @@ -0,0 +1,131 @@ +"""Behave environment hooks — Databricks SDK integration. + +Tested against azure-east workspace. +""" +from __future__ import annotations + +import logging +import os +from datetime import datetime + +from behave.model import Feature, Scenario, Step +from behave.runner import Context + +logger = logging.getLogger("behave.databricks") + + +def before_all(context: Context) -> None: + """Initialize Databricks clients and create ephemeral test schema.""" + from databricks.sdk import WorkspaceClient + + # Use profile from env or default + profile = os.environ.get("DATABRICKS_CONFIG_PROFILE", "azure-east") + context.workspace = WorkspaceClient(profile=profile) + + # Fix host URL — some profiles include ?o= which breaks SDK API paths + if context.workspace.config.host and "?" in context.workspace.config.host: + clean_host = context.workspace.config.host.split("?")[0].rstrip("/") + context.workspace = WorkspaceClient(profile=profile, host=clean_host) + + me = context.workspace.current_user.me() + context.current_user = me.user_name + logger.info("Authenticated as: %s", context.current_user) + + # Warehouse — from -D userdata, env var, or auto-discover + userdata = context.config.userdata + wh_id = userdata.get("warehouse_id", "auto") + if wh_id == "auto": + wh_id = os.environ.get("DATABRICKS_WAREHOUSE_ID") or _discover_warehouse( + context.workspace + ) + context.warehouse_id = wh_id + logger.info("Using warehouse: %s", context.warehouse_id) + + # Catalog + context.test_catalog = userdata.get("catalog", "main") + + # Create ephemeral schema + ts = datetime.now().strftime("%Y%m%d_%H%M%S") + context.test_schema = f"{context.test_catalog}.behave_test_{ts}" + + _execute_sql(context, f"CREATE SCHEMA IF NOT EXISTS {context.test_schema}") + logger.info("Created test schema: %s", context.test_schema) + + +def after_all(context: Context) -> None: + """Drop ephemeral test schema.""" + if hasattr(context, "test_schema"): + try: + _execute_sql( + context, f"DROP SCHEMA IF EXISTS {context.test_schema} CASCADE" + ) + logger.info("Dropped test schema: %s", context.test_schema) + except Exception as e: + logger.warning("Failed to drop test schema %s: %s", context.test_schema, e) + + +def before_feature(context: Context, feature: Feature) -> None: + logger.info("▶ Feature: %s", feature.name) + if "skip" in feature.tags: + feature.skip("Marked with @skip") + + +def after_feature(context: Context, feature: Feature) -> None: + logger.info("◀ Feature: %s [%s]", feature.name, feature.status) + + +def before_scenario(context: Context, scenario: Scenario) -> None: + logger.info(" ▶ Scenario: %s", scenario.name) + if "wip" in scenario.tags: + scenario.skip("Work in progress") + return + context.scenario_cleanup_sql = [] + + +def after_scenario(context: Context, scenario: Scenario) -> None: + for sql in getattr(context, "scenario_cleanup_sql", []): + try: + _execute_sql(context, sql) + except Exception as e: + logger.warning("Cleanup SQL failed: %s — %s", sql, e) + if scenario.status == "failed": + logger.error(" ✗ FAILED: %s", scenario.name) + + +def before_step(context: Context, step: Step) -> None: + context._step_start = datetime.now() + + +def after_step(context: Context, step: Step) -> None: + elapsed = (datetime.now() - context._step_start).total_seconds() + if elapsed > 10: + logger.warning(" Slow step (%.1fs): %s %s", elapsed, step.keyword, step.name) + if step.status == "failed": + logger.error( + " ✗ %s %s\n %s", step.keyword, step.name, step.error_message + ) + + +# ─── Helpers ──────────────────────────────────────────────────── + + +def _execute_sql(context: Context, sql: str) -> object: + """Execute a SQL statement via the Statement Execution API.""" + return context.workspace.statement_execution.execute_statement( + warehouse_id=context.warehouse_id, + statement=sql, + wait_timeout="30s", + ) + + +def _discover_warehouse(workspace) -> str: + """Find the first available SQL warehouse.""" + from databricks.sdk.service.sql import State + + warehouses = list(workspace.warehouses.list()) + for wh in warehouses: + if wh.state == State.RUNNING: + return wh.id + if warehouses: + return warehouses[0].id + raise RuntimeError("No SQL warehouses found") diff --git a/.claude/skills/bdd-scaffold/test-suite/features/sql/data_operations.feature b/.claude/skills/bdd-scaffold/test-suite/features/sql/data_operations.feature new file mode 100644 index 00000000..8e02c689 --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/features/sql/data_operations.feature @@ -0,0 +1,48 @@ +@sql @smoke +Feature: SQL data operations via Databricks + As a data engineer + I want to verify SQL operations work correctly against the warehouse + So that I can trust my data transformations + + Background: + Given a Databricks workspace connection is established + And a test schema is provisioned + + Scenario: Create a table and verify it exists + Given a managed table "smoke_test" exists + Then the managed table "smoke_test" should exist + + Scenario: Insert and count rows + Given a managed table "customers" with data: + | customer_id | name | email | + | 1 | Alice | alice@example.com | + | 2 | Bob | bob@example.com | + | 3 | Charlie | charlie@example.com | + Then the managed table "customers" should have 3 rows + + Scenario: Aggregate query returns correct results + Given a managed table "orders" with data: + | order_id | customer_id | amount | + | 101 | 1 | 50 | + | 102 | 1 | 75 | + | 103 | 2 | 100 | + When I execute a query on the test schema: + """ + SELECT customer_id, COUNT(*) as order_count + FROM {schema}.orders + GROUP BY customer_id + HAVING COUNT(*) > 1 + """ + Then the query result should have 1 rows + And the first result column "customer_id" should be "1" + + Scenario: Query with no matching rows returns zero + Given a managed table "statuses" with data: + | id | status | + | 1 | active | + | 2 | active | + When I execute a query on the test schema: + """ + SELECT * FROM {schema}.statuses WHERE status = 'inactive' + """ + Then the query result should have 0 rows diff --git a/.claude/skills/bdd-scaffold/test-suite/features/steps/catalog_steps.py b/.claude/skills/bdd-scaffold/test-suite/features/steps/catalog_steps.py new file mode 100644 index 00000000..da966aec --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/features/steps/catalog_steps.py @@ -0,0 +1,84 @@ +"""Step definitions for Unity Catalog operations. + +Table names in Gherkin are short ("customers"). +Schema resolution happens in _fqn() via context.test_schema. +""" +from __future__ import annotations + +from behave import when, then +from behave.runner import Context + + +@when("I list tables in the test schema") +def step_list_tables(context: Context) -> None: + catalog, schema = context.test_schema.split(".", 1) + context.table_list = list( + context.workspace.tables.list(catalog_name=catalog, schema_name=schema) + ) + + +@then("the table list should include {table_count:d} tables") +def step_table_count(context: Context, table_count: int) -> None: + actual = len(context.table_list) + assert actual == table_count, ( + f"Expected {table_count} tables, got {actual}: " + f"{[t.name for t in context.table_list]}" + ) + + +@then('the table list should include "{table_name}"') +def step_table_in_list(context: Context, table_name: str) -> None: + names = [t.name for t in context.table_list] + assert table_name in names, f"Table '{table_name}' not in list: {names}" + + +@when('I grant {privilege} on managed table "{table_name}" to group "{group}"') +def step_grant_privilege( + context: Context, privilege: str, table_name: str, group: str +) -> None: + """Grant privilege using SQL — more stable across SDK versions than the grants API.""" + from databricks.sdk.service.sql import StatementState + + fqn = f"{context.test_schema}.{table_name}" + result = context.workspace.statement_execution.execute_statement( + warehouse_id=context.warehouse_id, + statement=f"GRANT {privilege} ON TABLE {fqn} TO `{group}`", + wait_timeout="30s", + ) + assert result.status.state == StatementState.SUCCEEDED, ( + f"GRANT failed: {result.status.error}" + ) + + +@then('the group "{group}" should have {privilege} on managed table "{table_name}"') +def step_verify_permission( + context: Context, group: str, privilege: str, table_name: str +) -> None: + """Verify privilege using SHOW GRANTS — stable across SDK versions.""" + from databricks.sdk.service.sql import StatementState + + fqn = f"{context.test_schema}.{table_name}" + result = context.workspace.statement_execution.execute_statement( + warehouse_id=context.warehouse_id, + statement=f"SHOW GRANTS ON TABLE {fqn}", + wait_timeout="30s", + ) + assert result.status.state == StatementState.SUCCEEDED, ( + f"SHOW GRANTS failed: {result.status.error}" + ) + # Parse result: columns are Principal, ActionType, ObjectType, ObjectKey (PascalCase) + rows = result.result.data_array or [] + columns = [c.name for c in result.manifest.schema.columns] + # Handle case variation — normalize to lowercase for lookup + col_lower = [c.lower() for c in columns] + principal_idx = col_lower.index("principal") + action_idx = col_lower.index("actiontype") + + found = any( + row[principal_idx] == group and row[action_idx] == privilege + for row in rows + ) + assert found, ( + f"Expected {group} to have {privilege} on {fqn}. " + f"Grants found: {[(r[principal_idx], r[action_idx]) for r in rows]}" + ) diff --git a/.claude/skills/bdd-scaffold/test-suite/features/steps/common_steps.py b/.claude/skills/bdd-scaffold/test-suite/features/steps/common_steps.py new file mode 100644 index 00000000..03507610 --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/features/steps/common_steps.py @@ -0,0 +1,130 @@ +"""Shared step definitions for Databricks BDD tests. + +Design principle: Gherkin uses short table names ("customers"). +Step definitions prepend context.test_schema internally. +This avoids {curly_brace} conflicts with Behave's parse library. +""" +from __future__ import annotations + +from behave import given, when, then, step +from behave.runner import Context +from databricks.sdk.service.sql import StatementState + + +# ─── Connection and setup ──────────────────────────────────────── + + +@given("a Databricks workspace connection is established") +def step_workspace_connection(context: Context) -> None: + assert hasattr(context, "workspace"), "No workspace client — check environment.py" + assert hasattr(context, "warehouse_id"), "No warehouse_id — check environment.py" + + +@given("a test schema is provisioned") +def step_test_schema(context: Context) -> None: + assert hasattr(context, "test_schema"), "No test_schema — check environment.py" + + +# ─── Table creation ────────────────────────────────────────────── + + +@given('a managed table "{table_name}" exists') +def step_ensure_table(context: Context, table_name: str) -> None: + fqn = _fqn(context, table_name) + _sql(context, f"CREATE TABLE IF NOT EXISTS {fqn} (id BIGINT)") + context.scenario_cleanup_sql.append(f"DROP TABLE IF EXISTS {fqn}") + + +@given('a managed table "{table_name}" with data:') +def step_create_with_data(context: Context, table_name: str) -> None: + fqn = _fqn(context, table_name) + headers = context.table.headings + rows = context.table.rows + + col_defs = ", ".join(f"`{h}` STRING" for h in headers) + _sql(context, f"CREATE OR REPLACE TABLE {fqn} ({col_defs})") + context.scenario_cleanup_sql.append(f"DROP TABLE IF EXISTS {fqn}") + + for row in rows: + values = ", ".join(f"'{cell}'" for cell in row) + _sql(context, f"INSERT INTO {fqn} VALUES ({values})") + + +# ─── SQL execution ─────────────────────────────────────────────── + + +@when("I execute a query on the test schema:") +def step_execute_sql(context: Context) -> None: + """Execute SQL from docstring. Use {schema} as placeholder for test schema.""" + sql = context.text.replace("{schema}", context.test_schema) + context.query_result = _sql(context, sql) + + +# ─── Table assertions ──────────────────────────────────────────── + + +@then('the managed table "{table_name}" should exist') +def step_table_exists(context: Context, table_name: str) -> None: + fqn = _fqn(context, table_name) + try: + context.workspace.tables.get(fqn) + except Exception as e: + raise AssertionError(f"Table {fqn} does not exist: {e}") + + +@then('the managed table "{table_name}" should have {expected:d} rows') +def step_row_count(context: Context, table_name: str, expected: int) -> None: + fqn = _fqn(context, table_name) + result = _sql(context, f"SELECT COUNT(*) AS cnt FROM {fqn}") + actual = int(result.result.data_array[0][0]) + assert actual == expected, f"Expected {expected} rows in {table_name}, got {actual}" + + +@then("the test schema should exist in Unity Catalog") +def step_schema_exists(context: Context) -> None: + try: + context.workspace.schemas.get(context.test_schema) + except Exception as e: + raise AssertionError(f"Schema {context.test_schema} does not exist: {e}") + + +# ─── Query result assertions ──────────────────────────────────── + + +@then("the query result should have {expected:d} rows") +def step_result_row_count(context: Context, expected: int) -> None: + rows = context.query_result.result.data_array or [] + actual = len(rows) + assert actual == expected, f"Expected {expected} result rows, got {actual}" + + +@then('the first result column "{col}" should be "{value}"') +def step_first_result_value(context: Context, col: str, value: str) -> None: + result = context.query_result + columns = [c.name for c in result.manifest.schema.columns] + assert col in columns, f"Column '{col}' not in result: {columns}" + col_idx = columns.index(col) + actual = result.result.data_array[0][col_idx] + assert str(actual) == value, f"Expected {col}='{value}', got '{actual}'" + + +# ─── Helpers ───────────────────────────────────────────────────── + + +def _fqn(context: Context, table_name: str) -> str: + """Build fully-qualified table name from short name.""" + return f"{context.test_schema}.{table_name}" + + +def _sql(context: Context, sql: str): + """Execute SQL and assert success.""" + result = context.workspace.statement_execution.execute_statement( + warehouse_id=context.warehouse_id, + statement=sql, + wait_timeout="30s", + ) + assert result.status.state == StatementState.SUCCEEDED, ( + f"SQL failed ({result.status.state}): {result.status.error}\n" + f"Statement: {sql[:200]}" + ) + return result diff --git a/.claude/skills/bdd-scaffold/test-suite/pyproject.toml b/.claude/skills/bdd-scaffold/test-suite/pyproject.toml new file mode 100644 index 00000000..1ecbbdb4 --- /dev/null +++ b/.claude/skills/bdd-scaffold/test-suite/pyproject.toml @@ -0,0 +1,19 @@ +[project] +name = "bdd-test-suite" +version = "0.1.0" +requires-python = ">=3.11" +dependencies = [ + "behave>=1.2.6", + "databricks-sdk>=0.40.0", + "httpx>=0.27.0", +] + +[tool.behave] +default_format = "pretty" +show_timings = true +default_tags = ["not @wip and not @skip"] +logging_level = "INFO" + +[tool.behave.userdata] +warehouse_id = "auto" +catalog = "main" diff --git a/.claude/skills/bdd-steps/SKILL.md b/.claude/skills/bdd-steps/SKILL.md new file mode 100644 index 00000000..a14a12b6 --- /dev/null +++ b/.claude/skills/bdd-steps/SKILL.md @@ -0,0 +1,109 @@ +--- +name: bdd-steps +description: "This skill should be used when the user asks to \"write step definitions\", \"implement BDD steps\", \"generate step code\", \"create Behave steps\", \"implement Given When Then\", \"write Python steps for Gherkin\", \"step definitions for Databricks\", or needs to create Python step implementations for existing Gherkin feature files." +user-invocable: true +--- + +# BDD Steps — Python Step Definition Generation + +Generate Python step definitions for Behave that implement Gherkin steps using the Databricks SDK. Read existing `.feature` files, identify undefined steps, and produce well-typed implementations. + +## When to use + +- Implementing step definitions for new or existing feature files +- Adding Databricks SDK calls to step implementations +- Refactoring step definitions for reusability across features + +## Process + +### 1. Identify undefined steps + +Read the target feature files, then run a dry-run to find undefined steps: + +```bash +uv run behave --dry-run features/.feature 2>&1 +``` + +Behave prints suggested snippets for each undefined step. Use these as the starting point. + +### 2. Write step definitions + +Place step files in `features/steps/` organized by domain: + +| File | Domain | Key SDK imports | +|------|--------|----------------| +| `common_steps.py` | Shared utilities | `WorkspaceClient`, `StatementState` | +| `catalog_steps.py` | Unity Catalog | `catalog.PermissionsChange`, `catalog.Privilege`, `catalog.SecurableType` | +| `pipeline_steps.py` | Lakeflow SDP | `pipelines.PipelineStateInfo` | +| `job_steps.py` | Jobs/Notebooks | `jobs.SubmitTask`, `jobs.NotebookTask`, `jobs.RunLifeCycleState` | +| `app_steps.py` | Databricks Apps | `httpx.Client` for HTTP assertions | +| `sql_steps.py` | SQL/Data quality | `sql.StatementState`, `sql.Disposition` | + +**Step definition structure:** + +```python +from __future__ import annotations + +from behave import given, when, then +from behave.runner import Context + + +@given('a descriptive step pattern with "{parameter}"') +def step_impl(context: Context, parameter: str) -> None: + """Docstring explaining what this step does.""" + # Implementation using context.workspace (set in environment.py) + ... +``` + +Refer to `references/step-library.md` for a comprehensive library of reusable Databricks step definitions covering: +- Workspace connection and SQL execution +- Table/schema existence and row count assertions +- Grant and permission verification +- Pipeline triggering and status polling +- Job submission and completion waiting +- HTTP endpoint testing with SSO header simulation + +### 3. Step writing principles + +**Use `context` for state passing.** Store results in `context` attributes so downstream `Then` steps can assert on them: + +```python +@when('I execute a query on "{table_name}"') +def step_execute(context: Context, table_name: str) -> None: + context.query_result = context.workspace.statement_execution.execute_statement(...) + +@then('the result should have {count:d} rows') +def step_check_rows(context: Context, count: int) -> None: + actual = len(context.query_result.result.data_array or []) + assert actual == count, f"Expected {count}, got {actual}" +``` + +**Type all parameters.** Use Behave's parse types (`{name:d}` for int, `{name:f}` for float) or register custom types. + +**Assertion messages must be diagnostic.** Always include expected vs. actual values: + +```python +assert actual == expected, f"Expected {expected}, got {actual}" +``` + +**Substitute `{test_schema}` references.** Feature files may use `{test_schema}` as a placeholder. Step definitions should resolve it from `context.test_schema`: + +```python +table_fqn = table_name.replace("{test_schema}", context.test_schema) +``` + +**Poll with timeout for async operations.** Jobs, pipelines, and app deployments need polling loops with configurable timeouts. + +### 4. Validate steps compile + +After writing, verify all steps resolve: + +```bash +uv run behave --dry-run +``` + +Zero undefined steps = ready to run. + +## Additional resources + +- **`references/step-library.md`** — Complete reusable step definition library for all Databricks domains diff --git a/.claude/skills/bdd-steps/references/step-library.md b/.claude/skills/bdd-steps/references/step-library.md new file mode 100644 index 00000000..11ddf766 --- /dev/null +++ b/.claude/skills/bdd-steps/references/step-library.md @@ -0,0 +1,660 @@ +# Reusable Step Definition Library + +Complete library of Databricks step definitions for Behave. Organized by domain. Copy relevant sections into `features/steps/` files. + +**Proven patterns used throughout:** + +- Step patterns use **short names** (e.g., `"{table_name}"`), never `{test_schema}.table` in the pattern +- Step code builds FQN internally: `fqn = f"{context.test_schema}.{table_name}"` +- SQL in docstrings uses `{schema}` placeholder, replaced via `context.text.replace("{schema}", context.test_schema)` +- Steps with data tables have a **trailing colon** in the decorator: `@given('... with data:')` +- Grants use **SQL**, not the SDK grants API (which breaks on recent SDK versions) +- Integer parameters use Behave's built-in `{count:d}` format, not custom type parsers + +--- + +## Common Steps (`common_steps.py`) + +Always include these. They provide workspace connection, SQL execution, and basic assertions. + +```python +"""Shared step definitions for Databricks BDD tests.""" +from __future__ import annotations + +import os +from datetime import datetime + +from behave import given, then, step +from behave.runner import Context +from databricks.sdk.service.sql import StatementState + + +# ─── Connection and setup steps ───────────────────────────────── + +@given("a Databricks workspace connection is established") +def step_workspace_connection(context: Context) -> None: + """Initialize workspace client. Usually handled by environment.py.""" + if not hasattr(context, "workspace"): + from databricks.sdk import WorkspaceClient + context.workspace = WorkspaceClient() + me = context.workspace.current_user.me() + context.current_user = me.user_name + + +@given("a test schema is provisioned") +def step_test_schema(context: Context) -> None: + """Verify test schema exists. Usually handled by environment.py.""" + assert hasattr(context, "test_schema"), ( + "No test_schema on context — check environment.py before_all" + ) + + +# ─── SQL execution steps ──────────────────────────────────────── + +@step("I execute the following SQL") +def step_execute_sql_docstring(context: Context) -> None: + """Execute SQL from a docstring (triple-quoted text in feature file). + + In feature files, use {schema} as the placeholder: + When I execute the following SQL + \"\"\" + SELECT * FROM {schema}.customers + \"\"\" + """ + sql = context.text.replace("{schema}", context.test_schema) + context.query_result = _execute_sql(context, sql) + + +@step('I execute SQL "{sql}"') +def step_execute_sql_inline(context: Context, sql: str) -> None: + """Execute inline SQL. The {schema} placeholder is replaced automatically.""" + sql = sql.replace("{schema}", context.test_schema) + context.query_result = _execute_sql(context, sql) + + +# ─── Table existence and row count assertions ─────────────────── + +@then('the table "{table_name}" should exist') +def step_table_exists(context: Context, table_name: str) -> None: + fqn = f"{context.test_schema}.{table_name}" + try: + context.workspace.tables.get(fqn) + except Exception as e: + raise AssertionError(f"Table {fqn} does not exist: {e}") + + +@then('the streaming table "{table_name}" should exist') +def step_streaming_table_exists(context: Context, table_name: str) -> None: + fqn = f"{context.test_schema}.{table_name}" + try: + info = context.workspace.tables.get(fqn) + assert info.table_type is not None, f"{fqn} exists but has no table_type" + except Exception as e: + raise AssertionError(f"Streaming table {fqn} does not exist: {e}") + + +@then('the materialized view "{table_name}" should exist') +def step_mv_exists(context: Context, table_name: str) -> None: + fqn = f"{context.test_schema}.{table_name}" + try: + context.workspace.tables.get(fqn) + except Exception as e: + raise AssertionError(f"Materialized view {fqn} does not exist: {e}") + + +@then('the table "{table_name}" should have {expected:d} rows') +def step_exact_row_count(context: Context, table_name: str, expected: int) -> None: + actual = _count_rows(context, table_name) + assert actual == expected, f"Expected {expected} rows in {table_name}, got {actual}" + + +@then('the table "{table_name}" should have more than {expected:d} rows') +def step_min_row_count(context: Context, table_name: str, expected: int) -> None: + actual = _count_rows(context, table_name) + assert actual > expected, f"Expected more than {expected} rows in {table_name}, got {actual}" + + +@then('the table "{table_name}" should have 0 rows') +def step_empty_table(context: Context, table_name: str) -> None: + actual = _count_rows(context, table_name) + assert actual == 0, f"Expected 0 rows in {table_name}, got {actual}" + + +# ─── Query result assertions ──────────────────────────────────── + +@then("the result should have {expected:d} rows") +def step_result_row_count(context: Context, expected: int) -> None: + rows = context.query_result.result.data_array or [] + actual = len(rows) + assert actual == expected, f"Expected {expected} rows, got {actual}" + + +@then("the result should have more than {expected:d} rows") +def step_result_min_rows(context: Context, expected: int) -> None: + rows = context.query_result.result.data_array or [] + actual = len(rows) + assert actual > expected, f"Expected more than {expected} rows, got {actual}" + + +@then('the first row column "{col}" should be "{value}"') +def step_first_row_value(context: Context, col: str, value: str) -> None: + result = context.query_result + columns = [c.name for c in result.manifest.schema.columns] + col_idx = columns.index(col) + actual = result.result.data_array[0][col_idx] + assert str(actual) == value, f"Expected {col}={value}, got {actual}" + + +# ─── Data setup steps ─────────────────────────────────────────── + +@given('the table "{table_name}" has been loaded') +def step_table_loaded(context: Context, table_name: str) -> None: + """Assert table exists and is not empty.""" + fqn = f"{context.test_schema}.{table_name}" + count = _count_rows(context, table_name) + assert count > 0, f"Table {fqn} exists but is empty" + + +@given('a managed table "{table_name}" exists') +def step_ensure_table_exists(context: Context, table_name: str) -> None: + fqn = f"{context.test_schema}.{table_name}" + try: + context.workspace.tables.get(fqn) + except Exception: + # Create a minimal table + _execute_sql(context, f"CREATE TABLE IF NOT EXISTS {fqn} (id BIGINT)") + context.scenario_cleanup_sql.append(f"DROP TABLE IF EXISTS {fqn}") + + +@given('a managed table "{table_name}" with data:') +def step_create_table_with_data(context: Context, table_name: str) -> None: + """Create a table and populate from the Gherkin data table. + + The trailing colon in the decorator is required — Behave matches it + as part of the step text when a data table follows. + + Example feature file usage: + Given a managed table "customers" with data: + | id | name | region | + | 1 | Acme | APAC | + | 2 | Contoso | EMEA | + """ + fqn = f"{context.test_schema}.{table_name}" + headers = context.table.headings + rows = context.table.rows + + # Infer types (simple heuristic — all STRING) + col_defs = ", ".join(f"{h} STRING" for h in headers) + _execute_sql(context, f"CREATE OR REPLACE TABLE {fqn} ({col_defs})") + context.scenario_cleanup_sql.append(f"DROP TABLE IF EXISTS {fqn}") + + # Insert rows + for row in rows: + values = ", ".join(f"'{cell}'" for cell in row) + _execute_sql(context, f"INSERT INTO {fqn} VALUES ({values})") + + +# ─── Helpers ──────────────────────────────────────────────────── + +def _execute_sql(context: Context, sql: str): + """Execute SQL and return result.""" + result = context.workspace.statement_execution.execute_statement( + warehouse_id=context.warehouse_id, + statement=sql, + wait_timeout="30s", + ) + assert result.status.state == StatementState.SUCCEEDED, ( + f"SQL failed: {result.status.error}\nStatement: {sql[:200]}" + ) + return result + + +def _count_rows(context: Context, table_name: str) -> int: + """Count rows in a table.""" + fqn = f"{context.test_schema}.{table_name}" + result = _execute_sql(context, f"SELECT COUNT(*) AS cnt FROM {fqn}") + return int(result.result.data_array[0][0]) +``` + +--- + +## Catalog Steps (`catalog_steps.py`) + +Uses SQL for grants instead of the SDK grants API. The SDK's `grants.update(securable_type=SecurableType.TABLE, ...)` fails with `SECURABLETYPE.TABLE is not a valid securable type` on recent SDK versions. + +```python +"""Step definitions for Unity Catalog permissions and security. + +Uses SQL for all grant operations. The SDK grants API is unreliable — +SecurableType.TABLE fails on recent databricks-sdk versions. +""" +from __future__ import annotations + +from behave import when, then +from behave.runner import Context +from databricks.sdk.service.sql import StatementState + + +@when('I grant {privilege} on table "{table_name}" to group "{group}"') +def step_grant(context: Context, privilege: str, table_name: str, group: str) -> None: + """Grant a privilege on a table using SQL. + + Example feature file usage: + When I grant SELECT on table "customers" to group "analysts" + """ + fqn = f"{context.test_schema}.{table_name}" + _execute_sql(context, f"GRANT {privilege} ON TABLE {fqn} TO `{group}`") + + +@when('I revoke {privilege} on table "{table_name}" from group "{group}"') +def step_revoke(context: Context, privilege: str, table_name: str, group: str) -> None: + """Revoke a privilege on a table using SQL.""" + fqn = f"{context.test_schema}.{table_name}" + _execute_sql(context, f"REVOKE {privilege} ON TABLE {fqn} FROM `{group}`") + + +@then('the group "{group}" should have {privilege} permission on "{table_name}"') +def step_verify_grant( + context: Context, group: str, privilege: str, table_name: str +) -> None: + """Verify a grant exists using SHOW GRANTS. + + SHOW GRANTS returns PascalCase columns: Principal, ActionType, ObjectType, ObjectKey. + """ + fqn = f"{context.test_schema}.{table_name}" + result = _execute_sql(context, f"SHOW GRANTS ON TABLE {fqn}") + columns = [c.name for c in result.manifest.schema.columns] + principal_idx = columns.index("Principal") + action_idx = columns.index("ActionType") + + found_privs = [] + for row in result.result.data_array or []: + if row[principal_idx] == group: + found_privs.append(row[action_idx]) + + assert privilege in found_privs, ( + f"Expected {group} to have {privilege} on {fqn}, " + f"found: {found_privs}" + ) + + +@then('the group "{group}" should not have {privilege} permission on "{table_name}"') +def step_verify_no_grant( + context: Context, group: str, privilege: str, table_name: str +) -> None: + """Verify a grant does NOT exist using SHOW GRANTS.""" + fqn = f"{context.test_schema}.{table_name}" + result = _execute_sql(context, f"SHOW GRANTS ON TABLE {fqn}") + columns = [c.name for c in result.manifest.schema.columns] + principal_idx = columns.index("Principal") + action_idx = columns.index("ActionType") + + found_privs = [] + for row in result.result.data_array or []: + if row[principal_idx] == group: + found_privs.append(row[action_idx]) + + assert privilege not in found_privs, ( + f"Expected {group} NOT to have {privilege} on {fqn}, " + f"but found: {found_privs}" + ) + + +def _execute_sql(context: Context, sql: str): + """Execute SQL and return result.""" + result = context.workspace.statement_execution.execute_statement( + warehouse_id=context.warehouse_id, + statement=sql, + wait_timeout="30s", + ) + assert result.status.state == StatementState.SUCCEEDED, ( + f"SQL failed: {result.status.error}\nStatement: {sql[:200]}" + ) + return result +``` + +--- + +## Pipeline Steps (`pipeline_steps.py`) + +```python +"""Step definitions for Lakeflow Spark Declarative Pipelines.""" +from __future__ import annotations + +import time + +from behave import given, when, then +from behave.runner import Context + + +@given('a pipeline "{name}" exists targeting "{schema}"') +def step_pipeline_exists(context: Context, name: str, schema: str) -> None: + pipelines = list( + context.workspace.pipelines.list_pipelines(filter=f'name LIKE "{name}"') + ) + if pipelines: + context.pipeline_id = pipelines[0].pipeline_id + else: + result = context.workspace.pipelines.create( + name=name, + target=schema, + catalog=context.test_catalog, + channel="CURRENT", + ) + context.pipeline_id = result.pipeline_id + context.scenario_cleanup_sql.append(None) # Mark for pipeline cleanup + + +@given('the pipeline "{name}" has completed a full refresh') +def step_pipeline_refreshed(context: Context, name: str) -> None: + """Ensure pipeline exists and has been refreshed at least once.""" + pipelines = list( + context.workspace.pipelines.list_pipelines(filter=f'name LIKE "{name}"') + ) + assert pipelines, f"Pipeline '{name}' not found" + context.pipeline_id = pipelines[0].pipeline_id + # Check latest update status + detail = context.workspace.pipelines.get(context.pipeline_id) + assert detail.latest_updates, f"Pipeline '{name}' has never been run" + + +@when("I trigger a full refresh of the pipeline") +def step_full_refresh(context: Context) -> None: + response = context.workspace.pipelines.start_update( + pipeline_id=context.pipeline_id, + full_refresh=True, + ) + context.update_id = response.update_id + + +@when("I trigger an incremental refresh of the pipeline") +def step_incremental_refresh(context: Context) -> None: + response = context.workspace.pipelines.start_update( + pipeline_id=context.pipeline_id, + full_refresh=False, + ) + context.update_id = response.update_id + + +@then("the pipeline update should succeed within {timeout:d} seconds") +def step_pipeline_success(context: Context, timeout: int) -> None: + _wait_for_pipeline(context, timeout, expect_success=True) + + +@then("the pipeline update should fail") +def step_pipeline_fail(context: Context) -> None: + _wait_for_pipeline(context, timeout=300, expect_success=False) + + +@then('the pipeline error should mention {keyword}') +def step_pipeline_error_contains(context: Context, keyword: str) -> None: + events = list(context.workspace.pipelines.list_pipeline_events( + pipeline_id=context.pipeline_id, + max_results=10, + )) + error_messages = " ".join( + str(e.message) for e in events if e.level == "ERROR" + ) + assert keyword.lower() in error_messages.lower(), ( + f"Expected pipeline error to mention '{keyword}', " + f"but errors were: {error_messages[:500]}" + ) + + +def _wait_for_pipeline( + context: Context, timeout: int, expect_success: bool +) -> None: + deadline = time.time() + timeout + while time.time() < deadline: + update = context.workspace.pipelines.get_update( + pipeline_id=context.pipeline_id, + update_id=context.update_id, + ) + state = update.update.state + if state in ("COMPLETED",): + if expect_success: + return + raise AssertionError("Expected pipeline to fail, but it succeeded") + if state in ("FAILED", "CANCELED"): + if not expect_success: + return + raise AssertionError( + f"Pipeline update {state}. Check update {context.update_id}" + ) + time.sleep(15) + raise TimeoutError(f"Pipeline did not complete within {timeout}s") +``` + +--- + +## Job Steps (`job_steps.py`) + +```python +"""Step definitions for Databricks Jobs and notebook runs.""" +from __future__ import annotations + +import time + +from behave import when, then +from behave.runner import Context +from databricks.sdk.service.jobs import ( + NotebookTask, + RunLifeCycleState, + SubmitTask, +) + + +@when('I run the notebook "{path}" with parameters:') +def step_run_notebook(context: Context, path: str) -> None: + """Run a notebook with parameters from a Gherkin data table. + + The trailing colon is required when a data table follows. + + Example feature file usage: + When I run the notebook "/Workspace/tests/etl" with parameters: + | key | value | + | schema | my_schema | + | mode | full | + """ + params = {} + for row in context.table: + value = row["value"].replace("{schema}", context.test_schema) + params[row["key"]] = value + + run = context.workspace.jobs.submit( + run_name=f"behave-{context.scenario.name[:50]}", + tasks=[ + SubmitTask( + task_key="main", + notebook_task=NotebookTask( + notebook_path=path, + base_parameters=params, + ), + ) + ], + ) + context.run_id = run.response.run_id + + +@then('the job should complete with status "{expected}" within {timeout:d} seconds') +def step_job_status(context: Context, expected: str, timeout: int) -> None: + deadline = time.time() + timeout + while time.time() < deadline: + run = context.workspace.jobs.get_run(context.run_id) + state = run.state + if state.life_cycle_state in ( + RunLifeCycleState.TERMINATED, + RunLifeCycleState.INTERNAL_ERROR, + RunLifeCycleState.SKIPPED, + ): + break + time.sleep(10) + else: + raise TimeoutError(f"Run {context.run_id} did not complete within {timeout}s") + + actual = state.result_state.value if state.result_state else "UNKNOWN" + assert actual == expected, ( + f"Expected {expected}, got {actual}. Message: {state.state_message}" + ) +``` + +--- + +## App Steps (`app_steps.py`) + +```python +"""Step definitions for Databricks Apps (FastAPI) testing.""" +from __future__ import annotations + +import subprocess +import os + +import httpx +from behave import given, when, then +from behave.runner import Context + + +@given('the app is running at "{base_url}"') +def step_app_running(context: Context, base_url: str) -> None: + context.app_client = httpx.Client(base_url=base_url, timeout=10) + + +@given('the test user is "{email}"') +def step_test_user(context: Context, email: str) -> None: + context.auth_headers = { + "X-Forwarded-Email": email, + "X-Forwarded-User": email.split("@")[0], + } + + +@when('I GET "{path}"') +def step_get(context: Context, path: str) -> None: + context.response = context.app_client.get(path) + + +@when('I GET "{path}" with auth headers') +def step_get_auth(context: Context, path: str) -> None: + context.response = context.app_client.get(path, headers=context.auth_headers) + + +@when('I GET "{path}" without auth headers') +def step_get_no_auth(context: Context, path: str) -> None: + context.response = context.app_client.get(path) + + +@when('I POST "{path}" with auth headers and body') +def step_post_auth(context: Context, path: str) -> None: + """POST with JSON body from a docstring. + + Example feature file usage: + When I POST "/api/items" with auth headers and body + \"\"\" + {"name": "test-item", "value": 42} + \"\"\" + """ + import json + body = json.loads(context.text) + context.response = context.app_client.post( + path, json=body, headers=context.auth_headers, + ) + + +@then("the response status should be {code:d}") +def step_status_code(context: Context, code: int) -> None: + assert context.response.status_code == code, ( + f"Expected {code}, got {context.response.status_code}: " + f"{context.response.text[:200]}" + ) + + +@then('the response JSON should contain "{key}" with value "{value}"') +def step_json_value(context: Context, key: str, value: str) -> None: + data = context.response.json() + assert key in data, f"Key '{key}' not in response: {list(data.keys())}" + assert str(data[key]) == value, f"Expected {key}='{value}', got '{data[key]}'" + + +@then("the response should be a JSON list") +def step_json_list(context: Context) -> None: + data = context.response.json() + assert isinstance(data, list), f"Expected list, got {type(data).__name__}" + + +# ─── Deployment steps ──────────────────────────────────────────── + +@when('I deploy using Asset Bundles with target "{target}"') +def step_deploy_bundle(context: Context, target: str) -> None: + result = subprocess.run( + ["databricks", "bundle", "deploy", "--target", target], + capture_output=True, + text=True, + env={**dict(os.environ), "DATABRICKS_BUNDLE_ENGINE": "direct"}, + timeout=300, + ) + context.deploy_result = result + + +@then("the deployment should succeed") +def step_deploy_success(context: Context) -> None: + r = context.deploy_result + assert r.returncode == 0, ( + f"Deploy failed (rc={r.returncode}):\n{r.stderr[:500]}" + ) +``` + +--- + +## Shell Command Steps (reusable) + +```python +"""Step definitions for running CLI commands (DABs, databricks CLI).""" +from __future__ import annotations + +import os +import subprocess + +from behave import when, then +from behave.runner import Context + + +@when('I run "{command}" with target "{target}"') +def step_run_command(context: Context, command: str, target: str) -> None: + full_cmd = f"{command} --target {target}" + context.cmd_result = subprocess.run( + full_cmd.split(), + capture_output=True, + text=True, + env={**dict(os.environ), "DATABRICKS_BUNDLE_ENGINE": "direct"}, + timeout=300, + ) + + +@when('I run "{command}" with target "{target}" and auto-approve') +def step_run_command_approve(context: Context, command: str, target: str) -> None: + full_cmd = f"{command} --target {target} --auto-approve" + context.cmd_result = subprocess.run( + full_cmd.split(), + capture_output=True, + text=True, + env={**dict(os.environ), "DATABRICKS_BUNDLE_ENGINE": "direct"}, + timeout=300, + ) + + +@then("the command should exit with code {code:d}") +def step_exit_code(context: Context, code: int) -> None: + actual = context.cmd_result.returncode + assert actual == code, ( + f"Expected exit code {code}, got {actual}.\n" + f"stdout: {context.cmd_result.stdout[:300]}\n" + f"stderr: {context.cmd_result.stderr[:300]}" + ) + + +@then("the command should succeed") +def step_command_success(context: Context) -> None: + assert context.cmd_result.returncode == 0, ( + f"Command failed (rc={context.cmd_result.returncode}):\n" + f"{context.cmd_result.stderr[:500]}" + ) +``` diff --git a/.claude/skills/dart-add-unit-test/SKILL.md b/.claude/skills/dart-add-unit-test/SKILL.md new file mode 100644 index 00000000..dc27083f --- /dev/null +++ b/.claude/skills/dart-add-unit-test/SKILL.md @@ -0,0 +1,122 @@ +--- +name: dart-add-unit-test +description: Write and organize unit tests for functions, methods, and classes using `package:test`. Use when creating new logic or fixing bugs to ensure code remains correct and regression-free. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:07:58 GMT +--- +# Testing Dart and Flutter Applications + +## Contents +- [Structuring Test Files](#structuring-test-files) +- [Writing Tests](#writing-tests) +- [Executing Tests](#executing-tests) +- [Test Implementation Workflow](#test-implementation-workflow) +- [Examples](#examples) + +## Structuring Test Files +Organize test files to mirror the `lib` directory structure to maintain predictability. + +* Place all test code within the `test` directory at the root of the package. +* Append `_test.dart` to the end of all test file names (e.g., `lib/src/utils.dart` should be tested in `test/src/utils_test.dart`). +* If writing integration tests, place them in an `integration_test` directory at the root of the package. + +## Writing Tests +Utilize `package:test` as the standard testing library for Dart applications. + +* Import `package:test/test.dart` (or `package:flutter_test/flutter_test.dart` for Flutter). +* Group related tests using the `group()` function to provide shared context. +* Define individual test cases using the `test()` function. +* Validate outcomes using the `expect()` function alongside matchers (e.g., `equals()`, `isTrue`, `throwsA()`). +* Write asynchronous tests using standard `async`/`await` syntax. The test runner automatically waits for the `Future` to complete. +* Manage test setup and teardown using `setUp()` and `tearDown()` callbacks. +* If testing code that relies on dependency injection, use `package:mockito` alongside `package:test` to generate mock objects, configure fixed scenarios, and verify interactions. + +## Executing Tests +Select the appropriate test runner based on the project type and test location. + +* If working on a pure Dart project, execute tests using the `dart test` command. +* If working on a Flutter project, execute tests using the `flutter test` command. +* If running integration tests, explicitly specify the directory path, as the default runner ignores it: `dart test integration_test` or `flutter test integration_test`. + +## Test Implementation Workflow + +Follow this sequential workflow when implementing new test suites. Copy the checklist to track your progress. + +### Task Progress +- [ ] 1. Create the test file in the `test/` directory, ensuring the `_test.dart` suffix. +- [ ] 2. Import `package:test/test.dart` and the target library. +- [ ] 3. Define a `main()` function. +- [ ] 4. Initialize shared resources or mocks using `setUp()`. +- [ ] 5. Write `test()` cases grouped by functionality using `group()`. +- [ ] 6. Execute the test suite using the appropriate CLI command. +- [ ] 7. **Feedback Loop**: Run test -> Review stack trace for failures -> Fix implementation or assertions -> Re-run until passing. + +## Examples + +### Standard Unit Test Suite +Demonstrates grouping, setup, synchronous, and asynchronous testing. + +```dart +import 'package:test/test.dart'; +import 'package:my_package/calculator.dart'; + +void main() { + group('Calculator', () { + late Calculator calc; + + setUp(() { + calc = Calculator(); + }); + + test('adds two numbers correctly', () { + expect(calc.add(2, 3), equals(5)); + }); + + test('handles asynchronous operations', () async { + final result = await calc.fetchRemoteValue(); + expect(result, isNotNull); + expect(result, greaterThan(0)); + }); + }); +} +``` + +### Mocking with Mockito +Demonstrates configuring a mock object for dependency injection testing. + +```dart +import 'package:test/test.dart'; +import 'package:mockito/mockito.dart'; +import 'package:mockito/annotations.dart'; +import 'package:my_package/api_client.dart'; +import 'package:my_package/data_service.dart'; + +// Generate the mock using build_runner: dart run build_runner build +@GenerateNiceMocks([MockSpec()]) +import 'data_service_test.mocks.dart'; + +void main() { + group('DataService', () { + late MockApiClient mockApiClient; + late DataService dataService; + + setUp(() { + mockApiClient = MockApiClient(); + dataService = DataService(apiClient: mockApiClient); + }); + + test('returns parsed data on successful API call', () async { + // Configure the mock + when(mockApiClient.get('/data')).thenAnswer((_) async => '{"id": 1}'); + + // Execute the system under test + final result = await dataService.fetchData(); + + // Verify outcomes and interactions + expect(result.id, equals(1)); + verify(mockApiClient.get('/data')).called(1); + }); + }); +} +``` diff --git a/.claude/skills/dart-build-cli-app/SKILL.md b/.claude/skills/dart-build-cli-app/SKILL.md new file mode 100644 index 00000000..239a892c --- /dev/null +++ b/.claude/skills/dart-build-cli-app/SKILL.md @@ -0,0 +1,185 @@ +--- +name: dart-build-cli-app +description: Entrypoint structure, exit codes, cross-platform scripts. Use when building command line utilities, scripts, or applications. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 04 May 2026 17:41:00 GMT +--- +# Building Dart CLI Applications + +## Contents +- [Project Setup & Architecture](#project-setup--architecture) +- [Argument Parsing & Command Routing](#argument-parsing--command-routing) +- [Execution & Error Handling](#execution--error-handling) +- [Testing CLI Applications](#testing-cli-applications) +- [Compilation & Distribution](#compilation--distribution) +- [Workflows](#workflows) +- [Examples](#examples) + +## Project Setup & Architecture + +Initialize new CLI projects using the official Dart template to ensure standard directory structures. + +* Run `dart create -t cli ` to scaffold a console application with basic argument parsing. +* Place executable entry points (files containing `main()`) exclusively in the `bin/` directory. +* Place internal implementation logic in `lib/src/` and expose public APIs via `lib/.dart`. +* Enforce formatting in CI environments by running `dart format . --set-exit-if-changed`. This returns exit code 1 if formatting violations exist. + +## Argument Parsing & Command Routing + +Import the `args` package to manage command-line arguments, flags, and subcommands. + +* If building a simple script: Use `ArgParser` directly to define flags (`addFlag`) and options (`addOption`). +* If building a complex, multi-command CLI (like `git`): Implement `CommandRunner` and extend `Command` for each subcommand. +* Define global arguments on the `CommandRunner.argParser` and command-specific arguments on the individual `Command.argParser`. +* Catch `UsageException` to gracefully handle invalid arguments and display the automatically generated help text. +* **Validate Help Text Accuracy**: Ensure the help text provides all necessary information to run the tool. If the help text references a compiled executable name, and the user needs to add it to their PATH to run it that way, provide clear instructions on how to do so in the help text or description. + +## Execution & Error Handling + +Leverage the `io` and `stack_trace` packages to build robust, production-ready CLI tools. + +* Use the `io` package's `ExitCode` enum to return standard POSIX exit codes (e.g., `ExitCode.success.code`, `ExitCode.usage.code`). +* Use `sharedStdIn` from the `io` package if multiple asynchronous listeners need sequential access to standard input. +* Wrap the application execution in `Chain.capture()` from the `stack_trace` package to track asynchronous stack chains. +* Format output stack traces using `Trace.terse` or `Chain.terse` to strip noisy core library frames and present readable errors to the user. +* **Do not swallow exceptions** in lower-level logic or storage classes unless recovery is possible. Let them bubble up or rethrow them so higher-level commands know operations failed. +* **Fail fast and with non-zero exit codes**: Ensure operation failures result in descriptive error messages to `stderr` and appropriate non-zero exit codes (e.g., using `exit(1)` or triggering a 64 exit code after a caught `UsageException`). + +## Testing CLI Applications + +> [!IMPORTANT] +> **All new commands and significant features must be covered by automated tests.** Manual verification is not sufficient for testing logic. However, manual verification of help text and user experience (UX) is still required to ensure the interface is intuitive and correct. + +Use `test_process` and `test_descriptor` to write high-fidelity integration tests for your CLI. + +* Define expected filesystem states using `test_descriptor` (`d.dir`, `d.file`). +* Create the mock filesystem before execution using `await d.Descriptor.create()`. +* Spawn the CLI process using `TestProcess.start('dart', ['run', 'bin/cli.dart', ...args])`. +* Validate standard output and error streams using `StreamQueue` matchers (e.g., `emitsThrough`, `emits`). +* Assert the final exit code using `await process.shouldExit(0)`. +* Validate resulting filesystem mutations using `await d.Descriptor.validate()`. + +## Compilation & Distribution + +Select the appropriate compilation target based on your distribution requirements. + +* **If testing locally during development:** Use `dart run bin/cli.dart`. This uses the JIT compiler for rapid iteration. +* **If bundling code assets and dynamic libraries:** Use `dart build cli`. This runs build hooks and outputs to `build/cli/_/bundle/`. +* **If distributing a standalone native executable:** Use `dart compile exe bin/cli.dart -o `. This bundles the Dart runtime and machine code into a single file. +* **If distributing multiple apps with strict disk space limits:** Use `dart compile aot-snapshot bin/cli.dart`. Run the resulting `.aot` file using `dartaotruntime`. + +
+Cross-Compilation Targets (Linux Only) + +Dart supports cross-compiling to Linux from macOS, Windows, or Linux hosts. +Use the `--target-os` and `--target-arch` flags with `dart compile exe` or `dart compile aot-snapshot`. + +* `--target-os=linux` (Only Linux is currently supported as a cross-compilation target) +* `--target-arch=arm64` (64-bit ARM) +* `--target-arch=x64` (x86-64) +* `--target-arch=arm` (32-bit ARM) +* `--target-arch=riscv64` (64-bit RISC-V) + +Example: `dart compile exe --target-os=linux --target-arch=arm64 bin/cli.dart` +
+ +## Workflows + +### Task Progress: Implement a New CLI Command +- [ ] Create a new class extending `Command` in `lib/src/commands/`. +- [ ] Define the `name` and `description` properties. +- [ ] Register command-specific flags in the constructor using `argParser.addFlag()` or `argParser.addOption()`. +- [ ] Implement the `run()` method with the core logic. +- [ ] Register the new command in the `CommandRunner` instance in `bin/cli.dart` using `addCommand()`. +- [ ] Create tests for the new command in the `test/` directory using `test_process` or standard tests. +- [ ] Run validator -> Execute `dart run bin/cli.dart help ` to verify help text generation. +- [ ] Verify final UX: Compile the application using `dart compile exe` and run the resulting executable to verify the target user experience (e.g., `./bin/cli `). + +### Task Progress: Compile and Release Native Executable +- [ ] Run validator -> Execute `dart format . --set-exit-if-changed` to ensure code formatting. +- [ ] Run validator -> Execute `dart analyze` to ensure no static analysis errors. +- [ ] Run validator -> Execute `dart test` to pass all integration tests. +- [ ] Compile for host OS: `dart compile exe bin/cli.dart -o build/cli-host` +- [ ] Compile for Linux (if host is macOS/Windows): `dart compile exe --target-os=linux --target-arch=x64 bin/cli.dart -o build/cli-linux-x64` + +## Examples + +### Example: CommandRunner Implementation + +```dart +import 'dart:io'; +import 'package:args/command_runner.dart'; +import 'package:stack_trace/stack_trace.dart'; + +class CommitCommand extends Command { + @override + final String name = 'commit'; + @override + final String description = 'Record changes to the repository.'; + + CommitCommand() { + argParser.addFlag('all', abbr: 'a', help: 'Commit all changed files.'); + } + + @override + Future run() async { + final commitAll = argResults?['all'] as bool? ?? false; + print('Committing... (All: $commitAll)'); + } +} + +void main(List args) { + Chain.capture(() async { + final runner = CommandRunner('dgit', 'Distributed version control.') + ..addCommand(CommitCommand()); + + await runner.run(args); + }, onError: (error, chain) { + if (error is UsageException) { + stderr.writeln(error.message); + stderr.writeln(error.usage); + exit(64); // ExitCode.usage.code + } else { + stderr.writeln('Fatal error: $error'); + stderr.writeln(chain.terse); + exit(1); + } + }); +} +``` + +### Example: Integration Testing with Subprocesses + +```dart +import 'package:test/test.dart'; +import 'package:test_process/test_process.dart'; +import 'package:test_descriptor/test_descriptor.dart' as d; + +void main() { + test('CLI formats output correctly and modifies filesystem', () async { + // 1. Setup mock filesystem + await d.dir('project', [ + d.file('config.json', '{"key": "value"}') + ]).create(); + + // 2. Spawn the CLI process + final process = await TestProcess.start( + 'dart', + ['run', 'bin/cli.dart', 'process', '--path', '${d.sandbox}/project'] + ); + + // 3. Validate stdout stream + await expectLater(process.stdout, emitsThrough('Processing complete.')); + + // 4. Validate exit code + await process.shouldExit(0); + + // 5. Validate filesystem mutations + await d.dir('project', [ + d.file('config.json', '{"key": "value"}'), + d.file('output.log', 'Success') + ]).validate(); + }); +} +``` diff --git a/.claude/skills/dart-collect-coverage/SKILL.md b/.claude/skills/dart-collect-coverage/SKILL.md new file mode 100644 index 00000000..60dad775 --- /dev/null +++ b/.claude/skills/dart-collect-coverage/SKILL.md @@ -0,0 +1,141 @@ +--- +name: dart-collect-coverage +description: Collect coverage using the coverage packge and create an LCOV report +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:14:32 GMT +--- +# Implementing Dart and Flutter Test Coverage + +## Contents +- [Testing Fundamentals](#testing-fundamentals) +- [Coverage Directives](#coverage-directives) +- [Workflow: Configuring and Generating Coverage Reports](#workflow-configuring-and-generating-coverage-reports) +- [Workflow: Advanced Manual Coverage Collection](#workflow-advanced-manual-coverage-collection) +- [Examples](#examples) + +## Testing Fundamentals + +Structure your test suites using the standard Dart testing paradigms. Use `package:test` for Dart projects and `flutter_test` for Flutter projects. + +- **Unit Tests:** Verify individual functions, methods, or classes. +- **Component/Widget Tests:** Verify component behavior, layout, and interaction using mock objects (`package:mockito`). +- **Integration Tests:** Verify entire app flows on simulated or real devices. + +## Coverage Directives + +Exclude specific lines, blocks, or entire files from coverage metrics using inline comments. Pass the `--check-ignore` flag during formatting to enforce these directives. + +- Ignore a single line: `// coverage:ignore-line` +- Ignore a block of code: `// coverage:ignore-start` and `// coverage:ignore-end` +- Ignore an entire file: `// coverage:ignore-file` + +## Workflow: Configuring and Generating Coverage Reports + +Follow this sequential workflow to add the coverage package, execute tests, and generate an LCOV report. + +**Task Progress Checklist:** +- [ ] 1. Add `coverage` as a `dev_dependency`. +- [ ] 2. Execute the automated coverage script. +- [ ] 3. Validate the LCOV output. + +### 1. Add Dependencies +Add the `coverage` package as a `dev_dependency` to your project. Do not add it to standard dependencies. + +If working in a standard Dart project: +```bash +dart pub add dev:coverage +``` + +If working in a Flutter project: +```bash +flutter pub add dev:coverage +``` + +### 2. Collect Coverage and Generate LCOV +Use the bundled `test_with_coverage` script. This script automatically runs all tests, collects the JSON coverage data from the Dart VM, and formats it into an LCOV report. + +```bash +dart run coverage:test_with_coverage +``` +*Note: If working within a Dart workspace (monorepo), specify the test directories explicitly (e.g., `dart run coverage:test_with_coverage -- pkgs/foo/test pkgs/bar/test`).* + +### 3. Feedback Loop: Validate Output +**Run validator -> review errors -> fix:** +1. Verify that the `coverage/` directory was created in the project root. +2. Ensure `coverage/coverage.json` (raw data) and `coverage/lcov.info` (formatted report) exist. +3. If coverage is missing for specific files, ensure they are imported and executed by your test files, or add `// coverage:ignore-file` if they are intentionally excluded. + +## Workflow: Advanced Manual Coverage Collection + +If you require granular control over the VM service, isolate pausing, or need branch/function-level coverage, use the manual collection workflow. + +**Task Progress Checklist:** +- [ ] 1. Run tests with VM service enabled. +- [ ] 2. Collect raw JSON coverage. +- [ ] 3. Format JSON to LCOV. + +### 1. Run Tests with VM Service +Execute tests while pausing isolates on exit and exposing the VM service on a specific port (e.g., 8181). + +```bash +dart run --pause-isolates-on-exit --disable-service-auth-codes --enable-vm-service=8181 test & +``` + +### 2. Collect Raw Coverage +Extract the coverage data from the running VM service and output it to a JSON file. + +```bash +dart run coverage:collect_coverage --wait-paused --uri=http://127.0.0.1:8181/ -o coverage/coverage.json --resume-isolates +``` +*Optional: Append `--function-coverage` and `--branch-coverage` to gather deeper metrics (requires Dart VM 2.17.0+).* + +### 3. Format to LCOV +Convert the raw JSON data into the standard LCOV format. + +```bash +dart run coverage:format_coverage --packages=.dart_tool/package_config.json --lcov -i coverage/coverage.json -o coverage/lcov.info --check-ignore +``` + +## Examples + +### Example: `pubspec.yaml` Configuration +Ensure your `pubspec.yaml` reflects the `coverage` package strictly under `dev_dependencies`. + +```yaml +name: my_dart_app +environment: + sdk: ^3.0.0 + +dependencies: + path: ^1.8.0 + +dev_dependencies: + test: ^1.24.0 + coverage: ^1.15.0 +``` + +### Example: Applying Ignore Directives +Use ignore directives to prevent generated code or untestable edge cases from lowering coverage scores. + +```dart +// coverage:ignore-file +import 'package:meta/meta.dart'; + +class SystemConfig { + final String env; + + SystemConfig(this.env); + + // coverage:ignore-start + void legacyInit() { + print('Deprecated initialization'); + } + // coverage:ignore-end + + bool isProduction() { + if (env == 'prod') return true; + return false; // coverage:ignore-line + } +} +``` diff --git a/.claude/skills/dart-fix-runtime-errors/SKILL.md b/.claude/skills/dart-fix-runtime-errors/SKILL.md new file mode 100644 index 00000000..1a7db851 --- /dev/null +++ b/.claude/skills/dart-fix-runtime-errors/SKILL.md @@ -0,0 +1,166 @@ +--- +name: dart-fix-runtime-errors +description: Uses get_runtime_errors and lsp to fetch an active stack trace, locate the failing line, apply a fix, and verify resolution via hot_reload. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:13:22 GMT +--- +# Resolving Dart Static Analysis Errors + +## Contents +- [Core Concepts & Guidelines](#core-concepts--guidelines) + - [Type System & Soundness](#type-system--soundness) + - [Null Safety](#null-safety) + - [Error Handling](#error-handling) +- [Workflows](#workflows) + - [Workflow: Static Analysis Resolution](#workflow-static-analysis-resolution) +- [Examples](#examples) + +## Core Concepts & Guidelines + +### Type System & Soundness +Enforce Dart's sound type system to prevent runtime invalid states. + +* **Method Overrides:** Maintain sound return types (covariant) and parameter types (contravariant). Never tighten a parameter type in a subclass unless explicitly marked with the `covariant` keyword. +* **Generics & Collections:** Add explicit type annotations to generic classes (e.g., `List`, `Map`). Never assign a `List` to a typed list (e.g., `List`). +* **Downcasting:** Avoid implicit downcasts from `dynamic`. Use explicit casts (e.g., `as List`) when necessary, but ensure the underlying runtime type matches to prevent `TypeError` exceptions. +* **Strict Casts:** Enable `strict-casts: true` in `analysis_options.yaml` under `analyzer: language:` to force explicit casting and catch implicit downcast errors at compile time. + +### Null Safety +Eliminate static errors related to null safety by correctly managing variable initialization and nullability. + +* **Modifiers:** Apply `?` for nullable types, `!` for null assertions, and `required` for named parameters that cannot be null. +* **Late Initialization:** Use the `late` keyword for non-nullable variables guaranteed to be initialized before use. Apply this specifically to top-level or instance variables where Dart's control flow analysis cannot definitively prove initialization. +* **Wildcards:** Use the `_` wildcard variable (Dart 3.7+) for non-binding local variables or parameters to avoid unused variable warnings. + +### Error Handling +Distinguish between recoverable exceptions and unrecoverable errors. + +* **Catching:** Catch `Exception` subtypes for recoverable failures. +* **Errors:** Never explicitly catch `Error` or its subtypes (e.g., `TypeError`, `ArgumentError`). Errors indicate programming bugs that must be fixed, not caught. Enforce this by enabling the `avoid_catching_errors` linter rule. +* **Rethrowing:** Use `rethrow` inside a `catch` block to propagate an exception while preserving its original stack trace. + +## Workflows + +### Workflow: Static Analysis Resolution + +Use this sequential workflow to identify, fix, and verify static analysis errors in a Dart project. Copy the checklist to track your progress. + +**Task Progress:** +- [ ] 1. Run static analyzer. +- [ ] 2. Apply automated fixes. +- [ ] 3. Resolve remaining errors manually. +- [ ] 4. Verify fixes (Feedback Loop). + +**1. Run static analyzer** +Execute the Dart analyzer to identify all static errors in the target directory or file. +```bash +dart analyze . --fatal-infos +``` + +**2. Apply automated fixes** +Use the `dart fix` tool to automatically resolve standard linting and analysis issues. +```bash +# Preview changes +dart fix --dry-run +# Apply changes +dart fix --apply +``` + +**3. Resolve remaining errors manually** +Review the remaining analyzer output and apply conditional logic based on the error type: + +* **If the error is a Null Safety issue (e.g., "Property cannot be accessed on a nullable receiver"):** + * Verify if the variable can logically be null. + * If yes, use optional chaining (`?.`) or provide a fallback (`??`). + * If no, and initialization is guaranteed elsewhere, mark the declaration with `late`. +* **If the error is a Type Mismatch (e.g., "The argument type 'List' can't be assigned..."):** + * Trace the variable's initialization. + * Add explicit generic type annotations to the instantiation (e.g., `[]` instead of `[]`). +* **If the error is an Invalid Override (e.g., "The parameter type doesn't match the overridden method"):** + * Widen the parameter type to match the superclass, OR + * Add the `covariant` keyword to the parameter if tightening the type is intentionally required by the domain logic. + +**4. Verify fixes (Feedback Loop)** +Run the validator. Review errors. Fix. +```bash +dart analyze . +dart test +``` +* **If `dart analyze` reports errors:** Return to Step 3. +* **If `dart test` fails with a `TypeError`:** You have introduced an invalid explicit cast (`as T`) or accessed an uninitialized `late` variable. Locate the runtime failure and correct the type hierarchy or initialization order. + +## Examples + +### Example: Fixing Dynamic List Assignments +**Input (Fails Static Analysis):** +```dart +void printInts(List a) => print(a); + +void main() { + final list = []; // Inferred as List + list.add(1); + list.add(2); + printInts(list); // Error: List can't be assigned to List +} +``` + +**Output (Passes Static Analysis):** +```dart +void printInts(List a) => print(a); + +void main() { + final list = []; // Explicitly typed + list.add(1); + list.add(2); + printInts(list); +} +``` + +### Example: Fixing Method Overrides (Contravariance) +**Input (Fails Static Analysis):** +```dart +class Animal { + void chase(Animal a) {} +} + +class Cat extends Animal { + @override + void chase(Mouse a) {} // Error: Tightening parameter type +} +``` + +**Output (Passes Static Analysis):** +```dart +class Animal { + void chase(Animal a) {} +} + +class Cat extends Animal { + @override + void chase(covariant Mouse a) {} // Explicitly marked covariant +} +``` + +### Example: Fixing Null Safety with `late` +**Input (Fails Static Analysis):** +```dart +class Thermometer { + String temperature; // Error: Non-nullable instance field must be initialized + + void read() { + temperature = '20C'; + } +} +``` + +**Output (Passes Static Analysis):** +```dart +class Thermometer { + late String temperature; // Defers initialization check to runtime + + void read() { + temperature = '20C'; + } +} +``` diff --git a/.claude/skills/dart-generate-test-mocks/SKILL.md b/.claude/skills/dart-generate-test-mocks/SKILL.md new file mode 100644 index 00000000..fcd6d8b5 --- /dev/null +++ b/.claude/skills/dart-generate-test-mocks/SKILL.md @@ -0,0 +1,155 @@ +--- +name: dart-generate-test-mocks +description: Define and generate mock objects for external dependencies using `package:mockito` and `build_runner`. Use when unit testing classes that depend on complex external services like APIs or databases. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:13:58 GMT +--- +# Testing and Mocking Dart Applications + +## Contents +- [Structuring Code for Testability](#structuring-code-for-testability) +- [Managing Dependencies](#managing-dependencies) +- [Generating Mocks](#generating-mocks) +- [Implementing Unit Tests](#implementing-unit-tests) +- [Workflow: Creating and Running Mocked Tests](#workflow-creating-and-running-mocked-tests) +- [Examples](#examples) + +## Structuring Code for Testability +Design Dart classes to support dependency injection. Isolate complex external dependencies (like API clients or databases) so they can be replaced with mock objects during testing. + +- Inject external services (e.g., `http.Client`) through class constructors. +- Represent URLs strictly as `Uri` objects using `Uri.parse(string)`. +- Utilize Dart's object-oriented features (classes, mixins) to define clear interfaces for external interactions. + +## Managing Dependencies +Configure the `pubspec.yaml` file with the necessary testing and code generation packages. + +- Add runtime dependencies (e.g., `package:http`) using `dart pub add http`. +- Add testing dependencies using `dart pub add dev:test dev:mockito dev:build_runner`. +- Import HTTP libraries with a prefix to avoid namespace collisions: `import 'package:http/http.dart' as http;`. + +## Generating Mocks +Use `package:mockito` and `build_runner` to automatically generate mock classes for fixed scenarios and behavior verification. + +- Always use the `@GenerateNiceMocks` annotation (preferable to `@GenerateMocks` to avoid missing stub exceptions). +- Place the annotation in the test file, passing a list of `MockSpec()` objects. +- Import the generated file using the `.mocks.dart` extension. +- Execute `build_runner` to generate the mock files: `dart run build_runner build`. + +## Implementing Unit Tests +Isolate the system under test using the generated mock objects. Use `package:test` to structure the test suite. + +- **Stubbing:** Configure mock behavior before interacting with the system under test. + - Use `when(mock.method()).thenReturn(value)` for synchronous methods. + - **CRITICAL:** Always use `thenAnswer((_) async => value)` for methods returning a `Future` or `Stream`. Never use `thenReturn` for asynchronous returns. +- **Verification:** Assert that the system under test interacted with the mock object correctly. + - Use `verify(mock.method()).called(1)` to check exact invocation counts. + - Use argument matchers like `any`, `anyNamed`, or `captureAny` for flexible verification. + +## Workflow: Creating and Running Mocked Tests + +Use the following checklist to implement and verify mocked unit tests. + +### Task Progress +- [ ] 1. Identify the external dependency to mock (e.g., `http.Client`). +- [ ] 2. Inject the dependency into the target class constructor. +- [ ] 3. Create a test file (e.g., `target_test.dart`) and add `@GenerateNiceMocks([MockSpec()])`. +- [ ] 4. Add the `part` or `import` directive for the generated `.mocks.dart` file. +- [ ] 5. Run `dart run build_runner build` to generate the mock classes. +- [ ] 6. Write the test cases using `group()` and `test()`. +- [ ] 7. Stub required behaviors using `when()`. +- [ ] 8. Execute the target method. +- [ ] 9. Verify interactions using `verify()` and assert outcomes using `expect()`. +- [ ] 10. Run the test suite using `dart test`. + +### Feedback Loop: Test Failures +If tests fail or `build_runner` encounters errors: +1. **Run validator:** Execute `dart test` or `dart run build_runner build`. +2. **Review errors:** Check for missing stubs, mismatched argument matchers, or syntax errors in the generated files. +3. **Fix:** + - If a mock method throws an unexpected null error, ensure you used `@GenerateNiceMocks`. + - If an async stub throws an `ArgumentError`, change `thenReturn` to `thenAnswer`. + - If `build_runner` fails, ensure the `.mocks.dart` import matches the file name exactly. +4. Repeat until all tests pass. + +## Examples + +### High-Fidelity Mocking and Testing Example + +**1. System Under Test (`lib/api_service.dart`)** +```dart +import 'dart:convert'; +import 'package:http/http.dart' as http; + +class ApiService { + final http.Client client; + + ApiService(this.client); + + Future fetchData(String urlString) async { + final uri = Uri.parse(urlString); + final response = await client.get(uri); + + if (response.statusCode == 200) { + return jsonDecode(response.body)['data']; + } else { + throw Exception('Failed to load data'); + } + } +} +``` + +**2. Test Implementation (`test/api_service_test.dart`)** +```dart +import 'package:test/test.dart'; +import 'package:mockito/annotations.dart'; +import 'package:mockito/mockito.dart'; +import 'package:http/http.dart' as http; +import 'package:my_app/api_service.dart'; + +// Generate the mock class for http.Client +@GenerateNiceMocks([MockSpec()]) +import 'api_service_test.mocks.dart'; + +void main() { + group('ApiService', () { + late ApiService apiService; + late MockClient mockHttpClient; + + setUp(() { + mockHttpClient = MockClient(); + apiService = ApiService(mockHttpClient); + }); + + test('returns data if the http call completes successfully', () async { + // Arrange: Stub the async HTTP GET request using thenAnswer + when(mockHttpClient.get(any)).thenAnswer( + (_) async => http.Response('{"data": "Success"}', 200), + ); + + // Act + final result = await apiService.fetchData('https://api.example.com/data'); + + // Assert + expect(result, 'Success'); + + // Verify the mock was called with the correct Uri + verify(mockHttpClient.get(Uri.parse('https://api.example.com/data'))).called(1); + }); + + test('throws an exception if the http call completes with an error', () { + // Arrange + when(mockHttpClient.get(any)).thenAnswer( + (_) async => http.Response('Not Found', 404), + ); + + // Act & Assert + expect( + apiService.fetchData('https://api.example.com/data'), + throwsException, + ); + }); + }); +} +``` diff --git a/.claude/skills/dart-migrate-to-checks-package/SKILL.md b/.claude/skills/dart-migrate-to-checks-package/SKILL.md new file mode 100644 index 00000000..dc398146 --- /dev/null +++ b/.claude/skills/dart-migrate-to-checks-package/SKILL.md @@ -0,0 +1,118 @@ +--- +name: dart-migrate-to-checks-package +description: Replace the usage of `expect` and similar functions from `package:matcher` to `package:checks` equivalents. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:15:22 GMT +--- +# Migrating Dart Tests to Package Checks + +## Contents +- [Dependency Management](#dependency-management) +- [Syntax Migration Guidelines](#syntax-migration-guidelines) +- [Utilizing Dart MCP Tools](#utilizing-dart-mcp-tools) +- [Migration Workflow](#migration-workflow) +- [Examples](#examples) + +## Dependency Management +Manage dependencies using the Dart Tooling MCP Server `pub` tool or standard CLI commands. + +- Add `package:checks` as a `dev_dependency` using `dart pub add dev:checks`. +- Remove `package:matcher` if it is explicitly listed in the `pubspec.yaml` (note: it is often transitively included by `package:test`, which is fine). +- Import `package:checks/checks.dart` in all test files undergoing migration. + +## Syntax Migration Guidelines +Transition test assertions from the `package:matcher` syntax to the literate API provided by `package:checks`. + +- **Basic Equality:** Replace `expect(actual, equals(expected))` or `expect(actual, expected)` with `check(actual).equals(expected)`. +- **Type Checking:** Replace `expect(actual, isA())` with `check(actual).isA()`. +- **Property Extraction:** Replace `expect(actual.property, expected)` with `check(actual).has((a) => a.property, 'property name').equals(expected)`. +- **Cascades for Multiple Checks:** Use Dart's cascade operator (`..`) to chain multiple expectations on a single subject. +- **Asynchronous Expectations:** + - If checking a `Future`, `await` the `check` call: `await check(someFuture).completes((r) => r.equals(expected));`. + - If checking a `Stream`, wrap it in a `StreamQueue` for multiple checks, or use `.withQueue` for single/broadcast checks. + +## Migration Workflow + +Copy and use the following checklist to track progress when migrating a test suite: + +- [ ] **Task Progress** + - [ ] Add `package:checks` as a dev dependency. + - [ ] Identify all test files using `package:matcher` (`expect` calls). + - [ ] Import `package:checks/checks.dart` in target test files. + - [ ] Rewrite all `expect(...)` statements to `check(...)` statements. + - [ ] Run static analyzer (`analyze_files`). + - [ ] Run tests (`run_tests`). + +### Feedback Loop: Static Analysis +1. Run the `analyze_files` tool on the modified test directories. +2. Review any static analysis warnings or errors (e.g., missing imports, incorrect generic types on `isA`, unawaited futures). +3. Fix the warnings. +4. Repeat until the analyzer returns zero issues. + +### Feedback Loop: Test Validation +1. Run the `run_tests` tool. +2. If tests fail, review the failure output. `package:checks` provides detailed context (e.g., `Which: has length of <2>`). +3. Adjust the `check()` expectations or the underlying code to resolve the failure. +4. Repeat until all tests pass. + +## Examples + +### Basic Assertions +**Input (`matcher`):** +```dart +expect(someList.length, 1); +expect(someString, startsWith('a')); +expect(someObject, isA()); +``` + +**Output (`checks`):** +```dart +check(someList).length.equals(1); +check(someString).startsWith('a'); +check(someObject).isA(); +``` + +### Composed Expectations +**Input (`matcher`):** +```dart +expect('foo,bar,baz', allOf([ + contains('foo'), + isNot(startsWith('bar')), + endsWith('baz') +])); +``` + +**Output (`checks`):** +```dart +check('foo,bar,baz') + ..contains('foo') + ..not((s) => s.startsWith('bar')) + ..endsWith('baz'); +``` + +### Asynchronous Futures +**Input (`matcher`):** +```dart +expect(Future.value(10), completion(equals(10))); +expect(Future.error('oh no'), throwsA(equals('oh no'))); +``` + +**Output (`checks`):** +```dart +await check(Future.value(10)).completes((it) => it.equals(10)); +await check(Future.error('oh no')).throws().equals('oh no'); +``` + +### Asynchronous Streams +**Input (`matcher`):** +```dart +var stdout = StreamQueue(Stream.fromIterable(['Ready', 'Go'])); +await expectLater(stdout, emitsThrough('Ready')); +``` + +**Output (`checks`):** +```dart +var stdout = StreamQueue(Stream.fromIterable(['Ready', 'Go'])); +await check(stdout).emitsThrough((it) => it.equals('Ready')); +``` diff --git a/.claude/skills/dart-resolve-package-conflicts/SKILL.md b/.claude/skills/dart-resolve-package-conflicts/SKILL.md new file mode 100644 index 00000000..9a7ffdc5 --- /dev/null +++ b/.claude/skills/dart-resolve-package-conflicts/SKILL.md @@ -0,0 +1,116 @@ +--- +name: dart-resolve-package-conflicts +description: Workflow for fixing package version conflicts. Use this when `pub get` fails due to incompatible package versions. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:11:14 GMT +--- +# Managing Dart Dependencies + +## Contents +- [Core Concepts](#core-concepts) +- [Version Constraints](#version-constraints) +- [Workflow: Auditing Dependencies](#workflow-auditing-dependencies) +- [Workflow: Upgrading Dependencies](#workflow-upgrading-dependencies) +- [Workflow: Resolving Version Conflicts](#workflow-resolving-version-conflicts) +- [Examples](#examples) + +## Core Concepts + +Dart enforces a strict single-version rule for dependencies: a project and all its transitive dependencies must resolve to a single, shared version of any given package. This prevents runtime type mismatches but introduces the risk of "version lock." + +To mitigate version lock, Dart relies on version constraints rather than pinned versions in the `pubspec.yaml`. The `pubspec.lock` file maintains the exact resolved versions for reproducible builds. + +Understand the output columns of `dart pub outdated`: +* **Current:** The version currently recorded in `pubspec.lock`. +* **Upgradable:** The latest version allowed by the constraints in `pubspec.yaml`. `dart pub upgrade` resolves to this. +* **Resolvable:** The absolute latest version that can be resolved when factoring in all other dependencies in the project. +* **Latest:** The latest published version of the package (excluding prereleases). + +## Version Constraints + +* **Use Caret Syntax:** Always use caret syntax (e.g., `^1.2.3`) for dependencies in `pubspec.yaml`. This allows `pub` to select newer, non-breaking versions (up to, but not including, the next major version) during resolution. +* **Tighten Dev Dependencies:** Set the lower bound of `dev_dependencies` to the exact version currently used. This reduces resolution complexity and prevents older, incompatible dev tools from being selected. +* **Enforce Lockfiles in CI:** Use `dart pub get --enforce-lockfile` in CI/CD pipelines to ensure the exact versions tested locally are used in production. + +## Workflow: Auditing Dependencies + +Run this workflow periodically to identify stale packages that may impact stability or performance. + +**Task Progress:** +- [ ] Run `dart pub outdated`. +- [ ] Review the **Upgradable** column to identify packages that can be updated without modifying `pubspec.yaml`. +- [ ] Review the **Resolvable** column to identify packages that require constraint modifications in `pubspec.yaml` to update. +- [ ] Identify any packages marked as retracted or discontinued. + +## Workflow: Upgrading Dependencies + +Use conditional logic based on the audit results to upgrade dependencies. + +**Task Progress:** +- [ ] **If updating to "Upgradable" versions:** + - [ ] Run `dart pub upgrade`. + - [ ] Run `dart pub upgrade --tighten` to automatically update the lower bounds in `pubspec.yaml` to match the newly resolved versions. +- [ ] **If updating to "Resolvable" versions (Major updates):** + - [ ] Manually edit `pubspec.yaml` to bump the version constraint to match the "Resolvable" column (e.g., change `^0.11.0` to `^0.12.1`). + - [ ] Run `dart pub upgrade` to resolve the new constraints and update `pubspec.lock`. +- [ ] **Feedback Loop:** + - [ ] Run `dart analyze` -> review errors -> fix breaking API changes. + - [ ] Run `dart test` -> review failures -> fix regressions. + +## Workflow: Resolving Version Conflicts + +When `pub` cannot find a set of concrete versions that satisfy all constraints, or when dealing with a retracted package version, manipulate the lockfile surgically. + +**NEVER** delete the entire `pubspec.lock` file and run `dart pub get`. This causes uncontrolled upgrades across the entire dependency graph. + +**Task Progress:** +- [ ] Open `pubspec.lock`. +- [ ] Locate the specific YAML block for the conflicting or retracted package. +- [ ] Delete ONLY that package's entry from the lockfile. +- [ ] Run `dart pub get` to fetch the newest compatible, non-retracted version for that specific package. +- [ ] **Feedback Loop:** + - [ ] Run `dart pub deps` -> verify the dependency graph resolves correctly. + - [ ] If resolution fails, identify the transitive dependency causing the lock, update its constraint in `pubspec.yaml`, and retry. + +## Examples + +### Tightening Constraints +When `dart pub outdated` shows a package is resolvable to a higher minor/patch version, use the `--tighten` flag to update the `pubspec.yaml` automatically. + +**Input (`pubspec.yaml`):** +```yaml +dependencies: + http: ^0.13.0 +``` + +**Command:** +```bash +dart pub upgrade --tighten http +``` + +**Output (`pubspec.yaml`):** +```yaml +dependencies: + http: ^0.13.5 +``` + +### Surgical Lockfile Removal +If `package_a` is retracted or locked in a conflict, remove only its block from `pubspec.lock`. + +**Before (`pubspec.lock`):** +```yaml +packages: + package_a: + dependency: "direct main" + description: + name: package_a + url: "https://pub.dev" + source: hosted + version: "1.0.0" # Retracted version + package_b: + dependency: "direct main" + # ... +``` + +**Action:** Delete the `package_a` block entirely. Leave `package_b` untouched. Run `dart pub get`. diff --git a/.claude/skills/dart-run-static-analysis/SKILL.md b/.claude/skills/dart-run-static-analysis/SKILL.md new file mode 100644 index 00000000..27ca6546 --- /dev/null +++ b/.claude/skills/dart-run-static-analysis/SKILL.md @@ -0,0 +1,104 @@ +--- +name: dart-run-static-analysis +description: Execute `dart analyze` to identify warnings and errors, and use `dart fix --apply` to automatically resolve mechanical lint issues. Use during development to ensure code quality and before committing changes. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:09:34 GMT +--- +# Analyzing and Fixing Dart Code + +## Contents +- [Analysis Configuration](#analysis-configuration) +- [Diagnostic Suppression](#diagnostic-suppression) +- [Workflow: Executing Static Analysis](#workflow-executing-static-analysis) +- [Workflow: Applying Automated Fixes](#workflow-applying-automated-fixes) +- [Examples](#examples) + +## Analysis Configuration + +Configure the Dart analyzer using the `analysis_options.yaml` file located at the package root. + +- **Base Configuration:** Always include a standard rule set (e.g., `package:lints/recommended.yaml` or `package:flutter_lints/flutter.yaml`) using the `include:` directive. +- **Strict Type Checks:** Enable strict type checks under the `analyzer: language:` node to prevent implicit downcasts and dynamic inferences. Set `strict-casts: true`, `strict-inference: true`, and `strict-raw-types: true`. +- **Linter Rules:** Explicitly enable or disable specific rules under the `linter: rules:` node. Use a key-value map (`rule_name: true/false`) when overriding included rules, or a list (`- rule_name`) when defining a fresh set. Do not mix list and map syntax in the same `rules` block. +- **Formatter Configuration:** Configure `dart format` behavior under the `formatter:` node. Set `page_width` (default 80) and `trailing_commas` (`automate` or `preserve`). +- **Analyzer Plugins:** Enable custom diagnostics by adding plugins under the `analyzer: plugins:` node. Ensure the plugin package is added as a `dev_dependency` in `pubspec.yaml`. + +## Diagnostic Suppression + +When a diagnostic (lint or warning) yields a false positive or applies to generated code, suppress it explicitly. + +- **File-level Exclusion:** Use the `analyzer: exclude:` node in `analysis_options.yaml` to exclude entire files or directories (e.g., `**/*.g.dart`) using glob patterns. +- **File-level Suppression:** Add `// ignore_for_file: ` at the top of a Dart file to suppress specific diagnostics for the entire file. Use `// ignore_for_file: type=lint` to suppress all linter rules. +- **Line-level Suppression:** Add `// ignore: ` on the line directly above the offending code, or appended to the end of the offending line. +- **Pubspec Suppression:** Add `# ignore: ` above the offending line in `pubspec.yaml` files (e.g., `# ignore: sort_pub_dependencies`). +- **Plugin Diagnostics:** Prefix the diagnostic code with the plugin name when suppressing plugin-specific issues (e.g., `// ignore: some_plugin/some_code`). + +## Workflow: Executing Static Analysis + +Use this workflow to identify type-related bugs, style violations, and potential runtime errors. + +**Task Progress:** +- [ ] 1. Verify `analysis_options.yaml` exists at the project root. +- [ ] 2. Run the analyzer using the `analyze_files` MCP tool (if available) or the CLI command `dart analyze `. +- [ ] 3. Review the diagnostic output. +- [ ] 4. If info-level issues must be treated as failures, append the `--fatal-infos` flag. +- [ ] 5. Resolve reported errors manually or proceed to the Automated Fixes workflow. + +## Workflow: Applying Automated Fixes + +Use this workflow to resolve outdated API usages, apply quick fixes, and migrate code (e.g., Dart 3 migrations). + +**Task Progress:** +- [ ] 1. Execute a dry run to preview proposed changes using the `dart_fix` MCP tool or CLI command `dart fix --dry-run`. +- [ ] 2. Review the proposed fixes to ensure they align with the intended architecture. +- [ ] 3. If additional fixes are required, verify that the corresponding linter rules are enabled in `analysis_options.yaml`. +- [ ] 4. Apply the fixes using the `dart_fix` MCP tool or CLI command `dart fix --apply`. +- [ ] 5. Format the modified code using the `dart_format` MCP tool or CLI command `dart format .`. +- [ ] 6. Run the static analysis workflow to verify all diagnostics are resolved. + +## Examples + +### Comprehensive `analysis_options.yaml` + +```yaml +include: package:flutter_lints/recommended.yaml + +analyzer: + exclude: + - "**/*.g.dart" + - "lib/generated/**" + language: + strict-casts: true + strict-inference: true + strict-raw-types: true + errors: + todo: ignore + invalid_assignment: warning + missing_return: error + +linter: + rules: + avoid_shadowing_type_parameters: false + await_only_futures: true + use_super_parameters: true + +formatter: + page_width: 100 + trailing_commas: preserve +``` + +### Inline Diagnostic Suppression + +```dart +// Suppress for the entire file +// ignore_for_file: unused_local_variable, dead_code + +void processData() { + // Suppress for a specific line + // ignore: invalid_assignment + int x = ''; + + const y = 10; // ignore: constant_identifier_names +} +``` diff --git a/.claude/skills/dart-use-pattern-matching/SKILL.md b/.claude/skills/dart-use-pattern-matching/SKILL.md new file mode 100644 index 00000000..74556201 --- /dev/null +++ b/.claude/skills/dart-use-pattern-matching/SKILL.md @@ -0,0 +1,146 @@ +--- +name: dart-use-pattern-matching +description: Use switch expressions and pattern matching where appropriate +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Fri, 24 Apr 2026 15:08:55 GMT +--- +# Implementing Dart Patterns + +## Contents +- [Pattern Selection Strategy](#pattern-selection-strategy) +- [Switch Statements vs. Expressions](#switch-statements-vs-expressions) +- [Core Pattern Implementations](#core-pattern-implementations) +- [Workflows](#workflows) +- [Examples](#examples) + +## Pattern Selection Strategy + +Apply specific pattern types based on the data structure and desired outcome. Follow these conditional guidelines: + +* **If validating and extracting from deserialized data (e.g., JSON):** Use Map and List patterns to simultaneously check structure and destructure key-value pairs. +* **If handling multiple return values:** Use Record patterns to destructure fields directly into local variables. +* **If executing type-specific behavior (Algebraic Data Types):** Use Object patterns combined with `sealed` classes to ensure exhaustiveness. +* **If matching numeric ranges or conditions:** Use Relational (`>=`, `<=`) and Logical-and (`&&`) patterns. +* **If multiple cases share logic:** Use Logical-or (`||`) patterns to share a single case body or guard clause. +* **If ignoring specific values:** Use the Wildcard pattern (`_`) or a non-matching Rest element (`...`) in collections. + +## Switch Statements vs. Expressions + +Select the appropriate switch construct based on the execution context: + +* **If producing a value:** Use a **switch expression**. + * Syntax: `switch (value) { pattern => expression, }` + * Rule: Each case must be a single expression. No implicit fallthrough. Must be exhaustive. +* **If executing statements or side effects:** Use a **switch statement**. + * Syntax: `switch (value) { case pattern: statements; }` + * Rule: Empty cases fall through to the next case. Non-empty cases implicitly break (no `break` keyword required). + +## Core Pattern Implementations + +Implement patterns using the following syntax and rules: + +* **Logical-or (`||`):** `pattern1 || pattern2`. Both branches must define the exact same set of variables. +* **Logical-and (`&&`):** `pattern1 && pattern2`. Branches must *not* define overlapping variables. +* **Relational:** `==`, `!=`, `<`, `>`, `<=`, `>=` followed by a constant expression. +* **Cast (`as`):** `pattern as Type`. Throws if the value does not match the type. Use to forcibly assert types during destructuring. +* **Null-check (`?`):** `pattern?`. Fails the match if the value is null. Binds the variable to the non-nullable base type. +* **Null-assert (`!`):** `pattern!`. Throws if the value is null. +* **Variable:** `var name` or `Type name`. Binds the matched value to a new local variable. +* **Wildcard (`_`):** Matches any value and discards it. +* **List:** `[pattern1, pattern2]`. Matches lists of exact length unless a Rest element (`...` or `...var rest`) is used. +* **Map:** `{"key": pattern}`. Matches maps containing the specified keys. Ignores unmatched keys. +* **Record:** `(pattern1, named: pattern2)`. Matches records of the exact shape. Use `:var name` to infer the getter name. +* **Object:** `ClassName(field: pattern)`. Matches instances of `ClassName`. Use `:var field` to infer the getter name. + +## Workflows + +### Task Progress: Implementing Pattern Matching +Copy this checklist to track progress when implementing complex pattern matching logic: + +- [ ] Identify the data structure being evaluated (JSON, Record, Class, Enum). +- [ ] Select the appropriate switch construct (Expression for values, Statement for side-effects). +- [ ] Define the required patterns (Object, Map, List, Record). +- [ ] Extract required data using Variable patterns (`var x`, `:var y`). +- [ ] Apply Guard clauses (`when condition`) for logic that cannot be expressed via patterns. +- [ ] Handle unmatched cases using a Wildcard (`_`) or `default` clause (if not using a sealed class). +- [ ] Run exhaustiveness validator. + +### Feedback Loop: Exhaustiveness Checking +When switching over `sealed` classes or enums, you must ensure all subtypes are handled. + +1. **Run validator:** Execute `dart analyze`. +2. **Review errors:** Look for "The type 'X' is not exhaustively matched by the switch cases" errors. +3. **Fix:** Add the missing Object patterns for the unhandled subtypes, or add a Wildcard (`_`) case if a default fallback is acceptable. + +## Examples + +### JSON Validation and Destructuring +Use Map and List patterns to validate structure and extract data in a single step. + +**Input:** +```dart +var data = { + 'user': ['Lily', 13], +}; +``` + +**Implementation:** +```dart +if (data case {'user': [String name, int age]}) { + print('User $name is $age years old.'); +} else { + print('Invalid JSON structure.'); +} +``` + +### Algebraic Data Types (Sealed Classes) +Use Object patterns with switch expressions to handle family types exhaustively. + +**Implementation:** +```dart +sealed class Shape {} + +class Square implements Shape { + final double length; + Square(this.length); +} + +class Circle implements Shape { + final double radius; + Circle(this.radius); +} + +// Switch expression guarantees exhaustiveness due to `sealed` modifier. +double calculateArea(Shape shape) => switch (shape) { + Square(length: var l) => l * l, + Circle(:var radius) => math.pi * radius * radius, +}; +``` + +### Variable Swapping and Destructuring +Use variable assignment patterns to swap values or extract record fields without temporary variables. + +**Implementation:** +```dart +var (a, b) = ('left', 'right'); +(b, a) = (a, b); // Swap values + +// Destructuring a function return +var (name, age) = getUserInfo(); +``` + +### Guard Clauses and Logical-or +Use `when` to evaluate arbitrary conditions after a pattern matches. + +**Implementation:** +```dart +switch (shape) { + case Square(size: var s) || Circle(size: var s) when s > 0: + print('Valid symmetric shape with size $s'); + case Square() || Circle(): + print('Invalid or empty shape'); + default: + print('Unknown shape'); +} +``` diff --git a/.claude/skills/agent-bricks/1-knowledge-assistants.md b/.claude/skills/databricks-agent-bricks/1-knowledge-assistants.md similarity index 80% rename from .claude/skills/agent-bricks/1-knowledge-assistants.md rename to .claude/skills/databricks-agent-bricks/1-knowledge-assistants.md index d81e351d..3adff469 100644 --- a/.claude/skills/agent-bricks/1-knowledge-assistants.md +++ b/.claude/skills/databricks-agent-bricks/1-knowledge-assistants.md @@ -25,12 +25,12 @@ Before creating a KA, you need documents in a Unity Catalog Volume: - Upload PDFs/text files to a Volume manually or via SDK **Option 2: Generate synthetic documents** -- Use the `unstructured-pdf-generation` skill to create realistic PDF documents +- Use the `databricks-unstructured-pdf-generation` skill to create realistic PDF documents - Each PDF gets a companion JSON file with question/guideline pairs for evaluation ## Creating a Knowledge Assistant -Use the `create_or_update_ka` tool: +Use the `manage_ka` tool with `action="create_or_update"`: - `name`: "HR Policy Assistant" - `volume_path`: "/Volumes/my_catalog/my_schema/raw_data/hr_docs" @@ -52,7 +52,7 @@ After creation, the KA endpoint needs to provision: | `ONLINE` | Ready to use | - | | `OFFLINE` | Not currently running | - | -Use `get_ka` to check the status: +Use `manage_ka` with `action="get"` to check the status: - `tile_id`: "" @@ -76,7 +76,7 @@ These are automatically added when `add_examples_from_volume=true` (default). ### Manual -Examples can also be specified in the `create_or_update_ka` call if needed. +Examples can also be specified in the `manage_ka` create_or_update call if needed. ## Best Practices @@ -101,12 +101,12 @@ Be helpful and professional. When answering: To update the indexed documents: 1. Add/remove/modify files in the volume -2. Call `create_or_update_ka` with the same name and `tile_id` +2. Call `manage_ka` with `action="create_or_update"`, the same name and `tile_id` 3. The KA will re-index the updated content ## Example Workflow -1. **Generate PDF documents** using `unstructured-pdf-generation` skill: +1. **Generate PDF documents** using `databricks-unstructured-pdf-generation` skill: - Creates PDFs in `/Volumes/catalog/schema/raw_data/pdf_documents` - Creates JSON files with question/guideline pairs @@ -120,13 +120,13 @@ To update the indexed documents: 5. **Test the KA** in the Databricks UI -## Using KA in Multi-Agent Supervisors +## Using KA in Supervisor Agents -Knowledge Assistants can be used as agents in a Multi-Agent Supervisor (MAS). Each KA has an associated model serving endpoint. +Knowledge Assistants can be used as agents in a Supervisor Agent (formerly Multi-Agent Supervisor, MAS). Each KA has an associated model serving endpoint. ### Finding the Endpoint Name -Use `get_ka` to retrieve the KA details. The response includes: +Use `manage_ka` with `action="get"` to retrieve the KA details. The response includes: - `tile_id`: The unique identifier for the KA - `name`: The KA name (sanitized) - `endpoint_status`: Current status (ONLINE, PROVISIONING, etc.) @@ -135,26 +135,27 @@ The endpoint name follows this pattern: `ka-{tile_id}-endpoint` ### Finding a KA by Name -If you know the KA name but not the tile_id, use `find_ka_by_name`: +If you know the KA name but not the tile_id, use `manage_ka` with `action="find_by_name"`: ```python -find_ka_by_name(name="HR_Policy_Assistant") +manage_ka(action="find_by_name", name="HR_Policy_Assistant") # Returns: {"found": True, "tile_id": "01abc...", "name": "HR_Policy_Assistant", "endpoint_name": "ka-01abc...-endpoint"} ``` -### Example: Adding KA to MAS +### Example: Adding KA to Supervisor Agent ```python # First, find the KA -ka_result = find_ka_by_name(name="HR_Policy_Assistant") +manage_ka(action="find_by_name", name="HR_Policy_Assistant") -# Then use it in a MAS -create_or_update_mas( - name="Support MAS", +# Then use the tile_id in a Supervisor Agent +manage_mas( + action="create_or_update", + name="Support_MAS", agents=[ { "name": "hr_agent", - "endpoint_name": ka_result["endpoint_name"], + "ka_tile_id": "", "description": "Answers HR policy questions from the employee handbook" } ] diff --git a/.claude/skills/databricks-agent-bricks/2-supervisor-agents.md b/.claude/skills/databricks-agent-bricks/2-supervisor-agents.md new file mode 100644 index 00000000..7121bfcf --- /dev/null +++ b/.claude/skills/databricks-agent-bricks/2-supervisor-agents.md @@ -0,0 +1,394 @@ +# Supervisor Agents (MAS) + +Supervisor Agents orchestrate multiple specialized agents, routing user queries to the most appropriate agent based on the query content. + +## What is a Supervisor Agent? + +A Supervisor Agent (formerly Multi-Agent Supervisor, MAS) acts as a traffic controller for multiple AI agents, routing user queries to the most appropriate agent. It supports five types of agents: + +1. **Knowledge Assistants (KA)**: Document-based Q&A from PDFs/files in Volumes +2. **Genie Spaces**: Natural language to SQL for data exploration +3. **Model Serving Endpoints**: Custom LLM agents, fine-tuned models, RAG applications +4. **Unity Catalog Functions**: Callable UC functions for data operations +5. **External MCP Servers**: JSON-RPC endpoints via UC HTTP Connections for external system integration + +When a user asks a question: +1. **Analyzes** the query to understand the intent +2. **Routes** to the most appropriate specialized agent +3. **Returns** the agent's response to the user + +This allows you to combine multiple specialized agents into a single unified interface. + +## When to Use + +Use a Supervisor Agent when: +- You have multiple specialized agents (billing, technical support, HR, etc.) +- Users shouldn't need to know which agent to ask +- You want to provide a unified conversational experience + +## Prerequisites + +Before creating a Supervisor Agent, you need agents of one or both types: + +**Model Serving Endpoints** (`endpoint_name`): +- Knowledge Assistant (KA) endpoints (e.g., `ka-abc123-endpoint`) +- Custom agents built with LangChain, LlamaIndex, etc. +- Fine-tuned models +- RAG applications + +**Genie Spaces** (`genie_space_id`): +- Existing Genie spaces for SQL-based data exploration +- Great for analytics, metrics, and data-driven questions +- No separate endpoint deployment required - reference the space directly +- To find a Genie space by name, use `find_genie_by_name(display_name="My Genie")` +- **Note**: There is NO system table for Genie spaces - do not try to query `system.ai.genie_spaces` + +## Unity Catalog Functions + +Unity Catalog Functions allow Supervisor Agents to call registered UC functions for data operations. + +### Prerequisites + +- UC Function already exists (use SQL `CREATE FUNCTION` or Python UDF) +- Agent service principal has `EXECUTE` privilege: + ```sql + GRANT EXECUTE ON FUNCTION catalog.schema.function_name TO ``; + ``` + +### Configuration + +```json +{ + "name": "data_enrichment", + "uc_function_name": "sales_analytics.utils.enrich_customer_data", + "description": "Enriches customer records with demographic and purchase history data" +} +``` + +**Field**: `uc_function_name` - Fully-qualified function name in format `catalog.schema.function_name` + +## External MCP Servers + +External MCP Servers enable Supervisor Agents to interact with external systems (ERP, CRM, etc.) via UC HTTP Connections. The MCP server implements a JSON-RPC 2.0 endpoint that exposes tools for the Supervisor Agent to call. + +### Prerequisites + +**1. MCP Server Endpoint**: Your external system must provide a JSON-RPC 2.0 endpoint (e.g., `/api/mcp`) that implements the MCP protocol: + +```python +# Example MCP server tool definition +TOOLS = [ + { + "name": "approve_invoice", + "description": "Approve a specific invoice", + "inputSchema": { + "type": "object", + "properties": { + "invoice_number": {"type": "string", "description": "Invoice number to approve"}, + "approver": {"type": "string", "description": "Name/email of approver"}, + }, + "required": ["invoice_number"], + }, + }, +] + +# JSON-RPC methods: initialize, tools/list, tools/call +``` + +**2. UC HTTP Connection**: Create a Unity Catalog HTTP Connection that points to your MCP endpoint: + +```sql +CREATE CONNECTION my_mcp_connection TYPE HTTP +OPTIONS ( + host 'https://my-app.databricksapps.com', -- Your MCP server URL + port '443', + base_path '/api/mcp', -- Path to JSON-RPC endpoint + client_id '', -- OAuth M2M credentials + client_secret '', + oauth_scope 'all-apis', + token_endpoint 'https://.azuredatabricks.net/oidc/v1/token', + is_mcp_connection 'true' -- REQUIRED: Identifies as MCP connection +); +``` + +**3. Grant Permissions**: Agent service principal needs access to the connection: + +```sql +GRANT USE CONNECTION ON my_mcp_connection TO ``; +``` + +### Configuration + +Reference the UC Connection using the `connection_name` field: + +```python +{ + "name": "external_operations", + "connection_name": "my_mcp_connection", + "description": "Execute external system operations: approve invoices, create records, trigger workflows" +} +``` + +**Field**: `connection_name` - the name of the Unity Catalog HTTP Connection configured as an MCP server + +**Important**: Make the description comprehensive - it guides the Supervisor Agent's routing decisions for when to call this agent. + +### Complete Example: Multi-System Supervisor + +Example showing integration of Genie, KA, and external MCP: + +```python +manage_mas( + action="create_or_update", + name="AP_Invoice_Supervisor", + agents=[ + { + "name": "billing_analyst", + "genie_space_id": "01abc123...", + "description": "SQL analytics on AP invoice data: spending trends, vendor analysis, aging reports" + }, + { + "name": "policy_expert", + "ka_tile_id": "f32c5f73...", + "description": "Answers questions about AP policies, approval workflows, and compliance requirements from policy documents" + }, + { + "name": "ap_operations", + "connection_name": "ap_invoice_mcp", + "description": ( + "Execute AP operations: approve/reject/flag invoices, search invoice details, " + "get vendor summaries, trigger batch workflows. Use for ANY action or write operation." + ) + } + ], + description="AP automation assistant with analytics, policy guidance, and operational actions", + instructions=""" + Route queries as follows: + - Data questions (invoice counts, spend analysis, vendor metrics) → billing_analyst + - Policy questions (thresholds, SLAs, compliance rules) → policy_expert + - Actions (approve, reject, flag, search, workflows) → ap_operations + + When a user asks to approve, reject, or flag an invoice, ALWAYS use ap_operations. + """ +) +``` + +### MCP Connection Testing + +Verify your connection before adding to MAS: + +```sql +-- Test tools/list method +SELECT http_request( + conn => 'my_mcp_connection', + method => 'POST', + path => '', + json => '{"jsonrpc":"2.0","method":"tools/list","id":1}' +); +``` + +### Resources + +- **MCP Protocol Spec**: [Model Context Protocol](https://modelcontextprotocol.io) + +## Creating a Supervisor Agent + +Use the `manage_mas` tool with `action="create_or_update"`: + +- `name`: "Customer Support MAS" +- `agents`: + ```json + [ + { + "name": "policy_agent", + "ka_tile_id": "f32c5f73-466b-4798-b3a0-5396b5ece2a5", + "description": "Answers questions about company policies and procedures from indexed documents" + }, + { + "name": "usage_analytics", + "genie_space_id": "01abc123-def4-5678-90ab-cdef12345678", + "description": "Answers data questions about usage metrics, trends, and statistics" + }, + { + "name": "custom_agent", + "endpoint_name": "my-custom-endpoint", + "description": "Handles specialized queries via custom model endpoint" + } + ] + ``` +- `description`: "Routes customer queries to specialized support agents" +- `instructions`: "Analyze the user's question and route to the most appropriate agent. If unclear, ask for clarification." + +This example shows mixing Knowledge Assistants (policy_agent), Genie spaces (usage_analytics), and custom endpoints (custom_agent). + +## Agent Configuration + +Each agent in the `agents` list needs: + +| Field | Required | Description | +|-------|----------|-------------| +| `name` | Yes | Internal identifier for the agent | +| `description` | Yes | What this agent handles (critical for routing) | +| `ka_tile_id` | One of these | Knowledge Assistant tile ID (for document Q&A agents) | +| `genie_space_id` | One of these | Genie space ID (for SQL-based data agents) | +| `endpoint_name` | One of these | Model serving endpoint name (for custom agents) | +| `uc_function_name` | One of these | Unity Catalog function name in format `catalog.schema.function_name` | +| `connection_name` | One of these | Unity Catalog connection name (for external MCP servers) | + +**Note**: Provide exactly one of: `ka_tile_id`, `genie_space_id`, `endpoint_name`, `uc_function_name`, or `connection_name`. + +To find a KA tile_id, use `manage_ka(action="find_by_name", name="Your KA Name")`. +To find a Genie space_id, use `find_genie_by_name(display_name="Your Genie Name")`. + +### Writing Good Descriptions + +The `description` field is critical for routing. Make it specific: + +**Good descriptions:** +- "Handles billing questions including invoices, payments, refunds, and subscription changes" +- "Answers technical questions about API errors, integration issues, and product bugs" +- "Provides information about HR policies, PTO, benefits, and employee handbook" + +**Bad descriptions:** +- "Billing agent" (too vague) +- "Handles stuff" (not helpful) +- "Technical" (not specific) + +## Provisioning Timeline + +After creation, the Supervisor Agent endpoint needs to provision: + +| Status | Meaning | Duration | +|--------|---------|----------| +| `PROVISIONING` | Creating the supervisor | 2-5 minutes | +| `ONLINE` | Ready to route queries | - | +| `OFFLINE` | Not currently running | - | + +Use `manage_mas` with `action="get"` to check the status. + +## Adding Example Questions + +Example questions help with evaluation and can guide routing optimization: + +```json +{ + "examples": [ + { + "question": "I haven't received my invoice for this month", + "guideline": "Should be routed to billing_agent" + }, + { + "question": "The API is returning a 500 error", + "guideline": "Should be routed to technical_agent" + }, + { + "question": "How many vacation days do I have?", + "guideline": "Should be routed to hr_agent" + } + ] +} +``` + +If the Supervisor Agent is not yet `ONLINE`, examples are queued and added automatically when ready. + +## Best Practices + +### Agent Design + +1. **Specialized agents**: Each agent should have a clear, distinct purpose +2. **Non-overlapping domains**: Avoid agents with similar descriptions +3. **Clear boundaries**: Define what each agent does and doesn't handle + +### Instructions + +Provide routing instructions: + +``` +You are a customer support supervisor. Your job is to route user queries to the right specialist: + +1. For billing, payments, or subscription questions → billing_agent +2. For technical issues, bugs, or API problems → technical_agent +3. For HR, benefits, or policy questions → hr_agent + +If the query is unclear or spans multiple domains, ask the user to clarify. +``` + +### Fallback Handling + +Consider adding a general-purpose agent for queries that don't fit elsewhere: + +```json +{ + "name": "general_agent", + "endpoint_name": "general-support-endpoint", + "description": "Handles general inquiries that don't fit other categories, provides navigation help" +} +``` + +## Example Workflow + +1. **Deploy specialized agents** as model serving endpoints: + - `billing-assistant-endpoint` + - `tech-support-endpoint` + - `hr-assistant-endpoint` + +2. **Create the MAS**: + - Configure agents with clear descriptions + - Add routing instructions + +3. **Wait for ONLINE status** (2-5 minutes) + +4. **Add example questions** for evaluation + +5. **Test routing** with various query types + +## Updating a Supervisor Agent + +To update an existing Supervisor Agent: + +1. **Add/remove agents**: Call `manage_mas` with `action="create_or_update"` and updated `agents` list +2. **Update descriptions**: Change agent descriptions to improve routing +3. **Modify instructions**: Update routing rules + +The tool finds the existing Supervisor Agent by name and updates it. + +## Troubleshooting + +### Queries routed to wrong agent + +- Review and improve agent descriptions +- Make descriptions more specific and distinct +- Add examples that demonstrate correct routing + +### Endpoint not responding + +- Verify each underlying model serving endpoint is running +- Check endpoint logs for errors +- Ensure endpoints accept the expected input format + +### Slow responses + +- Check latency of underlying endpoints +- Consider endpoint scaling settings +- Monitor for cold start issues + +## Advanced: Hierarchical Routing + +For complex scenarios, you can create multiple levels of Supervisor Agents: + +``` +Top-level Supervisor +├── Customer Support Supervisor +│ ├── billing_agent +│ ├── technical_agent +│ └── general_agent +├── Sales Supervisor +│ ├── pricing_agent +│ ├── demo_agent +│ └── contract_agent +└── Internal Supervisor + ├── hr_agent + └── it_helpdesk_agent +``` + +Each sub-supervisor is deployed as an endpoint and configured as an agent in the top-level supervisor. diff --git a/.claude/skills/databricks-agent-bricks/SKILL.md b/.claude/skills/databricks-agent-bricks/SKILL.md new file mode 100644 index 00000000..026f204a --- /dev/null +++ b/.claude/skills/databricks-agent-bricks/SKILL.md @@ -0,0 +1,212 @@ +--- +name: databricks-agent-bricks +description: "Create and manage Databricks Agent Bricks: Knowledge Assistants (KA) for document Q&A, Genie Spaces for SQL exploration, and Supervisor Agents (MAS) for multi-agent orchestration. Use when building conversational AI applications on Databricks." +--- + +# Agent Bricks + +Create and manage Databricks Agent Bricks - pre-built AI components for building conversational applications. + +## Overview + +Agent Bricks are three types of pre-built AI tiles in Databricks: + +| Brick | Purpose | Data Source | +|-------|---------|-------------| +| **Knowledge Assistant (KA)** | Document-based Q&A using RAG | PDF/text files in Volumes | +| **Genie Space** | Natural language to SQL | Unity Catalog tables | +| **Supervisor Agent (MAS)** | Multi-agent orchestration | Model serving endpoints | + +## Prerequisites + +Before creating Agent Bricks, ensure you have the required data: + +### For Knowledge Assistants +- **Documents in a Volume**: PDF, text, or other files stored in a Unity Catalog volume +- Generate synthetic documents using the `databricks-unstructured-pdf-generation` skill if needed + +### For Genie Spaces +- **See the `databricks-genie` skill** for comprehensive Genie Space guidance +- Tables in Unity Catalog with the data to explore +- Generate raw data using the `databricks-synthetic-data-gen` skill +- Create tables using the `databricks-spark-declarative-pipelines` skill + +### For Supervisor Agents +- **Model Serving Endpoints**: Deployed agent endpoints (KA endpoints, custom agents, fine-tuned models) +- **Genie Spaces**: Existing Genie spaces can be used directly as agents for SQL-based queries +- Mix and match endpoint-based and Genie-based agents in the same Supervisor Agent + +### For Unity Catalog Functions +- **Existing UC Function**: Function already registered in Unity Catalog +- Agent service principal has `EXECUTE` privilege on the function + +### For External MCP Servers +- **Existing UC HTTP Connection**: Connection configured with `is_mcp_connection: 'true'` +- Agent service principal has `USE CONNECTION` privilege on the connection + +## MCP Tools + +### Knowledge Assistant Tool + +**manage_ka** - Manage Knowledge Assistants (KA) +- `action`: "create_or_update", "get", "find_by_name", or "delete" +- `name`: Name for the KA (for create_or_update, find_by_name) +- `volume_path`: Path to documents (e.g., `/Volumes/catalog/schema/volume/folder`) (for create_or_update) +- `description`: (optional) What the KA does (for create_or_update) +- `instructions`: (optional) How the KA should answer (for create_or_update) +- `tile_id`: The KA tile ID (for get, delete, or update via create_or_update) +- `add_examples_from_volume`: (optional, default: true) Auto-add examples from JSON files (for create_or_update) + +Actions: +- **create_or_update**: Requires `name`, `volume_path`. Optionally pass `tile_id` to update. +- **get**: Requires `tile_id`. Returns tile_id, name, description, endpoint_status, knowledge_sources, examples_count. +- **find_by_name**: Requires `name` (exact match). Returns found, tile_id, name, endpoint_name, endpoint_status. Use this to look up an existing KA when you know the name but not the tile_id. +- **delete**: Requires `tile_id`. + +### Genie Space Tools + +**For comprehensive Genie guidance, use the `databricks-genie` skill.** + +Use `manage_genie` with actions: +- `create_or_update` - Create or update a Genie Space +- `get` - Get Genie Space details +- `list` - List all Genie Spaces +- `delete` - Delete a Genie Space +- `export` / `import` - For migration + +See `databricks-genie` skill for: +- Table inspection workflow +- Sample question best practices +- Curation (instructions, certified queries) + +**IMPORTANT**: There is NO system table for Genie spaces (e.g., `system.ai.genie_spaces` does not exist). Use `manage_genie(action="list")` to find spaces. + +### Supervisor Agent Tool + +**manage_mas** - Manage Supervisor Agents (MAS) +- `action`: "create_or_update", "get", "find_by_name", or "delete" +- `name`: Name for the Supervisor Agent (for create_or_update, find_by_name) +- `agents`: List of agent configurations (for create_or_update), each with: + - `name`: Agent identifier (required) + - `description`: What this agent handles - critical for routing (required) + - `ka_tile_id`: Knowledge Assistant tile ID (use for document Q&A agents - recommended for KAs) + - `genie_space_id`: Genie space ID (use for SQL-based data agents) + - `endpoint_name`: Model serving endpoint name (for custom agents) + - `uc_function_name`: Unity Catalog function name in format `catalog.schema.function_name` + - `connection_name`: Unity Catalog connection name (for external MCP servers) + - Note: Provide exactly one of: `ka_tile_id`, `genie_space_id`, `endpoint_name`, `uc_function_name`, or `connection_name` +- `description`: (optional) What the Supervisor Agent does (for create_or_update) +- `instructions`: (optional) Routing instructions for the supervisor (for create_or_update) +- `tile_id`: The Supervisor Agent tile ID (for get, delete, or update via create_or_update) +- `examples`: (optional) List of example questions with `question` and `guideline` fields (for create_or_update) + +Actions: +- **create_or_update**: Requires `name`, `agents`. Optionally pass `tile_id` to update. +- **get**: Requires `tile_id`. Returns tile_id, name, description, endpoint_status, agents, examples_count. +- **find_by_name**: Requires `name` (exact match). Returns found, tile_id, name, endpoint_status, agents_count. Use this to look up an existing Supervisor Agent when you know the name but not the tile_id. +- **delete**: Requires `tile_id`. + +## Typical Workflow + +### 1. Generate Source Data + +Before creating Agent Bricks, generate the required source data: + +**For KA (document Q&A)**: +``` +1. Use `databricks-unstructured-pdf-generation` skill to generate PDFs +2. PDFs are saved to a Volume with companion JSON files (question/guideline pairs) +``` + +**For Genie (SQL exploration)**: +``` +1. Use `databricks-synthetic-data-gen` skill to create raw parquet data +2. Use `databricks-spark-declarative-pipelines` skill to create bronze/silver/gold tables +``` + +### 2. Create the Agent Brick + +Use `manage_ka(action="create_or_update", ...)` or `manage_mas(action="create_or_update", ...)` with your data sources. + +### 3. Wait for Provisioning + +Newly created KA and MAS tiles need time to provision. The endpoint status will progress: +- `PROVISIONING` - Being created (can take 2-5 minutes) +- `ONLINE` - Ready to use +- `OFFLINE` - Not running + +### 4. Add Examples (Automatic) + +For KA, if `add_examples_from_volume=true`, examples are automatically extracted from JSON files in the volume and added once the endpoint is `ONLINE`. + +## Best Practices + +1. **Use meaningful names**: Names are sanitized automatically (spaces become underscores) +2. **Provide descriptions**: Helps users understand what the brick does +3. **Add instructions**: Guide the AI's behavior and tone +4. **Include sample questions**: Shows users how to interact with the brick +5. **Use the workflow**: Generate data first, then create the brick + +## Example: Multi-Modal Supervisor Agent + +```python +manage_mas( + action="create_or_update", + name="Enterprise Support Supervisor", + agents=[ + { + "name": "knowledge_base", + "ka_tile_id": "f32c5f73-466b-...", + "description": "Answers questions about company policies, procedures, and documentation from indexed files" + }, + { + "name": "analytics_engine", + "genie_space_id": "01abc123...", + "description": "Runs SQL analytics on usage metrics, performance stats, and operational data" + }, + { + "name": "ml_classifier", + "endpoint_name": "custom-classification-endpoint", + "description": "Classifies support tickets and predicts resolution time using custom ML model" + }, + { + "name": "data_enrichment", + "uc_function_name": "support.utils.enrich_ticket_data", + "description": "Enriches support ticket data with customer history and context" + }, + { + "name": "ticket_operations", + "connection_name": "ticket_system_mcp", + "description": "Creates, updates, assigns, and closes support tickets in external ticketing system" + } + ], + description="Comprehensive enterprise support agent with knowledge retrieval, analytics, ML, data enrichment, and ticketing operations", + instructions=""" + Route queries as follows: + 1. Policy/procedure questions → knowledge_base + 2. Data analysis requests → analytics_engine + 3. Ticket classification → ml_classifier + 4. Customer context lookups → data_enrichment + 5. Ticket creation/updates → ticket_operations + + If a query spans multiple domains, chain agents: + - First gather information (analytics_engine or knowledge_base) + - Then take action (ticket_operations) + """ +) +``` + +## Related Skills + +- **[databricks-genie](../databricks-genie/SKILL.md)** - Comprehensive Genie Space creation, curation, and Conversation API guidance +- **[databricks-unstructured-pdf-generation](../databricks-unstructured-pdf-generation/SKILL.md)** - Generate synthetic PDFs to feed into Knowledge Assistants +- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** - Create raw data for Genie Space tables +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - Build bronze/silver/gold tables consumed by Genie Spaces +- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** - Deploy custom agent endpoints used as MAS agents +- **[databricks-vector-search](../databricks-vector-search/SKILL.md)** - Build vector indexes for RAG applications paired with KAs + +## See Also + +- `1-knowledge-assistants.md` - Detailed KA patterns and examples +- `databricks-genie` skill - Detailed Genie patterns, curation, and examples +- `2-supervisor-agents.md` - Detailed MAS patterns and examples diff --git a/.claude/skills/databricks-ai-functions/1-task-functions.md b/.claude/skills/databricks-ai-functions/1-task-functions.md new file mode 100644 index 00000000..a94159ea --- /dev/null +++ b/.claude/skills/databricks-ai-functions/1-task-functions.md @@ -0,0 +1,385 @@ +# Task-Specific AI Functions — Full Reference + +These functions require no model endpoint selection. They call pre-configured Foundation Model APIs optimized for each task. All require DBR 15.1+ (15.4 ML LTS for batch); `ai_parse_document` requires DBR 17.1+. + +--- + +## `ai_analyze_sentiment` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_analyze_sentiment + +Returns one of: `positive`, `negative`, `neutral`, `mixed`, or `NULL`. + +```sql +SELECT ai_analyze_sentiment(review_text) AS sentiment +FROM customer_reviews; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("customer_reviews") +df.withColumn("sentiment", expr("ai_analyze_sentiment(review_text)")).display() +``` + +--- + +## `ai_classify` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_classify + +**Syntax:** `ai_classify(content, labels [, options])` +- `content`: VARIANT | STRING — raw text, or VARIANT from `ai_parse_document` / `ai_extract` +- `labels`: STRING — JSON labels definition: + - Simple array: `'["urgent", "not_urgent", "spam"]'` + - With descriptions: `'{"billing_error": "Payment, invoice, or refund issues", "product_defect": "Any malfunction or bug"}'` (descriptions up to 1000 chars each) + - 2–500 labels, each 1–100 characters +- `options`: optional MAP\: + - `instructions`: task context to improve accuracy (max 20,000 chars) + - `multilabel`: `"true"` to return multiple matching labels (default `"false"`) + +Returns VARIANT. Returns `NULL` if content is `NULL`. + +```sql +-- simple labels +SELECT ticket_text, + ai_classify(ticket_text, '["urgent", "not urgent", "spam"]') AS priority +FROM support_tickets; +-- {"response": ["urgent"], "error_message": null} + +-- labels with descriptions +SELECT ticket_text, + ai_classify( + ticket_text, + '{"billing_error": "Payment, invoice, or refund issues", + "product_defect": "Any malfunction, bug, or breakage", + "account_issue": "Login failures, password resets"}', + MAP('instructions', 'Customer support tickets for a SaaS product') + ) AS category +FROM support_tickets; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("support_tickets") +df.withColumn( + "priority", + expr("ai_classify(ticket_text, '[\"urgent\", \"not urgent\", \"spam\"]')") +).display() +``` + +**Tips:** +- Use label descriptions for ambiguous categories — they significantly improve accuracy +- `multilabel: "true"` enables multi-label classification without running multiple calls +- Up to 500 labels supported + +--- + +## `ai_extract` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_extract + +**Syntax:** `ai_extract(content, schema [, options])` +- `content`: VARIANT | STRING — raw text, or VARIANT from `ai_parse_document` +- `schema`: STRING — JSON schema definition: + - Simple (field names only): `'["invoice_id", "vendor_name", "total_amount"]'` + - Advanced (with types and descriptions): + ```json + { + "invoice_id": {"type": "string"}, + "total_amount": {"type": "number"}, + "currency": {"type": "enum", "labels": ["USD", "EUR", "GBP"]}, + "line_items": {"type": "array", "items": {"type": "object", "properties": {...}}} + } + ``` + - Supported types: `string`, `integer`, `number`, `boolean`, `enum` + - Max 128 fields, 7 nesting levels, 500 enum values +- `options`: optional MAP\: + - `instructions`: task context to improve extraction quality (max 20,000 chars) + +Returns VARIANT `{"response": {...}, "error_message": null}`. Returns `NULL` if content is `NULL`. + +```sql +-- simple schema +SELECT ai_extract( + 'Invoice #12345 from Acme Corp for $1,250.00', + '["invoice_id", "vendor_name", "total_amount"]' +) AS extracted; +-- {"response": {"invoice_id": "12345", "vendor_name": "Acme Corp", ...}, "error_message": null} + +-- composable with ai_parse_document +WITH parsed AS ( + SELECT ai_parse_document(content, MAP('version', '2.0')) AS parsed + FROM READ_FILES('/Volumes/finance/invoices/', format => 'binaryFile') +) +SELECT ai_extract( + parsed, + '["invoice_id", "vendor_name", "total_amount"]', + MAP('instructions', 'These are vendor invoices.') +) AS invoice_data +FROM parsed; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("messages") +df = df.withColumn( + "entities", + expr("ai_extract(message, '[\"person\", \"location\", \"date\"]')") +) +df.display() +``` + +--- + +## `ai_fix_grammar` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_fix_grammar + +**Syntax:** `ai_fix_grammar(content)` — Returns corrected STRING. + +Optimized for English. Useful for cleaning user-generated content before downstream processing. + +```sql +SELECT ai_fix_grammar(user_comment) AS corrected FROM user_feedback; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("user_feedback") +df.withColumn("corrected", expr("ai_fix_grammar(user_comment)")).display() +``` + +--- + +## `ai_gen` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_gen + +**Syntax:** `ai_gen(prompt)` — Returns a generated STRING. + +Use for free-form text generation where the output format doesn't need to be structured. For structured JSON output, use `ai_query` with `responseFormat`. + +```sql +SELECT product_name, + ai_gen(CONCAT('Write a one-sentence marketing tagline for: ', product_name)) AS tagline +FROM products; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("products") +df.withColumn( + "tagline", + expr("ai_gen(concat('Write a one-sentence marketing tagline for: ', product_name))") +).display() +``` + +--- + +## `ai_mask` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_mask + +**Syntax:** `ai_mask(content, labels)` +- `content`: STRING — text with sensitive data +- `labels`: ARRAY\ — entity types to redact + +Returns text with identified entities replaced by `[MASKED]`. + +Common label values: `'person'`, `'email'`, `'phone'`, `'address'`, `'ssn'`, `'credit_card'` + +```sql +SELECT ai_mask( + message_body, + ARRAY('person', 'email', 'phone', 'address') +) AS message_safe +FROM customer_messages; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("customer_messages") +df.withColumn( + "message_safe", + expr("ai_mask(message_body, array('person', 'email', 'phone'))") +).write.format("delta").mode("append").saveAsTable("catalog.schema.messages_safe") +``` + +--- + +## `ai_similarity` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_similarity + +**Syntax:** `ai_similarity(expr1, expr2)` — Returns a FLOAT between 0.0 and 1.0. + +Use for fuzzy deduplication, search result ranking, or item matching across datasets. + +```sql +-- Deduplicate company names (similarity > 0.85 = likely duplicate) +SELECT a.id, b.id, a.name, b.name, + ai_similarity(a.name, b.name) AS score +FROM companies a +JOIN companies b ON a.id < b.id +WHERE ai_similarity(a.name, b.name) > 0.85 +ORDER BY score DESC; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("product_search") +df.withColumn( + "match_score", + expr("ai_similarity(search_query, product_title)") +).orderBy("match_score", ascending=False).display() +``` + +--- + +## `ai_summarize` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_summarize + +**Syntax:** `ai_summarize(content [, max_words])` +- `content`: STRING — text to summarize +- `max_words`: INTEGER (optional) — word limit; default 50; use `0` for uncapped + +```sql +-- Default (50 words) +SELECT ai_summarize(article_body) AS summary FROM news_articles; + +-- Custom word limit +SELECT ai_summarize(article_body, 20) AS brief FROM news_articles; +SELECT ai_summarize(article_body, 0) AS full FROM news_articles; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("news_articles") +df.withColumn("summary", expr("ai_summarize(article_body, 30)")).display() +``` + +--- + +## `ai_translate` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_translate + +**Syntax:** `ai_translate(content, to_lang)` +- `content`: STRING — source text +- `to_lang`: STRING — target language code + +**Supported languages:** `en`, `de`, `fr`, `it`, `pt`, `hi`, `es`, `th` + +For unsupported languages, use `ai_query` with a multilingual model endpoint. + +```sql +-- Single language +SELECT ai_translate(product_description, 'es') AS description_es FROM products; + +-- Multi-language fanout +SELECT + description, + ai_translate(description, 'fr') AS description_fr, + ai_translate(description, 'de') AS description_de +FROM products; +``` + +```python +from pyspark.sql.functions import expr +df = spark.table("products") +df.withColumn( + "description_es", + expr("ai_translate(product_description, 'es')") +).display() +``` + +--- + +## `ai_parse_document` + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_parse_document + +**Requires:** DBR 17.1+ + +**Syntax:** `ai_parse_document(content [, options])` +- `content`: BINARY — document content loaded from `read_files()` or `spark.read.format("binaryFile")` +- `options`: MAP\ (optional) — parsing configuration + +**Supported formats:** PDF, JPG/JPEG, PNG, DOCX, PPTX + +Returns a VARIANT with pages, elements (text paragraphs, tables, figures, headers, footers), bounding boxes, and error metadata. + +**Options:** + +| Key | Values | Description | +|-----|--------|-------------| +| `version` | `'2.0'` | Output schema version | +| `imageOutputPath` | Volume path | Save rendered page images | +| `descriptionElementTypes` | `''`, `'figure'`, `'*'` | AI-generated descriptions (default: `'*'` for all) | + +**Output schema:** + +``` +document +├── pages[] -- page id, image_uri +└── elements[] -- extracted content + ├── type -- "text", "table", "figure", etc. + ├── content -- extracted text + ├── bbox -- bounding box coordinates + └── description -- AI-generated description +metadata -- file info, schema version +error_status[] -- errors per page (if any) +``` + +```sql +-- Parse and extract text blocks +SELECT + path, + parsed:pages[*].elements[*].content AS text_blocks, + parsed:error AS parse_error +FROM ( + SELECT path, ai_parse_document(content) AS parsed + FROM read_files('/Volumes/catalog/schema/landing/docs/', format => 'binaryFile') +); + +-- Parse with options (image output + descriptions) +SELECT ai_parse_document( + content, + map( + 'version', '2.0', + 'imageOutputPath', '/Volumes/catalog/schema/volume/images/', + 'descriptionElementTypes', '*' + ) +) AS parsed +FROM read_files('/Volumes/catalog/schema/volume/invoices/', format => 'binaryFile'); +``` + +```python +from pyspark.sql.functions import expr + +df = ( + spark.read.format("binaryFile") + .load("/Volumes/catalog/schema/landing/docs/") + .withColumn("parsed", expr("ai_parse_document(content)")) + .selectExpr( + "path", + "parsed:pages[*].elements[*].content AS text_blocks", + "parsed:error AS parse_error", + ) + .filter("parse_error IS NULL") +) + +# Chain with task-specific functions on the extracted text +df = ( + df.withColumn("summary", expr("ai_summarize(text_blocks, 50)")) + .withColumn("entities", expr("ai_extract(text_blocks, array('date', 'amount', 'vendor'))")) + .withColumn("category", expr("ai_classify(text_blocks, array('invoice', 'contract', 'report'))")) +) +df.display() +``` + +**Limitations:** +- Processing is slow for dense or low-resolution documents +- Suboptimal for non-Latin alphabets and digitally signed PDFs +- Custom models not supported — always uses the built-in parsing model diff --git a/.claude/skills/databricks-ai-functions/2-ai-query.md b/.claude/skills/databricks-ai-functions/2-ai-query.md new file mode 100644 index 00000000..60d860fa --- /dev/null +++ b/.claude/skills/databricks-ai-functions/2-ai-query.md @@ -0,0 +1,223 @@ +# `ai_query` — Full Reference + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_query + +> Use `ai_query` only when no task-specific function fits. See the function selection table in [SKILL.md](SKILL.md). + +## When to Use `ai_query` + +- Output schema has **nested arrays or deeply nested STRUCTs** (e.g., `itens: [{codigo, descricao, qtde}]`) +- Calling a **custom Model Serving endpoint** (your own fine-tuned model) +- **Multimodal input** — passing binary image files via `files =>` +- **Cross-document reasoning** — prompt includes content from multiple sources +- Need **sampling parameters** (`temperature`, `max_tokens`) control + +## Syntax + +```sql +ai_query( + endpoint, + request + [, returnType => ddl_schema] + [, failOnError => boolean] + [, modelParameters => named_struct(...)] + [, responseFormat => json_string] + [, files => binary_column] +) +``` + +## Parameters + +| Parameter | Type | Runtime | Description | +|---|---|---|---| +| `endpoint` | STRING literal | — | Foundation Model name or custom endpoint name. Never guess — use exact names from the [model serving docs](https://docs.databricks.com/aws/en/machine-learning/foundation-models/supported-models.html). | +| `request` | STRING or STRUCT | — | Prompt string for chat models; STRUCT for custom ML endpoints | +| `returnType` | DDL schema (optional) | 15.2+ | Structures the parsed response like `from_json` | +| `failOnError` | BOOLEAN (optional, default `true`) | 15.3+ | If `false`, returns STRUCT `{response, error}` instead of raising on failure | +| `modelParameters` | STRUCT (optional) | 15.3+ | Sampling params: `temperature`, `max_tokens`, `top_p`, etc. | +| `responseFormat` | JSON string (optional) | 15.4+ | Forces structured JSON output: `'{"type":"json_object"}'` | +| `files` | binary column (optional) | — | Pass binary images directly (JPEG/PNG) — no upload step needed | + +## Foundation Model Names (Do Not Guess) + +| Use case | Endpoint name | +|---|---| +| General reasoning / extraction | `databricks-claude-sonnet-4` | +| Fast / cheap tasks | `databricks-meta-llama-3-1-8b-instruct` | +| Large context / complex | `databricks-meta-llama-3-3-70b-instruct` | +| Multimodal (vision + text) | `databricks-llama-4-maverick` | +| Embeddings | `databricks-gte-large-en` | + +## Patterns + +### Basic — single prompt + +```sql +SELECT ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + 'Describe Databricks SQL in 30 words.' +) AS response; +``` + +### Applied to a table column + +```sql +SELECT ticket_id, + ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + CONCAT('Summarize in one sentence: ', ticket_body) + ) AS summary +FROM support_tickets; +``` + +### Structured JSON output (`responseFormat`) + +Preferred over `returnType` for chat models (requires Runtime 15.4+): + +```sql +SELECT ai_query( + 'databricks-claude-sonnet-4', + CONCAT('Extract invoice fields as JSON. Fields: numero, fornecedor, total, ' + 'itens:[{codigo, descricao, qtde, vlrUnit}]. Input: ', text_blocks), + responseFormat => '{"type":"json_object"}', + failOnError => false +) AS ai_response +FROM parsed_documents; +``` + +Then parse with `from_json`: + +```python +from pyspark.sql.functions import from_json, col + +df = df.withColumn( + "invoice", + from_json( + col("ai_response.response"), + "STRUCT>>" + ) +) +# Access fields +df.select("invoice.numero", "invoice.total", "invoice.itens").display() +``` + +### With `failOnError` (always use in batch pipelines) + +```sql +SELECT + id, + ai_response.response, + ai_response.error +FROM ( + SELECT id, + ai_query( + 'databricks-claude-sonnet-4', + CONCAT('Classify: ', text), + failOnError => false + ) AS ai_response + FROM documents +) +-- Route errors to a separate table downstream +``` + +### With `modelParameters` (control sampling) + +```sql +SELECT ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + CONCAT('Extract entities from: ', text), + failOnError => false, + modelParameters => named_struct('temperature', CAST(0.0 AS DOUBLE), 'max_tokens', 500) +) AS result +FROM documents; +``` + +### Multimodal — image files (`files =>`) + +No file upload step needed. Pass the binary column directly: + +```sql +SELECT + path, + ai_query( + 'databricks-llama-4-maverick', + 'Describe what is in this image in detail.', + files => content + ) AS description +FROM read_files('/Volumes/catalog/schema/images/', format => 'binaryFile'); +``` + +```python +from pyspark.sql.functions import expr + +df = ( + spark.read.format("binaryFile") + .load("/Volumes/catalog/schema/images/") + .withColumn("description", expr(""" + ai_query( + 'databricks-llama-4-maverick', + 'Describe the contents of this image.', + files => content + ) + """)) +) +``` + +### As a reusable SQL UDF + +```sql +CREATE FUNCTION catalog.schema.extract_invoice(text STRING) +RETURNS STRING +RETURN ai_query( + 'databricks-claude-sonnet-4', + CONCAT('Extract invoice JSON from: ', text), + responseFormat => '{"type":"json_object"}' +); + +SELECT extract_invoice(document_text) FROM raw_documents; +``` + +### PySpark with `expr` + +```python +from pyspark.sql.functions import expr + +df = spark.table("documents") +df = df.withColumn("result", expr(""" + ai_query( + 'databricks-claude-sonnet-4', + concat('Extract structured data from: ', content), + responseFormat => '{"type":"json_object"}', + failOnError => false + ) +""")) +``` + +## Error Handling Pattern for Batch Pipelines + +Always use `failOnError => false` in batch jobs. Write errors to a sidecar table: + +```python +import dlt +from pyspark.sql.functions import expr, col + +@dlt.table(comment="AI extraction results") +def extracted(): + return ( + dlt.read("raw") + .withColumn("ai_response", expr(""" + ai_query('databricks-claude-sonnet-4', prompt, + responseFormat => '{"type":"json_object"}', + failOnError => false) + """)) + ) + +@dlt.table(comment="Rows that failed AI extraction") +def extraction_errors(): + return ( + dlt.read("extracted") + .filter(col("ai_response.error").isNotNull()) + .select("id", "prompt", col("ai_response.error").alias("error")) + ) +``` diff --git a/.claude/skills/databricks-ai-functions/3-ai-forecast.md b/.claude/skills/databricks-ai-functions/3-ai-forecast.md new file mode 100644 index 00000000..9c1f9b1f --- /dev/null +++ b/.claude/skills/databricks-ai-functions/3-ai-forecast.md @@ -0,0 +1,162 @@ +# `ai_forecast` — Full Reference + +**Docs:** https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_forecast + +> `ai_forecast` is a **table-valued function** — it returns a table of rows, not a scalar. Call it with `SELECT * FROM ai_forecast(...)`. + +## Requirements + +- **Pro or Serverless SQL warehouse** — not available on Classic or Starter +- Input data must have a DATE or TIMESTAMP time column and at least one numeric value column + +## Syntax + +```sql +SELECT * +FROM ai_forecast( + observed => TABLE(...) or query, + horizon => 'YYYY-MM-DD' or TIMESTAMP, + time_col => 'column_name', + value_col => 'column_name', + [group_col => 'column_name'], + [prediction_interval_width => 0.95] +) +``` + +## Parameters + +| Parameter | Type | Description | +|---|---|---| +| `observed` | TABLE reference or subquery | Training data with time + value columns | +| `horizon` | DATE, TIMESTAMP, or STRING | End date/time for the forecast period | +| `time_col` | STRING | Name of the DATE or TIMESTAMP column in `observed` | +| `value_col` | STRING | One or more numeric columns to forecast (up to 100 per group) | +| `group_col` | STRING (optional) | Column to partition forecasts by — produces one forecast series per group value | +| `prediction_interval_width` | DOUBLE (optional, default 0.95) | Confidence interval width between 0 and 1 | + +## Output Columns + +For each `value_col` named `metric`, the output includes: + +| Column | Type | Description | +|---|---|---| +| time_col | DATE or TIMESTAMP | The forecast timestamp (same type as input) | +| `metric_forecast` | DOUBLE | Point forecast | +| `metric_upper` | DOUBLE | Upper confidence bound | +| `metric_lower` | DOUBLE | Lower confidence bound | +| group_col | original type | Present when `group_col` is specified | + +## Patterns + +### Single Metric Forecast + +```sql +SELECT * +FROM ai_forecast( + observed => TABLE(SELECT order_date, revenue FROM daily_revenue), + horizon => '2026-12-31', + time_col => 'order_date', + value_col => 'revenue' +); +-- Returns: order_date, revenue_forecast, revenue_upper, revenue_lower +``` + +### Multi-Group Forecast + +Produces one forecast series per distinct value of `group_col`: + +```sql +SELECT * +FROM ai_forecast( + observed => TABLE(SELECT date, region, sales FROM regional_sales), + horizon => '2026-12-31', + time_col => 'date', + value_col => 'sales', + group_col => 'region' +); +-- Returns: date, region, sales_forecast, sales_upper, sales_lower +-- One row per date per region +``` + +### Multiple Value Columns + +```sql +SELECT * +FROM ai_forecast( + observed => TABLE(SELECT date, units, revenue FROM daily_kpis), + horizon => '2026-06-30', + time_col => 'date', + value_col => 'units,revenue' -- comma-separated +); +-- Returns: date, units_forecast, units_upper, units_lower, +-- revenue_forecast, revenue_upper, revenue_lower +``` + +### Custom Confidence Interval + +```sql +SELECT * +FROM ai_forecast( + observed => TABLE(SELECT ts, sensor_value FROM iot_readings), + horizon => '2026-03-31', + time_col => 'ts', + value_col => 'sensor_value', + prediction_interval_width => 0.80 -- narrower interval = less conservative +); +``` + +### Filtering Input Data (Subquery) + +```sql +SELECT * +FROM ai_forecast( + observed => TABLE( + SELECT date, sales + FROM daily_sales + WHERE region = 'BR' AND date >= '2024-01-01' + ), + horizon => '2026-12-31', + time_col => 'date', + value_col => 'sales' +); +``` + +### PySpark — Use `spark.sql()` + +`ai_forecast` is a table-valued function and must be called through `spark.sql()`: + +```python +result = spark.sql(""" + SELECT * + FROM ai_forecast( + observed => TABLE(SELECT date, sales FROM catalog.schema.daily_sales), + horizon => '2026-12-31', + time_col => 'date', + value_col => 'sales' + ) +""") +result.display() +``` + +### Save Forecast to Delta Table + +```python +result = spark.sql(""" + SELECT * + FROM ai_forecast( + observed => TABLE(SELECT date, region, revenue FROM catalog.schema.sales), + horizon => '2026-12-31', + time_col => 'date', + value_col => 'revenue', + group_col => 'region' + ) +""") +result.write.format("delta").mode("overwrite").saveAsTable("catalog.schema.revenue_forecast") +``` + +## Notes + +- The underlying model is a **prophet-like piecewise linear + seasonality model** — suitable for business time series with trend and weekly/yearly seasonality +- Handles "any number of groups" but up to **100 metrics per group** +- Output time column preserves the input type (DATE stays DATE, TIMESTAMP stays TIMESTAMP) +- Value columns are always cast to DOUBLE in output regardless of input type diff --git a/.claude/skills/databricks-ai-functions/4-document-processing-pipeline.md b/.claude/skills/databricks-ai-functions/4-document-processing-pipeline.md new file mode 100644 index 00000000..37498f49 --- /dev/null +++ b/.claude/skills/databricks-ai-functions/4-document-processing-pipeline.md @@ -0,0 +1,505 @@ +# Document Processing Pipeline with AI Functions + +End-to-end patterns for building batch document processing pipelines using AI Functions in a Lakeflow Declarative Pipeline (DLT). Covers function selection, `config.yml` centralization, error handling, and guidance on near-real-time variants with DSPy or LangChain. + +> For workflow migration context (e.g., migrating from n8n, LangChain, or other orchestration tools), see the companion skill `n8n-to-databricks`. + +--- + +## Function Selection for Document Pipelines + +When processing documents with AI Functions, apply this order of preference for each stage: + +| Stage | Preferred function | Use `ai_query` when... | +|---|---|---| +| Parse binary docs (PDF, DOCX, images) | `ai_parse_document` | Need image-level reasoning | +| Extract fields from text (flat or nested) | `ai_extract` | Schema exceeds 128 fields or 7 nesting levels | +| Classify document type or status | `ai_classify` | More than 20 categories | +| Score item similarity / matching | `ai_similarity` | Need cross-document reasoning | +| Summarize long sections | `ai_summarize` | — | +| Extract deeply nested JSON | `ai_query` with `responseFormat` | Schema exceeds `ai_extract` limits (128 fields, 7 levels) | + +--- + +## Centralized Configuration (`config.yml`) + +**Always centralize model names, volume paths, and prompts in a `config.yml`.** This makes model swaps a one-line change and keeps pipeline code free of hardcoded strings. + +```yaml +# config.yml +models: + default: "databricks-claude-sonnet-4" + mini: "databricks-meta-llama-3-1-8b-instruct" + vision: "databricks-llama-4-maverick" + +catalog: + name: "my_catalog" + schema: "document_processing" + +volumes: + input: "/Volumes/my_catalog/document_processing/landing/" + tmp: "/Volumes/my_catalog/document_processing/tmp/" + +output_tables: + results: "my_catalog.document_processing.processed_docs" + errors: "my_catalog.document_processing.processing_errors" + +prompts: + extract_invoice: | + Extract invoice fields and return ONLY valid JSON. + Fields: invoice_number, vendor_name, vendor_tax_id (digits only), + issue_date (dd/mm/yyyy), total_amount (numeric), + line_items: [{item_code, description, quantity, unit_price, total}]. + Return null for missing fields. + + classify_doc: | + Classify this document into exactly one category. +``` + +```python +# config_loader.py +import yaml + +def load_config(path: str = "config.yml") -> dict: + with open(path) as f: + return yaml.safe_load(f) + +CFG = load_config() +ENDPOINT = CFG["models"]["default"] +ENDPOINT_MINI = CFG["models"]["mini"] +VOLUME_INPUT = CFG["volumes"]["input"] +PROMPT_INV = CFG["prompts"]["extract_invoice"] +``` + +--- + +## Batch Pipeline — Lakeflow Declarative Pipeline + +Each logical step in your document workflow maps to a `@dlt.table` stage. Data flows through Delta tables between stages. + +``` +[Landing Volume] → Stage 1: ai_parse_document + → Stage 2: ai_classify (document type) + → Stage 3: ai_extract (flat fields) + ai_query (nested JSON) + → Stage 4: ai_similarity (item matching) + → Stage 5: Final Delta output table +``` + +### `pipeline.py` + +```python +import dlt +import yaml +from pyspark.sql.functions import expr, col, from_json + +CFG = yaml.safe_load(open("/Workspace/path/to/config.yml")) +ENDPOINT = CFG["models"]["default"] +VOL_IN = CFG["volumes"]["input"] +PROMPT = CFG["prompts"]["extract_invoice"] + + +# ── Stage 1: Parse binary documents ────────────────────────────────────────── +# Preferred: ai_parse_document — no model selection, no ai_query needed + +@dlt.table(comment="Parsed document text from all file types in the landing volume") +def raw_parsed(): + return ( + spark.read.format("binaryFile").load(VOL_IN) + .withColumn("parsed", expr("ai_parse_document(content, MAP('version', '2.0'))")) + .withColumn("text_blocks", expr(""" + concat_ws('\n', transform( + parsed:document:elements, + e -> e:content::STRING + )) + """)) + .selectExpr( + "path", + "text_blocks", + "parsed:error_status AS parse_error", + ) + .filter("parse_error IS NULL") + ) + + +# ── Stage 2: Classify document type ────────────────────────────────────────── +# Preferred: ai_classify — cheap, no endpoint selection + +@dlt.table(comment="Document type classification") +def classified_docs(): + return ( + dlt.read("raw_parsed") + .withColumn( + "doc_type", + expr(""" + ai_classify( + text_blocks, + '["invoice", "purchase_order", "receipt", "contract", "other"]', + MAP('version', '2.0') + ):response[0]::STRING + """) + ) + ) + + +# ── Stage 3a: Flat field extraction ────────────────────────────────────────── +# Preferred: ai_extract for flat fields (vendor, date, total) + +@dlt.table(comment="Flat header fields extracted from documents") +def extracted_flat(): + return ( + dlt.read("classified_docs") + .filter("doc_type = 'invoice'") + .filter("text_blocks IS NOT NULL") + .withColumn( + "result", + expr(""" + ai_extract( + text_blocks, + '{ + "invoice_number": {"type": "string"}, + "vendor_name": {"type": "string"}, + "issue_date": {"type": "string", "description": "dd/mm/yyyy"}, + "total_amount": {"type": "number"}, + "tax_id": {"type": "string"} + }', + MAP('version', '2.0') + ) + """) + ) + .selectExpr( + "path", "doc_type", "text_blocks", + "result:response AS header", + "result:error_message::STRING AS extract_error" + ) + ) + + +# ── Stage 3b: Nested JSON extraction (last resort: ai_query) ───────────────── +# Use ai_query only for deeply nested schemas that exceed ai_extract's 7-level limit + +@dlt.table(comment="Nested line items extracted — ai_query used for array schema only") +def extracted_line_items(): + return ( + dlt.read("extracted_flat") + .filter("extract_error IS NULL") + .withColumn( + "ai_response", + expr(f""" + ai_query( + '{ENDPOINT}', + concat('{PROMPT.strip()}', '\\n\\nDocument text:\\n', LEFT(text_blocks, 6000)), + responseFormat => '{{"type":"json_object"}}', + failOnError => false + ) + """) + ) + .withColumn( + "line_items", + from_json( + col("ai_response.response"), + "STRUCT>>" + ) + ) + .select("path", "doc_type", "header", "line_items", col("ai_response.error").alias("extraction_error")) + ) + + +# ── Stage 4: Similarity matching ───────────────────────────────────────────── +# Preferred: ai_similarity for fuzzy matching between extracted fields + +@dlt.table(comment="Vendor name similarity vs reference master data") +def vendor_matched(): + extracted = dlt.read("extracted_line_items") + # Join against a reference vendor table for fuzzy matching + vendors = spark.table("my_catalog.document_processing.vendor_master").select("vendor_id", "vendor_name") + + return ( + extracted.crossJoin(vendors) + .withColumn( + "name_similarity", + expr("ai_similarity(header:vendor_name::STRING, vendor_name)") + ) + .filter("name_similarity > 0.80") + .orderBy("name_similarity", ascending=False) + ) + + +# ── Stage 5: Final output + error sidecar ──────────────────────────────────── + +@dlt.table( + comment="Final processed documents ready for downstream consumption", + table_properties={"delta.enableChangeDataFeed": "true"}, +) +def processed_docs(): + return ( + dlt.read("extracted_line_items") + .filter("extraction_error IS NULL") + .selectExpr( + "path", + "doc_type", + "header:invoice_number::STRING AS invoice_number", + "header:vendor_name::STRING AS vendor_name", + "header:issue_date::STRING AS issue_date", + "header:total_amount::DOUBLE AS total_amount", + "line_items.line_items AS items", + ) + ) + + +@dlt.table(comment="Rows that failed at any extraction stage — review and reprocess") +def processing_errors(): + return ( + dlt.read("extracted_flat") + .filter("extract_error IS NOT NULL") + .select("path", "doc_type", col("extract_error").alias("error")) + .unionByName( + dlt.read("extracted_line_items") + .filter("extraction_error IS NOT NULL") + .select("path", "doc_type", col("extraction_error").alias("error")) + ) + ) +``` + +--- + +## Custom RAG Pipeline — Parse → Chunk → Index → Query + +When the goal is retrieval-augmented generation rather than field extraction, use this pipeline to parse documents, chunk them into a Delta table, and index with Vector Search. + +### Step 1 — Parse and Chunk into a Delta Table + +`ai_parse_document` returns a VARIANT. Use `variant_get` with an explicit `ARRAY` cast before calling `explode`, since `explode()` does not accept raw VARIANT values. + +```sql +CREATE OR REPLACE TABLE catalog.schema.parsed_chunks AS +WITH parsed AS ( + SELECT + path, + ai_parse_document(content) AS doc + FROM read_files('/Volumes/catalog/schema/volume/docs/', format => 'binaryFile') +), +elements AS ( + SELECT + path, + explode(variant_get(doc, '$.document.elements', 'ARRAY')) AS element + FROM parsed +) +SELECT + md5(concat(path, variant_get(element, '$.content', 'STRING'))) AS chunk_id, + path AS source_path, + variant_get(element, '$.content', 'STRING') AS content, + variant_get(element, '$.type', 'STRING') AS element_type, + current_timestamp() AS parsed_at +FROM elements +WHERE variant_get(element, '$.content', 'STRING') IS NOT NULL + AND length(trim(variant_get(element, '$.content', 'STRING'))) > 10; +``` + +### Step 1a (Production) — Incremental Parsing with Structured Streaming + +For production pipelines where new documents arrive over time, use Structured Streaming with checkpoints for exactly-once processing. Each run processes only new files, then stops with `trigger(availableNow=True)`. + +See the official bundle example: +[databricks/bundle-examples/contrib/job_with_ai_parse_document](https://github.com/databricks/bundle-examples/tree/main/contrib/job_with_ai_parse_document) + +**Stage 1 — Parse raw documents (streaming):** + +```python +from pyspark.sql.functions import col, current_timestamp, expr + +files_df = ( + spark.readStream.format("binaryFile") + .option("pathGlobFilter", "*.{pdf,jpg,jpeg,png}") + .option("recursiveFileLookup", "true") + .load("/Volumes/catalog/schema/volume/docs/") +) + +parsed_df = ( + files_df + .repartition(8, expr("crc32(path) % 8")) + .withColumn("parsed", expr(""" + ai_parse_document(content, map( + 'version', '2.0', + 'descriptionElementTypes', '*' + )) + """)) + .withColumn("parsed_at", current_timestamp()) + .select("path", "parsed", "parsed_at") +) + +( + parsed_df.writeStream.format("delta") + .outputMode("append") + .option("checkpointLocation", "/Volumes/catalog/schema/checkpoints/01_parse") + .option("mergeSchema", "true") + .trigger(availableNow=True) + .toTable("catalog.schema.parsed_documents_raw") +) +``` + +**Stage 2 — Extract text from parsed VARIANT (streaming):** + +Uses `transform()` to extract element content from the VARIANT array, and `try_cast` for safe access. Error rows are preserved but flagged. + +```python +from pyspark.sql.functions import col, concat_ws, expr, lit, when + +parsed_stream = spark.readStream.format("delta").table("catalog.schema.parsed_documents_raw") + +text_df = ( + parsed_stream + .withColumn("text", + when( + expr("try_cast(parsed:error_status AS STRING)").isNotNull(), lit(None) + ).otherwise( + concat_ws("\n\n", expr(""" + transform( + try_cast(parsed:document:elements AS ARRAY), + element -> try_cast(element:content AS STRING) + ) + """)) + ) + ) + .withColumn("error_status", expr("try_cast(parsed:error_status AS STRING)")) + .select("path", "text", "error_status", "parsed_at") +) + +( + text_df.writeStream.format("delta") + .outputMode("append") + .option("checkpointLocation", "/Volumes/catalog/schema/checkpoints/02_text") + .option("mergeSchema", "true") + .trigger(availableNow=True) + .toTable("catalog.schema.parsed_documents_text") +) +``` + +Key techniques: +- **`repartition` by file hash** — parallelizes `ai_parse_document` across workers +- **`trigger(availableNow=True)`** — processes all pending files then stops (batch-like) +- **Checkpoints** — exactly-once guarantee; no re-parsing on re-runs +- **`transform()` + `try_cast`** — safer than `explode` + `variant_get` for text extraction +- **Separate stages with independent checkpoints** — parse and text extraction can fail/retry independently + +### Step 1b — Enable Change Data Feed + +Required for Vector Search Delta Sync: + +```sql +ALTER TABLE catalog.schema.parsed_chunks +SET TBLPROPERTIES (delta.enableChangeDataFeed = true); +``` + +### Step 2 — Create a Vector Search Index and Query It + +Use the **[databricks-vector-search](../databricks-vector-search/SKILL.md)** skill to create a Delta Sync index on the chunked table and query it. Ensure CDF is enabled first (Step 1b above). + +### RAG-Specific Issues + +| Issue | Solution | +|-------|----------| +| `explode()` fails with VARIANT | `explode()` requires ARRAY, not VARIANT. Use `variant_get(doc, '$.document.elements', 'ARRAY')` to cast before exploding | +| Short/noisy chunks | Filter with `length(trim(...)) > 10` — parsing produces tiny fragments (page numbers, headers) that pollute the index | +| Re-parsing unchanged documents | Use Structured Streaming with checkpoints — see Step 1a above | +| Region not supported | US/EU regions only, or enable cross-geography routing | + +--- + +## Near-Real-Time Variant — DSPy + MLflow Agent + +When the pipeline must respond in seconds (triggered by a user action, API call, or form submission), use DSPy with an MLflow ChatAgent instead of a DLT pipeline. + +**When to use DSPy vs LangChain:** + +| Scenario | Stack | +|---|---| +| Fixed pipeline steps, well-defined I/O, want prompt optimization | **DSPy** | +| Needs tool-calling, memory, or multi-agent coordination | **LangChain LCEL** + MLflow ChatAgent | +| Single LLM call, simple task | Direct AI Function or `ai_query` in a notebook | + +### DSPy Signatures (replace LangChain agent system prompts) + +```python +# pip install dspy-ai mlflow databricks-sdk +import dspy, yaml + +CFG = yaml.safe_load(open("config.yml")) +lm = dspy.LM( + model=f"databricks/{CFG['models']['default']}", + api_base="https:///serving-endpoints", + api_key=dbutils.secrets.get("scope", "databricks-token"), +) +dspy.configure(lm=lm) + + +class ExtractInvoiceHeader(dspy.Signature): + """Extract invoice header fields from document text.""" + document_text: str = dspy.InputField(desc="Raw text from the document") + invoice_number: str = dspy.OutputField(desc="Invoice number, or null") + vendor_name: str = dspy.OutputField(desc="Vendor/supplier name, or null") + issue_date: str = dspy.OutputField(desc="Date as dd/mm/yyyy, or null") + total_amount: float = dspy.OutputField(desc="Total amount as float, or null") + + +class ClassifyDocument(dspy.Signature): + """Classify a document into one of the provided categories.""" + document_text: str = dspy.InputField() + category: str = dspy.OutputField( + desc="One of: invoice, purchase_order, receipt, contract, other" + ) + + +class DocumentPipeline(dspy.Module): + def __init__(self): + self.classify = dspy.Predict(ClassifyDocument) + self.extract = dspy.Predict(ExtractInvoiceHeader) + + def forward(self, document_text: str): + doc_type = self.classify(document_text=document_text).category + if doc_type == "invoice": + header = self.extract(document_text=document_text) + return {"doc_type": doc_type, "header": header.__dict__} + return {"doc_type": doc_type, "header": None} + + +pipeline = DocumentPipeline() +``` + +### Wrap and Register with MLflow + +```python +import mlflow, json + +class DSPyDocumentAgent(mlflow.pyfunc.PythonModel): + def load_context(self, context): + import dspy, yaml + cfg = yaml.safe_load(open(context.artifacts["config"])) + lm = dspy.LM(model=f"databricks/{cfg['models']['default']}") + dspy.configure(lm=lm) + self.pipeline = DocumentPipeline() + + def predict(self, context, model_input): + text = model_input.iloc[0]["document_text"] + return json.dumps(self.pipeline(document_text=text), ensure_ascii=False) + +mlflow.set_registry_uri("databricks-uc") +with mlflow.start_run(): + mlflow.pyfunc.log_model( + artifact_path="document_agent", + python_model=DSPyDocumentAgent(), + artifacts={"config": "config.yml"}, + registered_model_name="my_catalog.document_processing.document_agent", + ) +``` + +--- + +## Tips + +1. **Parse first, enrich second** — always run `ai_parse_document` as the first stage. Feed its text output to task-specific functions; never pass raw binary to `ai_query`. +2. **Flat or nested fields → `ai_extract`; deeply nested JSON exceeding 7 levels → `ai_query`** — pass `MAP('version', '2.0')` and access results through `:response`. +3. **`failOnError => false` is mandatory in batch** — write errors to a sidecar `_errors` table rather than crashing the pipeline. +4. **Truncate before sending to `ai_query`** — use `LEFT(text, 6000)` or chunk long documents to stay within context window limits. +5. **Prompts belong in `config.yml`** — never hardcode prompt strings in pipeline code. A prompt change should be a config change, not a code change. +6. **DSPy for agents** — when migrating from LangChain agent-based tools, DSPy typed `Signature` classes give you structured I/O contracts, testability, and optional prompt compilation/optimization. diff --git a/.claude/skills/databricks-ai-functions/SKILL.md b/.claude/skills/databricks-ai-functions/SKILL.md new file mode 100644 index 00000000..19897d8a --- /dev/null +++ b/.claude/skills/databricks-ai-functions/SKILL.md @@ -0,0 +1,195 @@ +--- +name: databricks-ai-functions +description: "Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → chunk → index → query)." +--- + +# Databricks AI Functions + +> **Official Docs:** https://docs.databricks.com/aws/en/large-language-models/ai-functions +> Individual function reference: https://docs.databricks.com/aws/en/sql/language-manual/functions/ + +## Overview + +Databricks AI Functions are built-in SQL and PySpark functions that call Foundation Model APIs directly from your data pipelines — no model endpoint setup, no API keys, no boilerplate. They operate on table columns as naturally as `UPPER()` or `LENGTH()`, and are optimized for batch inference at scale. + +There are three categories: + +| Category | Functions | Use when | +|---|---|---| +| **Task-specific** | `ai_analyze_sentiment`, `ai_classify`, `ai_extract`, `ai_fix_grammar`, `ai_gen`, `ai_mask`, `ai_similarity`, `ai_summarize`, `ai_translate`, `ai_parse_document` | The task is well-defined — prefer these always | +| **General-purpose** | `ai_query` | Complex nested JSON, custom endpoints, multimodal — **last resort only** | +| **Table-valued** | `ai_forecast` | Time series forecasting | + +**Function selection rule — always prefer a task-specific function over `ai_query`:** + +| Task | Use this | Fall back to `ai_query` when... | +|---|---|---| +| Sentiment scoring | `ai_analyze_sentiment` | Never | +| Fixed-label routing | `ai_classify` (2–500 labels; add descriptions for accuracy) | Never | +| Entity / field extraction | `ai_extract` | Never | +| Summarization | `ai_summarize` | Never — use `max_words=0` for uncapped | +| Grammar correction | `ai_fix_grammar` | Never | +| Translation | `ai_translate` | Target language not in the supported list | +| PII redaction | `ai_mask` | Never | +| Free-form generation | `ai_gen` | Need structured JSON output | +| Semantic similarity | `ai_similarity` | Never | +| PDF / document parsing | `ai_parse_document` | Need image-level reasoning | +| Complex JSON / reasoning | — | **This is the intended use case for `ai_query`** | + +## Prerequisites + +- Databricks SQL warehouse (**not Classic**) or cluster with DBR **15.1+** +- DBR **15.4 ML LTS** recommended for batch workloads +- DBR **17.1+** required for `ai_parse_document` +- `ai_forecast` requires a **Pro or Serverless** SQL warehouse +- Workspace in a supported AWS/Azure region for batch AI inference +- Models run under Apache 2.0 or LLAMA 3.3 Community License — customers are responsible for compliance + +## Quick Start + +Classify, extract, and score sentiment from a text column in a single query: + +```sql +SELECT + ticket_id, + ticket_text, + ai_classify(ticket_text, ARRAY('urgent', 'not urgent', 'spam')) AS priority, + ai_extract(ticket_text, ARRAY('product', 'error_code', 'date')) AS entities, + ai_analyze_sentiment(ticket_text) AS sentiment +FROM support_tickets; +``` + +```python +from pyspark.sql.functions import expr + +df = spark.table("support_tickets") +df = ( + df.withColumn("priority", expr("ai_classify(ticket_text, array('urgent', 'not urgent', 'spam'))")) + .withColumn("entities", expr("ai_extract(ticket_text, array('product', 'error_code', 'date'))")) + .withColumn("sentiment", expr("ai_analyze_sentiment(ticket_text)")) +) +# Access nested STRUCT fields from ai_extract +df.select("ticket_id", "priority", "sentiment", + "entities.product", "entities.error_code", "entities.date").display() +``` + +## Common Patterns + +### Pattern 1: Text Analysis Pipeline + +Chain multiple task-specific functions to enrich a text column in one pass: + +```sql +SELECT + id, + content, + ai_analyze_sentiment(content) AS sentiment, + ai_summarize(content, 30) AS summary, + ai_classify(content, + ARRAY('technical', 'billing', 'other')) AS category, + ai_fix_grammar(content) AS content_clean +FROM raw_feedback; +``` + +### Pattern 2: PII Redaction Before Storage + +```python +from pyspark.sql.functions import expr + +df_clean = ( + spark.table("raw_messages") + .withColumn( + "message_safe", + expr("ai_mask(message, array('person', 'email', 'phone', 'address'))") + ) +) +df_clean.write.format("delta").mode("append").saveAsTable("catalog.schema.messages_safe") +``` + +### Pattern 3: Document Ingestion from a Unity Catalog Volume + +Parse PDFs/Office docs, then enrich with task-specific functions: + +```python +from pyspark.sql.functions import expr + +df = ( + spark.read.format("binaryFile") + .load("/Volumes/catalog/schema/landing/documents/") + .withColumn("parsed", expr("ai_parse_document(content)")) + .selectExpr("path", + "parsed:pages[*].elements[*].content AS text_blocks", + "parsed:error AS parse_error") + .filter("parse_error IS NULL") + .withColumn("summary", expr("ai_summarize(text_blocks, 50)")) + .withColumn("entities", expr("ai_extract(text_blocks, array('date', 'amount', 'vendor'))")) +) +``` + +### Pattern 4: Semantic Matching / Deduplication + +```sql +-- Find near-duplicate company names +SELECT a.id, b.id, ai_similarity(a.name, b.name) AS score +FROM companies a +JOIN companies b ON a.id < b.id +WHERE ai_similarity(a.name, b.name) > 0.85; +``` + +### Pattern 5: Complex JSON Extraction with `ai_query` (last resort) + +Use only when the output schema has nested arrays or requires multi-step reasoning that no task-specific function handles: + +```python +from pyspark.sql.functions import expr, from_json, col + +df = ( + spark.table("parsed_documents") + .withColumn("ai_response", expr(""" + ai_query( + 'databricks-claude-sonnet-4', + concat('Extract invoice as JSON with nested itens array: ', text_blocks), + responseFormat => '{"type":"json_object"}', + failOnError => false + ) + """)) + .withColumn("invoice", from_json( + col("ai_response.response"), + "STRUCT>>" + )) +) +``` + +### Pattern 6: Time Series Forecasting + +```sql +SELECT * +FROM ai_forecast( + observed => TABLE(SELECT date, sales FROM daily_sales), + horizon => '2026-12-31', + time_col => 'date', + value_col => 'sales' +); +-- Returns: date, sales_forecast, sales_upper, sales_lower +``` + +## Reference Files + +- [1-task-functions.md](1-task-functions.md) — Full syntax, parameters, SQL + PySpark examples for all 9 task-specific functions (`ai_analyze_sentiment`, `ai_classify`, `ai_extract`, `ai_fix_grammar`, `ai_gen`, `ai_mask`, `ai_similarity`, `ai_summarize`, `ai_translate`) and `ai_parse_document` +- [2-ai-query.md](2-ai-query.md) — `ai_query` complete reference: all parameters, structured output with `responseFormat`, multimodal `files =>`, UDF patterns, and error handling +- [3-ai-forecast.md](3-ai-forecast.md) — `ai_forecast` parameters, single-metric, multi-group, multi-metric, and confidence interval patterns +- [4-document-processing-pipeline.md](4-document-processing-pipeline.md) — End-to-end batch document processing pipeline using AI Functions in a Lakeflow Declarative Pipeline; includes `config.yml` centralization, function selection logic, custom RAG pipeline (parse → chunk → Vector Search), and DSPy/LangChain guidance for near-real-time variants + +## Common Issues + +| Issue | Solution | +|---|---| +| `ai_parse_document` not found | Requires DBR **17.1+**. Check cluster runtime. | +| `ai_forecast` fails | Requires **Pro or Serverless** SQL warehouse — not available on Classic or Starter. | +| All functions return NULL | Input column is NULL. Filter with `WHERE col IS NOT NULL` before calling. | +| `ai_translate` fails for a language | Supported: English, German, French, Italian, Portuguese, Hindi, Spanish, Thai. Use `ai_query` with a multilingual model for others. | +| `ai_classify` returns unexpected labels | Use clear, mutually exclusive label names. Fewer labels (2–5) produces more reliable results. | +| `ai_query` raises on some rows in a batch job | Add `failOnError => false` — returns a STRUCT with `.response` and `.error` instead of raising. | +| Batch job runs slowly | Use DBR **15.4 ML LTS** cluster (not serverless or interactive) for optimized batch inference throughput. | +| Want to swap models without editing pipeline code | Store all model names and prompts in `config.yml` — see [4-document-processing-pipeline.md](4-document-processing-pipeline.md) for the pattern. | diff --git a/.claude/skills/databricks-aibi-dashboards/1-widget-specifications.md b/.claude/skills/databricks-aibi-dashboards/1-widget-specifications.md new file mode 100644 index 00000000..8f23b166 --- /dev/null +++ b/.claude/skills/databricks-aibi-dashboards/1-widget-specifications.md @@ -0,0 +1,341 @@ +# Widget Specifications + +Core widget types for AI/BI dashboards. For advanced visualizations (area, scatter, choropleth map, combo), see [2-advanced-widget-specifications.md](2-advanced-widget-specifications.md). + +## Widget Naming and Display + +- `widget.name`: alphanumeric + hyphens + underscores ONLY (max 60 characters) +- `frame.title`: human-readable title (any characters allowed) +- `frame.showTitle`: always set to `true` so users understand the widget +- `displayName`: use in encodings to label axes/values clearly (e.g., "Revenue ($)", "Growth Rate (%)") +- `widget.queries[].name`: use `"main_query"` for chart/counter/table widgets. Filter widgets with multiple queries can use descriptive names (see [3-filters.md](3-filters.md)) + +**Always format values appropriately** - use `format` for currency, percentages, and large numbers (see [Axis Formatting](#axis-formatting)). + +## Version Requirements + +| Widget Type | Version | File | +|-------------|---------|------| +| text | N/A | this file | +| counter | 2 | this file | +| table | 2 | this file | +| bar | 3 | this file | +| line | 3 | this file | +| pie | 3 | this file | +| area | 3 | [2-advanced-widget-specifications.md](2-advanced-widget-specifications.md) | +| scatter | 3 | [2-advanced-widget-specifications.md](2-advanced-widget-specifications.md) | +| combo | 1 | [2-advanced-widget-specifications.md](2-advanced-widget-specifications.md) | +| choropleth-map | 1 | [2-advanced-widget-specifications.md](2-advanced-widget-specifications.md) | +| filter-* | 2 | [3-filters.md](3-filters.md) | + +--- + +## Text (Headers/Descriptions) + +- **CRITICAL: Text widgets do NOT use a spec block** - use `multilineTextboxSpec` directly +- Supports markdown: `#`, `##`, `###`, `**bold**`, `*italic*` +- **CRITICAL: Multiple items in the `lines` array are concatenated on a single line, NOT displayed as separate lines!** +- For title + subtitle, use **separate text widgets** at different y positions + +```json +// CORRECT: Separate widgets for title and subtitle +{ + "widget": { + "name": "title", + "multilineTextboxSpec": {"lines": ["## Dashboard Title"]} + }, + "position": {"x": 0, "y": 0, "width": 12, "height": 1} +}, +{ + "widget": { + "name": "subtitle", + "multilineTextboxSpec": {"lines": ["Description text here"]} + }, + "position": {"x": 0, "y": 1, "width": 12, "height": 1} +} + +// WRONG: Multiple lines concatenate into one line! +{ + "widget": { + "name": "title-widget", + "multilineTextboxSpec": { + "lines": ["## Dashboard Title", "Description text here"] // Becomes "## Dashboard TitleDescription text here" + } + }, + "position": {"x": 0, "y": 0, "width": 12, "height": 2} +} +``` + +--- + +## Counter (KPI) + +- `version`: **2** (NOT 3!) +- `widgetType`: "counter" +- Percent values must be 0-1 in the data (not 0-100) + +### Number Formatting + +```json +"encodings": { + "value": { + "fieldName": "revenue", + "displayName": "Total Revenue", + "format": { + "type": "number-currency", + "currencyCode": "USD", + "abbreviation": "compact", + "decimalPlaces": {"type": "max", "places": 2} + } + } +} +``` + +Format types: `number`, `number-currency`, `number-percent` + +### Counter Patterns + +**Pre-aggregated dataset (1 row)** - use `disaggregated: true`: +```json +{ + "widget": { + "name": "total-revenue", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "summary_ds", + "fields": [{"name": "revenue", "expression": "`revenue`"}], + "disaggregated": true + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": {"fieldName": "revenue", "displayName": "Total Revenue"} + }, + "frame": {"showTitle": true, "title": "Total Revenue"} + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 3} +} +``` + +**Multi-row dataset with aggregation (supports filters)** - use `disaggregated: false`: +- Dataset returns multiple rows (e.g., grouped by a filter dimension) +- Use `"disaggregated": false` and aggregation expression +- **CRITICAL**: Field `name` MUST match `fieldName` exactly (e.g., `"sum(spend)"`) + +```json +{ + "widget": { + "name": "total-spend", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "by_category", + "fields": [{"name": "sum(spend)", "expression": "SUM(`spend`)"}], + "disaggregated": false + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": {"fieldName": "sum(spend)", "displayName": "Total Spend"} + }, + "frame": {"showTitle": true, "title": "Total Spend"} + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 3} +} +``` + +--- + +## Table + +- `version`: **2** (NOT 1 or 3!) +- `widgetType`: "table" +- **Columns only need `fieldName` and `displayName`** - no other properties required +- Use `"disaggregated": true` for raw rows +- Default sort: use `ORDER BY` in dataset SQL + +```json +{ + "widget": { + "name": "details-table", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "details_ds", + "fields": [ + {"name": "name", "expression": "`name`"}, + {"name": "value", "expression": "`value`"} + ], + "disaggregated": true + } + }], + "spec": { + "version": 2, + "widgetType": "table", + "encodings": { + "columns": [ + {"fieldName": "name", "displayName": "Name"}, + {"fieldName": "value", "displayName": "Value"} + ] + }, + "frame": {"showTitle": true, "title": "Details"} + } + }, + "position": {"x": 0, "y": 0, "width": 12, "height": 6} +} +``` + +--- + +## Line / Bar Charts + +- `version`: **3** +- `widgetType`: "line" or "bar" +- Use `x`, `y`, optional `color` encodings +- `scale.type`: `"temporal"` (dates), `"quantitative"` (numbers), `"categorical"` (strings) +- Use `"disaggregated": true` with pre-aggregated dataset data + +**Multiple series - two approaches:** + +1. **Multi-Y Fields** (different metrics): +```json +"y": { + "scale": {"type": "quantitative"}, + "fields": [ + {"fieldName": "sum(orders)", "displayName": "Orders"}, + {"fieldName": "sum(returns)", "displayName": "Returns"} + ] +} +``` + +2. **Color Grouping** (same metric split by dimension): +```json +"y": {"fieldName": "sum(revenue)", "scale": {"type": "quantitative"}}, +"color": {"fieldName": "region", "scale": {"type": "categorical"}} +``` + +### Bar Chart Modes + +| Mode | Configuration | +|------|---------------| +| Stacked (default) | No `mark` field | +| Grouped | `"mark": {"layout": "group"}` | + +### Horizontal Bar Chart + +Swap `x` and `y` - put quantitative on `x`, categorical/temporal on `y`: +```json +"encodings": { + "x": {"scale": {"type": "quantitative"}, "fields": [...]}, + "y": {"fieldName": "category", "scale": {"type": "categorical"}} +} +``` + +### Color Scale + +> **CRITICAL**: For bar/line/pie, color scale ONLY supports `type` and `sort`. +> Do NOT use `scheme`, `colorRamp`, or `mappings` (only for choropleth-map). + +--- + +## Pie Chart + +- `version`: **3** +- `widgetType`: "pie" +- `angle`: quantitative field +- `color`: categorical dimension +- **Limit to 3-8 categories for readability** + +```json +"spec": { + "version": 3, + "widgetType": "pie", + "encodings": { + "angle": {"fieldName": "revenue", "scale": {"type": "quantitative"}}, + "color": {"fieldName": "category", "scale": {"type": "categorical"}} + } +} +``` + +--- + +## Axis Formatting + +Add `format` to any encoding to display values appropriately: + +| Data Type | Format Type | Example | +|-----------|-------------|---------| +| Currency | `number-currency` | $1.2M | +| Percentage | `number-percent` | 45.2% (data must be 0-1, not 0-100) | +| Large numbers | `number` with `abbreviation` | 1.5K, 2.3M | + +```json +"value": { + "fieldName": "revenue", + "displayName": "Revenue", + "format": { + "type": "number-currency", + "currencyCode": "USD", + "abbreviation": "compact", + "decimalPlaces": {"type": "max", "places": 2} + } +} +``` + +**Options:** +- `abbreviation`: `"compact"` (K/M/B) or omit for full numbers +- `decimalPlaces`: `{"type": "max", "places": N}` or `{"type": "fixed", "places": N}` + +--- + +## Dataset Parameters + +Use `:param` syntax in SQL for dynamic filtering: + +```json +{ + "name": "revenue_by_category", + "queryLines": ["SELECT ... WHERE returns_usd > :threshold GROUP BY category"], + "parameters": [{ + "keyword": "threshold", + "dataType": "INTEGER", + "defaultSelection": {} + }] +} +``` + +**Parameter types:** +- Single value: `"dataType": "INTEGER"` / `"DECIMAL"` / `"STRING"` +- Multi-select: Add `"complexType": "MULTI"` +- Range: `"dataType": "DATE", "complexType": "RANGE"` - use `:param.min` / `:param.max` + +--- + +## Widget Field Expressions + +Allowed in `query.fields` (no CAST or complex SQL): + +```json +// Aggregations +{"name": "sum(revenue)", "expression": "SUM(`revenue`)"} +{"name": "avg(price)", "expression": "AVG(`price`)"} +{"name": "count(id)", "expression": "COUNT(`id`)"} +{"name": "countdistinct(id)", "expression": "COUNT(DISTINCT `id`)"} + +// Date truncation +{"name": "daily(date)", "expression": "DATE_TRUNC(\"DAY\", `date`)"} +{"name": "weekly(date)", "expression": "DATE_TRUNC(\"WEEK\", `date`)"} +{"name": "monthly(date)", "expression": "DATE_TRUNC(\"MONTH\", `date`)"} + +// Simple reference +{"name": "category", "expression": "`category`"} +``` + +For conditional logic, compute in dataset SQL instead. diff --git a/.claude/skills/databricks-aibi-dashboards/2-advanced-widget-specifications.md b/.claude/skills/databricks-aibi-dashboards/2-advanced-widget-specifications.md new file mode 100644 index 00000000..3f658609 --- /dev/null +++ b/.claude/skills/databricks-aibi-dashboards/2-advanced-widget-specifications.md @@ -0,0 +1,177 @@ +# Advanced Widget Specifications + +Advanced visualization types for AI/BI dashboards. For core widgets (text, counter, table, bar, line, pie), see [1-widget-specifications.md](1-widget-specifications.md). + +--- + +## Area Chart + +- `version`: **3** +- `widgetType`: "area" +- Same structure as line chart - useful for showing cumulative values or emphasizing volume + +```json +"spec": { + "version": 3, + "widgetType": "area", + "encodings": { + "x": {"fieldName": "week_start", "scale": {"type": "temporal"}}, + "y": { + "scale": {"type": "quantitative"}, + "fields": [ + {"fieldName": "revenue_usd", "displayName": "Revenue"}, + {"fieldName": "returns_usd", "displayName": "Returns"} + ] + } + } +} +``` + +--- + +## Scatter Plot / Bubble Chart + +- `version`: **3** +- `widgetType`: "scatter" +- `x`, `y`: quantitative or temporal +- `size`: optional quantitative field for bubble size +- `color`: optional categorical or quantitative for grouping + +```json +"spec": { + "version": 3, + "widgetType": "scatter", + "encodings": { + "x": {"fieldName": "return_date", "scale": {"type": "temporal"}}, + "y": {"fieldName": "daily_returns", "scale": {"type": "quantitative"}}, + "size": {"fieldName": "count(*)", "scale": {"type": "quantitative"}}, + "color": {"fieldName": "category", "scale": {"type": "categorical"}} + } +} +``` + +--- + +## Combo Chart (Bar + Line) + +Combines bar and line visualizations on the same chart - useful for showing related metrics with different scales. + +- `version`: **1** +- `widgetType`: "combo" +- `y.primary`: bar chart fields +- `y.secondary`: line chart fields + +```json +{ + "widget": { + "name": "revenue-and-growth", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "metrics_ds", + "fields": [ + {"name": "daily(date)", "expression": "DATE_TRUNC(\"DAY\", `date`)"}, + {"name": "sum(revenue)", "expression": "SUM(`revenue`)"}, + {"name": "avg(growth_rate)", "expression": "AVG(`growth_rate`)"} + ], + "disaggregated": false + } + }], + "spec": { + "version": 1, + "widgetType": "combo", + "encodings": { + "x": {"fieldName": "daily(date)", "scale": {"type": "temporal"}}, + "y": { + "scale": {"type": "quantitative"}, + "primary": { + "fields": [{"fieldName": "sum(revenue)", "displayName": "Revenue ($)"}] + }, + "secondary": { + "fields": [{"fieldName": "avg(growth_rate)", "displayName": "Growth Rate"}] + } + }, + "label": {"show": false} + }, + "frame": {"title": "Revenue & Growth Rate", "showTitle": true} + } + }, + "position": {"x": 0, "y": 0, "width": 12, "height": 5} +} +``` + +--- + +## Choropleth Map + +Displays geographic regions colored by aggregate values. Requires a field with geographic names (state names, country names, etc.). + +- `version`: **1** +- `widgetType`: "choropleth-map" +- `region`: defines the geographic area mapping +- `color`: quantitative field for coloring regions + +```json +"spec": { + "version": 1, + "widgetType": "choropleth-map", + "encodings": { + "region": { + "regionType": "mapbox-v4-admin", + "admin0": { + "type": "value", + "value": "United States", + "geographicRole": "admin0-name" + }, + "admin1": { + "fieldName": "state_name", + "type": "field", + "geographicRole": "admin1-name" + } + }, + "color": { + "fieldName": "sum(revenue)", + "scale": {"type": "quantitative"} + } + } +} +``` + +### Region Configuration + +**Region levels:** +- `admin0`: Country level - use `"type": "value"` with fixed country name +- `admin1`: State/Province level - use `"type": "field"` with your data column +- `admin2`: County/District level + +**Geographic roles:** +- `admin0-name`, `admin1-name`, `admin2-name` - match by name +- `admin0-iso`, `admin1-iso` - match by ISO code + +**Supported countries for admin1:** United States, Japan (prefectures), and others. + +### Color Scale for Maps + +> **Note**: Unlike other charts, choropleth-map supports additional color scale properties: +> - `scheme`: color scheme name (e.g., "YIGnBu") +> - `colorRamp`: custom color gradient +> - `mappings`: explicit value-to-color mappings + +--- + +## Other Visualization Types + +The following visualization types are available in Databricks AI/BI dashboards but are less commonly used. Refer to [Databricks documentation](https://docs.databricks.com/aws/en/visualizations/visualization-types) for details: + +| Widget Type | Description | +|-------------|-------------| +| heatmap | Color intensity grid for numerical data | +| histogram | Frequency distribution with configurable bins | +| funnel | Stage-based metric analysis | +| sankey | Flow visualization between value sets | +| box | Distribution summary with quartiles | +| marker-map | Latitude/longitude point markers | +| pivot | Drag-and-drop aggregation table | +| word-cloud | Word frequency visualization | +| sunburst | Hierarchical data in concentric circles | +| cohort | Group outcome analysis over time | diff --git a/.claude/skills/databricks-aibi-dashboards/3-examples.md b/.claude/skills/databricks-aibi-dashboards/3-examples.md new file mode 100644 index 00000000..cb8791dd --- /dev/null +++ b/.claude/skills/databricks-aibi-dashboards/3-examples.md @@ -0,0 +1,308 @@ +# Complete Dashboard Examples + +Production-ready templates you can adapt for your use case. + +## Basic Dashboard (NYC Taxi) + +```python +import json + +# Step 1: Check table schema +table_info = get_table_stats_and_schema(catalog="samples", schema="nyctaxi") + +# Step 2: Test queries +execute_sql("SELECT COUNT(*) as trips, AVG(fare_amount) as avg_fare, AVG(trip_distance) as avg_distance FROM samples.nyctaxi.trips") +execute_sql(""" + SELECT pickup_zip, COUNT(*) as trip_count + FROM samples.nyctaxi.trips + GROUP BY pickup_zip + ORDER BY trip_count DESC + LIMIT 10 +""") + +# Step 3: Build dashboard JSON +dashboard = { + "datasets": [ + { + "name": "summary", + "displayName": "Summary Stats", + "queryLines": [ + "SELECT COUNT(*) as trips, AVG(fare_amount) as avg_fare, ", + "AVG(trip_distance) as avg_distance ", + "FROM samples.nyctaxi.trips " + ] + }, + { + "name": "by_zip", + "displayName": "Trips by ZIP", + "queryLines": [ + "SELECT pickup_zip, COUNT(*) as trip_count ", + "FROM samples.nyctaxi.trips ", + "GROUP BY pickup_zip ", + "ORDER BY trip_count DESC ", + "LIMIT 10 " + ] + } + ], + "pages": [{ + "name": "overview", + "displayName": "NYC Taxi Overview", + "pageType": "PAGE_TYPE_CANVAS", + "layoutVersion": "GRID_V1", + "layout": [ + # Text header - NO spec block! Use SEPARATE widgets for title and subtitle! + { + "widget": { + "name": "title", + "multilineTextboxSpec": { + "lines": ["## NYC Taxi Dashboard"] + } + }, + "position": {"x": 0, "y": 0, "width": 12, "height": 1} + }, + { + "widget": { + "name": "subtitle", + "multilineTextboxSpec": { + "lines": ["Trip statistics and analysis"] + } + }, + "position": {"x": 0, "y": 1, "width": 12, "height": 1} + }, + # Counter - version 2, width 4! + { + "widget": { + "name": "total-trips", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "summary", + "fields": [{"name": "trips", "expression": "`trips`"}], + "disaggregated": True + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": {"fieldName": "trips", "displayName": "Total Trips"} + }, + "frame": {"title": "Total Trips", "showTitle": True} + } + }, + "position": {"x": 0, "y": 2, "width": 4, "height": 3} + }, + { + "widget": { + "name": "avg-fare", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "summary", + "fields": [{"name": "avg_fare", "expression": "`avg_fare`"}], + "disaggregated": True + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": {"fieldName": "avg_fare", "displayName": "Avg Fare"} + }, + "frame": {"title": "Average Fare", "showTitle": True} + } + }, + "position": {"x": 4, "y": 2, "width": 4, "height": 3} + }, + { + "widget": { + "name": "total-distance", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "summary", + "fields": [{"name": "avg_distance", "expression": "`avg_distance`"}], + "disaggregated": True + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": {"fieldName": "avg_distance", "displayName": "Avg Distance"} + }, + "frame": {"title": "Average Distance", "showTitle": True} + } + }, + "position": {"x": 8, "y": 2, "width": 4, "height": 3} + }, + # Bar chart - version 3 + { + "widget": { + "name": "trips-by-zip", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "by_zip", + "fields": [ + {"name": "pickup_zip", "expression": "`pickup_zip`"}, + {"name": "trip_count", "expression": "`trip_count`"} + ], + "disaggregated": True + } + }], + "spec": { + "version": 3, + "widgetType": "bar", + "encodings": { + "x": {"fieldName": "pickup_zip", "scale": {"type": "categorical"}, "displayName": "ZIP"}, + "y": {"fieldName": "trip_count", "scale": {"type": "quantitative"}, "displayName": "Trips"} + }, + "frame": {"title": "Trips by Pickup ZIP", "showTitle": True} + } + }, + "position": {"x": 0, "y": 5, "width": 12, "height": 5} + }, + # Table - version 2, minimal column props! + { + "widget": { + "name": "zip-table", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "by_zip", + "fields": [ + {"name": "pickup_zip", "expression": "`pickup_zip`"}, + {"name": "trip_count", "expression": "`trip_count`"} + ], + "disaggregated": True + } + }], + "spec": { + "version": 2, + "widgetType": "table", + "encodings": { + "columns": [ + {"fieldName": "pickup_zip", "displayName": "ZIP Code"}, + {"fieldName": "trip_count", "displayName": "Trip Count"} + ] + }, + "frame": {"title": "Top ZIP Codes", "showTitle": True} + } + }, + "position": {"x": 0, "y": 10, "width": 12, "height": 5} + } + ] + }] +} + +# Step 4: Deploy +result = manage_dashboard( + action="create_or_update", + display_name="NYC Taxi Dashboard", + parent_path="/Workspace/Users/me/dashboards", + serialized_dashboard=json.dumps(dashboard), + warehouse_id=manage_warehouse(action="get_best"), +) +print(result["url"]) +``` + +## Dashboard with Global Filters + +```python +import json + +# Dashboard with a global filter for region +dashboard_with_filters = { + "datasets": [ + { + "name": "sales", + "displayName": "Sales Data", + "queryLines": [ + "SELECT region, SUM(revenue) as total_revenue ", + "FROM catalog.schema.sales ", + "GROUP BY region" + ] + } + ], + "pages": [ + { + "name": "overview", + "displayName": "Sales Overview", + "pageType": "PAGE_TYPE_CANVAS", + "layoutVersion": "GRID_V1", + "layout": [ + { + "widget": { + "name": "total-revenue", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "sales", + "fields": [{"name": "total_revenue", "expression": "`total_revenue`"}], + "disaggregated": True + } + }], + "spec": { + "version": 2, # Version 2 for counters! + "widgetType": "counter", + "encodings": { + "value": {"fieldName": "total_revenue", "displayName": "Total Revenue"} + }, + "frame": {"title": "Total Revenue", "showTitle": True} + } + }, + "position": {"x": 0, "y": 0, "width": 12, "height": 3} + } + ] + }, + { + "name": "filters", + "displayName": "Filters", + "pageType": "PAGE_TYPE_GLOBAL_FILTERS", # Required for global filter page! + "layoutVersion": "GRID_V1", + "layout": [ + { + "widget": { + "name": "filter_region", + "queries": [{ + "name": "ds_sales_region", + "query": { + "datasetName": "sales", + "fields": [ + {"name": "region", "expression": "`region`"} + # DO NOT use associative_filter_predicate_group - causes SQL errors! + ], + "disaggregated": False # False for filters! + } + }], + "spec": { + "version": 2, # Version 2 for filters! + "widgetType": "filter-multi-select", # NOT "filter"! + "encodings": { + "fields": [{ + "fieldName": "region", + "displayName": "Region", + "queryName": "ds_sales_region" # Must match query name! + }] + }, + "frame": {"showTitle": True, "title": "Region"} # Always show title! + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 2} + } + ] + } + ] +} + +# Deploy with filters +result = manage_dashboard( + action="create_or_update", + display_name="Sales Dashboard with Filters", + parent_path="/Workspace/Users/me/dashboards", + serialized_dashboard=json.dumps(dashboard_with_filters), + warehouse_id=manage_warehouse(action="get_best"), +) +print(result["url"]) +``` diff --git a/.claude/skills/databricks-aibi-dashboards/3-filters.md b/.claude/skills/databricks-aibi-dashboards/3-filters.md new file mode 100644 index 00000000..5a4ab497 --- /dev/null +++ b/.claude/skills/databricks-aibi-dashboards/3-filters.md @@ -0,0 +1,242 @@ +# Filters (Global vs Page-Level) + +> **CRITICAL**: Filter widgets use DIFFERENT widget types than charts! +> - Valid types: `filter-multi-select`, `filter-single-select`, `filter-date-range-picker` +> - **DO NOT** use `widgetType: "filter"` - this does not exist and will cause errors +> - Filters use `spec.version: 2` +> - **ALWAYS include `frame` with `showTitle: true`** for filter widgets + +**Filter widget types:** +- `filter-date-range-picker`: for DATE/TIMESTAMP fields (date range selection) +- `filter-single-select`: categorical with single selection +- `filter-multi-select`: categorical with multiple selections (preferred for drill-down) + +> **Performance note**: Global filters automatically apply `WHERE` clauses to dataset queries at runtime. You don't need to pre-filter data in your SQL - the dashboard engine handles this efficiently. + +--- + +## Global Filters vs Page-Level Filters + +| Type | Placement | Scope | Use Case | +|------|-----------|-------|----------| +| **Global Filter** | Dedicated page with `"pageType": "PAGE_TYPE_GLOBAL_FILTERS"` | Affects ALL pages that have datasets with the filter field | Cross-dashboard filtering (e.g., date range, campaign) | +| **Page-Level Filter** | Regular page with `"pageType": "PAGE_TYPE_CANVAS"` | Affects ONLY widgets on that same page | Page-specific filtering (e.g., platform filter on breakdown page only) | + +**Key Insight**: A filter only affects datasets that contain the filter field. To have a filter affect only specific pages: +1. Include the filter dimension in datasets for pages that should be filtered +2. Exclude the filter dimension from datasets for pages that should NOT be filtered + +--- + +## Filter Widget Structure + +> **CRITICAL**: Do NOT use `associative_filter_predicate_group` - it causes SQL errors! +> Use a simple field expression instead. + +```json +{ + "widget": { + "name": "filter_region", + "queries": [{ + "name": "ds_data_region", // Query name - must match queryName in encodings! + "query": { + "datasetName": "ds_data", + "fields": [ + {"name": "region", "expression": "`region`"} + ], + "disaggregated": false // CRITICAL: Always false for filters! + } + }], + "spec": { + "version": 2, + "widgetType": "filter-multi-select", + "encodings": { + "fields": [{ + "fieldName": "region", + "displayName": "Region", + "queryName": "ds_data_region" // Must match queries[].name above! + }] + }, + "frame": {"showTitle": true, "title": "Region"} + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 2} +} +``` + +--- + +## Global Filter Example + +Place on a dedicated filter page: + +```json +{ + "name": "filters", + "displayName": "Filters", + "pageType": "PAGE_TYPE_GLOBAL_FILTERS", + "layoutVersion": "GRID_V1", + "layout": [ + { + "widget": { + "name": "filter_campaign", + "queries": [{ + "name": "ds_campaign", + "query": { + "datasetName": "overview", + "fields": [{"name": "campaign_name", "expression": "`campaign_name`"}], + "disaggregated": false + } + }], + "spec": { + "version": 2, + "widgetType": "filter-multi-select", + "encodings": { + "fields": [{ + "fieldName": "campaign_name", + "displayName": "Campaign", + "queryName": "ds_campaign" + }] + }, + "frame": {"showTitle": true, "title": "Campaign"} + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 2} + } + ] +} +``` + +--- + +## Page-Level Filter Example + +Place filter widget directly on a `PAGE_TYPE_CANVAS` page (same widget structure as global filter, but only affects that page): + +```json +{ + "name": "platform_breakdown", + "displayName": "Platform Breakdown", + "pageType": "PAGE_TYPE_CANVAS", + "layoutVersion": "GRID_V1", + "layout": [ + {"widget": {...}, "position": {...}}, + { + "widget": { + "name": "filter_platform", + "queries": [{"name": "ds_platform", "query": {"datasetName": "platform_data", "fields": [{"name": "platform", "expression": "`platform`"}], "disaggregated": false}}], + "spec": { + "version": 2, + "widgetType": "filter-multi-select", + "encodings": {"fields": [{"fieldName": "platform", "displayName": "Platform", "queryName": "ds_platform"}]}, + "frame": {"showTitle": true, "title": "Platform"} + } + }, + "position": {"x": 8, "y": 0, "width": 4, "height": 2} + } + ] +} +``` + +--- + +## Date Range Filtering + +> **Best Practice**: Most dashboards should include a date range filter. However, metrics that are not based on a time range (like "MRR" or "All-Time Total") should NOT be date-filtered - omit them from the filter's queries. + +**Two binding approaches** (can be combined in one filter): +- **Field-based**: Bind to a date column in SELECT → filter auto-applies `IN_RANGE()` +- **Parameter-based**: Use `:param.min`/`:param.max` in WHERE clause for pre-aggregation filtering + +```json +// Dataset with parameter (for aggregated queries) +{ + "name": "revenue_by_category", + "queryLines": [ + "SELECT category, SUM(revenue) as revenue FROM catalog.schema.orders ", + "WHERE order_date BETWEEN :date_range.min AND :date_range.max ", + "GROUP BY category" + ], + "parameters": [{ + "keyword": "date_range", "dataType": "DATE", "complexType": "RANGE", + "defaultSelection": {"range": {"dataType": "DATE", "min": {"value": "now-12M/M"}, "max": {"value": "now/M"}}} + }] +} + +// Filter widget binding to both field and parameter +{ + "widget": { + "name": "date_range_filter", + "queries": [ + {"name": "q_trend", "query": {"datasetName": "weekly_trend", "fields": [{"name": "week_start", "expression": "`week_start`"}], "disaggregated": false}}, + {"name": "q_category", "query": {"datasetName": "revenue_by_category", "parameters": [{"name": "date_range", "keyword": "date_range"}], "disaggregated": false}} + ], + "spec": { + "version": 2, + "widgetType": "filter-date-range-picker", + "encodings": { + "fields": [ + {"fieldName": "week_start", "queryName": "q_trend"}, + {"parameterName": "date_range", "queryName": "q_category"} + ] + }, + "frame": {"showTitle": true, "title": "Date Range"} + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 2} +} +``` + +--- + +## Multi-Dataset Filters + +When a filter should affect multiple datasets (e.g., "Region" filter for both sales and customers data), add multiple queries - one per dataset: + +```json +{ + "widget": { + "name": "filter_region", + "queries": [ + { + "name": "sales_region", + "query": { + "datasetName": "sales", + "fields": [{"name": "region", "expression": "`region`"}], + "disaggregated": false + } + }, + { + "name": "customers_region", + "query": { + "datasetName": "customers", + "fields": [{"name": "region", "expression": "`region`"}], + "disaggregated": false + } + } + ], + "spec": { + "version": 2, + "widgetType": "filter-multi-select", + "encodings": { + "fields": [ + {"fieldName": "region", "displayName": "Region (Sales)", "queryName": "sales_region"}, + {"fieldName": "region", "displayName": "Region (Customers)", "queryName": "customers_region"} + ] + }, + "frame": {"showTitle": true, "title": "Region"} + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 2} +} +``` + +Each `queryName` in `encodings.fields` binds the filter to that specific dataset. Datasets not bound will not be filtered. + +--- + +## Filter Layout Guidelines + +- Global filters: Position on dedicated filter page, stack vertically at `x=0` +- Page-level filters: Position in header area of page (e.g., top-right corner) +- Typical sizing: `width: 4, height: 2` diff --git a/.claude/skills/databricks-aibi-dashboards/4-examples.md b/.claude/skills/databricks-aibi-dashboards/4-examples.md new file mode 100644 index 00000000..8fa49c5b --- /dev/null +++ b/.claude/skills/databricks-aibi-dashboards/4-examples.md @@ -0,0 +1,498 @@ +# Complete Dashboard Example + +This is a **reference example** to understand the JSON structure and layout patterns. **Always adapt to what the user requests** - use their tables, metrics, and visualizations. This example demonstrates the correct syntax; your dashboard should reflect the user's actual requirements. + +## Key Patterns (Read First) + +### 1. Page Types (Required) +- `PAGE_TYPE_CANVAS` - Main content page with widgets +- `PAGE_TYPE_GLOBAL_FILTERS` - Dedicated filter page that affects all canvas pages + +### 2. Widget Versions (Critical!) +| Widget Type | Version | +|-------------|---------| +| `counter`, `table` | **2** | +| `bar`, `line`, `area`, `pie` | **3** | +| `filter-*` | **2** | + +### 3. KPI Counter with Currency Formatting +```json +"format": { + "type": "number-currency", + "currencyCode": "USD", + "abbreviation": "compact", + "decimalPlaces": {"type": "max", "places": 1} +} +``` + +### 4. Filter Binding to Multiple Datasets +Each filter query binds the filter to one dataset. Add multiple queries to filter multiple datasets: +```json +"queries": [ + {"name": "ds1_region", "query": {"datasetName": "dataset1", ...}}, + {"name": "ds2_region", "query": {"datasetName": "dataset2", ...}} +] +``` + +### 5. Layout Grid (12 columns) +``` +y=0: Header with title + description (w=12, h=2) +y=2: KPI(w=4,h=3) | KPI(w=4,h=3) | KPI(w=4,h=3) ← fills 12 +y=5: Section header (w=12, h=1) +y=6: Area chart (w=12, h=5) +y=11: Section header (w=12, h=1) +y=12: Pie(w=4,h=5) | Bar chart(w=8,h=5) ← fills 12 +``` + +Use `\n\n` in text widget lines array to create line breaks within a single widget. + +--- + +## Full Dashboard: Sales Analytics + +This example shows a complete dashboard with: +- Title and subtitle text widgets +- 3 KPI counters with currency/number formatting +- Area chart for time series trends +- Pie chart for category breakdown +- Bar chart with color grouping by region +- Data table for detailed records +- Global filters (date range, region, category) + +```json +{ + "datasets": [ + { + "name": "ds_daily_sales", + "displayName": "Daily Sales", + "queryLines": [ + "SELECT sale_date, region, department, total_orders, total_units, total_revenue, total_cost, profit_margin ", + "FROM catalog.schema.gold_daily_sales ", + "ORDER BY sale_date" + ] + }, + { + "name": "ds_products", + "displayName": "Product Performance", + "queryLines": [ + "SELECT product_id, product_name, department, region, units_sold, revenue, cost, profit ", + "FROM catalog.schema.gold_product_performance" + ] + } + ], + "pages": [ + { + "name": "sales_overview", + "displayName": "Sales Overview", + "pageType": "PAGE_TYPE_CANVAS", + "layoutVersion": "GRID_V1", + "layout": [ + { + "widget": { + "name": "header", + "multilineTextboxSpec": { + "lines": ["# Sales Dashboard\n\nMonitor daily sales, revenue, and profit margins across regions and departments."] + } + }, + "position": {"x": 0, "y": 0, "width": 12, "height": 2} + }, + { + "widget": { + "name": "kpi_revenue", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "ds_daily_sales", + "fields": [{"name": "sum(total_revenue)", "expression": "SUM(`total_revenue`)"}], + "disaggregated": false + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": { + "fieldName": "sum(total_revenue)", + "displayName": "Total Revenue", + "format": { + "type": "number-currency", + "currencyCode": "USD", + "abbreviation": "compact", + "decimalPlaces": {"type": "max", "places": 1} + } + } + }, + "frame": {"title": "Total Revenue", "showTitle": true, "description": "For the selected period", "showDescription": true} + } + }, + "position": {"x": 0, "y": 2, "width": 4, "height": 3} + }, + { + "widget": { + "name": "kpi_orders", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "ds_daily_sales", + "fields": [{"name": "sum(total_orders)", "expression": "SUM(`total_orders`)"}], + "disaggregated": false + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": { + "fieldName": "sum(total_orders)", + "displayName": "Total Orders", + "format": { + "type": "number", + "abbreviation": "compact", + "decimalPlaces": {"type": "max", "places": 0} + } + } + }, + "frame": {"title": "Total Orders", "showTitle": true, "description": "For the selected period", "showDescription": true} + } + }, + "position": {"x": 4, "y": 2, "width": 4, "height": 3} + }, + { + "widget": { + "name": "kpi_profit", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "ds_daily_sales", + "fields": [{"name": "avg(profit_margin)", "expression": "AVG(`profit_margin`)"}], + "disaggregated": false + } + }], + "spec": { + "version": 2, + "widgetType": "counter", + "encodings": { + "value": { + "fieldName": "avg(profit_margin)", + "displayName": "Avg Profit Margin", + "format": { + "type": "number-percent", + "decimalPlaces": {"type": "max", "places": 1} + } + } + }, + "frame": {"title": "Profit Margin", "showTitle": true, "description": "Average for period", "showDescription": true} + } + }, + "position": {"x": 8, "y": 2, "width": 4, "height": 3} + }, + { + "widget": { + "name": "section_trends", + "multilineTextboxSpec": { + "lines": ["## Revenue Trend"] + } + }, + "position": {"x": 0, "y": 5, "width": 12, "height": 1} + }, + { + "widget": { + "name": "chart_revenue_trend", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "ds_daily_sales", + "fields": [ + {"name": "sale_date", "expression": "`sale_date`"}, + {"name": "sum(total_revenue)", "expression": "SUM(`total_revenue`)"} + ], + "disaggregated": false + } + }], + "spec": { + "version": 3, + "widgetType": "area", + "encodings": { + "x": { + "fieldName": "sale_date", + "scale": {"type": "temporal"}, + "axis": {"title": "Date"}, + "displayName": "Date" + }, + "y": { + "fieldName": "sum(total_revenue)", + "scale": {"type": "quantitative"}, + "format": { + "type": "number-currency", + "currencyCode": "USD", + "abbreviation": "compact" + }, + "axis": {"title": "Revenue ($)"}, + "displayName": "Revenue ($)" + } + }, + "frame": { + "title": "Daily Revenue", + "showTitle": true, + "description": "Track daily revenue trends" + } + } + }, + "position": {"x": 0, "y": 6, "width": 12, "height": 5} + }, + { + "widget": { + "name": "section_breakdown", + "multilineTextboxSpec": { + "lines": ["## Breakdown"] + } + }, + "position": {"x": 0, "y": 11, "width": 12, "height": 1} + }, + { + "widget": { + "name": "chart_by_department", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "ds_daily_sales", + "fields": [ + {"name": "department", "expression": "`department`"}, + {"name": "sum(total_revenue)", "expression": "SUM(`total_revenue`)"} + ], + "disaggregated": false + } + }], + "spec": { + "version": 3, + "widgetType": "pie", + "encodings": { + "angle": { + "fieldName": "sum(total_revenue)", + "scale": {"type": "quantitative"}, + "displayName": "Revenue" + }, + "color": { + "fieldName": "department", + "scale": {"type": "categorical"}, + "displayName": "Department" + }, + "label": {"show": true} + }, + "frame": {"title": "Revenue by Department", "showTitle": true} + } + }, + "position": {"x": 0, "y": 12, "width": 4, "height": 5} + }, + { + "widget": { + "name": "chart_by_region", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "ds_daily_sales", + "fields": [ + {"name": "sale_date", "expression": "`sale_date`"}, + {"name": "region", "expression": "`region`"}, + {"name": "sum(total_revenue)", "expression": "SUM(`total_revenue`)"} + ], + "disaggregated": false + } + }], + "spec": { + "version": 3, + "widgetType": "bar", + "encodings": { + "x": { + "fieldName": "sale_date", + "scale": {"type": "temporal"}, + "axis": {"title": "Date"}, + "displayName": "Date" + }, + "y": { + "fieldName": "sum(total_revenue)", + "scale": {"type": "quantitative"}, + "format": { + "type": "number-currency", + "currencyCode": "USD", + "abbreviation": "compact" + }, + "axis": {"title": "Revenue ($)"}, + "displayName": "Revenue ($)" + }, + "color": { + "fieldName": "region", + "scale": {"type": "categorical"}, + "displayName": "Region" + } + }, + "frame": {"title": "Revenue by Region", "showTitle": true} + } + }, + "position": {"x": 4, "y": 12, "width": 8, "height": 5} + }, + { + "widget": { + "name": "section_products", + "multilineTextboxSpec": { + "lines": ["## Top Products"] + } + }, + "position": {"x": 0, "y": 17, "width": 12, "height": 1} + }, + { + "widget": { + "name": "table_products", + "queries": [{ + "name": "main_query", + "query": { + "datasetName": "ds_products", + "fields": [ + {"name": "product_name", "expression": "`product_name`"}, + {"name": "department", "expression": "`department`"}, + {"name": "units_sold", "expression": "`units_sold`"}, + {"name": "revenue", "expression": "`revenue`"}, + {"name": "profit", "expression": "`profit`"} + ], + "disaggregated": true + } + }], + "spec": { + "version": 2, + "widgetType": "table", + "encodings": { + "columns": [ + {"fieldName": "product_name", "displayName": "Product"}, + {"fieldName": "department", "displayName": "Department"}, + {"fieldName": "units_sold", "displayName": "Units Sold"}, + {"fieldName": "revenue", "displayName": "Revenue ($)"}, + {"fieldName": "profit", "displayName": "Profit ($)"} + ] + }, + "frame": { + "title": "Product Performance", + "showTitle": true, + "description": "Top products by revenue" + } + } + }, + "position": {"x": 0, "y": 18, "width": 12, "height": 6} + } + ] + }, + { + "name": "global_filters", + "displayName": "Filters", + "pageType": "PAGE_TYPE_GLOBAL_FILTERS", + "layoutVersion": "GRID_V1", + "layout": [ + { + "widget": { + "name": "filter_date_range", + "queries": [ + { + "name": "ds_sales_date", + "query": { + "datasetName": "ds_daily_sales", + "fields": [{"name": "sale_date", "expression": "`sale_date`"}], + "disaggregated": false + } + } + ], + "spec": { + "version": 2, + "widgetType": "filter-date-range-picker", + "encodings": { + "fields": [ + {"fieldName": "sale_date", "displayName": "Date", "queryName": "ds_sales_date"} + ] + }, + "selection": { + "defaultSelection": { + "range": { + "dataType": "DATE", + "min": {"value": "now/y"}, + "max": {"value": "now/y"} + } + } + }, + "frame": {"showTitle": true, "title": "Date Range"} + } + }, + "position": {"x": 0, "y": 0, "width": 4, "height": 2} + }, + { + "widget": { + "name": "filter_region", + "queries": [ + { + "name": "ds_sales_region", + "query": { + "datasetName": "ds_daily_sales", + "fields": [{"name": "region", "expression": "`region`"}], + "disaggregated": false + } + }, + { + "name": "ds_products_region", + "query": { + "datasetName": "ds_products", + "fields": [{"name": "region", "expression": "`region`"}], + "disaggregated": false + } + } + ], + "spec": { + "version": 2, + "widgetType": "filter-multi-select", + "encodings": { + "fields": [ + {"fieldName": "region", "displayName": "Region", "queryName": "ds_sales_region"}, + {"fieldName": "region", "displayName": "Region", "queryName": "ds_products_region"} + ] + }, + "frame": {"showTitle": true, "title": "Region"} + } + }, + "position": {"x": 4, "y": 0, "width": 4, "height": 2} + }, + { + "widget": { + "name": "filter_department", + "queries": [ + { + "name": "ds_sales_dept", + "query": { + "datasetName": "ds_daily_sales", + "fields": [{"name": "department", "expression": "`department`"}], + "disaggregated": false + } + }, + { + "name": "ds_products_dept", + "query": { + "datasetName": "ds_products", + "fields": [{"name": "department", "expression": "`department`"}], + "disaggregated": false + } + } + ], + "spec": { + "version": 2, + "widgetType": "filter-multi-select", + "encodings": { + "fields": [ + {"fieldName": "department", "displayName": "Department", "queryName": "ds_sales_dept"}, + {"fieldName": "department", "displayName": "Department", "queryName": "ds_products_dept"} + ] + }, + "frame": {"showTitle": true, "title": "Department"} + } + }, + "position": {"x": 8, "y": 0, "width": 4, "height": 2} + } + ] + } + ] +} +``` diff --git a/.claude/skills/databricks-aibi-dashboards/5-troubleshooting.md b/.claude/skills/databricks-aibi-dashboards/5-troubleshooting.md new file mode 100644 index 00000000..f6477c05 --- /dev/null +++ b/.claude/skills/databricks-aibi-dashboards/5-troubleshooting.md @@ -0,0 +1,100 @@ +# Troubleshooting + +Common errors and fixes for AI/BI dashboards. + +## Structural Errors (JSON Parse Failures) + +These errors occur when the JSON structure is wrong: + +| Error | Cause | Fix | +|-------|-------|-----| +| "failed to parse serialized dashboard" | Wrong JSON structure | Check: `queryLines` is array (not `"query": "string"`), widgets inline in `layout[].widget`, `pageType` on every page | +| "no selected fields to visualize" | `fields[].name` ≠ `encodings.fieldName` | Names must match exactly (e.g., both `"sum(spend)"`) | +| Widgets in wrong location | Used separate `"widgets"` array | Widgets must be INLINE: `layout[]: {widget: {...}, position: {...}}` | +| Missing page content | Omitted `pageType` | Add `"pageType": "PAGE_TYPE_CANVAS"` or `"PAGE_TYPE_GLOBAL_FILTERS"` | + +--- + +## Widget shows "no selected fields to visualize" + +**This is a field name mismatch error.** The `name` in `query.fields` must exactly match the `fieldName` in `encodings`. + +**Fix:** Ensure names match exactly: +```json +// WRONG - names don't match +"fields": [{"name": "spend", "expression": "SUM(`spend`)"}] +"encodings": {"value": {"fieldName": "sum(spend)", ...}} // ERROR! + +// CORRECT - names match +"fields": [{"name": "sum(spend)", "expression": "SUM(`spend`)"}] +"encodings": {"value": {"fieldName": "sum(spend)", ...}} // OK! +``` + +## Widget shows "Invalid widget definition" + +**Check version numbers:** +- Counters: `version: 2` (NOT 3!) +- Tables: `version: 2` (NOT 1 or 3!) +- Filters: `version: 2` +- Bar/Line/Pie/Area/Scatter charts: `version: 3` +- Combo/Choropleth-map: `version: 1` + +**Text widget errors:** +- Text widgets must NOT have a `spec` block +- Use `multilineTextboxSpec` directly on the widget object +- Do NOT use `widgetType: "text"` - this is invalid + +**Table widget errors:** +- Use `version: 2` (NOT 1 or 3) +- Column objects only need `fieldName` and `displayName` +- Do NOT add `type`, `numberFormat`, or other column properties + +**Counter widget errors:** +- Use `version: 2` (NOT 3) +- Ensure dataset returns exactly 1 row for `disaggregated: true` + +## Dashboard shows empty widgets + +- Run the dataset SQL query directly to check data exists +- Verify column aliases match widget field expressions +- Check `disaggregated` flag: + - `true` for pre-aggregated data (1 row) + - `false` when widget performs aggregation (multi-row) + +## Layout has gaps + +- Ensure each row sums to width=12 +- Check that y positions don't skip values + +## Filter shows "Invalid widget definition" + +- Check `widgetType` is one of: `filter-multi-select`, `filter-single-select`, `filter-date-range-picker` +- **DO NOT** use `widgetType: "filter"` - this is invalid +- Verify `spec.version` is `2` +- Ensure `queryName` in encodings matches the query `name` +- Confirm `disaggregated: false` in filter queries +- Ensure `frame` with `showTitle: true` is included + +## Filter not affecting expected pages + +- **Global filters** (on `PAGE_TYPE_GLOBAL_FILTERS` page) affect all datasets containing the filter field +- **Page-level filters** (on `PAGE_TYPE_CANVAS` page) only affect widgets on that same page +- A filter only works on datasets that include the filter dimension column + +## Filter shows "UNRESOLVED_COLUMN" error for `associative_filter_predicate_group` + +- **DO NOT** use `COUNT_IF(\`associative_filter_predicate_group\`)` in filter queries +- This internal expression causes SQL errors when the dashboard executes queries +- Use a simple field expression instead: `{"name": "field", "expression": "\`field\`"}` + +## Text widget shows title and description on same line + +- Multiple items in the `lines` array are **concatenated**, not displayed on separate lines +- Use **separate text widgets** for title and subtitle at different y positions +- Example: title at y=0 with height=1, subtitle at y=1 with height=1 + +## Chart unreadable (too many categories) + +- Use TOP-N + "Other" bucketing in dataset SQL +- Aggregate to a higher level (region instead of store) +- Use a table widget instead of a chart for high-cardinality data diff --git a/.claude/skills/databricks-aibi-dashboards/SKILL.md b/.claude/skills/databricks-aibi-dashboards/SKILL.md new file mode 100644 index 00000000..426e6024 --- /dev/null +++ b/.claude/skills/databricks-aibi-dashboards/SKILL.md @@ -0,0 +1,226 @@ +--- +name: databricks-aibi-dashboards +description: "Create Databricks AI/BI dashboards. Use when creating, updating, or deploying Lakeview dashboards. CRITICAL: You MUST test ALL SQL queries via execute_sql BEFORE deploying. Follow guidelines strictly." +--- + +# AI/BI Dashboard Skill + +Create Databricks AI/BI dashboards (formerly Lakeview dashboards). **Follow these guidelines strictly.** + +## CRITICAL: MANDATORY VALIDATION WORKFLOW + +**You MUST follow this workflow exactly. Skipping validation causes broken dashboards.** + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ STEP 1: Get table schemas via get_table_stats_and_schema(catalog, schema) │ +├─────────────────────────────────────────────────────────────────────┤ +│ STEP 2: Write SQL queries for each dataset │ +├─────────────────────────────────────────────────────────────────────┤ +│ STEP 3: TEST EVERY QUERY via execute_sql() ← DO NOT SKIP! │ +│ - If query fails, FIX IT before proceeding │ +│ - Verify column names match what widgets will reference │ +│ - Verify data types are correct (dates, numbers, strings) │ +├─────────────────────────────────────────────────────────────────────┤ +│ STEP 4: Build dashboard JSON using ONLY verified queries │ +├─────────────────────────────────────────────────────────────────────┤ +│ STEP 5: Deploy via manage_dashboard(action="create_or_update") │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +**WARNING: If you deploy without testing queries, widgets WILL show "Invalid widget definition" errors!** + +## Available MCP Tools + +| Tool | Description | +|------|-------------| +| `get_table_stats_and_schema` | **STEP 1**: Get table schemas for designing queries | +| `execute_sql` | **STEP 3**: Test SQL queries - MANDATORY before deployment! | +| `manage_warehouse` (action="get_best") | Get available warehouse ID | +| `manage_dashboard` | **STEP 5**: Dashboard lifecycle management (see actions below) | + +### manage_dashboard Actions + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create_or_update` | Deploy dashboard JSON (only after validation!) | display_name, parent_path, serialized_dashboard, warehouse_id | +| `get` | Get dashboard details by ID | dashboard_id | +| `list` | List all dashboards | (none) | +| `delete` | Move dashboard to trash | dashboard_id | +| `publish` | Publish a dashboard | dashboard_id, warehouse_id | +| `unpublish` | Unpublish a dashboard | dashboard_id | + +**Example usage:** +```python +# Create/update dashboard +manage_dashboard( + action="create_or_update", + display_name="Sales Dashboard", + parent_path="/Workspace/Users/me/dashboards", + serialized_dashboard=dashboard_json, + warehouse_id="abc123", + publish=True # auto-publish after create +) + +# Get dashboard details +manage_dashboard(action="get", dashboard_id="dashboard_123") + +# List all dashboards +manage_dashboard(action="list") +``` + +## Reference Files + +| What are you building? | Reference | +|------------------------|-----------| +| Any widget (text, counter, table, chart) | [1-widget-specifications.md](1-widget-specifications.md) | +| Dashboard with filters (global or page-level) | [2-filters.md](2-filters.md) | +| Need a complete working template to adapt | [3-examples.md](3-examples.md) | +| Debugging a broken dashboard | [4-troubleshooting.md](4-troubleshooting.md) | + +--- + +## Implementation Guidelines + +### 1) DATASET ARCHITECTURE + +- **One dataset per domain** (e.g., orders, customers, products) +- **Exactly ONE valid SQL query per dataset** (no multiple queries separated by `;`) +- Always use **fully-qualified table names**: `catalog.schema.table_name` +- SELECT must include all dimensions needed by widgets and all derived columns via `AS` aliases +- Put ALL business logic (CASE/WHEN, COALESCE, ratios) into the dataset SELECT with explicit aliases +- **Contract rule**: Every widget `fieldName` must exactly match a dataset column or alias + +### 2) WIDGET FIELD EXPRESSIONS + +> **CRITICAL: Field Name Matching Rule** +> The `name` in `query.fields` MUST exactly match the `fieldName` in `encodings`. +> If they don't match, the widget shows "no selected fields to visualize" error! + +**Correct pattern for aggregations:** +```json +// In query.fields: +{"name": "sum(spend)", "expression": "SUM(`spend`)"} + +// In encodings (must match!): +{"fieldName": "sum(spend)", "displayName": "Total Spend"} +``` + +**WRONG - names don't match:** +```json +// In query.fields: +{"name": "spend", "expression": "SUM(`spend`)"} // name is "spend" + +// In encodings: +{"fieldName": "sum(spend)", ...} // ERROR: "sum(spend)" ≠ "spend" +``` + +Allowed expressions in widget queries (you CANNOT use CAST or other SQL in expressions): + +**For numbers:** +```json +{"name": "sum(revenue)", "expression": "SUM(`revenue`)"} +{"name": "avg(price)", "expression": "AVG(`price`)"} +{"name": "count(orders)", "expression": "COUNT(`order_id`)"} +{"name": "countdistinct(customers)", "expression": "COUNT(DISTINCT `customer_id`)"} +{"name": "min(date)", "expression": "MIN(`order_date`)"} +{"name": "max(date)", "expression": "MAX(`order_date`)"} +``` + +**For dates** (use daily for timeseries, weekly/monthly for grouped comparisons): +```json +{"name": "daily(date)", "expression": "DATE_TRUNC(\"DAY\", `date`)"} +{"name": "weekly(date)", "expression": "DATE_TRUNC(\"WEEK\", `date`)"} +{"name": "monthly(date)", "expression": "DATE_TRUNC(\"MONTH\", `date`)"} +``` + +**Simple field reference** (for pre-aggregated data): +```json +{"name": "category", "expression": "`category`"} +``` + +If you need conditional logic or multi-field formulas, compute a derived column in the dataset SQL first. + +### 3) SPARK SQL PATTERNS + +- Date math: `date_sub(current_date(), N)` for days, `add_months(current_date(), -N)` for months +- Date truncation: `DATE_TRUNC('DAY'|'WEEK'|'MONTH'|'QUARTER'|'YEAR', column)` +- **AVOID** `INTERVAL` syntax - use functions instead + +### 4) LAYOUT (12-Column Grid, NO GAPS) + +**Every page must include `"layoutVersion": "GRID_V1"`** alongside `pageType`. + +```json +{ + "name": "overview", + "displayName": "Overview", + "pageType": "PAGE_TYPE_CANVAS", + "layoutVersion": "GRID_V1", + "layout": [...] +} +``` + +Each widget has a position: `{"x": 0, "y": 0, "width": 4, "height": 4}` + +**CRITICAL**: Each row must fill width=12 exactly. No gaps allowed. + +**Recommended widget sizes:** + +| Widget Type | Width | Height | Notes | +|-------------|-------|--------|-------| +| Text header | 12 | 1 | Full width; use SEPARATE widgets for title and subtitle | +| Counter/KPI | 4 | **3-4** | **NEVER height=2** - too cramped! | +| Line/Bar chart | 6 | **5-6** | Pair side-by-side to fill row | +| Pie chart | 6 | **5-6** | Needs space for legend | +| Full-width chart | 12 | 5-7 | For detailed time series | +| Table | 12 | 5-8 | Full width for readability | + +**Standard dashboard structure:** +```text +y=0: Title (w=12, h=1) - Dashboard title (use separate widget!) +y=1: Subtitle (w=12, h=1) - Description (use separate widget!) +y=2: KPIs (w=4 each, h=3) - 3 key metrics side-by-side +y=5: Section header (w=12, h=1) - "Trends" or similar +y=6: Charts (w=6 each, h=5) - Two charts side-by-side +y=11: Section header (w=12, h=1) - "Details" +y=12: Table (w=12, h=6) - Detailed data +``` + +### 5) CARDINALITY & READABILITY (CRITICAL) + +**Dashboard readability depends on limiting distinct values:** + +| Dimension Type | Max Values | Examples | +|----------------|------------|----------| +| Chart color/groups | **3-8** | 4 regions, 5 product lines, 3 tiers | +| Filters | 4-10 | 8 countries, 5 channels | +| High cardinality | **Table only** | customer_id, order_id, SKU | + +**Before creating any chart with color/grouping:** +1. Check column cardinality (use `get_table_stats_and_schema` to see distinct values) +2. If >10 distinct values, aggregate to higher level OR use TOP-N + "Other" bucket +3. For high-cardinality dimensions, use a table widget instead of a chart + +### 6) QUALITY CHECKLIST + +Before deploying, verify: +1. All widget names use only alphanumeric + hyphens + underscores +2. **Every page has `"layoutVersion": "GRID_V1"`** +3. All rows sum to width=12 with no gaps +4. KPIs use height 3-4, charts use height 5-6 +5. Chart dimensions have ≤8 distinct values +6. All widget fieldNames match dataset columns exactly +7. **Field `name` in query.fields matches `fieldName` in encodings exactly** (e.g., both `"sum(spend)"`) +8. Counter datasets: use `disaggregated: true` for 1-row datasets, `disaggregated: false` with aggregation for multi-row +9. Percent values are 0-1 (not 0-100) +10. SQL uses Spark syntax (date_sub, not INTERVAL) +11. **All SQL queries tested via `execute_sql` and return expected data** + +--- + +## Related Skills + +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - for querying the underlying data and system tables +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - for building the data pipelines that feed dashboards +- **[databricks-jobs](../databricks-jobs/SKILL.md)** - for scheduling dashboard data refreshes diff --git a/.claude/skills/databricks-app-apx/SKILL.md b/.claude/skills/databricks-app-apx/SKILL.md deleted file mode 100644 index 54c2767d..00000000 --- a/.claude/skills/databricks-app-apx/SKILL.md +++ /dev/null @@ -1,246 +0,0 @@ ---- -name: databricks-app-apx -description: "Build full-stack Databricks applications using APX framework (FastAPI + React)." ---- - -# Databricks APX Application - -Build full-stack Databricks applications using APX framework (FastAPI + React). - -## Trigger Conditions - -**Invoke when user requests**: -- "Databricks app" or "Databricks application" -- Full-stack app for Databricks without specifying framework -- Mentions APX framework - -**Do NOT invoke if user specifies**: Streamlit, Dash, Node.js, Shiny, Gradio, Flask, or other frameworks. - -## Prerequisites Check - -Option A) -Repository configured for use with APX. -1.. Verify APX MCP available: `mcp-cli tools | grep apx` -2. Verify shadcn MCP available: `mcp-cli tools | grep shadcn` -3. Confirm APX project (check `pyproject.toml`) - -Option B) -Install APX -1. Verify uv available or prompt for install. On Mac, suggest: `brew install uv`. -2. Verify bun available or prompt for install. On Mac, suggest: -``` -brew tap oven-sh/bun -brew install bun -``` -3. Verify git available or prompt for install. -4. Run APX setup commands: -``` -uvx --from git+https://github.com/databricks-solutions/apx.git apx init -``` - - -## Workflow Overview - -Total time: 55-70 minutes - -1. **Initialize** (5 min) - Start servers, create todos -2. **Backend** (15-20 min) - Models + routes with mock data -3. **Frontend** (20-25 min) - Components + pages -4. **Test** (5-10 min) - Type check + manual verification -5. **Document** (10 min) - README + code structure guide - -## Phase 1: Initialize - -```bash -# Start APX development server -mcp-cli call apx/start '{}' -mcp-cli call apx/status '{}' -``` - -Create TodoWrite with tasks: -- Start servers ✓ -- Design models -- Create API routes -- Add UI components -- Create pages -- Test & document - -## Phase 2: Backend Development - -### Create Pydantic Models - -In `src/{app_name}/backend/models.py`: - -**Follow 3-model pattern**: -- `EntityIn` - Input validation -- `EntityOut` - Complete output with computed fields -- `EntityListOut` - Performance-optimized summary - -**See [backend-patterns.md](backend-patterns.md) for complete code templates.** - -### Create API Routes - -In `src/{app_name}/backend/router.py`: - -**Critical requirements**: -- Always include `response_model` (enables OpenAPI generation) -- Always include `operation_id` (becomes frontend hook name) -- Use naming pattern: `listX`, `getX`, `createX`, `updateX`, `deleteX` -- Initialize 3-4 mock data samples for testing - -**See [backend-patterns.md](backend-patterns.md) for complete CRUD templates.** - -### Type Check - -```bash -mcp-cli call apx/dev_check '{}' -``` - -Fix any Python type errors reported by basedpyright. - -## Phase 3: Frontend Development - -**Wait 5-10 seconds** after backend changes for OpenAPI client regeneration. - -### Add UI Components - -```bash -# Get shadcn add command -mcp-cli call shadcn/get_add_command_for_items '{ - "items": ["@shadcn/button", "@shadcn/card", "@shadcn/table", - "@shadcn/badge", "@shadcn/select", "@shadcn/skeleton"] -}' -``` - -Run the command from project root with `--yes` flag. - -### Create Pages - -**List page**: `src/{app_name}/ui/routes/_sidebar/{entity}.tsx` -- Table view with all entities -- Suspense boundaries with skeleton fallback -- Formatted data (currency, dates, status colors) - -**Detail page**: `src/{app_name}/ui/routes/_sidebar/{entity}.$id.tsx` -- Complete entity view with cards -- Update/delete mutations -- Back navigation - -**See [frontend-patterns.md](frontend-patterns.md) for complete page templates.** - -### Update Navigation - -In `src/{app_name}/ui/routes/_sidebar/route.tsx`, add new item to `navItems` array. - -## Phase 4: Testing - -```bash -# Type check both backend and frontend -mcp-cli call apx/dev_check '{}' - -# Test API endpoints -curl http://localhost:8000/api/{entities} | jq . -curl http://localhost:8000/api/{entities}/{id} | jq . - -# Get frontend URL -mcp-cli call apx/get_frontend_url '{}' -``` - -Manually verify in browser: -- List page displays data -- Detail page shows complete info -- Mutations work (update, delete) -- Loading states work (skeletons) -- Browser console errors are automatically captured in APX dev logs - -## Phase 5: Deployment & Monitoring - -### Deploy to Databricks - -Use DABs to deploy your APX application to Databricks. See the `asset-bundles` skill for complete deployment guidance. - -### Monitor Application Logs - -**Automated log checking with APX MCP:** - -The APX MCP server can automatically check deployed application logs. Simply ask: -"Please check the deployed app logs for " - - -The APX MCP will retrieve logs and identify issues automatically, including: -- Deployment status and errors -- Runtime exceptions and stack traces -- Both `[SYSTEM]` (deployment) and `[APP]` (application) logs -- Browser console errors (now included in APX dev logs) - -**Manual log checking (reference):** - -For direct CLI access: -```bash -databricks apps logs --profile -``` - -**Key patterns to look for:** -- ✅ `Deployment successful` - App deployed correctly -- ✅ `App started successfully` - Application is running -- ❌ `Error:` - Check stack traces for issues - -## Phase 6: Documentation - -Create two markdown files: - -**README.md**: -- Features overview -- Technology stack -- How app was created (AI tools + MCP servers used) -- Application architecture -- Getting started instructions -- API documentation -- Development workflow - -**CODE_STRUCTURE.md**: -- Directory structure explanation -- Backend structure (models, routes, patterns) -- Frontend structure (routes, components, hooks) -- Auto-generated files warnings -- Guide for adding new features -- Best practices -- Common patterns -- Troubleshooting guide - -## Key Patterns - -### Backend -- **3-model pattern**: Separate In, Out, and ListOut models -- **operation_id naming**: `listEntities` → `useListEntities()` -- **Type hints everywhere**: Enable validation and IDE support - -### Frontend -- **Suspense hooks**: `useXSuspense(selector())` -- **Suspense boundaries**: Always provide skeleton fallback -- **Formatters**: Currency, dates, status colors -- **Never edit**: `lib/api.ts` or `types/routeTree.gen.ts` - -## Success Criteria - -- [ ] Type checking passes (`apx dev check` succeeds) -- [ ] API endpoints return correct data (curl verification) -- [ ] Frontend displays and mutates data correctly -- [ ] Loading states work (skeletons display) -- [ ] Documentation complete - -## Common Issues - -**Deployed app not working**: Ask to check deployed app logs (APX MCP will automatically retrieve and analyze them) or manually use `databricks apps logs ` -**Python type errors**: Use explicit casting for dict access, check Optional fields -**TypeScript errors**: Wait for OpenAPI regen, verify hook names match operation_ids -**OpenAPI not updating**: Check watcher status with `apx dev status`, restart if needed -**Components not added**: Run shadcn from project root with `--yes` flag - -## Reference Materials - -- **[backend-patterns.md](backend-patterns.md)** - Complete backend code templates -- **[frontend-patterns.md](frontend-patterns.md)** - Complete frontend page templates -- **[best-practices.md](best-practices.md)** - Best practices, anti-patterns, debugging - -Read these files only when actively writing that type of code or debugging issues. diff --git a/.claude/skills/databricks-app-apx/backend-patterns.md b/.claude/skills/databricks-app-apx/backend-patterns.md deleted file mode 100644 index 1b8d6d07..00000000 --- a/.claude/skills/databricks-app-apx/backend-patterns.md +++ /dev/null @@ -1,225 +0,0 @@ -# Backend Code Patterns for APX - -Reference templates for backend development. **Only consult when writing backend code.** - -## Pydantic Models (models.py) - -### 3-Model Pattern - -```python -from pydantic import BaseModel, Field -from datetime import datetime -from enum import Enum -from typing import Optional - -# Enum for status -class EntityStatus(str, Enum): - STATUS_1 = "status_1" - STATUS_2 = "status_2" - -# Nested models -class ItemIn(BaseModel): - name: str - value: float = Field(gt=0) - -class ItemOut(BaseModel): - id: str - name: str - value: float - created_at: datetime - -# Main entity models -class EntityIn(BaseModel): - """Input for creating entities""" - title: str - items: list[ItemIn] - notes: Optional[str] = None - -class EntityOut(BaseModel): - """Complete entity output""" - id: str - entity_number: str - title: str - status: EntityStatus - items: list[ItemOut] - total: float # Computed field - notes: Optional[str] = None - created_at: datetime - updated_at: datetime - -class EntityListOut(BaseModel): - """Summary for list views (performance)""" - id: str - entity_number: str - title: str - status: EntityStatus - total: float - created_at: datetime -``` - -## API Routes (router.py) - -### Basic CRUD Structure - -```python -from typing import Annotated -from fastapi import APIRouter, Depends, HTTPException -from .models import EntityIn, EntityOut, EntityListOut, EntityStatus -from .config import conf -from datetime import datetime -import uuid - -api = APIRouter(prefix=conf.api_prefix) - -# In-memory storage (replace with database) -_entities_db: dict[str, EntityOut] = {} - -# List all -@api.get("/entities", response_model=list[EntityListOut], operation_id="listEntities") -async def list_entities(): - """Get all entities (summary view)""" - return [ - EntityListOut( - id=e.id, - entity_number=e.entity_number, - title=e.title, - status=e.status, - total=e.total, - created_at=e.created_at, - ) - for e in sorted(_entities_db.values(), key=lambda x: x.created_at, reverse=True) - ] - -# Get one -@api.get("/entities/{entity_id}", response_model=EntityOut, operation_id="getEntity") -async def get_entity(entity_id: str): - """Get a specific entity by ID""" - if entity_id not in _entities_db: - raise HTTPException(status_code=404, detail="Entity not found") - return _entities_db[entity_id] - -# Create -@api.post("/entities", response_model=EntityOut, operation_id="createEntity") -async def create_entity(entity_in: EntityIn): - """Create a new entity""" - entity_id = str(uuid.uuid4()) - - # Process items - items = [ - ItemOut( - id=str(uuid.uuid4()), - name=item.name, - value=item.value, - created_at=datetime.now() - ) - for item in entity_in.items - ] - - # Calculate total - total = sum(item.value for item in items) - - entity = EntityOut( - id=entity_id, - entity_number=f"ENT-{datetime.now().strftime('%Y%m%d')}-{len(_entities_db) + 1:04d}", - title=entity_in.title, - status=EntityStatus.STATUS_1, - items=items, - total=total, - notes=entity_in.notes, - created_at=datetime.now(), - updated_at=datetime.now(), - ) - - _entities_db[entity_id] = entity - return entity - -# Update -@api.patch("/entities/{entity_id}", response_model=EntityOut, operation_id="updateEntity") -async def update_entity(entity_id: str, entity_update: EntityIn): - """Update an entity""" - if entity_id not in _entities_db: - raise HTTPException(status_code=404, detail="Entity not found") - - entity = _entities_db[entity_id] - # Apply updates - entity.title = entity_update.title - entity.updated_at = datetime.now() - - return entity - -# Delete -@api.delete("/entities/{entity_id}", operation_id="deleteEntity") -async def delete_entity(entity_id: str): - """Delete an entity""" - if entity_id not in _entities_db: - raise HTTPException(status_code=404, detail="Entity not found") - - del _entities_db[entity_id] - return {"message": "Entity deleted successfully"} -``` - -### Mock Data Initialization - -```python -def _init_mock_data(): - """Initialize with sample data""" - if _entities_db: - return - - mock_data = [ - { - "title": "Sample Entity 1", - "status": EntityStatus.STATUS_1, - "items": [ - {"name": "Item A", "value": 100.0}, - {"name": "Item B", "value": 50.0}, - ], - "notes": "Sample note", - }, - # Add 2-3 more samples - ] - - for idx, data in enumerate(mock_data): - entity_id = str(uuid.uuid4()) - - items = [ - ItemOut( - id=str(uuid.uuid4()), - name=item["name"], - value=item["value"], - created_at=datetime.now() - ) - for item in data["items"] - ] - - entity = EntityOut( - id=entity_id, - entity_number=f"ENT-{datetime.now().strftime('%Y%m%d')}-{idx + 1:04d}", - title=data["title"], - status=data["status"], - items=items, - total=sum(item.value for item in items), - notes=data.get("notes"), - created_at=datetime.now(), - updated_at=datetime.now(), - ) - - _entities_db[entity_id] = entity - -# Call at module level -_init_mock_data() -``` - -## Naming Conventions - -### operation_id → Frontend Hook Name - -| operation_id | Generated Hook | -|--------------|----------------| -| `listEntities` | `useListEntities()`, `useListEntitiesSuspense()` | -| `getEntity` | `useGetEntity(id)`, `useGetEntitySuspense(id)` | -| `createEntity` | `useCreateEntity()` | -| `updateEntity` | `useUpdateEntity()` | -| `deleteEntity` | `useDeleteEntity()` | - -**Pattern**: Verb + EntityName in camelCase diff --git a/.claude/skills/databricks-app-apx/best-practices.md b/.claude/skills/databricks-app-apx/best-practices.md deleted file mode 100644 index ef71f0d8..00000000 --- a/.claude/skills/databricks-app-apx/best-practices.md +++ /dev/null @@ -1,318 +0,0 @@ -# APX Best Practices & Anti-Patterns - -Guidelines for building high-quality APX applications. **Consult only when needed.** - -## Critical Rules - -### Backend - -1. **Always include `response_model` and `operation_id`** - ```python - # ✅ Correct - @api.get("/entities", response_model=list[EntityOut], operation_id="listEntities") - - # ❌ Wrong - missing both - @api.get("/entities") - ``` - -2. **Follow 3-model pattern** - - `EntityIn` - Input validation - - `EntityOut` - Complete output - - `EntityListOut` - Performance-optimized summary - -3. **Use descriptive operation_ids** - - Pattern: `` (camelCase) - - Examples: `listOrders`, `getOrder`, `createOrder`, `updateOrderStatus` - -4. **Always use type hints** - ```python - # ✅ Correct - def get_entity(entity_id: str) -> EntityOut: - - # ❌ Wrong - no types - def get_entity(entity_id): - ``` - -5. **Handle errors with HTTPException** - ```python - if entity_id not in db: - raise HTTPException(status_code=404, detail="Not found") - ``` - -### Frontend - -1. **Always use Suspense hooks** - ```typescript - // ✅ Correct - }> - - - - function DataComponent() { - const { data } = useListEntitiesSuspense(selector()); - return
{data.map(...)}
; - } - - // ❌ Wrong - no Suspense - const { data, isLoading } = useListEntities(); - if (isLoading) return
Loading...
; - ``` - -2. **Use selector() for destructuring** - ```typescript - // ✅ Correct - const { data: entities } = useListEntitiesSuspense(selector()); - - // ❌ Wrong - verbose - const result = useListEntitiesSuspense(); - const entities = result.data; - ``` - -3. **Provide matching skeleton fallbacks** - - Skeleton should mirror actual content structure - - Use same table/card layout - -4. **Never edit auto-generated files** - - `lib/api.ts` - Generated by Orval - - `types/routeTree.gen.ts` - Generated by TanStack Router - -5. **Implement proper formatters** - - Currency: `Intl.NumberFormat` - - Dates: `toLocaleDateString` - - Status colors: Tailwind classes with dark mode support - -## Anti-Patterns - -### Backend - -**❌ Missing response_model** -```python -@api.get("/entities") # OpenAPI won't generate correctly -async def list_entities(): - return [] -``` - -**❌ Generic operation_id** -```python -@api.get("/entities", operation_id="get") # Too generic -``` - -**❌ No type safety** -```python -def process(data): # Can't validate, no IDE support - return data["field"] -``` - -**❌ Using plain dicts instead of Pydantic** -```python -def create_entity(data: dict): # No validation - return {"id": "123", **data} -``` - -### Frontend - -**❌ Not using Suspense** -```typescript -const { data, isLoading } = useListEntities(); -if (isLoading) return ; // Manual loading state -``` - -**❌ Editing generated files** -```typescript -// In lib/api.ts -export function useListEntities() { - // Custom changes ❌ -} -``` - -**❌ No skeleton fallback** -```typescript - {/* No fallback - will show nothing */} - - -``` - -**❌ Inline styles or classes** -```typescript -
{/* No dark mode support */} -``` - -## Type Safety - -### Python Type Errors - -**Problem**: Dict access typing -```python -# ❌ Problem -item_data["field"] # Type checker doesn't know structure -``` - -**Solution**: Explicit casting -```python -# ✅ Solution -if not isinstance(item_data, dict): - continue -item_dict: dict[str, Any] = item_data -value = str(item_dict.get("field", "")) -``` - -**Problem**: Optional fields -```python -# ❌ Problem -entity.notes.upper() # notes is Optional[str] -``` - -**Solution**: Check before access -```python -# ✅ Solution -if entity.notes: - entity.notes.upper() -``` - -### TypeScript Type Errors - -**Problem**: Wrong destructuring -```typescript -// ❌ Problem -const { data: response } = useListEntitiesSuspense(selector()); -const entities = response.data; // response.data doesn't exist -``` - -**Solution**: Direct destructuring -```typescript -// ✅ Solution -const { data: entities } = useListEntitiesSuspense(selector()); -``` - -## Performance - -### Backend - -1. **Use EntityListOut for lists** - Don't return full EntityOut for performance -2. **Implement pagination** - For large datasets -3. **Use async** - For I/O operations -4. **Index database queries** - When replacing mock data - -### Frontend - -1. **Use EntityListOut endpoints** - Lists should use summary endpoints -2. **Implement virtual scrolling** - For very long lists -3. **Lazy load detail views** - Don't preload all details -4. **Use React.memo** - Only when profiling shows benefit - -## Code Organization - -### Backend - -``` -backend/ -├── models.py # All Pydantic models -├── router.py # All API routes -├── dependencies.py # Shared dependencies -├── config.py # Configuration -└── utils.py # Helper functions -``` - -**Don't**: Split models/routes across multiple files unless >1000 lines - -### Frontend - -``` -ui/ -├── routes/ -│ └── _sidebar/ -│ ├── entities.tsx # List page -│ └── entities.$entityId.tsx # Detail page -├── components/ -│ ├── ui/ # shadcn components (don't edit) -│ └── apx/ # Custom components -└── lib/ - ├── api.ts # Auto-generated (don't edit) - ├── utils.ts # Helpers (cn, etc.) - └── selector.ts # Query selector -``` - -**Do**: Keep list and detail pages together -**Don't**: Create deep nested route folders - -## Error Messages - -### Backend - -```python -# ✅ Descriptive -raise HTTPException( - status_code=404, - detail=f"Entity with ID {entity_id} not found" -) - -# ❌ Generic -raise HTTPException(status_code=404, detail="Not found") -``` - -### Frontend - -```typescript -// ✅ User-friendly -console.error("Failed to delete order:", error); -// Show toast/alert to user - -// ❌ Silent failure -try { - await deleteEntity.mutateAsync({ entityId }); -} catch {} // Silently swallows error -``` - -## Testing - -### Backend - -```bash -# Type check -uv run basedpyright --level error - -# Test endpoints -curl http://localhost:8000/api/entities | jq . -curl http://localhost:8000/api/entities/{id} | jq . -``` - -### Frontend - -```bash -# Type check -bun run tsc -b --incremental - -# Both -uv run apx dev check -``` - -## Common Pitfalls - -1. **Forgetting to wait for OpenAPI regeneration** - Wait 5-10 seconds after backend changes -2. **Running shadcn from wrong directory** - Must run from project root -3. **Not using --yes flag** - Shadcn will prompt for confirmation -4. **Editing auto-generated files** - Changes will be overwritten -5. **Not implementing skeleton fallbacks** - Page will appear broken while loading -6. **Inconsistent status colors** - Use same color scheme throughout -7. **No dark mode support** - Always use Tailwind dark: classes - -## Debugging Checklist - -**Backend issues**: -- [ ] All models use type hints -- [ ] All routes have response_model + operation_id -- [ ] Mock data initialized correctly -- [ ] Type checking passes - -**Frontend issues**: -- [ ] OpenAPI client regenerated (check timestamp on lib/api.ts) -- [ ] Using Suspense hooks -- [ ] Suspense boundaries in place -- [ ] Hook names match operation_ids -- [ ] Type checking passes - -**Integration issues**: -- [ ] Backend servers running (apx dev status) -- [ ] OpenAPI watcher running -- [ ] API returns correct data (curl test) -- [ ] Frontend URL accessible diff --git a/.claude/skills/databricks-app-apx/frontend-patterns.md b/.claude/skills/databricks-app-apx/frontend-patterns.md deleted file mode 100644 index 29b96851..00000000 --- a/.claude/skills/databricks-app-apx/frontend-patterns.md +++ /dev/null @@ -1,376 +0,0 @@ -# Frontend Code Patterns for APX - -Reference templates for frontend development. **Only consult when writing frontend code.** - -## List Page Template (routes/_sidebar/entities.tsx) - -```typescript -import { createFileRoute, Link } from "@tanstack/react-router"; -import { Suspense } from "react"; -import { useListEntitiesSuspense, EntityStatus } from "@/lib/api"; -import { selector } from "@/lib/selector"; -import { - Table, - TableBody, - TableCell, - TableHead, - TableHeader, - TableRow, -} from "@/components/ui/table"; -import { Badge } from "@/components/ui/badge"; -import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { Skeleton } from "@/components/ui/skeleton"; - -export const Route = createFileRoute("/_sidebar/entities")({ - component: () => ( -
- - - Entities - - - }> - - - - -
- ), -}); - -function EntitiesTable() { - const { data: entities } = useListEntitiesSuspense(selector()); - - return ( -
- - - - Number - Title - Status - Total - Created - Actions - - - - {entities.length === 0 ? ( - - - No items found - - - ) : ( - entities.map((entity) => ( - - {entity.entity_number} - {entity.title} - - - {entity.status} - - - {formatCurrency(entity.total)} - {formatDate(entity.created_at)} - - - View - - - - )) - )} - -
-
- ); -} - -function TableSkeleton() { - return ( -
- - - - Number - Title - Status - Total - Created - Actions - - - - {[...Array(4)].map((_, i) => ( - - - - - - - - - ))} - -
-
- ); -} - -// Helper functions -const getStatusColor = (status: EntityStatus) => { - const colors = { - status_1: "bg-yellow-100 text-yellow-800 dark:bg-yellow-900 dark:text-yellow-300", - status_2: "bg-green-100 text-green-800 dark:bg-green-900 dark:text-green-300", - }; - return colors[status] || "bg-gray-100 text-gray-800"; -}; - -const formatDate = (dateString: string) => { - return new Date(dateString).toLocaleDateString("en-US", { - year: "numeric", - month: "short", - day: "numeric", - hour: "2-digit", - minute: "2-digit", - }); -}; - -const formatCurrency = (amount: number) => { - return new Intl.NumberFormat("en-US", { - style: "currency", - currency: "USD", - }).format(amount); -}; -``` - -## Detail Page Template (routes/_sidebar/entities.$entityId.tsx) - -```typescript -import { createFileRoute, Link, useNavigate } from "@tanstack/react-router"; -import { Suspense } from "react"; -import { useGetEntitySuspense, useUpdateEntity, useDeleteEntity } from "@/lib/api"; -import { selector } from "@/lib/selector"; -import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; -import { Button } from "@/components/ui/button"; -import { Skeleton } from "@/components/ui/skeleton"; -import { ArrowLeft } from "lucide-react"; - -export const Route = createFileRoute("/_sidebar/entities/$entityId")({ - component: () => ( -
- }> - - -
- ), -}); - -function EntityDetail() { - const { entityId } = Route.useParams(); - const navigate = useNavigate(); - const { data: entity } = useGetEntitySuspense(entityId, selector()); - - const updateMutation = useUpdateEntity(); - const deleteMutation = useDeleteEntity(); - - const handleDelete = async () => { - if (!confirm("Are you sure you want to delete this item?")) return; - - try { - await deleteMutation.mutateAsync({ entityId: entity.id }); - navigate({ to: "/entities" }); - } catch (error) { - console.error("Failed to delete:", error); - } - }; - - return ( -
- {/* Header */} -
-
- - - -
-

{entity.entity_number}

-

Entity Details

-
-
- -
- - {/* Content Cards */} -
- - - Information - - -
-

Title

-

{entity.title}

-
-
-

Status

-

{entity.status}

-
- - - - - - Items - - -
- {entity.items.map((item) => ( -
- {item.name} - {formatCurrency(item.value)} -
- ))} -
- - -
-
- ); -} - -function DetailSkeleton() { - return ( -
-
- -
- - -
-
-
- {[...Array(2)].map((_, i) => ( - - - - - - - - - - ))} -
-
- ); -} - -const formatCurrency = (amount: number) => { - return new Intl.NumberFormat("en-US", { - style: "currency", - currency: "USD", - }).format(amount); -}; -``` - -## Navigation Update (routes/_sidebar/route.tsx) - -Add to `navItems` array: - -```typescript -import { Package } from "lucide-react"; // Choose appropriate icon - -const navItems = [ - { - to: "/entities", - label: "Entities", - icon: , - match: (path: string) => path.startsWith("/entities"), - }, - // ... existing items -]; -``` - -## Common Formatters - -```typescript -// Currency -const formatCurrency = (amount: number) => { - return new Intl.NumberFormat("en-US", { - style: "currency", - currency: "USD", - }).format(amount); -}; - -// Date with time -const formatDate = (dateString: string) => { - return new Date(dateString).toLocaleDateString("en-US", { - year: "numeric", - month: "short", - day: "numeric", - hour: "2-digit", - minute: "2-digit", - }); -}; - -// Date only -const formatDateOnly = (dateString: string) => { - return new Date(dateString).toLocaleDateString("en-US", { - year: "numeric", - month: "long", - day: "numeric", - }); -}; - -// Number with commas -const formatNumber = (num: number) => { - return new Intl.NumberFormat("en-US").format(num); -}; -``` - -## Status Badge Colors - -```typescript -const getStatusColor = (status: string) => { - const colors: Record = { - pending: "bg-yellow-100 text-yellow-800 dark:bg-yellow-900 dark:text-yellow-300", - processing: "bg-blue-100 text-blue-800 dark:bg-blue-900 dark:text-blue-300", - active: "bg-green-100 text-green-800 dark:bg-green-900 dark:text-green-300", - completed: "bg-green-100 text-green-800 dark:bg-green-900 dark:text-green-300", - cancelled: "bg-red-100 text-red-800 dark:bg-red-900 dark:text-red-300", - inactive: "bg-gray-100 text-gray-800 dark:bg-gray-900 dark:text-gray-300", - }; - return colors[status] || "bg-gray-100 text-gray-800"; -}; -``` - -## Mutation Pattern with Error Handling - -```typescript -const createMutation = useCreateEntity(); - -const handleCreate = async (data: EntityIn) => { - try { - const result = await createMutation.mutateAsync({ data }); - // Success - navigate or show message - navigate({ to: `/entities/${result.data.id}` }); - } catch (error) { - console.error("Failed to create:", error); - // Show error to user - } -}; -``` diff --git a/.claude/skills/databricks-app-python/README.md b/.claude/skills/databricks-app-python/README.md deleted file mode 100644 index 63b30a6b..00000000 --- a/.claude/skills/databricks-app-python/README.md +++ /dev/null @@ -1,157 +0,0 @@ -# Databricks Python App Skill - -Claude Agent skill for building Python-based Databricks applications with various frameworks. - -## Structure - -``` -databricks-app-python/ -├── SKILL.md # Main skill file with core patterns -├── dash.md # Dash framework specific guide -├── streamlit.md # Streamlit guide (coming soon) -├── flask.md # Flask guide (coming soon) -└── README.md # This file -``` - -## Overview - -This skill provides comprehensive guidance for building Python applications for Databricks, including: - -### Core Components (SKILL.md) -- Architecture patterns -- Pydantic data models -- Mock and real backend patterns -- Databricks connectivity -- Unity Catalog integration -- Environment configuration -- Best practices - -### Framework-Specific Guides - -#### Dash (dash.md) -- Complete Dash application structure -- Component patterns (cards, tables, charts, modals) -- Callback patterns and best practices -- Plotly chart examples -- Bootstrap styling -- Common pitfalls and solutions - -#### Coming Soon -- **streamlit.md** - Streamlit patterns for rapid prototyping -- **flask.md** - Flask patterns for custom web apps -- **gradio.md** - Gradio for ML model interfaces - -## Usage - -When user requests a Python app for Databricks: - -1. **SKILL.md** provides the foundation: - - Data model design - - Backend architecture (mock + real) - - Databricks connectivity patterns - - Database setup - -2. **Framework-specific file** provides implementation: - - UI component patterns - - Framework-specific callbacks/routing - - Styling and theming - - Deployment configuration - -## Design Philosophy - -### Separation of Concerns -- **Core patterns** (SKILL.md) - Framework-agnostic -- **Framework details** (dash.md, etc.) - Implementation specifics - -### Progressive Complexity -- Start with mock backend (rapid development) -- Add real backend (production ready) -- Scale with Unity Catalog - -### Consistent Architecture -All apps follow same pattern: -``` -models.py → Data definitions -backend_mock.py → Sample data -backend_real.py → Databricks SQL -{framework}_app.py → UI implementation -setup_database.py → Schema initialization -``` - -## Example Applications - -### Order Management (Dash) -Location: `/example-app-dash/` - -Features: -- Dashboard with statistics and charts -- Filterable orders table -- Customer and product management -- Order details modal -- Mock and real backend support - -To run: -```bash -cd example-app-dash -uv pip install -r requirements.txt -USE_MOCK_BACKEND=true uv run python dash_app.py -``` - -## Adding New Frameworks - -To add support for a new framework: - -1. Create `{framework}.md` in this directory -2. Follow the structure of `dash.md`: - - When to use - - Dependencies - - Project structure - - Component patterns - - Best practices - - Common pitfalls - - Example code - -3. Update `SKILL.md` to reference new framework -4. Create example app in `/example-app-{framework}/` - -## Contributing - -When updating this skill: - -1. **SKILL.md changes**: Update if affecting all frameworks - - New backend patterns - - Database connectivity - - Environment configuration - - Pydantic model patterns - -2. **Framework-specific changes**: Update individual files - - New component patterns - - Framework version updates - - Best practices - - Bug fixes - -3. Keep example apps in sync with documentation - -## Testing - -Before committing changes: - -1. Test example apps run without errors -2. Verify all code examples are syntactically correct -3. Check cross-references between files work -4. Ensure new patterns follow existing conventions - -## Related Skills - -- **databricks-app-apx** - APX framework (FastAPI + React) -- **databricks-dev** - General Databricks development -- **python-dev** - Python development standards -- **asset-bundles** - Databricks Asset Bundles - -## Support - -For issues or questions: -1. Check framework-specific documentation -2. Review example applications -3. Consult Databricks documentation -4. Check framework-specific communities diff --git a/.claude/skills/databricks-app-python/SKILL.md b/.claude/skills/databricks-app-python/SKILL.md deleted file mode 100644 index 1bbdeaeb..00000000 --- a/.claude/skills/databricks-app-python/SKILL.md +++ /dev/null @@ -1,812 +0,0 @@ ---- -name: databricks-app-python -description: "Build Python-based Databricks applications using Dash, Streamlit, or Flask." ---- - -# Databricks Python Application - -Build Python-based Databricks applications using frameworks like Dash, Streamlit, Flask, or other Python web frameworks. - -## Trigger Conditions - -**Invoke when user requests**: -- "Dash app" or "Dash application" -- "Streamlit app" or "Streamlit application" -- "Python web app" for Databricks -- Building data visualization or dashboard apps -- Order management, analytics dashboard, etc. - -**Do NOT invoke if user specifies**: APX, React, Node.js, or other non-Python frameworks. - -## Framework Selection - -Ask user which framework to use if not specified: -- **Dash** - Rich interactive dashboards, Bootstrap components, Plotly charts -- **Streamlit** - Rapid prototyping, simple syntax, data science focus, automatic reactivity -- **Flask** - Lightweight, flexible, custom web apps (coming soon) - -### Dash vs Streamlit Comparison - -| Aspect | Dash | Streamlit | -|--------|------|-----------| -| **Development Speed** | Moderate (more boilerplate) | Fast (script-based) | -| **Learning Curve** | Steeper (callbacks, components) | Gentle (Pythonic, intuitive) | -| **Layout Control** | High (Bootstrap grid, custom CSS) | Medium (columns, containers) | -| **Styling** | Extensive (Bootstrap themes, CSS) | Limited (custom CSS via markdown) | -| **Callbacks** | Explicit (Input/Output decorators) | Automatic (reruns on interaction) | -| **State Management** | Manual (via callbacks) | Built-in (st.session_state) | -| **Performance** | Better for complex interactions | Slower (full page reruns) | -| **Best For** | Production dashboards, BI tools | Prototypes, data science demos | -| **Multi-page Apps** | Better routing support | Simpler but less flexible | -| **Data Science Fit** | Good (requires more setup) | Excellent (notebook-like) | -| **Code Complexity** | ~600 lines for full app | ~400 lines for full app | - -**Choose Dash when:** -- Building production-grade business intelligence dashboards -- Need precise control over layout and styling -- Require complex callback chains and interactions -- Want Bootstrap components and themes -- Building for non-technical business users - -**Choose Streamlit when:** -- Rapid prototyping and POCs -- Data science team building internal tools -- Simple data exploration and visualization -- ML model demos and experiments -- Prefer notebook-like development workflow - -For framework-specific details, see: -- **[dash.md](dash.md)** - Complete Dash implementation guide -- **[streamlit.md](streamlit.md)** - Complete Streamlit implementation guide -- **flask.md** - Flask patterns (coming soon) - -## Prerequisites Check - -1. Verify Python environment: `python --version` (3.9+) -2. Check for `uv` package manager: `uv --version` -3. Verify Databricks connectivity (if using real backend): - - `DATABRICKS_WAREHOUSE_ID` (required for SQL backend) - - Databricks CLI configured profile (SDK Config handles auth automatically) - - **Note:** No explicit tokens needed when using SDK Config approach - -## Core Architecture - -All Python Databricks apps follow this pattern: - -``` -app-directory/ -├── models.py # Pydantic data models -├── backend_mock.py # Mock backend with sample data -├── backend_real.py # Real Databricks backend -├── {framework}_app.py # Main application (dash_app.py, streamlit_app.py, etc.) -├── setup_database.py # Database initialization -├── requirements.txt # Python dependencies -├── app.yaml # Databricks Apps configuration -├── .env # Environment configuration -└── README.md # Documentation -``` - -### Framework-Specific Requirements - -**Dash (dash_app.py):** -```txt -dash>=2.14.0 -dash-bootstrap-components>=1.5.0 -pandas>=2.0.0 -plotly>=5.17.0 -pydantic>=2.0.0 -python-dotenv>=1.0.0 -databricks-sdk>=0.12.0 -databricks-sql-connector>=3.0.0 -``` - -**Streamlit (streamlit_app.py):** -```txt -streamlit>=1.28.0 -pandas>=2.0.0 -plotly>=5.17.0 -pydantic>=2.0.0 -python-dotenv>=1.0.0 -databricks-sdk>=0.12.0 -databricks-sql-connector>=3.0.0 -``` - -**Key Difference:** Dash requires `dash-bootstrap-components`, Streamlit doesn't need any additional UI libraries. - -## Workflow Overview - -### Phase 1: Planning & Models (10-15 min) -1. Understand requirements -2. Design data models -3. Create Pydantic models with validation -4. Create TodoWrite to track progress - -### Phase 2: Mock Backend (10-15 min) -1. Generate realistic sample data -2. Implement filtering and search -3. Create statistics methods -4. Test data generation - -### Phase 3: Application UI (20-30 min) -1. Set up framework structure -2. Create consistent styling -3. Build main pages/views -4. Add interactivity (filters, charts) -5. Implement data tables - -### Phase 4: Real Backend (15-20 min) -1. Design Unity Catalog schema -2. Implement SQL queries -3. Create database initialization -4. Add data seeding from mock - -### Phase 5: Testing & Documentation (10-15 min) -1. Test with mock backend -2. Test with real backend -3. Create comprehensive README -4. Add deployment instructions - -## Databricks Connectivity Patterns - -### Environment Configuration - -```python -# Standard environment variables -USE_MOCK_BACKEND=true|false # Toggle backend mode -DATABRICKS_WAREHOUSE_ID=... # SQL Warehouse ID (required) -DATABRICKS_CATALOG=main # Unity Catalog -DATABRICKS_SCHEMA=app_schema # Schema name -DATABRICKS_APP_PORT=8080 # Application port -DEBUG=false # Debug mode - -# Note: No DATABRICKS_TOKEN needed when using SDK Config -# Authentication handled automatically via: -# - Databricks CLI profile (local development) -# - Service principal (Databricks Apps) -``` - -### Backend Toggle Pattern - -```python -import os - -USE_MOCK = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true" - -if USE_MOCK: - from backend_mock import MockBackend - backend = MockBackend() -else: - from backend_real import RealBackend - backend = RealBackend() -``` - -### Pydantic Models Pattern - -```python -from pydantic import BaseModel, Field, field_validator -from decimal import Decimal -from datetime import datetime -from enum import Enum -from typing import List, Optional - -class StatusEnum(str, Enum): - """Status enumeration""" - ACTIVE = "active" - INACTIVE = "inactive" - -class Entity(BaseModel): - """Main entity model""" - id: str = Field(..., description="Unique identifier") - name: str = Field(..., description="Entity name") - created_at: datetime = Field(default_factory=datetime.utcnow) - status: StatusEnum = Field(default=StatusEnum.ACTIVE) - amount: Decimal = Field(..., description="Monetary amount", gt=0) - - @field_validator('amount', mode='before') - @classmethod - def validate_amount(cls, v): - """Ensure amount is a valid Decimal""" - if isinstance(v, (int, float, str)): - return Decimal(str(v)) - return v - - class Config: - json_schema_extra = { - "example": { - "id": "ENT-001", - "name": "Example Entity", - "status": "active", - "amount": "99.99" - } - } -``` - -### Mock Backend Pattern - -```python -from typing import List, Optional -from models import Entity - -class MockBackend: - """Mock backend with sample data""" - - def __init__(self): - self.entities = self._generate_entities() - - def _generate_entities(self) -> List[Entity]: - """Generate sample data""" - return [ - Entity(id="ENT-001", name="Entity 1", amount=Decimal("100.00")), - Entity(id="ENT-002", name="Entity 2", amount=Decimal("200.00")), - ] - - def get_entities(self, filter_criteria: Optional[dict] = None) -> List[Entity]: - """Get entities with optional filtering""" - results = self.entities - - if filter_criteria: - # Apply filters - if filter_criteria.get("status"): - results = [e for e in results if e.status == filter_criteria["status"]] - - return results - - def get_entity(self, entity_id: str) -> Optional[Entity]: - """Get specific entity""" - for entity in self.entities: - if entity.id == entity_id: - return entity - return None - - def get_statistics(self) -> dict: - """Get aggregated statistics""" - return { - "total_count": len(self.entities), - "total_amount": float(sum(e.amount for e in self.entities)) - } -``` - -### Real Backend Pattern (Databricks SQL) - -**Important:** For SQL Warehouse connection examples, see the Databricks Apps Cookbook: -- **Tables Read Example**: https://apps-cookbook.dev/docs/dash/tables/tables_read -- Shows proper service principal authentication using SDK Config - -```python -import os -from databricks import sql -from databricks.sdk import WorkspaceClient -from databricks.sdk.core import Config -from typing import List, Optional -from models import Entity - -class RealBackend: - """Real backend using Databricks SQL with SDK Config authentication""" - - def __init__(self, catalog: Optional[str] = None, schema: Optional[str] = None): - self.catalog = catalog or os.getenv("DATABRICKS_CATALOG", "main") - self.schema = schema or os.getenv("DATABRICKS_SCHEMA", "app_schema") - self.warehouse_id = os.getenv("DATABRICKS_WAREHOUSE_ID") - - if not self.warehouse_id: - raise ValueError("DATABRICKS_WAREHOUSE_ID required") - - self.config = Config() # Automatically handles authentication - self._connection = None - - def _get_connection(self): - """Get or create database connection using SDK Config""" - if self._connection is None: - self._connection = sql.connect( - server_hostname=self.config.host, - http_path=f"/sql/1.0/warehouses/{self.warehouse_id}", - credentials_provider=lambda: self.config.authenticate - ) - return self._connection - - def _execute_query(self, query: str, params: Optional[dict] = None) -> List[dict]: - """Execute SQL query and return results""" - connection = self._get_connection() - cursor = connection.cursor() - - try: - cursor.execute(query, params or {}) - columns = [desc[0] for desc in cursor.description] - results = [] - for row in cursor.fetchall(): - results.append(dict(zip(columns, row))) - return results - finally: - cursor.close() - - def get_entities(self, filter_criteria: Optional[dict] = None) -> List[Entity]: - """Get entities with optional filtering""" - query = f""" - SELECT * FROM {self.catalog}.{self.schema}.entities - WHERE 1=1 - """ - - params = {} - if filter_criteria and filter_criteria.get("status"): - query += " AND status = :status" - params["status"] = filter_criteria["status"] - - query += " ORDER BY created_at DESC" - - results = self._execute_query(query, params) - return [Entity(**row) for row in results] - - def initialize_schema(self): - """Initialize database schema""" - self._execute_query(f""" - CREATE TABLE IF NOT EXISTS {self.catalog}.{self.schema}.entities ( - id STRING NOT NULL, - name STRING NOT NULL, - created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP(), - status STRING NOT NULL, - amount DECIMAL(10, 2) NOT NULL, - PRIMARY KEY (id) - ) - """) - - def close(self): - """Close database connection""" - if self._connection: - self._connection.close() - self._connection = None -``` - -### Database Setup Script Pattern - -```python -"""Database setup script""" -import os -import argparse -from dotenv import load_dotenv -from backend_mock import MockBackend -from backend_real import RealBackend - -def setup_database(seed_data: bool = False): - """Initialize database and optionally seed data""" - load_dotenv() - - # Verify environment - required_vars = ["DATABRICKS_SERVER_HOSTNAME", "DATABRICKS_TOKEN", "DATABRICKS_WAREHOUSE_ID"] - missing = [v for v in required_vars if not os.getenv(v)] - if missing: - print(f"Missing: {', '.join(missing)}") - return 1 - - # Initialize backend - backend = RealBackend() - backend.initialize_schema() - - # Seed if requested - if seed_data: - mock = MockBackend() - # Copy data from mock to real backend - for entity in mock.entities: - backend.insert_entity(entity) - - backend.close() - return 0 - -if __name__ == "__main__": - parser = argparse.ArgumentParser() - parser.add_argument("--seed", action="store_true") - args = parser.parse_args() - exit(setup_database(seed_data=args.seed)) -``` - -## Best Practices - -### Data Models -- Use Pydantic for validation -- Include proper type hints -- Add `json_schema_extra` examples -- Handle Decimal for currency -- Use Enums for status fields - -### Backend Design -- Create both mock and real backends -- Use consistent interface between them -- Implement filtering and pagination -- Provide statistics/aggregations -- Use parameterized queries (security) - -### Error Handling -- Validate environment variables -- Handle connection failures gracefully -- Provide clear error messages -- Log errors appropriately - -### Configuration -- Use `.env` files for configuration -- Never commit secrets -- Provide `.env.example` template -- Support environment variable overrides - -### Testing Strategy -1. Start with mock backend (rapid development) -2. Test all features with sample data -3. Initialize real database with `--seed` -4. Test with real backend -5. Verify performance at scale - -## Common Patterns - -### Decimal Handling -```python -# Always convert to Decimal for monetary values -@field_validator('price', 'total', mode='before') -@classmethod -def validate_decimal(cls, v): - if isinstance(v, (int, float, str)): - return Decimal(str(v)) - return v -``` - -### Date Formatting -```python -# Consistent date formatting -order.order_date.strftime("%Y-%m-%d %H:%M") -order.created_at.isoformat() -``` - -### Status Colors -```python -# Map status to visual indicators -STATUS_COLORS = { - Status.ACTIVE: "#2CA02C", # Green - Status.PENDING: "#FF7F0E", # Orange - Status.FAILED: "#D62728", # Red -} -``` - -### Filtering Pattern -```python -# Reusable filter criteria model -class FilterCriteria(BaseModel): - status: Optional[Status] = None - date_from: Optional[datetime] = None - date_to: Optional[datetime] = None - search: Optional[str] = None -``` - -## Success Criteria - -- [ ] Pydantic models with proper validation -- [ ] Mock backend with realistic data -- [ ] Framework UI with consistent styling -- [ ] Real backend with Unity Catalog -- [ ] Database initialization script -- [ ] Environment configuration -- [ ] Comprehensive documentation -- [ ] Both backends tested and working - -## Troubleshooting - -**First Step: Check Application Logs** -```bash -# Always check logs first when troubleshooting -databricks apps logs --profile - -# Examples: -databricks apps logs order-management-dash-dev -p DEFAULT -databricks apps logs order-management-streamlit-dev -p DEFAULT -``` - -Logs reveal: -- Deployment errors and stack traces -- Backend connection status (look for "✅ Initialized real backend") -- Missing dependencies or import errors -- SQL connection failures -- App startup issues - -**Connection Issues** -- Verify Databricks CLI profile is configured: `databricks auth profiles` -- Check `DATABRICKS_WAREHOUSE_ID` exists and is accessible -- Ensure warehouse is running: `databricks warehouses get ` -- Verify network connectivity to workspace -- For service principal: Check permissions on warehouse and catalog -- **Check logs for connection errors:** `databricks apps logs ` - -**Data Type Errors** -- Use Decimal for monetary values -- Handle None/Optional properly -- Validate datetime parsing - -**Performance Issues** -- Add database indexes -- Implement pagination -- Use query result caching -- Optimize SQL queries - -## Deployment to Databricks - -### Ask User for Deployment Preference - -**IMPORTANT:** Before deploying, ask the user which deployment method they prefer: - -1. **Databricks CLI** - Simple, direct deployment using `databricks apps` commands -2. **Databricks Asset Bundles (DABs)** - Infrastructure-as-code approach with version control - -Example: "Would you like to deploy using Databricks CLI or Databricks Asset Bundles (DABs)?" - -### Option 1: Deploy with Databricks CLI - -**Prerequisites:** -- Databricks CLI installed -- Authenticated profile configured -- SQL Warehouse ID available - -**Steps:** - -1. **Create app.yaml** - -**For Dash apps:** -```yaml -command: - - "python" - - "dash_app.py" - -env: - - name: USE_MOCK_BACKEND - value: "false" - - name: DATABRICKS_WAREHOUSE_ID - value: "your-warehouse-id" - - name: DATABRICKS_CATALOG - value: "main" - - name: DATABRICKS_SCHEMA - value: "app_schema" - - name: DATABRICKS_APP_PORT - value: "8080" - - name: DEBUG - value: "false" -``` - -**For Streamlit apps:** -```yaml -command: - - "streamlit" - - "run" - - "streamlit_app.py" - - "--server.port" - - "8080" - - "--server.address" - - "0.0.0.0" - -env: - - name: USE_MOCK_BACKEND - value: "false" - - name: DATABRICKS_WAREHOUSE_ID - value: "your-warehouse-id" - - name: DATABRICKS_CATALOG - value: "main" - - name: DATABRICKS_SCHEMA - value: "app_schema" -``` - -**Note:** Streamlit uses `streamlit run` command, while Dash uses `python`. Streamlit doesn't need `DATABRICKS_APP_PORT` env var as it's specified in the command. - -2. **Initialize database schema** -```bash -# Run setup script locally (requires profile configured) -python setup_database.py --seed -``` - -3. **Create Databricks app** -```bash -databricks apps create --profile -``` - -4. **Upload source code to workspace** -```bash -databricks workspace mkdirs /Workspace/Users//apps/ --profile -databricks workspace import-dir . /Workspace/Users//apps/ --profile -``` - -5. **Deploy the app** -```bash -databricks apps deploy \ - --source-code-path /Workspace/Users//apps/ \ - --profile -``` - -6. **Get app URL** -```bash -databricks apps get --profile -``` - -**Redeployment:** -```bash -# Update workspace files -databricks workspace delete /Workspace/Users//apps/ --recursive --profile -databricks workspace mkdirs /Workspace/Users//apps/ --profile -databricks workspace import-dir . /Workspace/Users//apps/ --profile - -# Redeploy -databricks apps deploy \ - --source-code-path /Workspace/Users//apps/ \ - --profile -``` - -### Option 2: Deploy with Databricks Asset Bundles (DABs) - -**Prerequisites:** -- Databricks CLI installed (v0.239.0+) -- App already deployed via CLI (recommended workflow) - -**Advantages:** -- Version controlled deployment -- Multi-environment support (dev/staging/prod) -- Declarative infrastructure -- Easier CI/CD integration - -**Recommended Workflow: CLI First, Then DABs** - -1. **Deploy app using CLI first** (see Option 1 above) - - This creates the app and validates everything works - - Easier to debug issues initially - -2. **Generate bundle configuration from existing app** -```bash -# This creates resources/*.app.yml and downloads source to src/app/ -databricks bundle generate app \ - --existing-app-name \ - --key \ - --profile - -# Example: -databricks bundle generate app \ - --existing-app-name order-management-dash \ - --key order_management_dash \ - --profile DEFAULT -``` - -**What gets generated:** -- `resources/.app.yml` - Minimal app resource definition -- `src/app/` - All app source files including `app.yaml` with env vars -- `databricks.yml` updated with bundle structure - -3. **Update generated configuration for multi-environment** - -**Edit `databricks.yml`:** -```yaml -bundle: - name: - -include: - - resources/*.yml - -variables: - warehouse_id: - default: "your-warehouse-id" - catalog: - default: "main" - schema: - default: "app_schema" - -targets: - dev: - default: true - mode: development - workspace: - profile: - variables: - warehouse_id: "dev-warehouse-id" - schema: "app_schema_dev" - - prod: - mode: production - workspace: - profile: - variables: - warehouse_id: "prod-warehouse-id" - schema: "app_schema_prod" -``` - -**Edit `resources/.app.yml`:** -```yaml -resources: - apps: - : - name: -${bundle.target} # Environment-specific naming - description: "Python ${framework} application" - source_code_path: ../src/app # Or .. if source in project root -``` - -**Important:** Environment variables are in `src/app/app.yaml`, NOT in databricks.yml: -```yaml -command: - - "python" - - "dash_app.py" - -env: - - name: USE_MOCK_BACKEND - value: "false" - - name: DATABRICKS_WAREHOUSE_ID - value: "your-warehouse-id" - - name: DATABRICKS_CATALOG - value: "main" - - name: DATABRICKS_SCHEMA - value: "app_schema" -``` - -4. **Deploy and run** -```bash -# Validate configuration -databricks bundle validate -t dev - -# Deploy to dev (creates/updates resource) -databricks bundle deploy -t dev - -# Start the app (required after deployment) -databricks bundle run -t dev - -# For production -databricks bundle deploy -t prod -databricks bundle run -t prod -``` - -**Key Differences from Other Resources:** -- Environment variables go in `app.yaml` (source dir), NOT databricks.yml -- Apps have minimal bundle configuration (name, description, path) -- Must run `databricks bundle run` to start the app after deployment - -**For complete DABs guidance, use the `asset-bundles` skill.** - -### Post-Deployment Steps - -1. **Verify deployment** - - Access app URL - - Check all pages load - - Verify data from Unity Catalog - -2. **Configure permissions** - - Set up user access - - Configure service principal permissions - - Grant warehouse access - -3. **Set up monitoring and view logs** - - **View application logs:** - ```bash - # View logs for your deployed app - databricks apps logs --profile - - # Examples: - databricks apps logs order-management-dash-dev --profile DEFAULT - databricks apps logs order-management-streamlit-dev --profile DEFAULT - ``` - - **What logs show:** - - `[SYSTEM]` - Deployment status, file updates, dependency installation - - `[APP]` - Application output (print statements, framework messages) - - Backend initialization messages - - Connection status to Unity Catalog - - Error messages and stack traces - - **Useful for debugging:** - - ✅ Verify real backend connection: Look for "✅ Initialized real backend: main.schema" - - ✅ Check dependency installation: "Requirements installed successfully" - - ✅ Confirm app start: "App started successfully" - - ✅ Diagnose connection errors: SQL connection failures - - ✅ Track deployments: Each deployment has unique ID - - **Additional monitoring:** - - Monitor warehouse usage in Databricks SQL - - Track app performance and response times - - Set up alerts for app failures - -4. **Documentation** - - Update README with deployment URL - - Document environment variables - - Add troubleshooting guide - -## Reference Materials - -For framework-specific implementation details: -- **[dash.md](dash.md)** - Complete Dash implementation guide with Bootstrap components -- **[streamlit.md](streamlit.md)** - Complete Streamlit implementation guide with caching patterns -- **flask.md** - Flask patterns (coming soon) diff --git a/.claude/skills/databricks-app-python/dash.md b/.claude/skills/databricks-app-python/dash.md deleted file mode 100644 index 82e42ecc..00000000 --- a/.claude/skills/databricks-app-python/dash.md +++ /dev/null @@ -1,553 +0,0 @@ -# Dash Framework Guide - -Complete guide for building Databricks applications with Plotly Dash framework. - -## When to Use Dash - -**Best for**: -- Interactive dashboards with rich charts -- Business intelligence applications -- Data visualization heavy apps -- Multi-page applications -- Apps requiring custom styling with Bootstrap - -**Alternatives**: -- Streamlit - Simpler syntax, faster prototyping -- APX - Full-stack with React frontend - -## Dependencies - -```txt -dash>=2.14.0 -dash-bootstrap-components>=1.5.0 -plotly>=5.18.0 -pandas>=2.0.0 -databricks-sdk>=0.35.0 -databricks-sql-connector>=3.0.0 -pydantic>=2.0.0 -python-dotenv>=1.0.0 -``` - -## Project Structure - -``` -dash-app/ -├── models.py # Pydantic data models -├── backend_mock.py # Mock backend with sample data -├── backend_real.py # Databricks SQL backend -├── dash_app.py # Main Dash application -├── setup_database.py # Database initialization -├── requirements.txt # Dependencies -├── .env.example # Environment template -├── run_app.sh # Quick start script -└── README.md # Documentation -``` - -## Dash Application Structure - -### Basic Setup - -```python -import os -import dash -from dash import dcc, html, dash_table, Input, Output, State, callback -import dash_bootstrap_components as dbc -import plotly.express as px -import pandas as pd - -from backend_mock import MockBackend -from models import Status - -# Initialize backend -USE_MOCK = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true" -backend = MockBackend() if USE_MOCK else RealBackend() - -# Initialize Dash app with Bootstrap theme -app = dash.Dash( - __name__, - external_stylesheets=[dbc.themes.BOOTSTRAP, dbc.icons.FONT_AWESOME], - suppress_callback_exceptions=True, - title="Application Name" -) - -# Define color scheme -COLORS = { - "primary": "#1F77B4", - "success": "#2CA02C", - "warning": "#FF7F0E", - "danger": "#D62728", - "info": "#17A2B8", - "light": "#F8F9FA", - "dark": "#343A40", -} - -# Status color mappings -STATUS_COLORS = { - Status.ACTIVE: COLORS["success"], - Status.PENDING: COLORS["warning"], - Status.FAILED: COLORS["danger"], -} - -# Bootstrap badge colors (for dbc.Badge) -STATUS_BADGE_COLORS = { - Status.ACTIVE: "success", - Status.PENDING: "warning", - Status.FAILED: "danger", -} -``` - -### Navigation Bar - -```python -def create_navbar(): - """Create navigation bar""" - return dbc.Navbar( - dbc.Container([ - dbc.Row([ - dbc.Col([ - html.I(className="fas fa-chart-line me-2"), - dbc.NavbarBrand("Application Name", className="ms-2"), - ], width="auto"), - ], align="center", className="g-0"), - dbc.Nav([ - dbc.NavItem(dbc.NavLink("Dashboard", href="/", active="exact")), - dbc.NavItem(dbc.NavLink("Orders", href="/orders", active="exact")), - dbc.NavItem(dbc.NavLink("Customers", href="/customers", active="exact")), - ], navbar=True, className="ms-auto"), - ], fluid=True), - color="dark", - dark=True, - className="mb-4" - ) -``` - -### Main Layout with Routing - -```python -app.layout = html.Div([ - dcc.Location(id='url', refresh=False), - create_navbar(), - html.Div(id='page-content', style={'minHeight': '80vh'}), - dcc.Store(id='selected-item-id'), # Client-side data storage -], style={'backgroundColor': COLORS["light"], 'minHeight': '100vh'}) - -@callback( - Output('page-content', 'children'), - Input('url', 'pathname') -) -def display_page(pathname): - """Route to different pages""" - if pathname == '/orders': - return create_orders_layout() - elif pathname == '/customers': - return create_customers_layout() - else: - return create_dashboard_layout() -``` - -## Component Patterns - -### Statistics Card - -```python -def create_stat_card(title, value, icon, color="primary", subtitle=None): - """Create a statistics card""" - return dbc.Card([ - dbc.CardBody([ - html.Div([ - html.Div([ - html.H6(title, className="text-muted mb-2"), - html.H3(value, className="mb-0"), - html.Small(subtitle, className="text-muted") if subtitle else None, - ], className="flex-grow-1"), - html.Div([ - html.I(className=f"fas {icon} fa-2x text-{color}") - ], className="ms-3"), - ], className="d-flex align-items-center"), - ]), - ], className="shadow-sm mb-3") - -# Usage -dbc.Row([ - dbc.Col(create_stat_card( - "Total Orders", - f"{stats['total_orders']:,}", - "fa-shopping-cart", - "primary" - ), md=3), - dbc.Col(create_stat_card( - "Total Revenue", - f"${stats['total_revenue']:,.2f}", - "fa-dollar-sign", - "success" - ), md=3), -]) -``` - -### Interactive Data Table - -```python -def create_data_table(data, table_id, selectable=False): - """Create interactive data table""" - if not data: - return html.Div("No data available.", className="text-muted") - - return dash_table.DataTable( - id=table_id, - data=data, - columns=[{"name": col, "id": col} for col in data[0].keys()], - page_size=20, - style_table={'overflowX': 'auto'}, - style_cell={ - 'textAlign': 'left', - 'padding': '12px', - 'fontFamily': 'Arial, sans-serif' - }, - style_header={ - 'backgroundColor': COLORS["dark"], - 'color': 'white', - 'fontWeight': 'bold' - }, - style_data_conditional=[ - { - 'if': {'row_index': 'odd'}, - 'backgroundColor': COLORS["light"] - } - ], - filter_action="native", - sort_action="native", - row_selectable='single' if selectable else False, - ) -``` - -### Plotly Charts - -```python -# Pie Chart -@callback( - Output('status-pie-chart', 'figure'), - Input('url', 'pathname') -) -def update_pie_chart(pathname): - """Create pie chart for status distribution""" - stats = backend.get_statistics() - status_data = stats['status_distribution'] - - fig = px.pie( - values=list(status_data.values()), - names=[s.title() for s in status_data.keys()], - color_discrete_sequence=px.colors.qualitative.Set3 - ) - fig.update_layout( - margin=dict(t=20, b=20, l=20, r=20), - showlegend=True, - height=300 - ) - return fig - -# Bar Chart -@callback( - Output('revenue-bar-chart', 'figure'), - Input('url', 'pathname') -) -def update_bar_chart(pathname): - """Create bar chart for revenue by category""" - data = backend.get_revenue_by_category() - - # Create color map with error handling - color_map = {} - for category in data.keys(): - color_map[category] = COLORS.get(category.lower(), COLORS["primary"]) - - fig = px.bar( - x=list(data.keys()), - y=list(data.values()), - labels={'x': 'Category', 'y': 'Revenue ($)'}, - color=list(data.keys()), - color_discrete_map=color_map - ) - fig.update_layout( - margin=dict(t=20, b=40, l=40, r=20), - showlegend=False, - height=300, - xaxis_title="", - yaxis_title="Revenue ($)" - ) - fig.update_xaxes(tickangle=-45) # Note: update_xaxes, not update_xaxis - return fig -``` - - -## Callback Patterns - -### Basic Callback - -```python -@callback( - Output('output-div', 'children'), - Input('input-button', 'n_clicks') -) -def update_output(n_clicks): - """Basic callback pattern""" - if n_clicks is None: - return "Click the button" - return f"Button clicked {n_clicks} times" -``` - -### Multiple Inputs - -```python -@callback( - Output('filtered-table', 'children'), - [Input('filter-status', 'value'), - Input('filter-date', 'value'), - Input('refresh-button', 'n_clicks')] -) -def update_table(status, date, n_clicks): - """Callback with multiple inputs""" - filter_criteria = { - "status": Status(status) if status else None, - "date": date - } - data = backend.get_data(filter_criteria) - return create_data_table(data, "result-table") -``` - -### Using State (Non-Triggering Inputs) - -```python -@callback( - Output('result', 'children'), - Input('submit-button', 'n_clicks'), - [State('input-field', 'value'), - State('dropdown', 'value')] -) -def process_form(n_clicks, input_value, dropdown_value): - """State doesn't trigger callback, only provides values""" - if n_clicks is None: - return "" - return f"Processing: {input_value}, {dropdown_value}" -``` - -### Callback Context - -```python -@callback( - Output('result', 'children'), - [Input('button1', 'n_clicks'), - Input('button2', 'n_clicks')] -) -def handle_multiple_buttons(n1, n2): - """Determine which input triggered the callback""" - ctx = dash.callback_context - - if not ctx.triggered: - return "No button clicked" - - trigger_id = ctx.triggered[0]['prop_id'].split('.')[0] - - if trigger_id == 'button1': - return "Button 1 clicked" - elif trigger_id == 'button2': - return "Button 2 clicked" - - return "Unknown trigger" -``` - -## Complete Page Example - -```python -def create_orders_layout(): - """Complete orders page with table, filters, and modal""" - return dbc.Container([ - html.H2("Orders", className="mb-4"), - - # Filters - create_filters(), - - # Data Table - dbc.Card([ - dbc.CardHeader(html.H5([ - html.I(className="fas fa-table me-2"), - "Order List" - ])), - dbc.CardBody([ - html.Div(id="orders-table") - ]), - ], className="shadow-sm mb-4"), - - # Detail Modal - create_detail_modal(), - ], fluid=True) - -@callback( - Output('orders-table', 'children'), - [Input('filter-status', 'value'), - Input('refresh-button', 'n_clicks')] -) -def update_orders_table(status, n_clicks): - """Update orders table with filters""" - filter_criteria = {"status": Status(status) if status else None} - orders = backend.get_orders(filter_criteria) - - if not orders: - return html.Div("No orders found.", className="text-muted") - - order_data = [ - { - "Order ID": o.order_id, - "Customer": o.customer_name, - "Date": o.order_date.strftime("%Y-%m-%d %H:%M"), - "Status": o.status.value.title(), - "Total": f"${float(o.total):.2f}", - } - for o in orders - ] - - return create_data_table(order_data, "orders-data-table", selectable=True) -``` - -## Best Practices - -### Performance -1. **Use `dcc.Store`** for client-side caching -2. **Implement pagination** for large datasets -3. **Use `prevent_initial_call=True`** for expensive operations -4. **Minimize callback dependencies** -5. **Cache backend queries** when appropriate - - -### Consistent Styling -```python -# Define color constants at top of file -COLORS = {...} -STATUS_COLORS = {...} -STATUS_BADGE_COLORS = {...} - -# Use consistently throughout app -dbc.Badge(status, color=STATUS_BADGE_COLORS[status]) -``` - -## Common Pitfalls - -### ❌ Wrong: Missing ID on dynamically created component -```python -def update_table(): - return dash_table.DataTable( - # Missing id! - data=data, - columns=columns - ) -``` - -### ✅ Correct: Always provide ID -```python -def update_table(): - return dash_table.DataTable( - id='dynamic-table', # Always include id - data=data, - columns=columns - ) -``` - -### ❌ Wrong: Accessing data before checking if exists -```python -@callback(...) -def toggle_modal(selected_rows, table_data, is_open): - item_id = table_data[selected_rows[0]]["id"] # May fail! -``` - -### ✅ Correct: Check before accessing -```python -@callback(..., prevent_initial_call=True) -def toggle_modal(selected_rows, table_data, is_open): - if not selected_rows or not table_data: - return False, "", "" - item_id = table_data[selected_rows[0]]["id"] # Safe -``` - -### ❌ Wrong: Using hex colors for Bootstrap badges -```python -dbc.Badge(status, color="#2CA02C") # Won't work! -``` - -### ✅ Correct: Use Bootstrap color names -```python -dbc.Badge(status, color="success") # Correct -``` - -### ❌ Wrong: Plotly method typo -```python -fig.update_xaxis(tickangle=-45) # AttributeError! -``` - -### ✅ Correct: Use plural form -```python -fig.update_xaxes(tickangle=-45) # Correct -``` - -## Running the App - -### Development Mode -```bash -# With uv -USE_MOCK_BACKEND=true DEBUG=true DATABRICKS_APP_PORT=8080 uv run python dash_app.py - -# With python directly -USE_MOCK_BACKEND=true DEBUG=true python dash_app.py -``` - -### Production Mode -```python -if __name__ == '__main__': - port = int(os.getenv("DATABRICKS_APP_PORT", "8080")) - debug = os.getenv("DEBUG", "false").lower() == "true" - - app.run(host='0.0.0.0', port=port, debug=debug) -``` - -## Deployment to Databricks - -### app.yaml -```yaml -command: - - "python" - - "dash_app.py" - -env: - - name: USE_MOCK_BACKEND - value: "false" - - name: DATABRICKS_CONFIG_PROFILE - value: "" - - name: DATABRICKS_APP_PORT - value: "8080" -``` - -### Deploy Commands -```bash -databricks apps deploy -databricks apps list -databricks apps logs -``` - - -## Additional Resources - -- **Plotly Dash Docs**: https://dash.plotly.com/ -- **Dash Bootstrap Components**: https://dash-bootstrap-components.opensource.faculty.ai/ -- **Plotly Charts**: https://plotly.com/python/ -- **Example App Snippets**: https://apps-cookbook.dev/docs/category/dash - -## Success Checklist - -- [ ] App runs with mock backend -- [ ] All pages render without errors -- [ ] Callbacks work correctly -- [ ] Filters update data tables -- [ ] Charts display properly -- [ ] Modals open and close -- [ ] Consistent styling throughout -- [ ] Empty states handled -- [ ] Error handling in callbacks -- [ ] Real backend tested -- [ ] Documentation complete diff --git a/.claude/skills/databricks-app-python/streamlit.md b/.claude/skills/databricks-app-python/streamlit.md deleted file mode 100644 index d50646e8..00000000 --- a/.claude/skills/databricks-app-python/streamlit.md +++ /dev/null @@ -1,790 +0,0 @@ -# Streamlit Framework Implementation Guide - -Complete guide for building Databricks applications with Streamlit framework. - -## Table of Contents - -- [When to Use Streamlit](#when-to-use-streamlit) -- [Core Architecture](#core-architecture) -- [Essential Patterns](#essential-patterns) -- [Best Practices](#best-practices) -- [Component Guide](#component-guide) -- [Common Pitfalls](#common-pitfalls) -- [Performance Optimization](#performance-optimization) - ---- - -## When to Use Streamlit - -**Choose Streamlit when you need:** -- Rapid prototyping and development -- Data science and ML-focused applications -- Simple, script-like development workflow -- Built-in widgets and forms -- Interactive data exploration tools -- ML model demos and POCs - -**Key Strengths:** -- ✅ Fastest development time (script-based) -- ✅ Excellent for data scientists (Pythonic) -- ✅ Built-in state management -- ✅ Automatic reactivity (reruns on interaction) -- ✅ Great for notebooks-to-apps workflow - -**Limitations:** -- ❌ Less control over layout (compared to Dash) -- ❌ Full page reruns can be slower -- ❌ Harder to build complex multi-page apps -- ❌ Limited styling customization - ---- - -## Core Architecture - -### Application Structure - -```python -""" -Streamlit App Structure -""" -import streamlit as st -from databricks.sdk.core import Config -from databricks import sql - -# 1. Page configuration (MUST be first Streamlit command) -st.set_page_config( - page_title="My App", - page_icon="📊", - layout="wide", # or "centered" - initial_sidebar_state="expanded" # or "collapsed" -) - -# 2. Backend initialization with caching -@st.cache_resource -def get_backend(): - """Initialize and cache backend connection""" - # Your backend initialization - return backend - -# 3. Initialize backend -backend = get_backend() - -# 4. Sidebar navigation -page = st.sidebar.radio("Navigation", ["Page 1", "Page 2"]) - -# 5. Page content -if page == "Page 1": - # Page 1 content - pass -elif page == "Page 2": - # Page 2 content - pass -``` - -### File Organization - -``` -streamlit-app/ -├── streamlit_app.py # Main application entry point -├── models.py # Pydantic data models -├── backend_mock.py # Mock backend with sample data -├── backend_real.py # Real Databricks backend -├── setup_database.py # Database initialization -├── requirements.txt # Python dependencies -├── app.yaml # Databricks Apps configuration -├── .env # Environment variables -└── README.md # Documentation -``` - ---- - -## Essential Patterns - -### 1. Connection Caching (Critical) - -**Always use `@st.cache_resource` for database connections:** - -```python -from databricks.sdk.core import Config -from databricks import sql - -@st.cache_resource(ttl=300, show_spinner=True) -def get_sql_connection(http_path: str): - """ - Create and cache SQL warehouse connection - - Args: - ttl: Time-to-live in seconds (5 minutes default) - show_spinner: Show loading indicator during initialization - """ - cfg = Config() # Reads DATABRICKS_HOST automatically - - return sql.connect( - server_hostname=cfg.host, - http_path=http_path, - credentials_provider=lambda: cfg.authenticate - ) - -# Usage -conn = get_sql_connection("/sql/1.0/warehouses/xxxxx") -``` - -**Why `@st.cache_resource`?** -- Persists across sessions and reruns -- Prevents connection exhaustion -- Improves performance dramatically -- Required for production apps - -### 2. Backend Toggle Pattern - -```python -import os - -@st.cache_resource -def get_backend(): - """Initialize backend based on environment""" - use_mock = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true" - - if use_mock: - from backend_mock import MockBackend - return MockBackend() - else: - from backend_real import RealBackend - catalog = os.getenv("DATABRICKS_CATALOG", "main") - schema = os.getenv("DATABRICKS_SCHEMA", "app_schema") - return RealBackend(catalog=catalog, schema=schema) -``` - -### 3. Session State Management - -**Use `st.session_state` to persist data across reruns:** - -```python -# Initialize state -if 'order_id' not in st.session_state: - st.session_state.order_id = None - -# Set state -if st.button("Load Order"): - st.session_state.order_id = "ORD-001" - -# Read state -if st.session_state.order_id: - st.write(f"Current order: {st.session_state.order_id}") -``` - -**Common State Patterns:** - -```python -# Form data -if 'form_data' not in st.session_state: - st.session_state.form_data = {} - -# Page navigation -if 'current_page' not in st.session_state: - st.session_state.current_page = "Dashboard" - -# Filter persistence -if 'filters' not in st.session_state: - st.session_state.filters = { - 'status': [], - 'date_from': None, - 'date_to': None - } -``` - -### 4. Data Display Patterns - -**DataFrames (Read-only):** - -```python -import pandas as pd - -df = backend.get_orders() -st.dataframe( - df, - use_container_width=True, # Expand to container width - hide_index=True, # Hide row numbers - column_config={ - "amount": st.column_config.NumberColumn( - "Amount", - format="$%.2f" - ), - "status": st.column_config.SelectColumn( - "Status", - options=["pending", "confirmed", "shipped"] - ) - } -) -``` - -**Data Editor (Editable):** - -```python -# For editable tables -edited_df = st.data_editor( - df, - num_rows="dynamic", # Allow add/delete rows - hide_index=True, - column_config={ - "amount": st.column_config.NumberColumn( - "Amount", - min_value=0, - max_value=10000, - step=0.01, - format="$%.2f" - ) - } -) - -# Detect changes -if st.button("Save Changes"): - # Compare original vs edited - df_diff = pd.concat([df, edited_df]).drop_duplicates(keep=False) - if not df_diff.empty: - backend.update_data(edited_df) - st.success("Changes saved!") -``` - -### 5. Sidebar Navigation - -```python -# Sidebar navigation pattern -st.sidebar.title("📊 My App") -st.sidebar.markdown("---") - -page = st.sidebar.radio( - "Navigation", - ["Dashboard", "Orders", "Customers", "Products"], - label_visibility="collapsed" # Hide "Navigation" label -) - -# Filters in sidebar -st.sidebar.markdown("---") -st.sidebar.markdown("### Filters") - -status_filter = st.sidebar.multiselect( - "Status", - options=["pending", "confirmed", "shipped"], - default=None -) - -date_range = st.sidebar.date_input( - "Date Range", - value=None -) -``` - -### 6. Metrics Display - -```python -# Four-column metrics -col1, col2, col3, col4 = st.columns(4) - -with col1: - st.metric( - label="Total Orders", - value="1,234", - delta="12%", # Optional change indicator - delta_color="normal" # "normal", "inverse", or "off" - ) - -with col2: - st.metric( - label="Revenue", - value="$45,678", - delta="-8%", - delta_color="inverse" # Red for negative when inverse - ) -``` - -### 7. Charts Integration - -**Plotly Charts (Recommended):** - -```python -import plotly.express as px -import plotly.graph_objects as go - -# Pie chart -fig = px.pie( - df, - values='count', - names='status', - color='status', - color_discrete_map={'pending': '#FFA500', 'confirmed': '#2CA02C'} -) -st.plotly_chart(fig, use_container_width=True) - -# Bar chart -fig = px.bar( - df, - x='month', - y='revenue', - color='category' -) -fig.update_layout(showlegend=False) -st.plotly_chart(fig, use_container_width=True) -``` - -**Native Streamlit Charts:** - -```python -# For simple charts (less customizable but faster) -st.line_chart(df[['date', 'revenue']]) -st.bar_chart(df[['category', 'count']]) -st.area_chart(df[['date', 'cumulative_revenue']]) -``` - -### 8. Forms and Inputs - -**Form Pattern (Prevents reruns on every input change):** - -```python -with st.form("order_form"): - st.write("Create New Order") - - customer = st.text_input("Customer Name") - product = st.selectbox("Product", ["Product A", "Product B"]) - quantity = st.number_input("Quantity", min_value=1, value=1) - notes = st.text_area("Notes") - - # Form is only submitted when button is clicked - submitted = st.form_submit_button("Create Order") - - if submitted: - # Process form data - backend.create_order(customer, product, quantity, notes) - st.success("Order created!") -``` - -**Without Forms (Immediate reactivity):** - -```python -# These trigger reruns on every change -name = st.text_input("Name") -age = st.slider("Age", 0, 100) - -if st.button("Submit"): - # Only runs when button clicked - st.write(f"{name} is {age} years old") -``` - ---- - -## Best Practices - -### Page Configuration - -**✅ ALWAYS set page config first:** - -```python -# MUST be the first Streamlit command -st.set_page_config( - page_title="Order Management", - page_icon="📦", - layout="wide", - initial_sidebar_state="expanded" -) -``` - -**❌ Common mistake:** - -```python -import streamlit as st - -st.title("My App") # ❌ Error: set_page_config must be first -st.set_page_config(...) # ❌ Too late! -``` - -### State Management - -**✅ Initialize state at the top:** - -```python -# Initialize all state variables together -if 'user_id' not in st.session_state: - st.session_state.user_id = None -if 'filters' not in st.session_state: - st.session_state.filters = {} -``` - -**❌ Don't initialize in conditionals:** - -```python -# ❌ Bad: state only initialized if condition is true -if some_condition: - if 'user_id' not in st.session_state: - st.session_state.user_id = None -``` - -### Caching - -**✅ Cache expensive operations:** - -```python -@st.cache_resource # For connections, models -def get_connection(): - return create_connection() - -@st.cache_data(ttl=60) # For data, with TTL -def load_data(): - return fetch_data() -``` - -**When to use which:** -- `@st.cache_resource`: Connections, ML models, non-serializable objects -- `@st.cache_data`: DataFrames, lists, dicts, serializable data - -### Layout Organization - -**✅ Use columns for horizontal layout:** - -```python -col1, col2, col3 = st.columns([2, 1, 1]) # Ratios: 2:1:1 - -with col1: - st.write("Main content") - -with col2: - st.write("Sidebar content") -``` - -**✅ Use expanders for collapsible sections:** - -```python -with st.expander("Advanced Filters"): - filter1 = st.selectbox("Filter 1", options) - filter2 = st.multiselect("Filter 2", options) -``` - -**✅ Use tabs for switching content:** - -```python -tab1, tab2, tab3 = st.tabs(["Tab 1", "Tab 2", "Tab 3"]) - -with tab1: - st.write("Tab 1 content") - -with tab2: - st.write("Tab 2 content") -``` - -### Error Handling - -**✅ Graceful error handling:** - -```python -try: - data = backend.get_data() - st.dataframe(data) -except Exception as e: - st.error(f"Error loading data: {str(e)}") - st.info("Please check your connection and try again") -``` - -**✅ Input validation:** - -```python -user_input = st.text_input("Enter warehouse ID") - -if user_input: - if not user_input.startswith("/sql/"): - st.warning("Warehouse path should start with /sql/") - else: - # Process input - pass -``` - ---- - -## Component Guide - -### Core Components - -| Component | Use Case | Example | -|-----------|----------|---------| -| `st.title()` | Page titles | `st.title("Dashboard")` | -| `st.header()` | Section headers | `st.header("Overview")` | -| `st.subheader()` | Subsection headers | `st.subheader("Metrics")` | -| `st.text()` | Plain text | `st.text("Simple text")` | -| `st.markdown()` | Formatted text | `st.markdown("**Bold** text")` | -| `st.write()` | Auto-formatted | `st.write("Text", df, chart)` | - -### Input Widgets - -| Widget | Use Case | Example | -|--------|----------|---------| -| `st.button()` | Actions | `if st.button("Submit"):` | -| `st.text_input()` | Single-line text | `name = st.text_input("Name")` | -| `st.text_area()` | Multi-line text | `notes = st.text_area("Notes")` | -| `st.number_input()` | Numbers | `age = st.number_input("Age", 0, 100)` | -| `st.selectbox()` | Single selection | `choice = st.selectbox("Pick", options)` | -| `st.multiselect()` | Multiple selection | `choices = st.multiselect("Pick", options)` | -| `st.slider()` | Range selection | `value = st.slider("Value", 0, 100)` | -| `st.date_input()` | Date picker | `date = st.date_input("Date")` | -| `st.checkbox()` | Boolean toggle | `if st.checkbox("Agree"):` | -| `st.radio()` | Single choice | `choice = st.radio("Pick", options)` | - -### Display Components - -| Component | Use Case | Example | -|-----------|----------|---------| -| `st.dataframe()` | Read-only tables | `st.dataframe(df)` | -| `st.data_editor()` | Editable tables | `edited = st.data_editor(df)` | -| `st.metric()` | KPI displays | `st.metric("Sales", "$1M", "+10%")` | -| `st.json()` | JSON display | `st.json({"key": "value"})` | -| `st.code()` | Code blocks | `st.code("print('hello')")` | - -### Layout Components - -| Component | Use Case | Example | -|-----------|----------|---------| -| `st.columns()` | Side-by-side | `col1, col2 = st.columns(2)` | -| `st.expander()` | Collapsible | `with st.expander("Details"):` | -| `st.tabs()` | Tabbed interface | `tab1, tab2 = st.tabs(["A", "B"])` | -| `st.container()` | Grouping | `with st.container():` | -| `st.empty()` | Placeholder | `placeholder = st.empty()` | - -### Status Components - -| Component | Use Case | Example | -|-----------|----------|---------| -| `st.success()` | Success message | `st.success("Saved!")` | -| `st.error()` | Error message | `st.error("Failed!")` | -| `st.warning()` | Warning message | `st.warning("Caution!")` | -| `st.info()` | Info message | `st.info("Note: ...")` | -| `st.spinner()` | Loading indicator | `with st.spinner("Loading..."):` | -| `st.progress()` | Progress bar | `st.progress(0.5)` | - ---- - -## Common Pitfalls - -### 1. Page Config Not First - -**❌ Wrong:** -```python -import streamlit as st -st.title("My App") -st.set_page_config(...) # Error! -``` - -**✅ Correct:** -```python -import streamlit as st -st.set_page_config(...) # Must be first! -st.title("My App") -``` - -### 2. Not Caching Connections - -**❌ Wrong:** -```python -def get_data(): - conn = sql.connect(...) # Creates new connection every rerun! - return conn.cursor().fetchall() -``` - -**✅ Correct:** -```python -@st.cache_resource -def get_connection(): - return sql.connect(...) - -def get_data(): - conn = get_connection() # Reuses cached connection - return conn.cursor().fetchall() -``` - -### 3. Expensive Operations in Main Flow - -**❌ Wrong:** -```python -# Runs on every rerun! -data = expensive_api_call() -processed_data = expensive_processing(data) -``` - -**✅ Correct:** -```python -@st.cache_data(ttl=300) -def get_processed_data(): - data = expensive_api_call() - return expensive_processing(data) - -data = get_processed_data() # Cached for 5 minutes -``` - -### 4. Not Using Forms for Multiple Inputs - -**❌ Wrong:** -```python -# Page reruns on EVERY input change -name = st.text_input("Name") # Rerun -email = st.text_input("Email") # Rerun -phone = st.text_input("Phone") # Rerun -``` - -**✅ Correct:** -```python -# Page only reruns on submit -with st.form("user_form"): - name = st.text_input("Name") - email = st.text_input("Email") - phone = st.text_input("Phone") - submitted = st.form_submit_button("Submit") -``` - -### 5. Modifying State During Render - -**❌ Wrong:** -```python -if st.button("Increment"): - st.session_state.count += 1 # ❌ Can cause issues - st.write(st.session_state.count) # May not update immediately -``` - -**✅ Correct:** -```python -if 'count' not in st.session_state: - st.session_state.count = 0 - -if st.button("Increment"): - st.session_state.count += 1 - -st.write(f"Count: {st.session_state.count}") # Display outside callback -``` - ---- - -## Performance Optimization - -### 1. Use Appropriate Caching - -```python -# For connections (persist across sessions) -@st.cache_resource -def get_db_connection(): - return sql.connect(...) - -# For data (serialize and cache with TTL) -@st.cache_data(ttl=600) -def load_orders(): - return fetch_orders() -``` - -### 2. Lazy Loading - -```python -# Don't load all data upfront -def load_page(): - if page == "Dashboard": - load_dashboard_data() # Only load what's needed - elif page == "Orders": - load_orders_data() -``` - -### 3. Pagination - -```python -# Don't display 10,000 rows at once -page_size = 50 -page_num = st.number_input("Page", min_value=1, value=1) - -start_idx = (page_num - 1) * page_size -end_idx = start_idx + page_size - -st.dataframe(df[start_idx:end_idx]) -``` - -### 4. Debouncing with Forms - -```python -# Use forms to prevent reruns on every keystroke -with st.form("search_form"): - search = st.text_input("Search") - submitted = st.form_submit_button("Search") - -if submitted and search: - results = backend.search(search) - st.dataframe(results) -``` - -### 5. Fragment Updates (Streamlit 1.24+) - -```python -@st.experimental_fragment -def render_chart(): - """Only this fragment reruns, not entire page""" - data = load_chart_data() - st.plotly_chart(create_chart(data)) - -# Main page doesn't rerun when fragment updates -st.title("Dashboard") -render_chart() -``` - ---- - -## Deployment Configuration - -### app.yaml for Databricks - -```yaml -command: - - "streamlit" - - "run" - - "streamlit_app.py" - - "--server.port" - - "8080" - - "--server.address" - - "0.0.0.0" - - "--server.headless" - - "true" - -env: - - name: USE_MOCK_BACKEND - value: "false" - - name: DATABRICKS_WAREHOUSE_ID - value: "your-warehouse-id" - - name: DATABRICKS_CATALOG - value: "main" - - name: DATABRICKS_SCHEMA - value: "app_schema" -``` - -### requirements.txt - -```txt -streamlit>=1.28.0 -pandas>=2.0.0 -plotly>=5.17.0 -databricks-sdk>=0.12.0 -databricks-sql-connector>=3.0.0 -pydantic>=2.0.0 -python-dotenv>=1.0.0 -``` - ---- - -## Reference Resources - -- **[Databricks Streamlit Tutorial](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/tutorial-streamlit)** - Official tutorial -- **[Databricks Apps Cookbook - Streamlit](https://apps-cookbook.dev/docs/category/streamlit/)** - Code examples -- **[Streamlit Read Delta Table](https://apps-cookbook.dev/docs/streamlit/tables/tables_read/)** - Connection patterns -- **[Streamlit Documentation](https://docs.streamlit.io/)** - Full API reference - ---- - -## Key Takeaways - -1. **Always cache resources** - Use `@st.cache_resource` for connections -2. **Page config first** - Must be the first Streamlit command -3. **Use forms** - Prevent reruns for multiple inputs -4. **Session state** - For data persistence across reruns -5. **SDK Config pattern** - For Databricks authentication -6. **Layout wisely** - Columns, expanders, tabs for organization -7. **Handle errors** - Graceful degradation and user feedback - -Streamlit is perfect for rapid development of data-focused applications on Databricks! diff --git a/.claude/skills/databricks-apps-python/1-authorization.md b/.claude/skills/databricks-apps-python/1-authorization.md new file mode 100644 index 00000000..0a84f629 --- /dev/null +++ b/.claude/skills/databricks-apps-python/1-authorization.md @@ -0,0 +1,150 @@ +# Authorization for Databricks Apps + +Databricks Apps supports two complementary authorization models. Use one or both depending on your app's needs. + +**Docs**: https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth + +--- + +## App Authorization (Service Principal) + +Each app gets a dedicated service principal. Databricks auto-injects credentials: + +- `DATABRICKS_CLIENT_ID` — OAuth client ID +- `DATABRICKS_CLIENT_SECRET` — OAuth client secret + +**You don't need to read these manually.** The SDK `Config()` detects them automatically: + +```python +from databricks.sdk.core import Config +from databricks import sql + +cfg = Config() # Auto-detects SP credentials from environment +conn = sql.connect( + server_hostname=cfg.host, + http_path="/sql/1.0/warehouses/", + credentials_provider=lambda: cfg.authenticate, +) +``` + +**Use for**: background tasks, shared data access, logging, external service calls. + +**Limitation**: all users share the same permissions — no per-user access control. + +--- + +## User Authorization (On-Behalf-Of) + +Allows the app to act with the identity of the current user. Databricks forwards the user's access token to the app via HTTP header. + +**Use for**: user-specific data queries, Unity Catalog row/column filters, audit trails. + +**Prerequisite**: workspace admin must enable user authorization (Public Preview). Add scopes when creating/editing the app in the UI. + +### Retrieving the User Token Per Framework + +```python +# Streamlit +import streamlit as st +user_token = st.context.headers.get("x-forwarded-access-token") + +# Dash / Flask +from flask import request +user_token = request.headers.get("x-forwarded-access-token") + +# Gradio +import gradio as gr +def handler(message, request: gr.Request): + user_token = request.headers.get("x-forwarded-access-token") + +# FastAPI +from fastapi import Request +async def endpoint(request: Request): + user_token = request.headers.get("x-forwarded-access-token") + +# Reflex +user_token = session.http_conn.headers.get("x-forwarded-access-token") +``` + +### Querying with User Token + +```python +from databricks.sdk.core import Config +from databricks import sql + +cfg = Config() +user_token = get_user_token() # Per-framework method above + +conn = sql.connect( + server_hostname=cfg.host, + http_path="/sql/1.0/warehouses/", + access_token=user_token, # User's token, not SP credentials +) +``` + +--- + +## Combining Both Models + +Use app auth for shared operations and user auth for user-specific data: + +```python +from databricks.sdk.core import Config +from databricks import sql + +cfg = Config() + +def get_app_connection(warehouse_http_path: str): + """App auth — shared data, logging, background tasks.""" + return sql.connect( + server_hostname=cfg.host, + http_path=warehouse_http_path, + credentials_provider=lambda: cfg.authenticate, + ) + +def get_user_connection(warehouse_http_path: str, user_token: str): + """User auth — respects Unity Catalog row/column filters.""" + return sql.connect( + server_hostname=cfg.host, + http_path=warehouse_http_path, + access_token=user_token, + ) +``` + +--- + +## OAuth Scopes + +When adding user authorization, select only the scopes your app needs: + +| Scope | Grants Access To | +|-------|-----------------| +| `sql` | SQL warehouse queries | +| `files.files` | Files and directories | +| `dashboards.genie` | Genie spaces | +| `iam.access-control:read` | Access control (default) | +| `iam.current-user:read` | Current user identity (default) | + +**Best practice**: request minimum required scopes. Databricks blocks access outside approved scopes even if the user has broader permissions. + +--- + +## When to Use Which + +| Scenario | Model | +|----------|-------| +| All users see same data | App auth only | +| User-specific row/column filters | User auth | +| Background jobs, logging | App auth | +| Audit trail per user | User auth | +| Mixed shared + personal data | Both | + +--- + +## Best Practices + +- Never log, print, or write tokens to files +- Grant service principal minimum required permissions on resources +- Use `CAN MANAGE` only for trusted developers; `CAN USE` for app users +- Enforce peer review for app code before production deployment +- Cookbook auth examples: [Streamlit](https://apps-cookbook.dev/docs/streamlit/authentication/users_get_current) · [Dash](https://apps-cookbook.dev/docs/dash/authentication/users_get_current) · [Reflex](https://apps-cookbook.dev/docs/reflex/authentication/users_get_current) diff --git a/.claude/skills/databricks-apps-python/2-app-resources.md b/.claude/skills/databricks-apps-python/2-app-resources.md new file mode 100644 index 00000000..dd911c4b --- /dev/null +++ b/.claude/skills/databricks-apps-python/2-app-resources.md @@ -0,0 +1,120 @@ +# App Resources and Communication Strategies + +Databricks Apps integrate with platform resources via managed connections. Use resources instead of hardcoding IDs for portability and security. + +**Docs**: https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources + +--- + +## Supported Resource Types + +| Resource | Default Key | Permissions | Use Case | +|----------|-------------|-------------|----------| +| SQL warehouse | `sql-warehouse` | Can use, Can manage | Querying Delta tables | +| Lakebase database | `database` | Can connect and create | Low-latency transactional data | +| Model serving endpoint | `serving-endpoint` | Can view, Can query, Can manage | AI/ML inference | +| Secret | `secret` | Can read, Can write, Can manage | API keys, tokens | +| Unity Catalog volume | `volume` | Can read, Can read and write | File storage | +| Vector search index | `vector-search-index` | Can select | Semantic search | +| Genie space | `genie-space` | Can view, Can run, Can edit | Natural language analytics | +| UC connection | `connection` | Use Connection | External data sources | +| UC function | `function` | Can execute | SQL/Python functions | +| MLflow experiment | `experiment` | Can read, Can edit | ML experiment tracking | +| Lakeflow job | `job` | Can view, Can manage run | Data pipelines | + +--- + +## Configuring Resources in app.yaml + +Use `valueFrom` to reference resources — never hardcode IDs: + +```yaml +env: + - name: DATABRICKS_WAREHOUSE_ID + valueFrom: sql-warehouse + + - name: SERVING_ENDPOINT_NAME + valueFrom: serving-endpoint + + - name: DB_CONNECTION_STRING + valueFrom: database +``` + +Add resources via the Databricks Apps UI when creating or editing an app: +1. Navigate to Configure step +2. Click **+ Add resource** +3. Select resource type and set permissions +4. Assign a key (referenced in `valueFrom`) + +--- + +## Communication Strategies + +Choose your data backend based on access pattern: + +| Strategy | When to Use | Library | Connection Pattern | +|----------|-------------|---------|-------------------| +| **SQL Warehouse** | Analytical queries on Delta tables | `databricks-sql-connector` | `sql.connect()` with `Config()` | +| **Lakebase (PostgreSQL)** | Low-latency transactional CRUD | `psycopg2` / `asyncpg` | Standard PostgreSQL via auto-injected env vars | +| **Databricks SDK** | Platform API calls (jobs, clusters, UC) | `databricks-sdk` | `WorkspaceClient()` | +| **Model Serving** | AI/ML inference requests | `requests` or SDK | REST call to serving endpoint | +| **Unity Catalog Functions** | Server-side compute (SQL/Python UDFs) | `databricks-sql-connector` | Execute via SQL warehouse | + +### SQL Warehouse Pattern + +```python +import os +from databricks.sdk.core import Config +from databricks import sql + +cfg = Config() +conn = sql.connect( + server_hostname=cfg.host, + http_path=f"/sql/1.0/warehouses/{os.getenv('DATABRICKS_WAREHOUSE_ID')}", + credentials_provider=lambda: cfg.authenticate, +) + +with conn.cursor() as cursor: + cursor.execute("SELECT * FROM catalog.schema.table LIMIT 100") + rows = cursor.fetchall() +``` + +### Model Serving Pattern + +```python +import os, requests +from databricks.sdk.core import Config + +cfg = Config() +headers = cfg.authenticate() +headers["Content-Type"] = "application/json" + +endpoint = os.getenv("SERVING_ENDPOINT_NAME") +response = requests.post( + f"https://{cfg.host}/serving-endpoints/{endpoint}/invocations", + headers=headers, + json={"inputs": [{"prompt": "Hello"}]}, +) +result = response.json() +``` + +### SDK Pattern + +```python +from databricks.sdk import WorkspaceClient + +w = WorkspaceClient() # Auto-detects credentials +for cluster in w.clusters.list(): + print(f"{cluster.cluster_name}: {cluster.state}") +``` + +For Lakebase patterns, see [5-lakebase.md](5-lakebase.md). + +--- + +## Best Practices + +- Always use `valueFrom` — keeps apps portable between environments +- Grant service principal minimum required permissions (e.g., `CAN USE` not `CAN MANAGE` for SQL warehouse) +- Use Lakebase for transactional workloads; SQL warehouse for analytical workloads +- For external services, use UC connections or secrets (never hardcode API keys) diff --git a/.claude/skills/databricks-apps-python/3-frameworks.md b/.claude/skills/databricks-apps-python/3-frameworks.md new file mode 100644 index 00000000..b8e76c89 --- /dev/null +++ b/.claude/skills/databricks-apps-python/3-frameworks.md @@ -0,0 +1,248 @@ +# Supported Frameworks + +All frameworks below are **pre-installed** in the Databricks Apps runtime. Claude already knows how to use them — this guide covers only **Databricks-specific** patterns. For full examples and recipes, see the **[Databricks Apps Cookbook](https://apps-cookbook.dev/)**. + +--- + +## Dash + +**Best for**: Production dashboards, BI tools, complex interactive visualizations. + +**Critical**: Always use `dash-bootstrap-components` for layout and styling. + +```python +import dash +import dash_bootstrap_components as dbc + +app = dash.Dash( + __name__, + external_stylesheets=[dbc.themes.BOOTSTRAP, dbc.icons.FONT_AWESOME], + title="My Dashboard", +) +``` + +| Detail | Value | +|--------|-------| +| Pre-installed version | 2.18.1 | +| app.yaml command | `["python", "app.py"]` | +| Default port | 8050 — override in code: `app.run(port=int(os.environ.get("DATABRICKS_APP_PORT", 8000)))` | +| Auth header | `request.headers.get('x-forwarded-access-token')` (Flask under the hood) | + +**Databricks tips**: +- Use `dbc.themes.BOOTSTRAP` and `dbc.icons.FONT_AWESOME` for consistent styling +- Use Bootstrap badge color names (`"success"`, `"danger"`), not hex colors, for `dbc.Badge` +- Use `prevent_initial_call=True` on expensive callbacks +- Use `dcc.Store` for client-side caching + +**Cookbook**: [apps-cookbook.dev/docs/category/dash](https://apps-cookbook.dev/docs/category/dash) — tables, volumes, AI/ML, workflows, dashboards, compute, auth, external services. + +--- + +## Streamlit + +**Best for**: Rapid prototyping, data science apps, internal tools, notebook-to-app workflow. + +**Critical**: Always use `@st.cache_resource` for database connections. + +```python +import streamlit as st +from databricks.sdk.core import Config +from databricks import sql + +st.set_page_config(page_title="My App", layout="wide") # Must be first! + +@st.cache_resource(ttl=300) +def get_connection(): + cfg = Config() + return sql.connect( + server_hostname=cfg.host, + http_path="/sql/1.0/warehouses/", + credentials_provider=lambda: cfg.authenticate, + ) +``` + +| Detail | Value | +|--------|-------| +| Pre-installed version | 1.38.0 | +| app.yaml command | `["streamlit", "run", "app.py"]` | +| Auth header | `st.context.headers.get('x-forwarded-access-token')` | + +**Databricks tips**: +- `st.set_page_config()` must be the **first** Streamlit command +- `@st.cache_resource` for connections/models; `@st.cache_data(ttl=...)` for query results +- Use `st.form()` to batch inputs and prevent reruns on every keystroke +- Use `st.column_config` for formatted DataFrames (currency, dates) + +**Cookbook**: [apps-cookbook.dev/docs/category/streamlit](https://apps-cookbook.dev/docs/category/streamlit) — tables, volumes, AI/ML, workflows, visualizations, dashboards, compute, auth, external services. + +--- + +## Gradio + +**Best for**: ML model demos, chat interfaces, image/audio/video processing UIs. + +**Critical**: Use `gr.Request` parameter to access auth headers. + +```python +import os +import gradio as gr +import requests +from databricks.sdk.core import Config + +cfg = Config() + +def predict(message, request: gr.Request): + user_token = request.headers.get("x-forwarded-access-token") + # Query model serving endpoint + headers = {**cfg.authenticate(), "Content-Type": "application/json"} + resp = requests.post( + f"https://{cfg.host}/serving-endpoints/my-model/invocations", + headers=headers, + json={"inputs": [{"prompt": message}]}, + ) + return resp.json()["predictions"][0] + +demo = gr.Interface(fn=predict, inputs="text", outputs="text") +port = int(os.environ.get("DATABRICKS_APP_PORT", 8000)) +demo.launch(server_name="0.0.0.0", server_port=port) +``` + +| Detail | Value | +|--------|-------| +| Pre-installed version | 4.44.0 | +| app.yaml command | `["python", "app.py"]` | +| Default port | 7860 — override in code: `server_port=int(os.environ.get("DATABRICKS_APP_PORT", 8000))` | +| Auth header | `request.headers.get('x-forwarded-access-token')` via `gr.Request` | + +**Databricks tips**: +- Natural fit for model serving endpoint integration +- Use `gr.ChatInterface` for conversational AI demos +- Use `gr.Blocks` for complex multi-component layouts + +**Docs**: [gradio.app/docs](https://www.gradio.app/docs) + +--- + +## Flask + +**Best for**: Custom REST APIs, lightweight web apps, webhook receivers. + +**Critical**: Deploy with Gunicorn — never use Flask's dev server in production. + +```python +from flask import Flask, request, jsonify +from databricks.sdk.core import Config +from databricks import sql + +app = Flask(__name__) +cfg = Config() + +@app.route("/api/data") +def get_data(): + conn = sql.connect( + server_hostname=cfg.host, + http_path="/sql/1.0/warehouses/", + credentials_provider=lambda: cfg.authenticate, + ) + with conn.cursor() as cursor: + cursor.execute("SELECT * FROM catalog.schema.table LIMIT 10") + return jsonify(cursor.fetchall()) +``` + +| Detail | Value | +|--------|-------| +| Pre-installed version | 3.0.3 | +| app.yaml command | `["gunicorn", "app:app", "-w", "4", "-b", "0.0.0.0:8000"]` | +| Auth header | `request.headers.get('x-forwarded-access-token')` | + +**Databricks tips**: +- Use connection pooling (Flask doesn't cache connections like Streamlit) +- Gunicorn workers (`-w 4`) handle concurrent requests +- Use `request.headers` for user authorization tokens + +--- + +## FastAPI + +**Best for**: Modern async APIs, auto-generated OpenAPI/Swagger docs, high-performance backends. + +**Critical**: Deploy with uvicorn. + +```python +from fastapi import FastAPI, Request +from databricks.sdk.core import Config +from databricks import sql + +app = FastAPI(title="My API") +cfg = Config() + +@app.get("/api/data") +async def get_data(request: Request): + user_token = request.headers.get("x-forwarded-access-token") + conn = sql.connect( + server_hostname=cfg.host, + http_path="/sql/1.0/warehouses/", + access_token=user_token, + ) + with conn.cursor() as cursor: + cursor.execute("SELECT * FROM catalog.schema.table LIMIT 10") + return cursor.fetchall() +``` + +| Detail | Value | +|--------|-------| +| Pre-installed version | 0.115.0 | +| app.yaml command | `["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]` | +| Auth header | `request.headers.get('x-forwarded-access-token')` via `Request` | + +**Databricks tips**: +- Auto-generates OpenAPI docs at `/docs` (Swagger) and `/redoc` +- Databricks SQL connector is synchronous — use `asyncio.to_thread()` for async endpoints +- Good choice for API backends that serve APX (FastAPI + React) apps + +**Cookbook**: [apps-cookbook.dev/docs/category/fastapi](https://apps-cookbook.dev/docs/category/fastapi) — getting started, endpoint examples. + +--- + +## Reflex + +**Best for**: Full-stack Python apps with reactive UIs, no JavaScript required. + +```python +import reflex as rx +from databricks.sdk.core import Config + +cfg = Config() + +class State(rx.State): + data: list[dict] = [] + + def load_data(self): + from databricks import sql + conn = sql.connect( + server_hostname=cfg.host, + http_path="/sql/1.0/warehouses/", + credentials_provider=lambda: cfg.authenticate, + ) + with conn.cursor() as cursor: + cursor.execute("SELECT * FROM catalog.schema.table LIMIT 10") + self.data = [dict(zip([d[0] for d in cursor.description], row)) for row in cursor.fetchall()] +``` + +| Detail | Value | +|--------|-------| +| app.yaml command | `["reflex", "run", "--env", "prod"]` | +| Auth header | `session.http_conn.headers.get('x-forwarded-access-token')` | + +**Cookbook**: [apps-cookbook.dev/docs/category/reflex](https://apps-cookbook.dev/docs/category/reflex) — tables, volumes, AI/ML, workflows, dashboards, compute, auth, external services. + +--- + +## Common: All Frameworks + +- All frameworks are **pre-installed** — no need to add them to `requirements.txt` +- Add only additional packages your app needs to `requirements.txt` +- SDK `Config()` auto-detects credentials from injected environment variables +- Apps must bind to `DATABRICKS_APP_PORT` env var (defaults to 8000). Streamlit is auto-configured by the runtime; for other frameworks, read the env var in code or hardcode 8000 in `app.yaml` command. **Never use 8080** +- For framework-specific deployment commands, see [4-deployment.md](4-deployment.md) +- For authorization integration, see [1-authorization.md](1-authorization.md) diff --git a/.claude/skills/databricks-apps-python/4-deployment.md b/.claude/skills/databricks-apps-python/4-deployment.md new file mode 100644 index 00000000..0d0ab9f2 --- /dev/null +++ b/.claude/skills/databricks-apps-python/4-deployment.md @@ -0,0 +1,150 @@ +# Deploying Databricks Apps + +Three deployment options: Databricks CLI (simplest), Asset Bundles (multi-environment), or MCP tools (programmatic). + +**Cookbook deployment guide**: https://apps-cookbook.dev/docs/deploy + +--- + +## Option 1: Databricks CLI + +**Best for**: quick deployments, single environment. + +### Step 1: Create app.yaml + +```yaml +command: + - "python" # Adjust per framework — see table below + - "app.py" + +env: + - name: DATABRICKS_WAREHOUSE_ID + valueFrom: sql-warehouse + - name: USE_MOCK_BACKEND + value: "false" +``` + +### app.yaml Commands Per Framework + +| Framework | Command | +|-----------|---------| +| Dash | `["python", "app.py"]` | +| Streamlit | `["streamlit", "run", "app.py"]` | +| Gradio | `["python", "app.py"]` | +| Flask | `["gunicorn", "app:app", "-w", "4", "-b", "0.0.0.0:8000"]` | +| FastAPI | `["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]` | +| Reflex | `["reflex", "run", "--env", "prod"]` | + +### Excluded directories + +When uploading via the SDK's `upload_folder()` / `upload_to_workspace()`, the following directories are automatically skipped to keep uploads fast: + +`node_modules`, `__pycache__`, `.venv`, `venv`, `.tox`, `.pytest_cache`, `.mypy_cache`, `.ruff_cache`, `dist`, `build`, `.eggs`, `*.egg-info` + +If you use `databricks workspace import-dir` directly, it does **not** apply these exclusions. Either clean the directory first or use the SDK upload functions instead. + +### Step 2: Create and Deploy + +```bash +# Create the app +databricks apps create + +# Upload source code (make sure to exclude node_modules, venv, etc.) +databricks workspace mkdirs /Workspace/Users//apps/ +databricks workspace import-dir . /Workspace/Users//apps/ + +# Deploy +databricks apps deploy \ + --source-code-path /Workspace/Users//apps/ + +# Add resources via UI (SQL warehouse, Lakebase, etc.) + +# Check status and URL +databricks apps get +``` + +### Redeployment + +```bash +databricks workspace delete /Workspace/Users//apps/ --recursive +databricks workspace import-dir . /Workspace/Users//apps/ +databricks apps deploy \ + --source-code-path /Workspace/Users//apps/ +``` + +--- + +## Option 2: Databricks Asset Bundles (DABs) + +**Best for**: multi-environment deployments (dev/staging/prod), version-controlled infrastructure. + +**Recommended workflow**: deploy via CLI first to validate, then generate bundle config. + +### Generate Bundle from Existing App + +```bash +databricks bundle generate app \ + --existing-app-name \ + --key +``` + +This creates: +- `resources/.app.yml` — app resource definition +- `src/app/` — app source files including `app.yaml` + +### Deploy with Bundles + +```bash +# Validate +databricks bundle validate -t dev + +# Deploy +databricks bundle deploy -t dev + +# Start the app (required after deployment) +databricks bundle run -t dev + +# Production +databricks bundle deploy -t prod +databricks bundle run -t prod +``` + +**Key difference from other resources**: environment variables go in `src/app/app.yaml`, not `databricks.yml`. + +For complete DABs guidance, use the **databricks-bundles** skill. + +--- + +## Option 3: MCP Tools + +For programmatic app lifecycle management, see [6-mcp-approach.md](6-mcp-approach.md). + +--- + +## Post-Deployment + +### Check Logs + +```bash +databricks apps logs +``` + +**Key patterns in logs**: +- `[SYSTEM]` — deployment status, file updates, dependency installation +- `[APP]` — application output, framework messages +- `Deployment successful` — app deployed correctly +- `App started successfully` — app is running +- `Error:` — check stack traces + +### Verify + +1. Access app URL (from `databricks apps get `) +2. Check all pages load correctly +3. Verify data connectivity (look for backend initialization messages in logs) +4. Test user authorization flow if enabled + +### Configure Permissions + +- Set `CAN USE` for approved users/groups +- Set `CAN MANAGE` only for trusted developers +- Verify service principal has required resource permissions diff --git a/.claude/skills/databricks-apps-python/5-lakebase.md b/.claude/skills/databricks-apps-python/5-lakebase.md new file mode 100644 index 00000000..c6615609 --- /dev/null +++ b/.claude/skills/databricks-apps-python/5-lakebase.md @@ -0,0 +1,141 @@ +# Lakebase (PostgreSQL) Connectivity + +Lakebase provides low-latency transactional storage for Databricks Apps via a managed PostgreSQL interface. + +**Docs**: https://docs.databricks.com/aws/en/dev-tools/databricks-apps/lakebase + +--- + +## When to Use Lakebase + +| Use Case | Recommended Backend | +|----------|-------------------| +| Analytical queries on Delta tables | SQL Warehouse | +| Low-latency transactional CRUD | **Lakebase** | +| App-specific metadata/config | **Lakebase** | +| User session data | **Lakebase** | +| Large-scale data exploration | SQL Warehouse | + +--- + +## Setup + +1. Add Lakebase as an app resource in the Databricks UI (resource type: **Lakebase database**) +2. Databricks auto-injects PostgreSQL connection env vars: + +| Variable | Description | +|----------|-------------| +| `PGHOST` | Database hostname | +| `PGDATABASE` | Database name | +| `PGUSER` | PostgreSQL role (created per app) | +| `PGPASSWORD` | Role password | +| `PGPORT` | Port (typically 5432) | + +3. Reference in `app.yaml`: + +```yaml +env: + - name: DB_CONNECTION_STRING + valueFrom: + resource: database +``` + +--- + +## Connection Patterns + +### psycopg2 (Synchronous) + +```python +import os +import psycopg2 + +conn = psycopg2.connect( + host=os.getenv("PGHOST"), + database=os.getenv("PGDATABASE"), + user=os.getenv("PGUSER"), + password=os.getenv("PGPASSWORD"), + port=os.getenv("PGPORT", "5432"), +) + +with conn.cursor() as cur: + cur.execute("SELECT * FROM my_table LIMIT 10") + rows = cur.fetchall() + +conn.close() +``` + +### asyncpg (Asynchronous) + +```python +import os +import asyncpg + +async def get_data(): + conn = await asyncpg.connect( + host=os.getenv("PGHOST"), + database=os.getenv("PGDATABASE"), + user=os.getenv("PGUSER"), + password=os.getenv("PGPASSWORD"), + port=int(os.getenv("PGPORT", "5432")), + ) + rows = await conn.fetch("SELECT * FROM my_table LIMIT 10") + await conn.close() + return rows +``` + +### SQLAlchemy + +```python +import os +from sqlalchemy import create_engine + +DATABASE_URL = ( + f"postgresql://{os.getenv('PGUSER')}:{os.getenv('PGPASSWORD')}" + f"@{os.getenv('PGHOST')}:{os.getenv('PGPORT', '5432')}" + f"/{os.getenv('PGDATABASE')}" +) + +engine = create_engine(DATABASE_URL) +``` + +--- + +## Streamlit with Lakebase + +```python +import streamlit as st +import psycopg2 + +@st.cache_resource +def get_db_connection(): + return psycopg2.connect( + host=os.getenv("PGHOST"), + database=os.getenv("PGDATABASE"), + user=os.getenv("PGUSER"), + password=os.getenv("PGPASSWORD"), + ) +``` + +--- + +## Critical: requirements.txt + +`psycopg2` and `asyncpg` are **NOT pre-installed** in the Databricks Apps runtime. You **MUST** include them in `requirements.txt` or the app will crash on startup: + +``` +psycopg2-binary +``` + +For async apps: +``` +asyncpg +``` + +**This is the most common cause of Lakebase app failures.** + +## Notes + +- Lakebase is in **Public Preview** +- Each app gets its own PostgreSQL role with `Can connect and create` permission +- Lakebase is ideal alongside SQL warehouse: use Lakebase for app state, SQL warehouse for analytics diff --git a/.claude/skills/databricks-apps-python/6-mcp-approach.md b/.claude/skills/databricks-apps-python/6-mcp-approach.md new file mode 100644 index 00000000..943c49ba --- /dev/null +++ b/.claude/skills/databricks-apps-python/6-mcp-approach.md @@ -0,0 +1,79 @@ +# MCP Tools for App Lifecycle + +Use MCP tools to create, deploy, and manage Databricks Apps programmatically. This mirrors the CLI workflow but can be invoked by AI agents. + +--- + +## manage_app - App Lifecycle Management + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create_or_update` | Idempotent create, deploys if source_code_path provided | name | +| `get` | Get app details (with optional logs) | name | +| `list` | List all apps | (none, optional name_contains filter) | +| `delete` | Delete an app | name | + +--- + +## Workflow + +### Step 1: Write App Files Locally + +Create your app files in a local folder: + +``` +my_app/ +├── app.py # Main application +├── models.py # Pydantic models +├── backend.py # Data access layer +├── requirements.txt # Additional dependencies +└── app.yaml # Databricks Apps configuration +``` + +### Step 2: Upload to Workspace + +```python +# MCP Tool: manage_workspace_files +manage_workspace_files( + action="upload", + local_path="/path/to/my_app", + workspace_path="/Workspace/Users/user@example.com/my_app" +) +``` + +### Step 3: Create and Deploy App + +```python +# MCP Tool: manage_app (creates if needed + deploys) +result = manage_app( + action="create_or_update", + name="my-dashboard", + description="Customer analytics dashboard", + source_code_path="/Workspace/Users/user@example.com/my_app" +) +# Returns: {"name": "my-dashboard", "url": "...", "created": True, "deployment": {...}} +``` + +### Step 4: Verify + +```python +# MCP Tool: manage_app (get with logs) +app = manage_app(action="get", name="my-dashboard", include_logs=True) +# Returns: {"name": "...", "url": "...", "status": "RUNNING", "logs": "...", ...} +``` + +### Step 5: Iterate + +1. Fix issues in local files +2. Re-upload with `manage_workspace_files(action="upload", ...)` +3. Re-deploy with `manage_app(action="create_or_update", ...)` (will update existing + deploy) +4. Check `manage_app(action="get", name=..., include_logs=True)` for errors +5. Repeat until app is healthy + +--- + +## Notes + +- Add resources (SQL warehouse, Lakebase, etc.) via the Databricks Apps UI after creating the app +- MCP tools use the service principal's permissions — ensure it has access to required resources +- For manual deployment, see [4-deployment.md](4-deployment.md) diff --git a/.claude/skills/databricks-apps-python/SKILL.md b/.claude/skills/databricks-apps-python/SKILL.md new file mode 100644 index 00000000..161dbd81 --- /dev/null +++ b/.claude/skills/databricks-apps-python/SKILL.md @@ -0,0 +1,259 @@ +--- +name: databricks-apps-python +description: "Builds Databricks applications. Prefers AppKit (TypeScript + React SDK) for new apps; falls back to Python frameworks (Dash, Streamlit, Gradio, Flask, FastAPI, Reflex) when Python is required. Handles OAuth authorization, app resources, SQL warehouse and Lakebase connectivity, model serving, foundation model APIs, and deployment. Use when building web apps, dashboards, ML demos, or REST APIs for Databricks, or when the user mentions AppKit, Streamlit, Dash, Gradio, Flask, FastAPI, Reflex, or Databricks app." +--- + +# Databricks Applications + +Build Python-based Databricks applications. For full examples and recipes, see the **[Databricks Apps Cookbook](https://apps-cookbook.dev/)**. + +--- + +## AppKit (Preferred for New Apps) + +**[AppKit](https://github.com/databricks/appkit)** is the recommended SDK for new Databricks apps. It is a TypeScript + React SDK with a plugin architecture, built-in caching, telemetry, and end-to-end type safety. + +### Requirements +- Node.js v22+ +- Databricks CLI v0.295.0+ + +### Scaffold a new app +```bash +databricks apps init +``` +This interactive command scaffolds the full project, installs dependencies, and optionally deploys. + +### Deploy +```bash +databricks apps deploy +``` + +### AppKit plugins +| Plugin | Purpose | +|--------|---------| +| **Analytics** | SQL queries against Databricks SQL Warehouses — file-based, typed, cached | +| **Genie** | Conversational AI/BI interface with natural language queries | +| **Files** | Browse/upload Unity Catalog Volumes | +| **Lakebase** | OLTP PostgreSQL via Lakebase with OAuth token management | + +### AI-assisted development +```bash +# Install agent skills for AI-powered scaffolding +databricks experimental aitools skills install + +# Query AppKit docs inline +npx @databricks/appkit docs "your question here" +``` + +### AppKit documentation +- **[AppKit Docs](https://databricks.github.io/appkit/docs/)** — getting started, plugins, API reference +- **[AI-assisted development](https://databricks.github.io/appkit/docs/development/ai-assisted-development)** — guidance for code assistants +- **[llms.txt](https://databricks.github.io/appkit/llms.txt)** — machine-readable docs for AI context + +--- + +## Python Apps (alternative) + +Use Python when: the team is Python-only, you need Streamlit/Dash/Gradio/Gradio, or you are extending an existing Python app. + +## Critical Rules for Python apps (always follow) + +- **MUST** confirm framework choice or use [Python Framework Selection](#python-framework-selection) below +- **MUST** use SDK `Config()` for authentication (never hardcode tokens) +- **MUST** use `app.yaml` `valueFrom` for resources (never hardcode resource IDs) +- **MUST** use `dash-bootstrap-components` for Dash app layout and styling +- **MUST** use `@st.cache_resource` for Streamlit database connections +- **MUST** deploy Flask with Gunicorn, FastAPI with uvicorn (not dev servers) + +## Required Steps for Python apps + +Copy this checklist and verify each item: +``` +- [ ] Framework selected +- [ ] Auth strategy decided: app auth, user auth, or both +- [ ] App resources identified (SQL warehouse, Lakebase, serving endpoint, etc.) +- [ ] Backend data strategy decided (SQL warehouse, Lakebase, or SDK) +- [ ] Deployment method: CLI or DABs +``` + +--- + +## Python Framework Selection + +| Framework | Best For | app.yaml Command | +|-----------|----------|------------------| +| **Dash** | Production dashboards, BI tools, complex interactivity | `["python", "app.py"]` | +| **Streamlit** | Rapid prototyping, data science apps, internal tools | `["streamlit", "run", "app.py"]` | +| **Gradio** | ML demos, model interfaces, chat UIs | `["python", "app.py"]` | +| **Flask** | Custom REST APIs, lightweight apps, webhooks | `["gunicorn", "app:app", "-w", "4", "-b", "0.0.0.0:8000"]` | +| **FastAPI** | Async APIs, auto-generated OpenAPI docs | `["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]` | +| **Reflex** | Full-stack Python apps without JavaScript | `["reflex", "run", "--env", "prod"]` | + +**Default**: Recommend **Streamlit** for prototypes, **Dash** for production dashboards, **FastAPI** for APIs, **Gradio** for ML demos. + +--- + +## Quick Reference + +| Concept | Details | +|---------|---------| +| **Runtime** | Python 3.11, Ubuntu 22.04, 2 vCPU, 6 GB RAM | +| **Pre-installed** | Dash 2.18.1, Streamlit 1.38.0, Gradio 4.44.0, Flask 3.0.3, FastAPI 0.115.0 | +| **Auth (app)** | Service principal via `Config()` — auto-injected `DATABRICKS_CLIENT_ID`/`DATABRICKS_CLIENT_SECRET` | +| **Auth (user)** | `x-forwarded-access-token` header — see [1-authorization.md](1-authorization.md) | +| **Resources** | `valueFrom` in app.yaml — see [2-app-resources.md](2-app-resources.md) | +| **Cookbook** | https://apps-cookbook.dev/ | +| **Docs** | https://docs.databricks.com/aws/en/dev-tools/databricks-apps/ | + +--- + +## Detailed Guides + +**Authorization**: Use [1-authorization.md](1-authorization.md) when configuring app or user authorization — covers service principal auth, on-behalf-of user tokens, OAuth scopes, and per-framework code examples. (Keywords: OAuth, service principal, user auth, on-behalf-of, access token, scopes) + +**App resources**: Use [2-app-resources.md](2-app-resources.md) when connecting your app to Databricks resources — covers SQL warehouses, Lakebase, model serving, secrets, volumes, and the `valueFrom` pattern. (Keywords: resources, valueFrom, SQL warehouse, model serving, secrets, volumes, connections) + +**Frameworks**: See [3-frameworks.md](3-frameworks.md) for Databricks-specific patterns per framework — covers Dash, Streamlit, Gradio, Flask, FastAPI, and Reflex with auth integration, deployment commands, and Cookbook links. (Keywords: Dash, Streamlit, Gradio, Flask, FastAPI, Reflex, framework selection) + +**Deployment**: Use [4-deployment.md](4-deployment.md) when deploying your app — covers Databricks CLI, Asset Bundles (DABs), app.yaml configuration, and post-deployment verification. (Keywords: deploy, CLI, DABs, asset bundles, app.yaml, logs) + +**Lakebase**: Use [5-lakebase.md](5-lakebase.md) when using Lakebase (PostgreSQL) as your app's data layer — covers auto-injected env vars, psycopg2/asyncpg patterns, and when to choose Lakebase vs SQL warehouse. (Keywords: Lakebase, PostgreSQL, psycopg2, asyncpg, transactional, PGHOST) + +**MCP tools**: Use [6-mcp-approach.md](6-mcp-approach.md) for managing app lifecycle via MCP tools — covers creating, deploying, monitoring, and deleting apps programmatically. (Keywords: MCP, create app, deploy app, app logs) + +**Foundation Models**: See [examples/llm_config.py](examples/llm_config.py) for calling Databricks foundation model APIs — covers OAuth M2M auth, OpenAI-compatible client wiring, and token caching. (Keywords: foundation model, LLM, OpenAI client, chat completions) + +--- + +## Workflow + +1. Determine the task type: + + **New app from scratch?** → Use [AppKit](#appkit-preferred-for-new-apps) (`databricks apps init`). Fall back to [Python Framework Selection](#python-framework-selection) only if Python is required. + **Setting up authorization?** → Read [1-authorization.md](1-authorization.md) + **Connecting to data/resources?** → Read [2-app-resources.md](2-app-resources.md) + **Using Lakebase (PostgreSQL)?** → Read [5-lakebase.md](5-lakebase.md) + **Deploying to Databricks?** → Read [4-deployment.md](4-deployment.md) + **Using MCP tools?** → Read [6-mcp-approach.md](6-mcp-approach.md) + **Calling foundation model/LLM APIs?** → See [examples/llm_config.py](examples/llm_config.py) + +2. Follow the instructions in the relevant guide +3. For full code examples, browse https://apps-cookbook.dev/ + +--- + +## Core Architecture + +All Python Databricks apps follow this pattern: + +``` +app-directory/ +├── app.py # Main application (or framework-specific name) +├── models.py # Pydantic data models +├── backend.py # Data access layer +├── requirements.txt # Additional Python dependencies +├── app.yaml # Databricks Apps configuration +└── README.md +``` + +### Backend Toggle Pattern + +```python +import os +from databricks.sdk.core import Config + +USE_MOCK = os.getenv("USE_MOCK_BACKEND", "true").lower() == "true" + +if USE_MOCK: + from backend_mock import MockBackend as Backend +else: + from backend_real import RealBackend as Backend + +backend = Backend() +``` + +### SQL Warehouse Connection (shared across all frameworks) + +```python +from databricks.sdk.core import Config +from databricks import sql + +cfg = Config() # Auto-detects credentials from environment +conn = sql.connect( + server_hostname=cfg.host, + http_path=f"/sql/1.0/warehouses/{os.getenv('DATABRICKS_WAREHOUSE_ID')}", + credentials_provider=lambda: cfg.authenticate, +) +``` + +### Pydantic Models + +```python +from pydantic import BaseModel, Field +from datetime import datetime +from enum import Enum + +class Status(str, Enum): + ACTIVE = "active" + PENDING = "pending" + +class EntityOut(BaseModel): + id: str + name: str + status: Status + created_at: datetime + +class EntityIn(BaseModel): + name: str = Field(..., min_length=1) + status: Status = Status.PENDING +``` + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **Connection exhausted** | Use `@st.cache_resource` (Streamlit) or connection pooling | +| **Auth token not found** | Check `x-forwarded-access-token` header — only available when deployed, not locally | +| **App won't start** | Check `app.yaml` command matches framework; check `databricks apps logs ` | +| **Resource not accessible** | Add resource via UI, verify SP has permissions, use `valueFrom` in app.yaml | +| **Import error on deploy** | Add missing packages to `requirements.txt` (pre-installed packages don't need listing) | +| **Lakebase app crashes on start** | `psycopg2`/`asyncpg` are NOT pre-installed — MUST add to `requirements.txt` | +| **Port conflict** | Apps must bind to `DATABRICKS_APP_PORT` env var (defaults to 8000). Never use 8080. Streamlit is auto-configured; for others, read the env var in code or use 8000 in app.yaml command | +| **Streamlit: set_page_config error** | `st.set_page_config()` must be the first Streamlit command | +| **Dash: unstyled layout** | Add `dash-bootstrap-components`; use `dbc.themes.BOOTSTRAP` | +| **Slow queries** | Use Lakebase for transactional/low-latency; SQL warehouse for analytical queries | + +--- + +## Platform Constraints + +| Constraint | Details | +|------------|---------| +| **Runtime** | Python 3.11, Ubuntu 22.04 LTS | +| **Compute** | 2 vCPUs, 6 GB memory (default) | +| **Pre-installed frameworks** | Dash, Streamlit, Gradio, Flask, FastAPI, Shiny | +| **Custom packages** | Add to `requirements.txt` in app root | +| **Network** | Apps can reach Databricks APIs; external access depends on workspace config | +| **User auth** | Public Preview — workspace admin must enable before adding scopes | + +--- + +## Official Documentation + +- **[AppKit](https://databricks.github.io/appkit/docs/)** — preferred SDK for new apps (TypeScript + React) +- **[Databricks Apps Overview](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/)** — main docs hub +- **[Apps Cookbook](https://apps-cookbook.dev/)** — ready-to-use code snippets (Streamlit, Dash, Reflex, FastAPI) +- **[Authorization](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth)** — app auth and user auth +- **[Resources](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources)** — SQL warehouse, Lakebase, serving, secrets +- **[app.yaml Reference](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/app-runtime)** — command and env config +- **[System Environment](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/system-env)** — pre-installed packages, runtime details + +## Related Skills + +- **[databricks-app-apx](../databricks-app-apx/SKILL.md)** - full-stack apps with FastAPI + React +- **[databricks-bundles](../databricks-bundles/SKILL.md)** - deploying apps via DABs +- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** - backend SDK integration +- **[databricks-lakebase-provisioned](../databricks-lakebase-provisioned/SKILL.md)** - adding persistent PostgreSQL state +- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** - serving ML models for app integration diff --git a/.claude/skills/databricks-apps-python/examples/fm-minimal-chat.py b/.claude/skills/databricks-apps-python/examples/fm-minimal-chat.py new file mode 100644 index 00000000..920a4051 --- /dev/null +++ b/.claude/skills/databricks-apps-python/examples/fm-minimal-chat.py @@ -0,0 +1,182 @@ +""" +Minimal Databricks Foundation Model Chat App + +A complete, deployable Streamlit app demonstrating Foundation Model API integration +in Databricks Apps. This is a working example extracted from databricksters-check-and-pub. + +Features: +- Validated dual-mode auth (OAuth M2M in Apps, PAT for local dev) +- OpenAI SDK wired to Databricks serving endpoints +- Token caching with expiry check +- Multi-turn chat with conversation history +- Viewer identity display +- Latency tracking + +Local Development: + export DATABRICKS_TOKEN="dapi..." + export DATABRICKS_SERVING_BASE_URL="https:///serving-endpoints" + export DATABRICKS_MODEL="" # See databricks-model-serving + streamlit run fm-minimal-chat.py + +Databricks Apps Deployment: + 1. Create app.yaml: + command: ["streamlit", "run", "fm-minimal-chat.py"] + env: + - name: DATABRICKS_SERVING_BASE_URL + value: "https:///serving-endpoints" + - name: DATABRICKS_MODEL + value: "" # See databricks-model-serving + + 2. Create requirements.txt: + streamlit>=1.38,<2.0 + openai>=1.30,<2.0 + requests>=2.31,<3.0 # Needed for endpoint validation and OAuth fallback + + 3. Deploy: + databricks apps create foundation-chat --source-code-path . + + 4. Add service principal via UI for OAuth M2M auth +""" + +import time +from typing import Dict, List, Optional, Tuple + +import streamlit as st +from openai import OpenAI + +from llm_config import create_foundation_model_client, get_model_name + + +def _get_forwarded_headers() -> Dict[str, str]: + try: + return dict(getattr(st, "context").headers) + except Exception: + return {} + + +def get_viewer_identity() -> Tuple[Optional[str], Optional[str]]: + headers = _get_forwarded_headers() + email = headers.get("X-Forwarded-Email") or headers.get("x-forwarded-email") + token = headers.get("X-Forwarded-Access-Token") or headers.get( + "x-forwarded-access-token" + ) + return email, token + + +# ============================================================================= +# LLM Helper +# ============================================================================= +def llm_chat( + client: OpenAI, + *, + model: str, + messages: List[Dict[str, str]], + max_tokens: int = 1000, + temperature: float = 0.7, +) -> Tuple[str, int]: + """Call foundation model and return (response, latency_ms).""" + t0 = time.perf_counter() + resp = client.chat.completions.create( + model=model, + messages=messages, + max_tokens=max_tokens, + temperature=temperature, + ) + elapsed_ms = int((time.perf_counter() - t0) * 1000) + content = resp.choices[0].message.content or "" + return content, elapsed_ms + + +# ============================================================================= +# Streamlit App +# ============================================================================= +def main(): + st.set_page_config( + page_title="Databricks Foundation Model Chat", + page_icon="💬", + layout="centered", + ) + + st.title("💬 Foundation Model Chat") + st.caption("Powered by Databricks Apps") + + # Sidebar: viewer identity + viewer_email, _ = get_viewer_identity() + if viewer_email: + st.sidebar.success(f"Logged in as: {viewer_email}") + else: + st.sidebar.info("Local dev mode (no viewer identity)") + + # Sidebar: model config + with st.sidebar: + st.subheader("Configuration") + st.code(f"Model: {get_model_name()}", language=None) + + if st.button("🗑️ Clear Chat History"): + st.session_state.messages = [] + st.rerun() + + with st.expander("ℹ️ About"): + st.markdown( + """ + This app demonstrates calling Databricks Foundation Model APIs + from a Streamlit app using: + - Shared dual-mode auth (PAT + OAuth M2M) + - Shared OpenAI client wiring + - Viewer identity extraction + """ + ) + + # Initialize chat history + if "messages" not in st.session_state: + st.session_state.messages = [] + + # Display chat history + for message in st.session_state.messages: + with st.chat_message(message["role"]): + st.markdown(message["content"]) + if message.get("latency_ms"): + st.caption(f"⏱️ {message['latency_ms']}ms") + + # Chat input + if prompt := st.chat_input("Ask me anything..."): + # Add user message to chat history + st.session_state.messages.append({"role": "user", "content": prompt}) + with st.chat_message("user"): + st.markdown(prompt) + + # Generate assistant response + with st.chat_message("assistant"): + with st.spinner("Thinking..."): + try: + client = create_foundation_model_client(cache=st.session_state) + + # Call foundation model + response, latency_ms = llm_chat( + client, + model=get_model_name(), + messages=st.session_state.messages, + max_tokens=1000, + temperature=0.7, + ) + + # Display response + st.markdown(response) + st.caption(f"⏱️ {latency_ms}ms") + + # Add to chat history + st.session_state.messages.append( + { + "role": "assistant", + "content": response, + "latency_ms": latency_ms, + } + ) + + except Exception as e: + st.error(f"Error calling foundation model: {e}") + st.session_state.messages.pop() # Remove failed user message + + +if __name__ == "__main__": + main() diff --git a/.claude/skills/databricks-apps-python/examples/fm-parallel-calls.py b/.claude/skills/databricks-apps-python/examples/fm-parallel-calls.py new file mode 100644 index 00000000..71cd81b6 --- /dev/null +++ b/.claude/skills/databricks-apps-python/examples/fm-parallel-calls.py @@ -0,0 +1,267 @@ +""" +Parallel Foundation Model Calls + +This example demonstrates how to make multiple foundation model API calls in parallel +for improved performance. It uses the same bounded job-runner pattern as the +production Databricks App, but keeps the example generic enough to reuse in +other review, extraction, or scoring workflows. + +Use cases: +- Document evaluation with multiple independent checks +- Batch processing of independent prompts +- Multi-aspect analysis of the same content +- A/B testing different prompts + +Performance impact: +- Serial: 5 calls × 2s each = 10s total +- Parallel (max_workers=5): ~2s to 3s total depending on endpoint overhead + +Configuration: +- LLM_MAX_CONCURRENCY env var controls parallelism (positive integer, default: 5) +- Balance between throughput and rate limits +- DATABRICKS_MODEL must be set to a valid serving endpoint name +""" + +import time +from typing import Any, Callable, Dict, List, Tuple + +from openai import OpenAI + +from llm_config import ( + create_foundation_model_client, + get_model_name, + run_jobs_parallel, +) + + +# ============================================================================= +# LLM Call Helper +# ============================================================================= +def llm_call( + client: OpenAI, + prompt: str, + model: str | None = None, + max_tokens: int = 500, +) -> Tuple[str, int]: + """Make a single LLM call and return (response, latency_ms).""" + t0 = time.perf_counter() + resp = client.chat.completions.create( + model=model or get_model_name(), + messages=[{"role": "user", "content": prompt}], + max_tokens=max_tokens, + temperature=0.2, + ) + elapsed_ms = int((time.perf_counter() - t0) * 1000) + content = resp.choices[0].message.content or "" + return content, elapsed_ms + + +# ============================================================================= +# Example: Generic Technical Document Checks +# ============================================================================= +def check_structure(client: OpenAI, text: str) -> Dict[str, Any]: + """Check if a technical document has clear section structure.""" + prompt = f"""Evaluate the structure of this technical document. Does it have clear section headings and a logical progression? + +DOCUMENT: +{text[:2000]} + +Answer with: PASS or FAIL, then brief explanation.""" + + response, latency_ms = llm_call(client, prompt) + passed = "PASS" in response.upper().split("\n")[0] + + return { + "check": "structure", + "passed": passed, + "response": response, + "latency_ms": latency_ms, + } + + +def check_summary(client: OpenAI, text: str) -> Dict[str, Any]: + """Check if content has a concise executive summary near the top.""" + prompt = f"""Does this technical document start with a concise summary or key takeaways section in the first 10 percent? + +DOCUMENT: +{text[:2000]} + +Answer with: PASS or FAIL, then brief explanation.""" + + response, latency_ms = llm_call(client, prompt) + passed = "PASS" in response.upper().split("\n")[0] + + return { + "check": "summary", + "passed": passed, + "response": response, + "latency_ms": latency_ms, + } + + +def check_examples(client: OpenAI, text: str) -> Dict[str, Any]: + """Check if content includes concrete examples.""" + prompt = f"""Does this technical document include concrete examples, code, or step-by-step guidance readers can adapt? + +DOCUMENT: +{text[:2000]} + +Answer with: PASS or FAIL, then brief explanation.""" + + response, latency_ms = llm_call(client, prompt) + passed = "PASS" in response.upper().split("\n")[0] + + return { + "check": "examples", + "passed": passed, + "response": response, + "latency_ms": latency_ms, + } + + +def check_troubleshooting(client: OpenAI, text: str) -> Dict[str, Any]: + """Check if content covers troubleshooting or failure modes.""" + prompt = f"""Does this technical document include troubleshooting guidance, failure modes, or common pitfalls? + +DOCUMENT: +{text[:2000]} + +Answer with: PASS or FAIL, then brief explanation.""" + + response, latency_ms = llm_call(client, prompt) + passed = "PASS" in response.upper().split("\n")[0] + + return { + "check": "troubleshooting", + "passed": passed, + "response": response, + "latency_ms": latency_ms, + } + + +def check_audience_fit(client: OpenAI, text: str) -> Dict[str, Any]: + """Check if content matches a technical practitioner audience.""" + prompt = f"""Does this technical document appear written for practitioners, with the right level of specificity and useful context? + +DOCUMENT: +{text[:2000]} + +Answer with: PASS or FAIL, then brief explanation.""" + + response, latency_ms = llm_call(client, prompt) + passed = "PASS" in response.upper().split("\n")[0] + + return { + "check": "audience_fit", + "passed": passed, + "response": response, + "latency_ms": latency_ms, + } + + +# ============================================================================= +# Example Usage: Parallel Execution +# ============================================================================= +if __name__ == "__main__": + # Sample technical document + sample_text = """ + Summary: This guide shows how to deploy a Databricks App in three steps. + + ## Introduction + Databricks Apps provides a way to deploy web applications... + + ## Step 1: Create Your App + First, create an app.py file... + + ## Step 2: Configure app.yaml + Next, set up your configuration... + + ## Step 3: Deploy + Finally, deploy using the CLI... + """ + + client = create_foundation_model_client() + + print("Making 5 parallel LLM calls...") + print(f"Model: {get_model_name()}\n") + + # Define independent parallel jobs + jobs = { + "structure": (check_structure, (client, sample_text), {}), + "summary": (check_summary, (client, sample_text), {}), + "examples": (check_examples, (client, sample_text), {}), + "troubleshooting": (check_troubleshooting, (client, sample_text), {}), + "audience_fit": (check_audience_fit, (client, sample_text), {}), + } + + # Execute in parallel using the shared bounded job runner. + start = time.perf_counter() + results, errors = run_jobs_parallel(jobs) + total_time = time.perf_counter() - start + + # Display results + print("=" * 60) + print(f"Completed in {total_time:.2f}s (parallel execution)") + print("=" * 60) + + if errors: + print("\nErrors encountered:") + for error in errors: + print(f" ❌ {error}") + + print("\nResults:") + for job_name, result in results.items(): + if result: + status = "✅ PASS" if result["passed"] else "❌ FAIL" + print(f"\n{job_name.upper()}: {status}") + print(f" Latency: {result['latency_ms']}ms") + print(f" Response: {result['response'][:150]}...") + else: + print(f"\n{job_name.upper()}: ❌ FAILED (see errors above)") + + # Calculate time saved + total_latency = sum(r["latency_ms"] for r in results.values() if r) + time_saved = (total_latency / 1000) - total_time + print(f"\n{'='*60}") + print(f"Time saved vs serial execution: {time_saved:.2f}s") + if total_time > 0: + print(f"Speedup: {(total_latency/1000) / total_time:.1f}×") + else: + print("Speedup: N/A (total_time below resolution)") + print(f"{'='*60}") + + +# ============================================================================= +# Production Best Practices +# ============================================================================= +# +# Best practices from databricksters-check-and-pub: +# +# 1. Configurable concurrency +# - Use LLM_MAX_CONCURRENCY env var (default: 5 in the production app) +# - Balance throughput vs rate limits +# - Too high = rate limit errors +# - Too low = underutilized resources +# +# 2. Error handling +# - Capture exceptions per job +# - Return None for failed jobs +# - Collect error messages for debugging +# - Continue execution even if some jobs fail +# +# 3. Bounded execution +# - Only parallelize independent checks +# - Cap concurrency with an env var rather than firing unlimited requests +# - Keep the job contract simple: name -> (callable, args, kwargs) +# +# 4. When to use parallel calls +# - Multiple independent evaluations of same content +# - Batch processing multiple documents +# - A/B testing different prompts +# - Multi-aspect analysis +# +# 5. When NOT to use parallel calls +# - Dependent/sequential operations +# - Single evaluation needed +# - Rate limits are very strict +# - Debugging (use serial for easier troubleshooting) diff --git a/.claude/skills/databricks-apps-python/examples/fm-structured-outputs.py b/.claude/skills/databricks-apps-python/examples/fm-structured-outputs.py new file mode 100644 index 00000000..90fe6d27 --- /dev/null +++ b/.claude/skills/databricks-apps-python/examples/fm-structured-outputs.py @@ -0,0 +1,337 @@ +""" +Structured Outputs and Robust Response Parsing + +Production patterns for getting structured data (JSON) from foundation models. +Extracted from databricksters-check-and-pub production app. + +Key patterns: +1. Robust JSON parsing (handles code fences, smart quotes, malformed JSON) +2. Retry logic on parse failure with stricter prompts +3. Content normalization (handles various response formats) +4. temperature=0.0 for deterministic structured outputs +5. Streamlit caching for expensive API calls +6. Consistent timeout handling + +Use cases: +- Content evaluation/scoring +- Data extraction from text +- Classification tasks +- Compliance checking +- Any task requiring structured model output + +Set `DATABRICKS_MODEL` to a valid serving endpoint name before running. +""" + +import json +import re +import time +from typing import Any, Dict, List, Tuple + +import streamlit as st +from openai import OpenAI + +from llm_config import create_foundation_model_client, get_model_name + + +# ============================================================================= +# Pattern 1: Content Normalization +# ============================================================================= +def _content_to_text(content: Any) -> str: + """Normalize model message content to a string. + + Handles various content types returned by foundation models: + - str: return as-is + - bytes: decode to UTF-8 + - list: extract text from content parts (handles multi-modal responses) + + This is critical for handling different response formats consistently. + """ + if isinstance(content, str): + return content + + if isinstance(content, (bytes, bytearray)): + return content.decode("utf-8", errors="replace") + + if isinstance(content, list): + parts: List[str] = [] + for item in content: + if isinstance(item, str): + parts.append(item) + elif isinstance(item, dict): + # Handle content part objects + if "text" in item and isinstance(item["text"], str): + parts.append(item["text"]) + elif "content" in item and isinstance(item["content"], str): + parts.append(item["content"]) + return "".join(parts) + + return str(content) + + +# ============================================================================= +# Pattern 2: Robust JSON Parsing +# ============================================================================= +def _parse_json_object(response_text: str) -> Dict[str, Any]: + """Best-effort parse of a JSON object from a model response. + + Handles common failure modes: + 1. Model wraps JSON in markdown code fences (```json ... ```) + 2. Model uses smart/curly quotes instead of straight quotes + 3. Model includes extra text before/after JSON + 4. Model returns malformed JSON + + This is THE critical pattern for production structured outputs. + """ + text = (response_text or "").strip() + if not text: + raise ValueError("Empty model response (expected JSON object)") + + # Strip markdown code fences if present + if text.startswith("```"): + text = re.sub(r"^```[a-zA-Z]*\n", "", text) + text = re.sub(r"```$", "", text).strip() + + # Try direct parse first + try: + obj = json.loads(text) + if isinstance(obj, dict): + return obj + except Exception: + pass + + # Extract first {...} block (handles extra text around JSON) + start = text.find("{") + end = text.rfind("}") + if start != -1 and end != -1 and end > start: + candidate = text[start : end + 1] + else: + candidate = text + + # Normalize smart quotes (common LLM formatting issue) + candidate = ( + candidate.replace("\u201c", '"') # Left double quote + .replace("\u201d", '"') # Right double quote + .replace("\u2018", "'") # Left single quote + .replace("\u2019", "'") # Right single quote + ) + + # Final parse attempt + obj = json.loads(candidate) + if not isinstance(obj, dict): + raise ValueError("Model did not return a JSON object") + return obj + + +# ============================================================================= +# Pattern 3: Structured LLM Call with Retry +# ============================================================================= +def llm_structured_call( + client: OpenAI, + system_prompt: str, + user_prompt: str, + model: str | None = None, +) -> Tuple[Dict[str, Any], int]: + """Call foundation model for structured output with retry on parse failure. + + Returns: + (parsed_json_dict, latency_ms) + + Critical pattern: + - Use temperature=0.0 for deterministic structured outputs + - If JSON parse fails, retry with stricter instructions + - Combine latencies from both attempts + """ + # First attempt + t0 = time.perf_counter() + response = client.chat.completions.create( + model=model or get_model_name(), + messages=[ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt}, + ], + max_tokens=2000, + temperature=0.0, # Deterministic for structured outputs + ) + elapsed_ms = int((time.perf_counter() - t0) * 1000) + + content = _content_to_text(response.choices[0].message.content) + + # Try to parse response + try: + return _parse_json_object(content), elapsed_ms + except Exception as e: + # Retry with stricter prompt + print(f"Parse failed (attempt 1): {e}. Retrying with stricter prompt...") + + t0_retry = time.perf_counter() + retry_response = client.chat.completions.create( + model=model or get_model_name(), + messages=[ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": "Return ONLY minified JSON object. Strings must be JSON-escaped. No extra text."}, + {"role": "user", "content": user_prompt}, + ], + max_tokens=2000, + temperature=0.0, + ) + retry_elapsed_ms = int((time.perf_counter() - t0_retry) * 1000) + + retry_content = _content_to_text(retry_response.choices[0].message.content) + return _parse_json_object(retry_content), elapsed_ms + retry_elapsed_ms + + +# ============================================================================= +# Pattern 4: Caching Expensive Calls (Streamlit) +# ============================================================================= +@st.cache_data(ttl=60 * 60) # Cache for 1 hour +def cached_structured_call( + prompt: str, + model: str | None = None, +) -> Dict[str, Any]: + """Cache expensive structured LLM calls. + + Use @st.cache_data with TTL for: + - Expensive/slow API calls + - Calls with same inputs (idempotent) + - Data that doesn't need real-time freshness + + TTL examples: + - 60 * 10 = 10 minutes (frequently changing data) + - 60 * 60 = 1 hour (moderate freshness) + - 60 * 60 * 24 = 24 hours (stable data) + """ + client = create_foundation_model_client() + system = "You are a data extraction assistant. Return ONLY valid JSON." + result, _ = llm_structured_call(client, system, prompt, model or get_model_name()) + return result + + +# ============================================================================= +# Example: Content Quality Evaluation +# ============================================================================= +def evaluate_content_quality( + client: OpenAI, text: str +) -> Tuple[Dict[str, Any], int]: + """Evaluate content quality with structured output.""" + + system_prompt = """You are a content quality evaluator. +You must return ONLY valid JSON that exactly matches the schema below. +No commentary. No markdown. No explanations.""" + + user_prompt = f"""Evaluate this content and return JSON with this exact schema: +{{ + "overall_score": 0-100, + "readability": "poor"|"fair"|"good"|"excellent", + "has_clear_structure": true|false, + "has_actionable_takeaways": true|false, + "strengths": ["string", "string"], + "weaknesses": ["string", "string"], + "suggestions": ["string", "string"] +}} + +Content to evaluate: +{text[:2000]} +""" + + return llm_structured_call(client, system_prompt, user_prompt) + + +# ============================================================================= +# Example: Entity Extraction +# ============================================================================= +def extract_entities(client: OpenAI, text: str) -> Tuple[Dict[str, Any], int]: + """Extract structured entities from text.""" + + system_prompt = """You are an entity extraction system. +Return ONLY valid JSON. Do not include explanations.""" + + user_prompt = f"""Extract entities from this text and return JSON: +{{ + "people": ["name1", "name2"], + "organizations": ["org1", "org2"], + "technologies": ["tech1", "tech2"], + "key_concepts": ["concept1", "concept2"] +}} + +Text: +{text[:2000]} +""" + + return llm_structured_call(client, system_prompt, user_prompt) + + +# ============================================================================= +# Example Usage +# ============================================================================= +if __name__ == "__main__": + sample_text = """ + Databricks Lakehouse Platform combines data warehousing and AI with open + data formats like Delta Lake. Apache Spark and MLflow are key components. + Jane Smith, VP of Engineering at Acme Corp, recently shared their migration story. + """ + + client = create_foundation_model_client() + + print("=" * 60) + print("Example 1: Content Quality Evaluation") + print("=" * 60) + try: + quality_data, latency_ms = evaluate_content_quality(client, sample_text) + print(f"✓ Completed in {latency_ms}ms") + print(json.dumps(quality_data, indent=2)) + except Exception as e: + print(f"❌ Error: {e}") + + print("\n" + "=" * 60) + print("Example 2: Entity Extraction") + print("=" * 60) + try: + entity_data, latency_ms = extract_entities(client, sample_text) + print(f"✓ Completed in {latency_ms}ms") + print(json.dumps(entity_data, indent=2)) + except Exception as e: + print(f"❌ Error: {e}") + + +# ============================================================================= +# Production Best Practices Summary +# ============================================================================= +""" +Key takeaways from databricksters-check-and-pub: + +1. Content Normalization (_content_to_text) + - Handle str, bytes, list content types + - Essential for multi-modal or varying response formats + +2. Robust JSON Parsing (_parse_json_object) + - Strip markdown code fences (```json) + - Normalize smart quotes + - Extract {...} from surrounding text + - This ONE function prevents 90% of parsing errors in production + +3. Retry on Parse Failure + - If first attempt fails to parse, retry with stricter prompt + - Add latencies together for accurate tracking + - Shows user total cost, not just successful attempt + +4. Temperature Settings + - Use temperature=0.0 for structured outputs (deterministic) + - Use temperature=0.2-0.7 for creative/generative tasks + - Compliance checks = 0.0, content generation = 0.7 + +5. Caching with TTL + - Use @st.cache_data(ttl=...) for expensive calls + - Choose TTL based on data freshness needs + - Dramatically improves app responsiveness + +6. Timeouts + - Set timeout=30 on all HTTP requests + - Prevents hanging connections + - Provides better error messages to users + +7. System Prompts for Structure + - Clearly state: "Return ONLY valid JSON" + - Provide exact schema in prompt + - Use examples when needed + - Be explicit about constraints +""" diff --git a/.claude/skills/databricks-apps-python/examples/llm_config.py b/.claude/skills/databricks-apps-python/examples/llm_config.py new file mode 100644 index 00000000..200aaef9 --- /dev/null +++ b/.claude/skills/databricks-apps-python/examples/llm_config.py @@ -0,0 +1,354 @@ +import concurrent.futures +import os +import threading +import time +from collections.abc import MutableMapping as MutableMappingABC +from dataclasses import dataclass +from typing import Any, Callable, Dict, MutableMapping, Tuple +from urllib.parse import urlsplit + +from openai import OpenAI + +CACHE_KEY = "dbx_oauth" +VALIDATION_TTL_SECONDS = 300 + + +class DatabricksLLMConfigError(RuntimeError): + """Raised when Databricks LLM configuration is invalid.""" + + +@dataclass(frozen=True) +class DatabricksLLMConfig: + serving_base_url: str + workspace_host: str + model: str + auth_mode: str + + +_token_lock = threading.Lock() +_token_cache: Dict[str, Any] = {} +_validation_cache: Dict[Tuple[str, str], int] = {} + + +def _requests_module(): + import requests + + return requests + + +def _normalize_host(raw_host: str) -> str: + host = (raw_host or "").strip().rstrip("/") + if not host: + raise DatabricksLLMConfigError("Databricks workspace host is empty.") + if not host.startswith(("http://", "https://")): + host = "https://" + host + parts = urlsplit(host) + if not parts.scheme or not parts.netloc: + raise DatabricksLLMConfigError(f"Invalid Databricks workspace host: {raw_host!r}") + return f"{parts.scheme}://{parts.netloc}" + + +def _normalize_serving_base_url(raw_url: str) -> str: + value = (raw_url or "").strip() + if not value: + raise DatabricksLLMConfigError( + "DATABRICKS_SERVING_BASE_URL must be set to https:///serving-endpoints." + ) + if not value.startswith(("http://", "https://")): + value = "https://" + value + parts = urlsplit(value) + if not parts.scheme or not parts.netloc: + raise DatabricksLLMConfigError(f"Invalid DATABRICKS_SERVING_BASE_URL: {raw_url!r}") + path = parts.path.rstrip("/") + if path != "/serving-endpoints": + raise DatabricksLLMConfigError( + "DATABRICKS_SERVING_BASE_URL must end with /serving-endpoints for the target workspace." + ) + return f"{parts.scheme}://{parts.netloc}/serving-endpoints" + + +def get_databricks_llm_config() -> DatabricksLLMConfig: + serving_base_url = _normalize_serving_base_url( + os.environ.get("DATABRICKS_SERVING_BASE_URL", "") + ) + workspace_host = serving_base_url[: -len("/serving-endpoints")] + + configured_host = os.environ.get("DATABRICKS_HOST", "").strip() + if configured_host: + normalized_host = _normalize_host(configured_host) + if normalized_host != workspace_host: + raise DatabricksLLMConfigError( + "DATABRICKS_HOST must match the workspace host in DATABRICKS_SERVING_BASE_URL." + ) + + model = os.environ.get("DATABRICKS_MODEL", "").strip() + if not model: + raise DatabricksLLMConfigError( + "DATABRICKS_MODEL must be set to a serving endpoint available in the workspace." + ) + + client_id = os.environ.get("DATABRICKS_CLIENT_ID", "").strip() + client_secret = os.environ.get("DATABRICKS_CLIENT_SECRET", "").strip() + token = os.environ.get("DATABRICKS_TOKEN", "").strip() + + if client_id and client_secret: + auth_mode = "oauth-m2m" + elif token: + auth_mode = "pat" + else: + raise DatabricksLLMConfigError( + "No Databricks auth configured. Set DATABRICKS_CLIENT_ID and " + "DATABRICKS_CLIENT_SECRET, or provide DATABRICKS_TOKEN." + ) + + return DatabricksLLMConfig( + serving_base_url=serving_base_url, + workspace_host=workspace_host, + model=model, + auth_mode=auth_mode, + ) + + +def get_serving_base_url() -> str: + return get_databricks_llm_config().serving_base_url + + +def get_model_name() -> str: + return get_databricks_llm_config().model + + +def _is_token_fresh(cache: MutableMapping[str, Any] | Dict[str, Any]) -> bool: + return bool( + cache.get("access_token") + and int(cache.get("expires_at", 0)) > int(time.time()) + 30 + ) + + +def _write_token_cache( + access_token: str, + expires_at: int, + config: DatabricksLLMConfig, + cache: MutableMapping[str, Any] | None = None, +) -> None: + token_record = { + "access_token": access_token, + "expires_at": expires_at, + "workspace_host": config.workspace_host, + "auth_mode": config.auth_mode, + "client_id": os.environ.get("DATABRICKS_CLIENT_ID", "").strip(), + } + _token_cache.clear() + _token_cache.update(token_record) + if cache is not None: + cache[CACHE_KEY] = dict(token_record) + + +def _token_cache_matches( + cache: MutableMapping[str, Any] | Dict[str, Any], + config: DatabricksLLMConfig, +) -> bool: + return bool( + cache.get("workspace_host") == config.workspace_host + and cache.get("auth_mode") == config.auth_mode + and cache.get("client_id", "") == os.environ.get("DATABRICKS_CLIENT_ID", "").strip() + ) + + +def get_databricks_bearer_token( + cache: MutableMapping[str, Any] | None = None, +) -> str: + config = get_databricks_llm_config() + + if config.auth_mode == "pat": + return os.environ["DATABRICKS_TOKEN"].strip() + + if cache: + cached = cache.get(CACHE_KEY, {}) + if ( + isinstance(cached, MutableMappingABC) + and _token_cache_matches(cached, config) + and _is_token_fresh(cached) + ): + _write_token_cache( + str(cached["access_token"]), + int(cached["expires_at"]), + config, + cache=cache, + ) + return str(cached["access_token"]) + + if _token_cache_matches(_token_cache, config) and _is_token_fresh(_token_cache): + access_token = str(_token_cache["access_token"]) + expires_at = int(_token_cache["expires_at"]) + _write_token_cache(access_token, expires_at, config, cache=cache) + return access_token + + with _token_lock: + if _token_cache_matches(_token_cache, config) and _is_token_fresh(_token_cache): + access_token = str(_token_cache["access_token"]) + expires_at = int(_token_cache["expires_at"]) + _write_token_cache(access_token, expires_at, config, cache=cache) + return access_token + + requests = _requests_module() + try: + response = requests.post( + f"{config.workspace_host}/oidc/v1/token", + headers={"Content-Type": "application/x-www-form-urlencoded"}, + data={"grant_type": "client_credentials", "scope": "all-apis"}, + auth=( + os.environ["DATABRICKS_CLIENT_ID"].strip(), + os.environ["DATABRICKS_CLIENT_SECRET"].strip(), + ), + timeout=30, + ) + except Exception as exc: + raise DatabricksLLMConfigError( + f"Could not reach Databricks OAuth token endpoint for " + f"{config.workspace_host}: {type(exc).__name__}: {str(exc)[:200]}" + ) from exc + if response.status_code >= 400: + raise DatabricksLLMConfigError( + f"Failed Databricks OAuth authentication for {config.workspace_host} " + f"(HTTP {response.status_code}). Check the service principal credentials " + "for that workspace." + ) + + payload = response.json() + access_token = payload.get("access_token") + expires_in = int(payload.get("expires_in", 300)) + if not access_token: + payload_keys = sorted(payload.keys()) if isinstance(payload, dict) else [] + raise DatabricksLLMConfigError( + "Token endpoint response is missing access_token " + f"(keys present: {payload_keys})" + ) + + expires_at = int(time.time()) + expires_in + _write_token_cache(str(access_token), expires_at, config, cache=cache) + return str(access_token) + + +def validate_databricks_llm_config( + cache: MutableMapping[str, Any] | None = None, +) -> DatabricksLLMConfig: + config = get_databricks_llm_config() + cache_key = (config.serving_base_url, config.model) + + cached_expiry = _validation_cache.get(cache_key, 0) + if cached_expiry > int(time.time()): + return config + + requests = _requests_module() + token = get_databricks_bearer_token(cache=cache) + headers = {"Authorization": f"Bearer {token}"} + endpoint_url = f"{config.workspace_host}/api/2.0/serving-endpoints/{config.model}" + try: + response = requests.get(endpoint_url, headers=headers, timeout=30) + except Exception as exc: + raise DatabricksLLMConfigError( + f"Could not validate DATABRICKS_MODEL={config.model!r} in workspace " + f"{config.workspace_host}: {type(exc).__name__}: {str(exc)[:200]}" + ) from exc + + if response.status_code == 404: + try: + list_response = requests.get( + f"{config.workspace_host}/api/2.0/serving-endpoints", + headers=headers, + timeout=30, + ) + except Exception: + list_response = None + available: list[str] = [] + if list_response is not None and list_response.status_code < 400: + try: + payload = list_response.json() + available = sorted( + endpoint.get("name", "").strip() + for endpoint in payload.get("endpoints", []) + if endpoint.get("name", "").strip() + ) + except Exception: + available = [] + available_text = ", ".join(available[:10]) if available else "no endpoints were returned" + raise DatabricksLLMConfigError( + f"DATABRICKS_MODEL={config.model!r} was not found in workspace " + f"{config.workspace_host}. Available endpoints include: {available_text}." + ) + + if response.status_code >= 400: + raise DatabricksLLMConfigError( + f"Failed to validate DATABRICKS_MODEL={config.model!r} in workspace " + f"{config.workspace_host} (HTTP {response.status_code})." + ) + + _validation_cache[cache_key] = int(time.time()) + VALIDATION_TTL_SECONDS + return config + + +def build_openai_client( + *, + validate: bool = True, + cache: MutableMapping[str, Any] | None = None, +) -> OpenAI: + config = ( + validate_databricks_llm_config(cache=cache) + if validate + else get_databricks_llm_config() + ) + token = get_databricks_bearer_token(cache=cache) + return OpenAI(api_key=token, base_url=config.serving_base_url) + + +def create_foundation_model_client( + cache: MutableMapping[str, Any] | None = None, +) -> OpenAI: + return build_openai_client(validate=True, cache=cache) + + +def resolve_bearer_token(cache: MutableMapping[str, Any] | None = None) -> str: + return get_databricks_bearer_token(cache=cache) + + +def run_jobs_parallel( + jobs: Dict[str, Tuple[Callable[..., Any], Tuple[Any, ...], Dict[str, Any]]], + max_workers: int | None = None, +) -> Tuple[Dict[str, Any], list[str]]: + """Run independent jobs in parallel and collect per-job failures.""" + if max_workers is None: + raw_worker_count = os.environ.get("LLM_MAX_CONCURRENCY", "5") + try: + worker_count = int(raw_worker_count) + except ValueError as exc: + raise DatabricksLLMConfigError( + "LLM_MAX_CONCURRENCY must be a positive integer." + ) from exc + else: + worker_count = max_workers + + if worker_count < 1: + raise DatabricksLLMConfigError( + "LLM_MAX_CONCURRENCY must be a positive integer." + ) + + results: Dict[str, Any] = {} + errors: list[str] = [] + + def _call(fn: Callable[..., Any], args: Tuple[Any, ...], kwargs: Dict[str, Any]) -> Any: + return fn(*args, **kwargs) + + with concurrent.futures.ThreadPoolExecutor(max_workers=worker_count) as executor: + futures = { + executor.submit(_call, fn, args, kwargs): name + for name, (fn, args, kwargs) in jobs.items() + } + concurrent.futures.wait(list(futures.keys())) + for future, name in [(future, futures[future]) for future in futures]: + try: + results[name] = future.result() + except Exception as exc: + errors.append(f"{name}: {type(exc).__name__}: {str(exc)[:200]}") + results[name] = None + + return results, errors diff --git a/.claude/skills/asset-bundles/SDP_guidance.md b/.claude/skills/databricks-bundles/SDP_guidance.md similarity index 100% rename from .claude/skills/asset-bundles/SDP_guidance.md rename to .claude/skills/databricks-bundles/SDP_guidance.md diff --git a/.claude/skills/asset-bundles/SKILL.md b/.claude/skills/databricks-bundles/SKILL.md similarity index 85% rename from .claude/skills/asset-bundles/SKILL.md rename to .claude/skills/databricks-bundles/SKILL.md index 02e11d00..3cff53a1 100644 --- a/.claude/skills/asset-bundles/SKILL.md +++ b/.claude/skills/databricks-bundles/SKILL.md @@ -1,9 +1,9 @@ --- -name: asset-bundles -description: "Create and configure Databricks Asset Bundles (DABs) with best practices for multi-environment deployments. Use when working with: (1) Creating new DAB projects, (2) Adding resources (dashboards, pipelines, jobs, alerts), (3) Configuring multi-environment deployments, (4) Setting up permissions, (5) Deploying or running bundle resources" +name: databricks-bundles +description: "Create and configure Declarative Automation Bundles (formerly Asset Bundles) with best practices for multi-environment deployments (CICD). Use when working with: (1) Creating new DAB projects, (2) Adding resources (dashboards, pipelines, jobs, alerts), (3) Configuring multi-environment deployments, (4) Setting up permissions, (5) Deploying or running bundle resources" --- -# Databricks Asset Bundle (DABs) Writer +# DABs Writer ## Overview Create DABs for multi-environment deployment (dev/staging/prod). @@ -61,6 +61,8 @@ targets: ### Dashboard Resources +**Support for dataset_catalog and dataset_schema parameters added in Databricks CLI 0.281.0 (January 2026)** + ```yaml resources: dashboards: @@ -68,6 +70,8 @@ resources: display_name: "[${bundle.target}] Dashboard Title" file_path: ../src/dashboards/dashboard.lvdash.json # Relative to resources/ warehouse_id: ${var.warehouse_id} + dataset_catalog: ${var.catalog} # Default catalog used by all datasets in the dashboard if not otherwise specified in the query + dataset_schema: ${var.schema} # Default schema used by all datasets in the dashboard if not otherwise specified in the query permissions: - level: CAN_RUN group_name: "users" @@ -289,7 +293,7 @@ databricks bundle destroy -t prod --auto-approve | **Catalog doesn't exist** | Create catalog first or update variable | | **"admins" group error on jobs** | Cannot modify admins permissions on jobs | | **Volume permissions** | Use `grants` not `permissions` for volumes | -| **Hardcoded catalog in dashboard** | Create environment-specific files or parameterize JSON | +| **Hardcoded catalog in dashboard** | Use dataset_catalog parameter (CLI v0.281.0+), create environment-specific files, or parameterize JSON | | **App not starting after deploy** | Apps require `databricks bundle run ` to start | | **App env vars not working** | Environment variables go in `app.yaml` (source dir), not databricks.yml | | **Wrong app source path** | Use `../` from resources/ dir if source is in project root | @@ -303,9 +307,17 @@ databricks bundle destroy -t prod --auto-approve 4. **Groups**: Use `"users"` for all workspace users 5. **Job permissions**: Verify custom groups exist; can't modify "admins" +## Related Skills + +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - pipeline definitions referenced by DABs +- **[databricks-app-apx](../databricks-app-apx/SKILL.md)** - app deployment via DABs +- **[databricks-apps-python](../databricks-apps-python/SKILL.md)** - Python app deployment via DABs +- **[databricks-config](../databricks-config/SKILL.md)** - profile and authentication setup for CLI/SDK +- **[databricks-jobs](../databricks-jobs/SKILL.md)** - job orchestration managed through bundles + ## Resources -- [Databricks Asset Bundles Documentation](https://docs.databricks.com/dev-tools/bundles/) +- [DABs Documentation](https://docs.databricks.com/dev-tools/bundles/) - [Bundle Resources Reference](https://docs.databricks.com/dev-tools/bundles/resources) - [Bundle Configuration Reference](https://docs.databricks.com/dev-tools/bundles/settings) - [Supported Resource Types](https://docs.databricks.com/aws/en/dev-tools/bundles/resources#resource-types) diff --git a/.claude/skills/asset-bundles/alerts_guidance.md b/.claude/skills/databricks-bundles/alerts_guidance.md similarity index 100% rename from .claude/skills/asset-bundles/alerts_guidance.md rename to .claude/skills/databricks-bundles/alerts_guidance.md diff --git a/.claude/skills/databricks-config/SKILL.md b/.claude/skills/databricks-config/SKILL.md index 12952293..118713d1 100644 --- a/.claude/skills/databricks-config/SKILL.md +++ b/.claude/skills/databricks-config/SKILL.md @@ -1,74 +1,22 @@ --- name: databricks-config -description: Configure Databricks profile and authenticate for Databricks Connect, Databricks CLI, and Databricks SDK. +description: "Manage Databricks workspace connections: check current workspace, switch profiles, list available workspaces, or authenticate to a new workspace. Use when the user mentions \"switch workspace\", \"which workspace\", \"current profile\", \"databrickscfg\", \"connect to workspace\", or \"databricks auth\"." --- -Configure the Databricks profile in ~/.databrickscfg for use with Databricks Connect. +Use the `manage_workspace` MCP tool for all workspace operations. Do NOT edit `~/.databrickscfg`, use Bash, or use the Databricks CLI. -**Usage:** `/databricks-config [profile_name|workspace_host]` +## Steps -Examples: -- `/databricks-config` - Configure DEFAULT profile (interactive) -- `/databricks-config DEFAULT` - Configure DEFAULT profile -- `/databricks-config my-workspace` - Configure profile named "my-workspace" -- `/databricks-config https://adb-1234567890123456.7.azuredatabricks.net/` - Configure using workspace host URL +1. Call `ToolSearch` with query `select:mcp__databricks__manage_workspace` to load the tool. -## Task +2. Map user intent to action: + - status / which workspace / current → `action="status"` + - list / available workspaces → `action="list"` + - switch to X → call `list` first to find the profile name, then `action="switch", profile=""` (or `host=""` if a URL was given) + - login / connect / authenticate → `action="login", host=""` -1. Determine the profile and host: - - If a parameter is provided and it starts with `https://`, treat it as a workspace host: - - Extract profile name from the host (e.g., `adb-1234567890123456.7.azuredatabricks.net` → `adb-1234567890123456`, `my-company-dev.cloud.databricks.com` → `my-company-dev`) - - Use this as the profile name and configure it with the provided host - - If a parameter is provided and it doesn't start with `https://`, treat it as a profile name - - If no parameter is provided, ask the user which profile they want to configure (default: DEFAULT) +3. Call `mcp__databricks__manage_workspace` with the action and any parameters. -2. Run `databricks auth login -p ` with the determined profile name - - If a workspace host was provided, add `--host ` to the command - - This ensures authentication is completed and the profile works -3. Check if the profile exists in ~/.databrickscfg -4. Ask the user to choose ONE of the following compute options: - - **Cluster ID**: Provide a specific cluster ID for an interactive/all-purpose cluster - - **Serverless**: Use serverless compute (sets `serverless_compute_id = auto`) -5. Update the profile in ~/.databrickscfg with the selected configuration -6. Verify the configuration by displaying the updated profile section +4. Present the result. For `status`/`switch`/`login`: show host, profile, username. For `list`: formatted table with the active profile marked. -## Important Notes - -- Use the AskUserQuestion tool to present the compute options as a choice -- Only add ONE of: `cluster_id` OR `serverless_compute_id` (never both) -- For serverless, set `serverless_compute_id = auto` (not just `serverless = true`) -- Preserve all existing settings in the profile (host, auth_type, etc.) -- Format the configuration file consistently with proper spacing -- The `databricks auth login` command will open a browser for OAuth authentication -- **SECURITY: NEVER print token values in plain text** - - When displaying configuration, redact any `token` field values (e.g., `token = [REDACTED]`) - - Inform the user they can view the full configuration at `~/.databrickscfg` - - This applies to any output showing the profile configuration - -## Example Configurations - -**With Cluster ID:** -``` -[DEFAULT] -host = https://adb-123456789.11.azuredatabricks.net/ -cluster_id = 1217-064531-c9c3ngyn -auth_type = databricks-cli -``` - -**With Serverless:** -``` -[DEFAULT] -host = https://adb-123456789.11.azuredatabricks.net/ -serverless_compute_id = auto -auth_type = databricks-cli -``` - -**With Token (display as redacted):** -``` -[DEFAULT] -host = https://adb-123456789.11.azuredatabricks.net/ -token = [REDACTED] -cluster_id = 1217-064531-c9c3ngyn - -View full configuration at: ~/.databrickscfg -``` +> **Note:** The switch is session-scoped — it resets on MCP server restart. For permanent profile setup, use `databricks auth login -p ` and update `~/.databrickscfg` with `cluster_id` or `serverless_compute_id = auto`. diff --git a/.claude/skills/databricks-dbsql/SKILL.md b/.claude/skills/databricks-dbsql/SKILL.md new file mode 100644 index 00000000..24bf2694 --- /dev/null +++ b/.claude/skills/databricks-dbsql/SKILL.md @@ -0,0 +1,300 @@ +--- +name: databricks-dbsql +description: >- + Databricks SQL (DBSQL) advanced features and SQL warehouse capabilities. + This skill MUST be invoked when the user mentions: "DBSQL", "Databricks SQL", + "SQL warehouse", "SQL scripting", "stored procedure", "CALL procedure", + "materialized view", "CREATE MATERIALIZED VIEW", "pipe syntax", "|>", + "geospatial", "H3", "ST_", "spatial SQL", "collation", "COLLATE", + "ai_query", "ai_classify", "ai_extract", "ai_gen", "AI function", + "http_request", "remote_query", "read_files", "Lakehouse Federation", + "recursive CTE", "WITH RECURSIVE", "multi-statement transaction", + "temp table", "temporary view", "pipe operator". + SHOULD also invoke when the user asks about SQL best practices, data modeling + patterns, or advanced SQL features on Databricks. +--- + +# Databricks SQL (DBSQL) - Advanced Features + +## Quick Reference + +| Feature | Key Syntax | Since | Reference | +|---------|-----------|-------|-----------| +| SQL Scripting | `BEGIN...END`, `DECLARE`, `IF/WHILE/FOR` | DBR 16.3+ | [sql-scripting.md](sql-scripting.md) | +| Stored Procedures | `CREATE PROCEDURE`, `CALL` | DBR 17.0+ | [sql-scripting.md](sql-scripting.md) | +| Recursive CTEs | `WITH RECURSIVE` | DBR 17.0+ | [sql-scripting.md](sql-scripting.md) | +| Transactions | `BEGIN ATOMIC...END` | Preview | [sql-scripting.md](sql-scripting.md) | +| Materialized Views | `CREATE MATERIALIZED VIEW` | Pro/Serverless | [materialized-views-pipes.md](materialized-views-pipes.md) | +| Temp Tables | `CREATE TEMPORARY TABLE` | All | [materialized-views-pipes.md](materialized-views-pipes.md) | +| Pipe Syntax | `\|>` operator | DBR 16.1+ | [materialized-views-pipes.md](materialized-views-pipes.md) | +| Geospatial (H3) | `h3_longlatash3()`, `h3_polyfillash3()` | DBR 11.2+ | [geospatial-collations.md](geospatial-collations.md) | +| Geospatial (ST) | `ST_Point()`, `ST_Contains()`, 80+ funcs | DBR 16.0+ | [geospatial-collations.md](geospatial-collations.md) | +| Collations | `COLLATE`, `UTF8_LCASE`, locale-aware | DBR 16.1+ | [geospatial-collations.md](geospatial-collations.md) | +| AI Functions | `ai_query()`, `ai_classify()`, 11+ funcs | DBR 15.1+ | [ai-functions.md](ai-functions.md) | +| http_request | `http_request(conn, ...)` | Pro/Serverless | [ai-functions.md](ai-functions.md) | +| remote_query | `SELECT * FROM remote_query(...)` | Pro/Serverless | [ai-functions.md](ai-functions.md) | +| read_files | `SELECT * FROM read_files(...)` | All | [ai-functions.md](ai-functions.md) | +| Data Modeling | Star schema, Liquid Clustering | All | [best-practices.md](best-practices.md) | + +--- + +## Common Patterns + +### SQL Scripting - Procedural ETL + +```sql +BEGIN + DECLARE v_count INT; + DECLARE v_status STRING DEFAULT 'pending'; + + SET v_count = (SELECT COUNT(*) FROM catalog.schema.raw_orders WHERE status = 'new'); + + IF v_count > 0 THEN + INSERT INTO catalog.schema.processed_orders + SELECT *, current_timestamp() AS processed_at + FROM catalog.schema.raw_orders + WHERE status = 'new'; + + SET v_status = 'completed'; + ELSE + SET v_status = 'skipped'; + END IF; + + SELECT v_status AS result, v_count AS rows_processed; +END +``` + +### Stored Procedure with Error Handling + +```sql +CREATE OR REPLACE PROCEDURE catalog.schema.upsert_customers( + IN p_source STRING, + OUT p_rows_affected INT +) +LANGUAGE SQL +SQL SECURITY INVOKER +BEGIN + DECLARE EXIT HANDLER FOR SQLEXCEPTION + BEGIN + SET p_rows_affected = -1; + SIGNAL SQLSTATE '45000' + SET MESSAGE_TEXT = concat('Upsert failed for source: ', p_source); + END; + + MERGE INTO catalog.schema.dim_customer AS t + USING (SELECT * FROM identifier(p_source)) AS s + ON t.customer_id = s.customer_id + WHEN MATCHED THEN UPDATE SET * + WHEN NOT MATCHED THEN INSERT *; + + SET p_rows_affected = (SELECT COUNT(*) FROM identifier(p_source)); +END; + +-- Invoke: +CALL catalog.schema.upsert_customers('catalog.schema.staging_customers', ?); +``` + +### Materialized View with Scheduled Refresh + +```sql +CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.daily_revenue + CLUSTER BY (order_date) + SCHEDULE EVERY 1 HOUR + COMMENT 'Hourly-refreshed daily revenue by region' +AS SELECT + order_date, + region, + SUM(amount) AS total_revenue, + COUNT(DISTINCT customer_id) AS unique_customers +FROM catalog.schema.fact_orders +JOIN catalog.schema.dim_store USING (store_id) +GROUP BY order_date, region; +``` + +### Pipe Syntax - Readable Transformations + +```sql +-- Traditional SQL rewritten with pipe syntax +FROM catalog.schema.fact_orders + |> WHERE order_date >= current_date() - INTERVAL 30 DAYS + |> AGGREGATE SUM(amount) AS total, COUNT(*) AS cnt GROUP BY region, product_category + |> WHERE total > 10000 + |> ORDER BY total DESC + |> LIMIT 20; +``` + +### AI Functions - Enrich Data with LLMs + +```sql +-- Classify support tickets +SELECT + ticket_id, + description, + ai_classify(description, ARRAY('billing', 'technical', 'account', 'feature_request')) AS category, + ai_analyze_sentiment(description) AS sentiment +FROM catalog.schema.support_tickets +LIMIT 100; + +-- Extract entities from text +SELECT + doc_id, + ai_extract(content, ARRAY('person_name', 'company', 'dollar_amount')) AS entities +FROM catalog.schema.contracts; + +-- General-purpose AI query with structured output +SELECT ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + concat('Summarize this customer feedback in JSON with keys: topic, sentiment, action_items. Feedback: ', feedback), + returnType => 'STRUCT>' +) AS analysis +FROM catalog.schema.customer_feedback +LIMIT 50; +``` + +### Geospatial - Proximity Search with H3 + +```sql +-- Find stores within 5km of each customer using H3 indexing +WITH customer_h3 AS ( + SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell + FROM catalog.schema.customers +), +store_h3 AS ( + SELECT *, h3_longlatash3(longitude, latitude, 7) AS h3_cell + FROM catalog.schema.stores +) +SELECT + c.customer_id, + s.store_id, + ST_Distance( + ST_Point(c.longitude, c.latitude), + ST_Point(s.longitude, s.latitude) + ) AS distance_m +FROM customer_h3 c +JOIN store_h3 s ON h3_ischildof(c.h3_cell, h3_toparent(s.h3_cell, 5)) +WHERE ST_Distance( + ST_Point(c.longitude, c.latitude), + ST_Point(s.longitude, s.latitude) +) < 5000; +``` + +### Collation - Case-Insensitive Search + +```sql +-- Create table with case-insensitive collation +CREATE TABLE catalog.schema.products ( + product_id BIGINT GENERATED ALWAYS AS IDENTITY, + name STRING COLLATE UTF8_LCASE, + category STRING COLLATE UTF8_LCASE, + price DECIMAL(10, 2) +); + +-- Queries automatically case-insensitive (no LOWER() needed) +SELECT * FROM catalog.schema.products +WHERE name = 'MacBook Pro'; -- matches 'macbook pro', 'MACBOOK PRO', etc. +``` + +### http_request - Call External APIs + +```sql +-- Set up connection first (one-time) +CREATE CONNECTION my_api_conn + TYPE HTTP + OPTIONS (host 'https://api.example.com', bearer_token secret('scope', 'token')); + +-- Call API from SQL +SELECT + order_id, + http_request( + conn => 'my_api_conn', + method => 'POST', + path => '/v1/validate', + json => to_json(named_struct('order_id', order_id, 'amount', amount)) + ).text AS api_response +FROM catalog.schema.orders +WHERE needs_validation = true; +``` + +### read_files - Ingest Raw Files + +```sql +-- Read JSON files from a Volume with schema hints +SELECT * +FROM read_files( + '/Volumes/catalog/schema/raw/events/', + format => 'json', + schemaHints => 'event_id STRING, timestamp TIMESTAMP, payload MAP', + pathGlobFilter => '*.json', + recursiveFileLookup => true +); + +-- Read CSV with options +SELECT * +FROM read_files( + '/Volumes/catalog/schema/raw/sales/', + format => 'csv', + header => true, + delimiter => '|', + dateFormat => 'yyyy-MM-dd', + schema => 'sale_id INT, sale_date DATE, amount DECIMAL(10,2), store STRING' +); +``` + +### Recursive CTE - Hierarchy Traversal + +```sql +WITH RECURSIVE org_chart AS ( + -- Anchor: top-level managers + SELECT employee_id, name, manager_id, 0 AS depth, ARRAY(name) AS path + FROM catalog.schema.employees + WHERE manager_id IS NULL + + UNION ALL + + -- Recursive: direct reports + SELECT e.employee_id, e.name, e.manager_id, o.depth + 1, array_append(o.path, e.name) + FROM catalog.schema.employees e + JOIN org_chart o ON e.manager_id = o.employee_id + WHERE o.depth < 10 -- safety limit +) +SELECT * FROM org_chart ORDER BY depth, name; +``` + +### remote_query - Federated Queries + +```sql +-- Query PostgreSQL via Lakehouse Federation +SELECT * +FROM remote_query( + 'my_postgres_connection', + database => 'my_database', + query => 'SELECT customer_id, email, created_at FROM customers WHERE active = true' +); +``` + +--- + +## Reference Files + +Load these for detailed syntax, full parameter lists, and advanced patterns: + +| File | Contents | When to Read | +|------|----------|--------------| +| [sql-scripting.md](sql-scripting.md) | SQL Scripting, Stored Procedures, Recursive CTEs, Transactions | User needs procedural SQL, error handling, loops, dynamic SQL | +| [materialized-views-pipes.md](materialized-views-pipes.md) | Materialized Views, Temp Tables/Views, Pipe Syntax | User needs MVs, refresh scheduling, temp objects, pipe operator | +| [geospatial-collations.md](geospatial-collations.md) | 39 H3 functions, 80+ ST functions, Collation types and hierarchy | User needs spatial analysis, H3 indexing, case/accent handling | +| [ai-functions.md](ai-functions.md) | 13 AI functions, http_request, remote_query, read_files (all options) | User needs AI enrichment, API calls, federation, file ingestion | +| [best-practices.md](best-practices.md) | Data modeling, performance, Liquid Clustering, anti-patterns | User needs architecture guidance, optimization, or modeling advice | + +--- + +## Key Guidelines + +- **Always use Serverless SQL warehouses** for AI functions, MVs, and http_request +- **Use `LIMIT` during development** with AI functions to control costs +- **Prefer Liquid Clustering over partitioning** for new tables (1-4 keys max) +- **Use `CLUSTER BY AUTO`** when unsure about clustering keys +- **Star schema in Gold layer** for BI; OBT acceptable in Silver +- **Define PK/FK constraints** on dimensional models for query optimization +- **Use `COLLATE UTF8_LCASE`** for user-facing string columns that need case-insensitive search +- **Use MCP tools** (`execute_sql`, `execute_sql_multi`) to test and validate all SQL before deploying diff --git a/.claude/skills/databricks-dbsql/ai-functions.md b/.claude/skills/databricks-dbsql/ai-functions.md new file mode 100644 index 00000000..0853c6bb --- /dev/null +++ b/.claude/skills/databricks-dbsql/ai-functions.md @@ -0,0 +1,1348 @@ +# AI Functions, http_request, remote_query, and read_files Reference + +Comprehensive reference for Databricks SQL advanced functions: built-in AI functions, HTTP requests, Lakehouse Federation remote queries, and file reading. + +--- + +## Table of Contents + +- [AI Functions Overview](#ai-functions-overview) +- [ai_query -- General-Purpose AI Function](#ai_query----general-purpose-ai-function) +- [Task-Specific AI Functions](#task-specific-ai-functions) + - [ai_gen](#ai_gen) + - [ai_classify](#ai_classify) + - [ai_extract](#ai_extract) + - [ai_analyze_sentiment](#ai_analyze_sentiment) + - [ai_similarity](#ai_similarity) + - [ai_summarize](#ai_summarize) + - [ai_translate](#ai_translate) + - [ai_fix_grammar](#ai_fix_grammar) + - [ai_mask](#ai_mask) +- [Document and Multimodal AI Functions](#document-and-multimodal-ai-functions) + - [ai_parse_document](#ai_parse_document) +- [Time Series AI Functions](#time-series-ai-functions) + - [ai_forecast](#ai_forecast) +- [Vector Search Function](#vector-search-function) + - [vector_search](#vector_search) +- [http_request Function](#http_request-function) +- [remote_query Function (Lakehouse Federation)](#remote_query-function-lakehouse-federation) +- [read_files Table-Valued Function](#read_files-table-valued-function) + +--- + +## AI Functions Overview + +Databricks AI Functions are built-in SQL functions that invoke state-of-the-art generative AI models directly from SQL. They run on Databricks Foundation Model APIs and are available from Databricks SQL, notebooks, Lakeflow Spark Declarative Pipelines, and Workflows. + +**Common Requirements for All AI Functions:** +- Workspace must be in a region supporting AI Functions optimized for batch inference +- Not available on Databricks SQL Classic (requires Serverless SQL Warehouse) +- Databricks Runtime 15.1+ for notebooks; 15.4 ML LTS recommended for batch workloads +- Models licensed under Apache 2.0 or LLAMA 3.3 Community License +- Currently tuned for English (underlying models support multiple languages) +- Public Preview, HIPAA compliant + +**Rate Limits and Billing:** +- AI Functions are subject to Foundation Model API rate limits +- Billed as Databricks SQL compute plus token usage on Foundation Model APIs +- Use `LIMIT` in queries during development to control costs + +--- + +## ai_query -- General-Purpose AI Function + +The most powerful and flexible AI function. Queries any serving endpoint (Foundation Models, external models, or custom ML models) for real-time or batch inference. + +### Syntax + +```sql +-- Basic invocation +ai_query(endpoint, request) + +-- Full invocation with all optional parameters +ai_query( + endpoint, + request, + returnType => type_expression, + failOnError => boolean, + modelParameters => named_struct(...), + responseFormat => format_string, + files => content_expression +) +``` + +### Parameters + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `endpoint` | STRING | Yes | Name of a Foundation Model, external model, or custom model serving endpoint in the same workspace | +| `request` | STRING or STRUCT | Yes | For LLM endpoints: STRING prompt. For custom ML endpoints: single column or STRUCT matching expected input features | +| `returnType` | Expression | No | Expected return type (DDL-style). Optional in Runtime 15.2+; required in 15.1 and below | +| `failOnError` | BOOLEAN | No | Default `true`. When `false`, returns STRUCT with `response` and `errorStatus` fields instead of failing | +| `modelParameters` | STRUCT | No | Model parameters via `named_struct()` (Runtime 15.3+) | +| `responseFormat` | STRING | No | Controls output format: `'text'`, `'json_object'`, or a DDL/JSON schema string (Runtime 15.4 LTS+, chat models only) | +| `files` | Expression | No | Multimodal file input for image processing (JPEG, PNG supported) | + +### Return Types + +| Scenario | Return Type | +|----------|-------------| +| `failOnError => true` (default) | Parsed response matching endpoint type or `returnType` | +| `failOnError => false` | `STRUCT` where T is the parsed type | +| With `responseFormat` | Structured output matching the specified schema | + +### Model Parameters + +```sql +-- Control generation with modelParameters +SELECT ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + 'Explain quantum computing in 3 sentences.', + modelParameters => named_struct( + 'max_tokens', 256, + 'temperature', 0.1, + 'top_p', 0.9 + ) +) AS response; +``` + +Common model parameters: +- `max_tokens` (INT) -- Maximum tokens to generate +- `temperature` (DOUBLE) -- Randomness (0.0 = deterministic, 2.0 = max random) +- `top_p` (DOUBLE) -- Nucleus sampling threshold +- `stop` (ARRAY) -- Stop sequences + +### Structured Output with responseFormat + +> **Note:** The top-level `responseFormat` STRUCT must contain exactly one field. To return multiple fields, wrap them in a single outer field. + +```sql +-- Force JSON output matching a schema (top-level STRUCT must have exactly one field) +SELECT ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + 'Extract the product name, price, and category from: "Sony WH-1000XM5 headphones, $348, Electronics"', + responseFormat => 'STRUCT>' +) AS extracted; +``` + +### Batch Inference on Tables + +```sql +-- Classify all rows in a table +SELECT + review_id, + review_text, + ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + CONCAT('Classify the following review as positive, negative, or neutral: ', review_text), + responseFormat => 'STRUCT>' + ) AS classification +FROM catalog.schema.product_reviews; +``` + +### Custom ML Model Inference + +```sql +-- Query a custom sklearn/MLflow model +SELECT ai_query( + endpoint => 'spam-classification-endpoint', + request => named_struct( + 'text', email_body, + 'subject', email_subject + ), + returnType => 'BOOLEAN' +) AS is_spam +FROM catalog.schema.inbox_messages; +``` + +### Multimodal (Image) Input + +```sql +-- Analyze images using a vision model +SELECT ai_query( + 'databricks-meta-llama-3-2-90b-instruct', + 'Describe the contents of this image.', + files => READ_FILES('/Volumes/catalog/schema/images/photo.jpg', format => 'binaryFile') +) AS description; +``` + +### Error Handling with failOnError + +```sql +-- Graceful error handling for batch processing +SELECT + id, + result.result AS answer, + result.errorMessage AS error +FROM ( + SELECT + id, + ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + question, + failOnError => false + ) AS result + FROM catalog.schema.questions +); +``` + +### Embedding Generation + +```sql +-- Generate embeddings using ai_query +SELECT + text, + ai_query('databricks-gte-large-en', text) AS embedding +FROM catalog.schema.documents; +``` + +--- + +## Task-Specific AI Functions + +These functions provide simplified, single-purpose interfaces that do not require specifying an endpoint or model. + +### ai_gen + +Generate text from a prompt. + +```sql +ai_gen(prompt) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `prompt` | STRING | The user's request/prompt | + +**Returns:** STRING + +```sql +-- Simple generation +SELECT ai_gen('Generate a concise, cheerful email title for a summer bike sale with 20% discount'); +-- Returns: "Summer Bike Sale: Grab Your Dream Bike at 20% Off!" + +-- Generation using table data +SELECT + question, + ai_gen('You are a teacher. Answer the students question in 50 words: ' || question) AS answer +FROM catalog.schema.questions +LIMIT 10; +``` + +--- + +### ai_classify + +Classify text into one of the provided labels. + +```sql +ai_classify(content, labels) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `content` | STRING | Text to classify | +| `labels` | ARRAY | Classification options (min 2, max 20 elements) | + +**Returns:** STRING matching one of the labels, or NULL if classification fails. + +```sql +-- Simple classification +SELECT ai_classify('My password is leaked.', ARRAY('urgent', 'not urgent')); +-- Returns: "urgent" + +-- Batch product categorization +SELECT + product_name, + description, + ai_classify(description, ARRAY('clothing', 'shoes', 'accessories', 'furniture')) AS category +FROM catalog.schema.products +LIMIT 100; + +-- Support ticket routing +SELECT + ticket_id, + ai_classify( + description, + ARRAY('billing', 'technical', 'account', 'feature_request', 'other') + ) AS department +FROM catalog.schema.support_tickets; +``` + +--- + +### ai_extract + +Extract named entities from text. + +```sql +ai_extract(content, labels) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `content` | STRING | Text to extract entities from | +| `labels` | ARRAY | Entity types to extract | + +**Returns:** STRUCT where each field corresponds to a label, containing the extracted entity as STRING. Returns NULL if content is NULL. + +```sql +-- Extract person, location, organization +SELECT ai_extract( + 'John Doe lives in New York and works for Acme Corp.', + ARRAY('person', 'location', 'organization') +); +-- Returns: {"person": "John Doe", "location": "New York", "organization": "Acme Corp."} + +-- Extract contact details +SELECT ai_extract( + 'Send an email to jane.doe@example.com about the meeting at 10am.', + ARRAY('email', 'time') +); +-- Returns: {"email": "jane.doe@example.com", "time": "10am"} + +-- Batch entity extraction from customer feedback +SELECT + feedback_id, + ai_extract(feedback_text, ARRAY('product', 'issue', 'person')) AS entities +FROM catalog.schema.customer_feedback; +``` + +--- + +### ai_analyze_sentiment + +Perform sentiment analysis on text. + +```sql +ai_analyze_sentiment(content) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `content` | STRING | Text to analyze | + +**Returns:** STRING -- one of `'positive'`, `'negative'`, `'neutral'`, or `'mixed'`. Returns NULL if sentiment cannot be determined. + +```sql +SELECT ai_analyze_sentiment('I am happy'); -- Returns: "positive" +SELECT ai_analyze_sentiment('I am sad'); -- Returns: "negative" +SELECT ai_analyze_sentiment('It is what it is'); -- Returns: "neutral" + +-- Aggregate sentiment by product +SELECT + product_id, + ai_analyze_sentiment(review_text) AS sentiment, + COUNT(*) AS review_count +FROM catalog.schema.reviews +GROUP BY product_id, ai_analyze_sentiment(review_text); +``` + +--- + +### ai_similarity + +Compute semantic similarity between two text strings. + +```sql +ai_similarity(expr1, expr2) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `expr1` | STRING | First text to compare | +| `expr2` | STRING | Second text to compare | + +**Returns:** FLOAT -- Semantic similarity score where 1.0 means identical. The score is relative and should only be used for ranking. + +```sql +-- Exact match +SELECT ai_similarity('Apache Spark', 'Apache Spark'); +-- Returns: 1.0 + +-- Find similar company names (fuzzy matching) +SELECT company_name, ai_similarity(company_name, 'Databricks') AS score +FROM catalog.schema.customers +ORDER BY score DESC +LIMIT 10; + +-- Duplicate detection +SELECT + a.id AS id_a, + b.id AS id_b, + ai_similarity(a.description, b.description) AS similarity +FROM catalog.schema.products a +JOIN catalog.schema.products b ON a.id < b.id +WHERE ai_similarity(a.description, b.description) > 0.85; +``` + +--- + +### ai_summarize + +Generate a summary of text. + +```sql +ai_summarize(content [, max_words]) +``` + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `content` | STRING | Yes | Text to summarize | +| `max_words` | INTEGER | No | Target word count for summary. Default: 50. Set to 0 for no limit | + +**Returns:** STRING. Returns NULL if content is NULL. + +```sql +-- Summarize with default 50-word limit +SELECT ai_summarize( + 'Apache Spark is a unified analytics engine for large-scale data processing. ' + || 'It provides high-level APIs in Java, Scala, Python and R, and an optimized ' + || 'engine that supports general execution graphs.' +); + +-- Summarize with custom word limit +SELECT ai_summarize(article_body, 100) AS summary +FROM catalog.schema.articles; + +-- Executive summaries for reports +SELECT + report_id, + report_title, + ai_summarize(report_body, 30) AS executive_summary +FROM catalog.schema.quarterly_reports; +``` + +--- + +### ai_translate + +Translate text to a target language. + +```sql +ai_translate(content, to_lang) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `content` | STRING | Text to translate | +| `to_lang` | STRING | Target language code | + +**Supported Languages:** English (`en`), German (`de`), French (`fr`), Italian (`it`), Portuguese (`pt`), Hindi (`hi`), Spanish (`es`), Thai (`th`). + +**Returns:** STRING. Returns NULL if content is NULL. + +```sql +-- English to Spanish +SELECT ai_translate('Hello, how are you?', 'es'); +-- Returns: "Hola, como estas?" + +-- Spanish to English +SELECT ai_translate('La vida es un hermoso viaje.', 'en'); +-- Returns: "Life is a beautiful journey." + +-- Translate product descriptions for localization +SELECT + product_id, + description AS original, + ai_translate(description, 'fr') AS french, + ai_translate(description, 'de') AS german +FROM catalog.schema.products; +``` + +--- + +### ai_fix_grammar + +Correct grammatical errors in text. + +```sql +ai_fix_grammar(content) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `content` | STRING | Text to correct | + +**Returns:** STRING with corrected grammar. Returns NULL if content is NULL. + +```sql +SELECT ai_fix_grammar('This sentence have some mistake'); +-- Returns: "This sentence has some mistakes" + +SELECT ai_fix_grammar('She dont know what to did.'); +-- Returns: "She doesn't know what to do." + +-- Clean up user-generated content +SELECT + comment_id, + original_text, + ai_fix_grammar(original_text) AS corrected_text +FROM catalog.schema.user_comments; +``` + +--- + +### ai_mask + +Mask specified entity types in text (PII redaction). + +```sql +ai_mask(content, labels) +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `content` | STRING | Text containing entities to mask | +| `labels` | ARRAY | Entity types to mask (e.g., `'person'`, `'email'`, `'phone'`, `'address'`, `'location'`, `'ssn'`, `'credit_card'`) | + +**Returns:** STRING with specified entities replaced by `[MASKED]`. Returns NULL if content is NULL. + +```sql +-- Mask personal information +SELECT ai_mask( + 'John Doe lives in New York. His email is john.doe@example.com.', + ARRAY('person', 'email') +); +-- Returns: "[MASKED] lives in New York. His email is [MASKED]." + +-- Mask contact details +SELECT ai_mask( + 'Contact me at 555-1234 or visit us at 123 Main St.', + ARRAY('phone', 'address') +); +-- Returns: "Contact me at [MASKED] or visit us at [MASKED]" + +-- Create anonymized dataset +CREATE TABLE catalog.schema.anonymized_feedback AS +SELECT + feedback_id, + ai_mask(feedback_text, ARRAY('person', 'email', 'phone', 'address')) AS masked_text, + category +FROM catalog.schema.customer_feedback; +``` + +--- + +## Document and Multimodal AI Functions + +### ai_parse_document + +Extract structured content from unstructured documents (PDF, DOCX, PPTX, images). + +```sql +ai_parse_document(content) +ai_parse_document(content, options_map) +``` + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `content` | BINARY | Yes | Document as binary blob data | +| `options` | MAP | No | Configuration options | + +**Options Map Keys:** + +| Key | Values | Description | +|-----|--------|-------------| +| `version` | `'2.0'` | Output schema version | +| `imageOutputPath` | Volume path | Path to save rendered page images in Unity Catalog volume | +| `descriptionElementTypes` | `''`, `'figure'`, `'*'` | Controls AI-generated descriptions. Default: `'*'` (all elements) | + +**Returns:** VARIANT with structure: +- `document.pages[]` -- Page metadata (id, image_uri) +- `document.elements[]` -- Extracted content (type, content, bbox, description) +- `error_status[]` -- Error details per page +- `metadata` -- File and schema version info + +**Supported Formats:** PDF, JPG/JPEG, PNG, DOC/DOCX, PPT/PPTX + +**Requirements:** Databricks Runtime 17.1+, US/EU region or cross-geography routing enabled. + +```sql +-- Basic document parsing +SELECT ai_parse_document(content) +FROM READ_FILES('/Volumes/catalog/schema/volume/docs/', format => 'binaryFile'); + +-- Parse with options (save images, version 2.0) +SELECT ai_parse_document( + content, + map( + 'version', '2.0', + 'imageOutputPath', '/Volumes/catalog/schema/volume/images/', + 'descriptionElementTypes', '*' + ) +) +FROM READ_FILES('/Volumes/catalog/schema/volume/invoices/', format => 'binaryFile'); + +-- Parse documents then extract structured data with ai_query +WITH parsed AS ( + SELECT + path, + ai_parse_document(content) AS doc + FROM READ_FILES('/Volumes/catalog/schema/volume/invoices/', format => 'binaryFile') +) +SELECT + path, + ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + CONCAT('Extract vendor name, invoice number, and total from: ', doc:document:elements[0]:content::STRING), + responseFormat => 'STRUCT' + ) AS invoice_data +FROM parsed; +``` + +--- + +## Time Series AI Functions + +### ai_forecast + +Forecast time series data using a built-in prophet-like model. This is a table-valued function (TVF). + +```sql +ai_forecast( + observed TABLE, + horizon DATE | TIMESTAMP | STRING, + time_col STRING, + value_col STRING | ARRAY, + group_col STRING | ARRAY | NULL DEFAULT NULL, + prediction_interval_width DOUBLE DEFAULT 0.95, + frequency STRING DEFAULT 'auto', + seed INTEGER | NULL DEFAULT NULL, + parameters STRING DEFAULT '{}' +) +``` + +### Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `observed` | TABLE | Required | Training data passed as `TABLE(subquery)` or `TABLE(table_name)` | +| `horizon` | DATE/TIMESTAMP/STRING | Required | Right-exclusive forecast end time | +| `time_col` | STRING | Required | Name of DATE or TIMESTAMP column in observed data | +| `value_col` | STRING or ARRAY | Required | One or more numeric columns to forecast | +| `group_col` | STRING, ARRAY, or NULL | NULL | Partition column(s) for independent per-group forecasts | +| `prediction_interval_width` | DOUBLE | 0.95 | Confidence level for prediction bounds (0 to 1) | +| `frequency` | STRING | `'auto'` | Time granularity. Auto-infers from recent data. For DATE columns use: `'day'`, `'week'`, `'month'`. For TIMESTAMP columns: `'D'`, `'W'`, `'M'`, `'H'`, etc. | +| `seed` | INTEGER or NULL | NULL | Random seed for reproducibility | +| `parameters` | STRING | `'{}'` | JSON-encoded advanced settings | + +**Advanced Parameters (JSON):** +- `global_cap` -- Upper bound for logistic growth +- `global_floor` -- Lower bound for logistic growth +- `daily_order` -- Fourier order for daily seasonality +- `weekly_order` -- Fourier order for weekly seasonality + +### Return Columns + +For each `value_col` named `v`, the output contains: +- `{v}_forecast` (DOUBLE) -- Point forecast +- `{v}_upper` (DOUBLE) -- Upper prediction bound +- `{v}_lower` (DOUBLE) -- Lower prediction bound +- Plus the original time column and group columns + +**Requirements:** Serverless SQL Warehouse. + +```sql +-- Basic revenue forecast +SELECT * FROM ai_forecast( + TABLE(SELECT ds, revenue FROM catalog.schema.daily_sales), + horizon => '2025-12-31', + time_col => 'ds', + value_col => 'revenue' +); + +-- Multi-metric forecast by group +SELECT * FROM ai_forecast( + TABLE( + SELECT date, zipcode, revenue, trip_count + FROM catalog.schema.regional_metrics + ), + horizon => '2025-06-30', + time_col => 'date', + value_col => ARRAY('revenue', 'trip_count'), + group_col => 'zipcode', + prediction_interval_width => 0.90, + frequency => 'D' +); + +-- Monthly forecast with growth constraints (use 'month' for DATE columns, not 'M') +SELECT * FROM ai_forecast( + TABLE(catalog.schema.monthly_kpis), + horizon => '2026-01-01', + time_col => 'month', + value_col => 'active_users', + frequency => 'month', + parameters => '{"global_floor": 0}' +); +``` + +--- + +## Vector Search Function + +### vector_search + +Query a Mosaic AI Vector Search index using SQL. This is a table-valued function. + +```sql +-- Databricks Runtime 15.3+ +SELECT * FROM vector_search( + index => index_name, + query_text => search_text, -- OR query_vector => embedding_array + num_results => max_results, + query_type => 'ANN' | 'HYBRID' +) +``` + +### Parameters (Named Arguments Required) + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `index` | STRING constant | Required | Fully qualified name of the vector search index | +| `query_text` | STRING | -- | Search string (for Delta Sync indexes with embedding source) | +| `query_vector` | ARRAY | -- | Pre-computed embedding vector to search | +| `num_results` | INTEGER | 10 | Max records returned (max 100) | +| `query_type` | STRING | `'ANN'` | `'ANN'` for approximate nearest neighbor, `'HYBRID'` for hybrid search | + +**Returns:** Table containing all index columns with top matching records. + +**Requirements:** Serverless SQL Warehouse, Select permission on the index. + +```sql +-- Text-based similarity search +SELECT * FROM vector_search( + index => 'catalog.schema.product_index', + query_text => 'wireless noise canceling headphones', + num_results => 5 +); + +-- Hybrid search (combines keyword + semantic) +SELECT * FROM vector_search( + index => 'catalog.schema.support_docs_index', + query_text => 'Wi-Fi connection issues with router model LMP-9R2', + query_type => 'HYBRID', + num_results => 3 +); + +-- Vector-based search with pre-computed embedding +SELECT * FROM vector_search( + index => 'catalog.schema.embeddings_index', + query_vector => ARRAY(0.45, -0.35, 0.78, 0.22), + num_results => 10 +); + +-- Batch search using LATERAL join +SELECT + q.query_text, + q.query_id, + results.* +FROM catalog.schema.search_queries q, +LATERAL ( + SELECT * FROM vector_search( + index => 'catalog.schema.knowledge_base_index', + query_text => q.query_text, + num_results => 3 + ) +) AS results; +``` + +--- + +## http_request Function + +Make HTTP requests to external services from SQL using Unity Catalog HTTP connections. + +### Syntax + +```sql +http_request( + CONN => connection_name, + METHOD => http_method, + PATH => path, + HEADERS => header_map, + PARAMS => param_map, + JSON => json_body +) +``` + +### Parameters + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `CONN` | STRING constant | Yes | Name of an existing HTTP connection | +| `METHOD` | STRING constant | Yes | HTTP method: `'GET'`, `'POST'`, `'PUT'`, `'DELETE'`, `'PATCH'` | +| `PATH` | STRING constant | Yes | Path appended to the connection's base_path. Cannot contain directory traversal (`../`) | +| `HEADERS` | MAP | No | Request headers. Default: NULL | +| `PARAMS` | MAP | No | Query parameters. Default: NULL | +| `JSON` | STRING expression | No | Request body as JSON string | + +### Return Type + +`STRUCT` +- `status_code` -- HTTP response status (e.g., 200, 403, 404) +- `text` -- Response body (typically JSON) + +**Requirements:** Databricks Runtime 16.2+, Unity Catalog enabled workspace, USE CONNECTION privilege. + +### Creating HTTP Connections + +```sql +-- Bearer token authentication +CREATE CONNECTION slack_conn TYPE HTTP +OPTIONS ( + host 'https://slack.com', + port '443', + base_path '/api/', + bearer_token secret('my-scope', 'slack-token') +); + +-- OAuth Machine-to-Machine +CREATE CONNECTION github_conn TYPE HTTP +OPTIONS ( + host 'https://api.github.com', + port '443', + base_path '/', + client_id secret('my-scope', 'github-client-id'), + client_secret secret('my-scope', 'github-client-secret'), + oauth_scope 'repo read:org', + token_endpoint 'https://github.com/login/oauth/access_token' +); +``` + +**Connection Options:** + +| Option | Type | Description | +|--------|------|-------------| +| `host` | STRING | Base URL of the external service | +| `port` | STRING | Network port (typically `'443'` for HTTPS) | +| `base_path` | STRING | Root path for API endpoints | +| `bearer_token` | STRING | Auth token (use `secret()` for security) | +| `client_id` | STRING | OAuth application identifier | +| `client_secret` | STRING | OAuth application secret | +| `oauth_scope` | STRING | Space-delimited OAuth scopes | +| `token_endpoint` | STRING | OAuth token endpoint URL | +| `authorization_endpoint` | STRING | OAuth authorization redirect URL | +| `oauth_credential_exchange_method` | STRING | `'header_and_body'`, `'body_only'`, or `'header_only'` | + +### Examples + +```sql +-- POST a Slack message +SELECT http_request( + CONN => 'slack_conn', + METHOD => 'POST', + PATH => '/chat.postMessage', + JSON => to_json(named_struct('channel', '#alerts', 'text', 'Pipeline completed successfully')) +); + +-- GET request with headers and params +SELECT http_request( + CONN => 'github_conn', + METHOD => 'GET', + PATH => '/repos/databricks/spark/issues', + HEADERS => map('Accept', 'application/vnd.github+json'), + PARAMS => map('state', 'open', 'per_page', '5') +); + +-- Parse JSON response +SELECT + response.status_code, + from_json(response.text, 'STRUCT') AS issue +FROM ( + SELECT http_request( + CONN => 'github_conn', + METHOD => 'GET', + PATH => '/repos/databricks/spark/issues/1' + ) AS response +); + +-- Webhook notification triggered by data changes +SELECT http_request( + CONN => 'webhook_conn', + METHOD => 'POST', + PATH => '/notify', + JSON => to_json(named_struct( + 'event', 'data_quality_alert', + 'table', 'catalog.schema.orders', + 'message', CONCAT('Null rate exceeded threshold: ', CAST(null_pct AS STRING)) + )) +) +FROM catalog.schema.data_quality_metrics +WHERE null_pct > 0.05; +``` + +--- + +## remote_query Function (Lakehouse Federation) + +Run SQL queries against external databases using their native SQL syntax, returning results as a table in Databricks SQL. This is a table-valued function. + +### Overview + +Lakehouse Federation enables querying external databases without migrating data. It supports two modes: +- **Query Federation** -- Queries are pushed down to external databases via JDBC +- **Catalog Federation** -- Queries access foreign tables directly in object storage + +### Syntax + +```sql +SELECT * FROM remote_query( + '', + => '' + [, ...] +) +``` + +### Supported Databases + +| Database | Connection Type | +|----------|----------------| +| PostgreSQL | `POSTGRESQL` | +| MySQL | `MYSQL` | +| Microsoft SQL Server | `SQLSERVER` | +| Oracle | `ORACLE` | +| Teradata | `TERADATA` | +| Amazon Redshift | `REDSHIFT` | +| Snowflake | `SNOWFLAKE` | +| Google BigQuery | `BIGQUERY` | +| Databricks | `DATABRICKS` | + +### Parameters by Database Type + +**PostgreSQL / MySQL / SQL Server / Redshift / Teradata:** + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `database` | STRING | Yes | Remote database name | +| `query` | STRING | One of query/dbtable | SQL query in the remote database's native syntax | +| `dbtable` | STRING | One of query/dbtable | Fully qualified table name | +| `fetchsize` | STRING | No | Number of rows to fetch per round trip | +| `partitionColumn` | STRING | No | Column used for parallel read partitioning | +| `lowerBound` | STRING | No | Lower bound for partition column | +| `upperBound` | STRING | No | Upper bound for partition column | +| `numPartitions` | STRING | No | Number of parallel partitions | + +**Oracle (uses `service_name` instead of `database`):** + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `service_name` | STRING | Yes | Oracle service name | +| `query` or `dbtable` | STRING | Yes (one required) | Query or table reference | + +**Snowflake:** + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `database` | STRING | Yes | Snowflake database | +| `schema` | STRING | No | Schema name (defaults to `public`) | +| `query` or `dbtable` | STRING | Yes (one required) | Query or table reference | +| `query_timeout` | STRING | No | Query timeout in seconds | +| `partition_size_in_mb` | STRING | No | Partition size for reads | + +**BigQuery:** + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `query` or `dbtable` | STRING | Yes (one required) | Query or table reference | +| `materializationDataset` | STRING | For views/complex queries | Dataset for materialization | +| `materializationProject` | STRING | No | GCP project for materialization | +| `parentProject` | STRING | No | Parent GCP project | + +### Pushdown Control + +| Option | Default | Description | +|--------|---------|-------------| +| `pushdown.limit.enabled` | `true` | Push LIMIT to remote | +| `pushdown.offset.enabled` | `true` | Push OFFSET to remote | +| `pushdown.filters.enabled` | `true` | Push WHERE filters to remote | +| `pushdown.aggregates.enabled` | `true` | Push aggregations to remote | +| `pushdown.sortLimit.enabled` | `true` | Push ORDER BY + LIMIT to remote | + +### Requirements + +- Unity Catalog enabled workspace +- Databricks Runtime 17.3+ (clusters) or SQL Warehouse 2025.35+ (Pro/Serverless) +- Network connectivity to target database +- `USE CONNECTION` privilege or `SELECT` on a wrapping view + +### Limitations + +- **Read-only**: Only SELECT queries supported (no INSERT, UPDATE, DELETE, MERGE, DDL, or stored procedures) + +### Creating Connections + +```sql +-- PostgreSQL connection +CREATE CONNECTION my_postgres TYPE POSTGRESQL +OPTIONS ( + host 'pg-server.example.com', + port '5432', + user secret('my-scope', 'pg-user'), + password secret('my-scope', 'pg-password') +); + +-- SQL Server connection +CREATE CONNECTION my_sqlserver TYPE SQLSERVER +OPTIONS ( + host 'sql-server.example.com', + port '1433', + user secret('my-scope', 'sql-user'), + password secret('my-scope', 'sql-password') +); +``` + +### Examples + +```sql +-- Basic query against PostgreSQL +SELECT * FROM remote_query( + 'my_postgres', + database => 'sales_db', + query => 'SELECT customer_id, name, email FROM customers WHERE active = true' +); + +-- Parallel read from SQL Server +SELECT * FROM remote_query( + 'my_sqlserver', + database => 'orders_db', + dbtable => 'dbo.transactions', + partitionColumn => 'transaction_id', + lowerBound => '0', + upperBound => '1000000', + numPartitions => '10' +); + +-- Join federated data with local Delta tables +SELECT + o.order_id, + o.amount, + c.name, + c.email +FROM catalog.schema.orders o +JOIN remote_query( + 'my_postgres', + database => 'crm_db', + query => 'SELECT customer_id, name, email FROM customers' +) c ON o.customer_id = c.customer_id; + +-- Access delegation via view +CREATE VIEW catalog.schema.federated_customers AS +SELECT * FROM remote_query( + 'my_postgres', + database => 'crm_db', + query => 'SELECT customer_id, name, region FROM customers' +); + +-- Users only need SELECT on the view, not USE CONNECTION +GRANT SELECT ON VIEW catalog.schema.federated_customers TO `analysts`; +``` + +--- + +## read_files Table-Valued Function + +Read files from cloud storage or Unity Catalog volumes directly in SQL, with automatic format detection and schema inference. + +### Syntax + +```sql +SELECT * FROM read_files( + path + [, option_key => option_value ] [...] +) +``` + +### Core Parameters + +| Parameter | Type | Required | Description | +|-----------|------|----------|-------------| +| `path` | STRING | Yes | URI of data location. Supports `s3://`, `abfss://`, `gs://`, `/Volumes/...` paths. Accepts glob patterns | + +### Common Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `format` | STRING | Auto-detected | File format: `'csv'`, `'json'`, `'parquet'`, `'avro'`, `'orc'`, `'text'`, `'binaryFile'`, `'xml'` | +| `schema` | STRING | Inferred | Explicit schema definition in DDL format | +| `schemaHints` | STRING | None | Override subset of inferred schema columns | +| `rescuedDataColumn` | STRING | `'_rescued_data'` | Column name for data that could not be parsed. Set to empty string to disable | +| `pathGlobFilter` / `fileNamePattern` | STRING | None | Glob pattern to filter files (e.g., `'*.csv'`) | +| `recursiveFileLookup` | BOOLEAN | `false` | Search nested directories | +| `modifiedAfter` | TIMESTAMP STRING | None | Only read files modified after this timestamp | +| `modifiedBefore` | TIMESTAMP STRING | None | Only read files modified before this timestamp | +| `partitionColumns` | STRING | Auto-detected | Comma-separated Hive-style partition columns. Empty string ignores all partitions | +| `useStrictGlobber` | BOOLEAN | `true` | Strict glob pattern matching | +| `inferColumnTypes` | BOOLEAN | `true` | Infer exact column types (vs treating all as STRING) | +| `schemaEvolutionMode` | STRING | -- | Schema evolution behavior: `'none'` to drop rescued data column | + +### CSV-Specific Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `sep` / `delimiter` | STRING | `','` | Field delimiter | +| `header` | BOOLEAN | `false` | First row contains column names | +| `encoding` | STRING | `'UTF-8'` | Character encoding | +| `quote` | STRING | `'"'` | Quote character | +| `escape` | STRING | `'\'` | Escape character | +| `nullValue` | STRING | `''` | String representation of null | +| `dateFormat` | STRING | `'yyyy-MM-dd'` | Date parsing format | +| `timestampFormat` | STRING | `'yyyy-MM-dd\'T\'HH:mm:ss...'` | Timestamp parsing format | +| `mode` | STRING | `'PERMISSIVE'` | Parse mode: `'PERMISSIVE'`, `'DROPMALFORMED'`, `'FAILFAST'` | +| `multiLine` | BOOLEAN | `false` | Allow records spanning multiple lines | +| `ignoreLeadingWhiteSpace` | BOOLEAN | `false` | Trim leading whitespace | +| `ignoreTrailingWhiteSpace` | BOOLEAN | `false` | Trim trailing whitespace | +| `comment` | STRING | None | Line comment character | +| `maxCharsPerColumn` | INTEGER | None | Max characters per column | +| `maxColumns` | INTEGER | None | Max number of columns | +| `mergeSchema` | BOOLEAN | `false` | Merge schemas across files | +| `enforceSchema` | BOOLEAN | `true` | Enforce specified schema | +| `locale` | STRING | `'US'` | Locale for number/date parsing | +| `charToEscapeQuoteEscaping` | STRING | None | Character to escape the quote escape character | +| `readerCaseSensitive` | BOOLEAN | `true` | Case-sensitive column name matching | + +### JSON-Specific Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `multiLine` | BOOLEAN | `false` | Parse multi-line JSON records | +| `allowComments` | BOOLEAN | `false` | Allow Java/C++ style comments | +| `allowSingleQuotes` | BOOLEAN | `true` | Allow single quotes for strings | +| `allowUnquotedFieldNames` | BOOLEAN | `false` | Allow unquoted field names | +| `allowBackslashEscapingAnyCharacter` | BOOLEAN | `false` | Allow backslash to escape any character | +| `allowNonNumericNumbers` | BOOLEAN | `true` | Allow NaN, Infinity, -Infinity | +| `encoding` | STRING | `'UTF-8'` | Character encoding | +| `dateFormat` | STRING | `'yyyy-MM-dd'` | Date parsing format | +| `timestampFormat` | STRING | -- | Timestamp parsing format | +| `inferTimestamp` | BOOLEAN | `false` | Infer timestamp types | +| `prefersDecimal` | BOOLEAN | `false` | Prefer DECIMAL over DOUBLE | +| `primitivesAsString` | BOOLEAN | `false` | Infer all primitives as STRING | +| `singleVariantColumn` | STRING | None | Read entire JSON as single VARIANT column | +| `locale` | STRING | `'US'` | Locale for parsing | +| `mode` | STRING | `'PERMISSIVE'` | Parse mode | +| `readerCaseSensitive` | BOOLEAN | `true` | Case-sensitive column matching | +| `timeZone` | STRING | Session timezone | Timezone for timestamp parsing | + +### XML-Specific Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `rowTag` | STRING | **Required** | XML tag that delimits rows | +| `attributePrefix` | STRING | `'_'` | Prefix for XML attributes | +| `valueTag` | STRING | `'_VALUE'` | Tag for element text content | +| `encoding` | STRING | `'UTF-8'` | Character encoding | +| `ignoreSurroundingSpaces` | BOOLEAN | `true` | Ignore whitespace around values | +| `ignoreNamespace` | BOOLEAN | `false` | Ignore XML namespaces | +| `mode` | STRING | `'PERMISSIVE'` | Parse mode | +| `dateFormat` | STRING | `'yyyy-MM-dd'` | Date parsing format | +| `timestampFormat` | STRING | -- | Timestamp parsing format | +| `locale` | STRING | `'US'` | Locale for parsing | +| `readerCaseSensitive` | BOOLEAN | `true` | Case-sensitive matching | +| `samplingRatio` | DOUBLE | `1.0` | Fraction of rows to sample for schema inference | + +### Parquet / Avro / ORC Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `mergeSchema` | BOOLEAN | `false` | Merge schemas across files | +| `readerCaseSensitive` | BOOLEAN | `true` | Case-sensitive column matching | +| `rescuedDataColumn` | STRING | -- | Column for rescued data | +| `datetimeRebaseMode` | STRING | -- | Rebase mode for datetime values | +| `int96RebaseMode` | STRING | -- | Rebase mode for INT96 timestamps (Parquet only) | + +### Streaming Options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `includeExistingFiles` | BOOLEAN | `true` | Process existing files on first run | +| `maxFilesPerTrigger` | INTEGER | None | Max files per micro-batch | +| `maxBytesPerTrigger` | STRING | None | Max bytes per micro-batch | +| `allowOverwrites` | BOOLEAN | `false` | Allow processing of overwritten files | +| `schemaEvolutionMode` | STRING | -- | Schema evolution behavior | +| `schemaLocation` | STRING | -- | Location to store inferred schema | + +### Requirements + +- Databricks Runtime 13.3 LTS and above +- Databricks SQL + +### Examples + +```sql +-- Auto-detect format and schema from cloud storage +SELECT * FROM read_files('s3://my-bucket/data/'); + +-- Read CSV with explicit schema +SELECT * FROM read_files( + '/Volumes/catalog/schema/volume/sales.csv', + format => 'csv', + header => true, + schema => 'order_id INT, customer_id INT, amount DOUBLE, order_date DATE' +); + +-- Read CSV with schema hints (override specific columns only) +SELECT * FROM read_files( + '/Volumes/catalog/schema/volume/events/', + format => 'csv', + header => true, + schemaHints => 'event_timestamp TIMESTAMP, amount DECIMAL(10,2)' +); + +-- Read JSON with multi-line support +SELECT * FROM read_files( + '/Volumes/catalog/schema/volume/api_responses/', + format => 'json', + multiLine => true +); + +-- Read Parquet with merged schema across files +SELECT * FROM read_files( + 's3://my-bucket/parquet-data/', + format => 'parquet', + mergeSchema => true +); + +-- Read XML with row tag +SELECT * FROM read_files( + '/Volumes/catalog/schema/volume/feed.xml', + format => 'xml', + rowTag => 'record' +); + +-- Read binary files (images, PDFs) for ai_parse_document +SELECT path, content FROM read_files( + '/Volumes/catalog/schema/volume/documents/', + format => 'binaryFile' +); + +-- Filter files by glob pattern and modification date +SELECT * FROM read_files( + 's3://my-bucket/logs/', + format => 'json', + pathGlobFilter => '*.json', + modifiedAfter => '2025-01-01T00:00:00Z', + modifiedBefore => '2025-02-01T00:00:00Z' +); + +-- Recursive directory scan with partition discovery +SELECT * FROM read_files( + '/Volumes/catalog/schema/volume/partitioned_data/', + recursiveFileLookup => true, + partitionColumns => 'year,month' +); + +-- Include file metadata +SELECT *, _metadata.file_path, _metadata.file_name, _metadata.file_size +FROM read_files('/Volumes/catalog/schema/volume/data/'); + +-- Create table from files +CREATE TABLE catalog.schema.imported_data AS +SELECT * FROM read_files( + '/Volumes/catalog/schema/volume/export.csv', + format => 'csv', + header => true +); + +-- Streaming table from cloud storage +CREATE STREAMING TABLE catalog.schema.streaming_events AS +SELECT * FROM STREAM read_files( + 's3://my-bucket/events/', + format => 'json', + includeExistingFiles => false, + maxFilesPerTrigger => 100 +); + +-- Read single VARIANT column for semi-structured JSON +SELECT * FROM read_files( + '/Volumes/catalog/schema/volume/complex.json', + format => 'json', + singleVariantColumn => 'raw_data' +); +``` + +--- + +## Combining Functions -- Production Patterns + +### AI-Enhanced ETL Pipeline + +```sql +-- Process customer feedback with multiple AI functions +CREATE OR REPLACE TABLE catalog.schema.enriched_feedback AS +SELECT + feedback_id, + feedback_text, + ai_analyze_sentiment(feedback_text) AS sentiment, + ai_classify(feedback_text, ARRAY('product', 'service', 'billing', 'other')) AS category, + ai_extract(feedback_text, ARRAY('product', 'issue')) AS entities, + ai_summarize(feedback_text, 20) AS summary, + ai_mask(feedback_text, ARRAY('person', 'email', 'phone')) AS anonymized_text +FROM catalog.schema.raw_feedback; +``` + +### Document Processing Pipeline + +```sql +-- Ingest, parse, and query documents +WITH raw_docs AS ( + SELECT path, content + FROM read_files('/Volumes/catalog/schema/volume/contracts/', format => 'binaryFile') +), +parsed AS ( + SELECT path, ai_parse_document(content, map('version', '2.0')) AS doc + FROM raw_docs +) +SELECT + path, + ai_query( + 'databricks-meta-llama-3-3-70b-instruct', + CONCAT('Extract the contract parties, effective date, and termination clause from: ', + doc:document:elements[0]:content::STRING), + responseFormat => 'STRUCT' + ) AS contract_info +FROM parsed; +``` + +### External API Integration with http_request + +```sql +-- Enrich data by calling an external API and joining results +SELECT + o.order_id, + o.tracking_number, + from_json( + tracking.text, + 'STRUCT' + ) AS tracking_info +FROM catalog.schema.orders o +CROSS JOIN LATERAL ( + SELECT http_request( + CONN => 'shipping_api_conn', + METHOD => 'GET', + PATH => CONCAT('/track/', o.tracking_number) + ) AS response +) tracking +WHERE tracking.response.status_code = 200; +``` + +### Federated Analytics + +```sql +-- Combine remote database data with local lakehouse data and AI +SELECT + remote_orders.customer_id, + remote_orders.total_spend, + local_profiles.segment, + ai_classify( + CONCAT('Customer spent $', CAST(remote_orders.total_spend AS STRING), + ' in segment ', local_profiles.segment), + ARRAY('high_value', 'medium_value', 'low_value', 'at_risk') + ) AS value_tier +FROM remote_query( + 'my_postgres', + database => 'sales_db', + query => 'SELECT customer_id, SUM(amount) as total_spend FROM orders GROUP BY customer_id' +) remote_orders +JOIN catalog.schema.customer_profiles local_profiles + ON remote_orders.customer_id = local_profiles.customer_id; +``` diff --git a/.claude/skills/databricks-dbsql/best-practices.md b/.claude/skills/databricks-dbsql/best-practices.md new file mode 100644 index 00000000..a33cdc2a --- /dev/null +++ b/.claude/skills/databricks-dbsql/best-practices.md @@ -0,0 +1,475 @@ +# Data Modeling and DBSQL Best Practices + +Comprehensive reference for data modeling patterns, DBSQL performance optimization, and operational best practices on the Databricks Lakehouse Platform. + +--- + +## Data Modeling Best Practices + +### Star Schema vs Denormalization in the Lakehouse + +The Databricks Lakehouse fully supports dimensional modeling. Star schemas translate well to Delta tables and often deliver superior performance compared to fully denormalized approaches. + +**Star Schema (Dimensional Modeling):** +- Central fact table linked to multiple denormalized dimension tables +- Optimizes for complex analytics and multi-dimensional aggregations +- Provides intuitive business process mapping and scales well with SCDs +- Supports up to ~10 filtering dimensions (5 tables x 2 clustering keys each) +- Clear separation of concerns enables fine-grained governance + +**One Big Table (OBT):** +- Single wide table with all attributes pre-joined +- Eliminates joins, simpler governance (one table to manage) +- Liquid Clustering limited to 1-4 keys, so effective filtering is limited to 1-3 dimensions +- Full table scans become bottlenecks as data grows +- Lacks structured business process mapping +- Complicates fine-grained access controls and data quality checks + +**Key finding:** In benchmarks, dimensional models outperformed OBT (2.6s vs 3.5s) despite requiring joins, because fewer files needed to be scanned. However, with Liquid Clustering applied, OBT achieved >3x improvement (down to 1.13s). Both approaches achieve sub-500ms with automatic caching. + +**Recommended approach:** Use a hybrid medallion architecture: +- Silver layer: OBT or Data Vault for rapid integration and cleansing +- Gold layer: Star schema dimensional models as the curated, business-ready presentation layer for BI and reporting + +### When to Normalize vs Denormalize + +| Use Case | Approach | +|---|---| +| Gold layer for BI reporting | Star schema (denormalized dimensions, normalized facts) | +| Silver layer data integration | Normalized or Data Vault | +| Single-use IoT/logging analytics | OBT (filter by 1-3 dimensions) | +| Multi-dimensional business analysis | Star schema | +| Rapidly evolving schemas | OBT in Silver, star schema in Gold | +| High-cardinality filtering (5+ dimensions) | Star schema with Liquid Clustering per table | + +**Rule of thumb:** Dimension tables should be highly denormalized (flatten many-to-one relationships within a single dimension table). Fact tables should remain normalized at the grain of the business event. + +### Kimball-Style Modeling in Databricks + +Kimball dimensional modeling is the recommended approach for the Gold layer in the Lakehouse: + +1. **Identify the business process** (sales, orders, shipments) +2. **Declare the grain** (one row per transaction, per day, etc.) +3. **Choose dimensions** (who, what, where, when, why, how) +4. **Identify facts** (measurable numeric values at the declared grain) + +**Databricks-specific implementation details:** +- Use Unity Catalog for organizing dimensional models (catalog.schema.table) +- Define PRIMARY KEY constraints on dimension surrogate keys +- Define FOREIGN KEY constraints on fact table dimension keys for query optimization +- Add COMMENT on all tables and columns for discoverability +- Apply TAGS for governance (e.g., PII tagging) to enable downstream AI/BI capabilities +- Use `ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS` on dimension keys to support Adaptive Query Execution + +**Key principle:** "The better you model your data upfront, the more easily you can leverage AI on top of it out of the box." Proper schema design enables downstream AI/BI capabilities. + +### Fact Table Patterns + +**Design rules:** +- Store quantitative, numeric measures at the most granular transactional level +- Use DECIMAL instead of floating-point numbers for financial data +- Include foreign keys referencing dimension tables +- Include degenerate dimensions (source-system identifiers like order numbers) +- Transactional fact tables are typically not updated or versioned +- Cluster fact tables by foreign keys to frequently joined dimensions + +**Types of fact tables:** +- **Transaction facts:** One row per event (most common) +- **Periodic snapshot facts:** One row per entity per time period +- **Accumulating snapshot facts:** One row per entity lifecycle, updated as milestones are reached + +**Fact table Liquid Clustering strategy:** +```sql +CREATE TABLE gold.sales.fact_orders ( + order_key BIGINT GENERATED ALWAYS AS IDENTITY, + customer_key BIGINT NOT NULL, + product_key BIGINT NOT NULL, + date_key INT NOT NULL, + order_amount DECIMAL(18,2), + quantity INT, + CONSTRAINT fk_customer FOREIGN KEY (customer_key) REFERENCES gold.sales.dim_customer(customer_key), + CONSTRAINT fk_product FOREIGN KEY (product_key) REFERENCES gold.sales.dim_product(product_key) +) +CLUSTER BY (date_key, customer_key); +``` + +### Dimension Table Patterns + +**Design rules:** +- Use `GENERATED ALWAYS AS IDENTITY` or hash values for surrogate keys +- Prefer integer surrogate keys over strings for join performance +- Highly denormalize: flatten many-to-one relationships within a single dimension table +- Support complex types: MAP for extensibility, STRUCT for nested attributes, ARRAY for multi-valued attributes +- Avoid using ARRAY/MAP columns as filter predicates (they lack column-level statistics for data skipping) +- Cluster dimension tables by primary key plus common filter columns + +**Dimension table example:** +```sql +CREATE TABLE gold.sales.dim_customer ( + customer_key BIGINT GENERATED ALWAYS AS IDENTITY, + customer_id STRING NOT NULL COMMENT 'Natural key from source system', + full_name STRING, + email STRING, + city STRING, + state STRING, + country STRING, + segment STRING, + effective_start_date TIMESTAMP, + effective_end_date TIMESTAMP, + is_current BOOLEAN, + CONSTRAINT pk_customer PRIMARY KEY (customer_key) +) +CLUSTER BY (customer_key, segment) +COMMENT 'Customer dimension with SCD Type 2 history tracking'; +``` + +### Slowly Changing Dimensions (SCD) Patterns + +**SCD Type 1 (Overwrite):** +- In-place updates without tracking history +- Use MERGE INTO with matched UPDATE +- Suitable for corrections or attributes where history is not needed + +**SCD Type 2 (History Tracking):** +- Version records with surrogate keys and metadata columns +- Include `effective_start_date`, `effective_end_date`, and `is_current` columns +- Use MERGE INTO for implementing SCD Type 2 logic in DBSQL + +**SCD Type 2 with MERGE:** +```sql +MERGE INTO gold.sales.dim_customer AS target +USING ( + SELECT * FROM silver.crm.customers_changes +) AS source +ON target.customer_id = source.customer_id AND target.is_current = TRUE +WHEN MATCHED AND ( + target.full_name != source.full_name OR + target.city != source.city +) THEN UPDATE SET + effective_end_date = current_timestamp(), + is_current = FALSE +WHEN NOT MATCHED THEN INSERT ( + customer_id, full_name, email, city, state, country, segment, + effective_start_date, effective_end_date, is_current +) VALUES ( + source.customer_id, source.full_name, source.email, + source.city, source.state, source.country, source.segment, + current_timestamp(), NULL, TRUE +); +-- Then insert new versions for changed records in a second pass +``` + +**Delta Lake Time Travel** enables historical data access within configured log retention periods as a complementary feature to SCD. + +### Partitioning Strategies + +**Databricks recommends Liquid Clustering over traditional partitioning for all new tables.** + +Traditional partitioning rules of thumb (when needed): +- Keep partition count under 10,000 (ideally under 5,000 distinct values) +- Each partition should contain at least 1 GB of data +- Partition by low-cardinality columns that are frequently used in WHERE clauses (e.g., date, region) +- Works best for highly selective single-partition queries (e.g., filter on one day) + +**When traditional partitioning may still be appropriate:** +- Very large tables (hundreds of terabytes) with a clear, stable partition key +- Queries consistently filter on the same low-cardinality column +- Data lifecycle management requires partition-level operations + +### Liquid Clustering vs Traditional Partitioning + +**Liquid Clustering is the default recommendation for all new Delta tables**, including streaming tables and materialized views. It replaces both partitioning and Z-ORDER. + +| Aspect | Liquid Clustering | Partitioning + Z-ORDER | +|---|---|---| +| Column flexibility | Change clustering keys anytime | Partition column fixed at creation | +| Maintenance | Incremental, automatic with predictive optimization | Manual OPTIMIZE + Z-ORDER required | +| Filter dimensions | Best with 1-4 clustering keys | One partition key + Z-ORDER columns | +| Write overhead | Minimal (only unclustered ZCubes reorganized) | Z-ORDER reorganizes entire table/partition | +| Best for | Most workloads, evolving access patterns | Very large tables with stable, low-cardinality filter | +| Performance | 30-60% query speed improvement for variable queries | Better for single-partition lookup queries | + +**Liquid Clustering key selection best practices:** +- Choose columns most frequently used in query filters and joins +- Limit to 1-4 keys (fewer is better for smaller tables under 10 TB) +- For fact tables: cluster by the most commonly filtered foreign keys +- For dimension tables: cluster by primary key + common filter columns +- Too many keys dilute data skipping benefits; for tables under 10 TB, 2 keys often outperform 4 + +**Important:** Liquid Clustering is not compatible with partitioning or Z-ORDER on the same table. + +### Z-Ordering Considerations + +Z-ORDER is the legacy approach, now superseded by Liquid Clustering: + +- Z-ORDER reorganizes the entire table/partition during optimization (heavier writes) +- Does not track ZCube IDs, so every OPTIMIZE re-sorts all data +- Better suited for read-heavy workloads where write overhead is acceptable +- For new tables, always prefer Liquid Clustering + +**Migration path:** When migrating existing partitioned + Z-ORDERed tables to Liquid Clustering: +1. Drop the partition specification +2. Enable Liquid Clustering with chosen keys +3. Run OPTIMIZE to incrementally cluster data +4. Allow predictive optimization to maintain layout going forward + +--- + +## DBSQL Performance + +### Query Optimization Tips + +**Engine-level optimizations (automatic in DBSQL Serverless):** +- **Predictive Query Execution (PQE):** Monitors tasks in real time, dynamically adjusts query execution to avoid skew, spills, and unnecessary work. Unlike Adaptive Query Execution (AQE) which re-plans only after a stage completes, PQE detects issues like data skew or memory spills as they occur and replans immediately. +- **Photon Vectorized Shuffle:** Keeps data in compact columnar format, sorts within CPU cache, and uses vectorized instructions for 1.5x higher shuffle throughput. Best for CPU-bound workloads (large joins, wide aggregations). +- **Low Shuffle Merge:** Optimized MERGE implementation that reduces shuffle overhead for most common workloads. + +**Manual optimization actions:** +- Run `ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS` on dimension keys and frequently filtered columns to support AQE and data skipping +- Set `'delta.dataSkippingStatsColumns'` table property to specify which columns collect statistics +- Define PRIMARY KEY and FOREIGN KEY constraints to help the query optimizer +- Use deterministic queries (avoid `NOW()`, `CURRENT_TIMESTAMP()` in filters) to benefit from query result caching +- Prefer `CREATE OR REPLACE TABLE` over delete-then-create patterns +- Use `DECIMAL` over `FLOAT`/`DOUBLE` for financial calculations + +**SQL writing tips for DBSQL:** +- Filter early, aggregate late: push WHERE clauses as close to the source as possible +- Prefer explicit column lists over SELECT * +- Use CTEs for readability but be aware the optimizer may inline them +- Avoid Python/Scala UDFs when native SQL functions exist (UDFs require serialization between Python and Spark, significantly slowing queries) +- Use window functions instead of self-joins where possible +- Leverage QUALIFY clause for row-level filtering after window functions + +### Warehouse Sizing Guidance + +**Databricks recommends serverless SQL warehouses for most workloads.** Serverless uses Intelligent Workload Management (IWM) to automatically manage query workloads. + +**Sizing strategy:** +- Start with a single larger warehouse and let serverless features manage concurrency +- Size down if needed rather than starting small and scaling up +- If queries spill to disk, increase the cluster size + +**Scaling configuration:** +- Low concurrency (1-2 queries): keep max_clusters low +- Unpredictable spikes: set max_num_clusters high with target_utilization ~70% +- For dashboards with variable/infrequent load: enable aggressive auto-scaling and auto-stopping + +**Serverless advantages:** +- Start and scale up in seconds +- Scale down earlier than non-serverless warehouses +- Pay only when queries are running +- 30-60 second cold start latency (savings from no idle time far outweigh this) +- All 2025 optimizations (PQE, Photon Vectorized Shuffle) are automatically available + +### Caching Strategies + +**Query Result Cache:** +- DBSQL caches results per-cluster for all queries +- Cache is invalidated when underlying Delta data changes +- To maximize cache hits, use deterministic queries (no `NOW()`, `RAND()`, etc.) +- Both OBT and star schema achieve sub-500ms with automatic caching after first run + +**Delta Cache (Disk Cache):** +- Automatically caches remote data on local SSD in columnar format +- Accelerates data reads without manual configuration on serverless warehouses +- Particularly effective for repeated scans of the same tables + +**Best practice:** Design dashboards and reports to use parameterized queries that hit the same underlying patterns, maximizing cache reuse. + +### Photon Engine Benefits + +Photon is a vectorized query engine written in C++ that runs natively on Databricks: + +- Enabled by default on all DBSQL serverless warehouses +- Processes data in columnar batches using CPU vector instructions (SIMD) +- Excels at: large joins, wide aggregations, string processing, data shuffles +- 2025 vectorized shuffle delivers 1.5x higher shuffle throughput +- Combined with PQE, delivers up to 25% faster queries on top of existing 5x gains + +### Recent Performance Improvements (2025) + +| Improvement | Impact | +|---|---| +| Overall production workloads | Up to 40% faster (automatic, no tuning) | +| Photon Vectorized Shuffle | 1.5x higher shuffle throughput | +| PQE + Photon Vectorized Shuffle combined | Up to 25% faster on top of existing 5x gains | +| Spatial SQL queries | Up to 17x faster (R-tree indexing, optimized spatial joins) | +| AI functions | Up to 85x faster for large batch workloads | +| End-to-end Unity Catalog latency | Up to 10x improvement | +| 3-year cumulative improvement | 5x faster across customer workloads | + +All improvements are live in DBSQL Serverless with nothing to enable. + +### Cost Optimization Patterns + +1. **Use serverless SQL warehouses:** Pay only when queries run, auto-scale and auto-stop +2. **Enable predictive optimization:** Automatically runs OPTIMIZE and VACUUM on Unity Catalog managed tables +3. **Right-size warehouses:** Start larger, scale down based on actual usage patterns +4. **Avoid idle warehouses:** Use aggressive auto-stop for dashboards with infrequent load +5. **Leverage caching:** Design deterministic queries to maximize result cache hits +6. **Use Liquid Clustering:** Reduces scan volume, fewer DBUs consumed per query +7. **Collect statistics:** `ANALYZE TABLE` enables better query plans, reducing wasted compute +8. **Monitor with Query Profile:** Identify expensive operations, spills, and skew +9. **Use materialized views** for frequently computed aggregations +10. **Avoid UDFs:** Native functions are dramatically faster, no serialization overhead + +--- + +## Delta Lake Optimization for DBSQL + +### OPTIMIZE, VACUUM, and ANALYZE + +**Recommended execution order:** OPTIMIZE -> VACUUM -> ANALYZE + +**OPTIMIZE:** +- Compacts small files into larger ones (target 1 GB by default) +- Run frequently on tables with many small files (especially after streaming writes) +- Configurable target size via `delta.targetFileSize` table property +- With Liquid Clustering: only reorganizes unclustered ZCubes (incremental) + +**VACUUM:** +- Removes old files no longer in the transaction log +- Reduces storage costs +- Use compute-optimized instances (AWS C5, Azure F-series, GCP C2) +- Default retention: 7 days (configurable via `delta.deletedFileRetentionDuration`) +- Never set retention below the longest-running query duration + +**ANALYZE TABLE:** +- Computes column-level statistics for query optimization +- Run immediately after table overwrites or major data changes +- Focus on columns used in WHERE clauses, JOINs, and GROUP BY + +**Predictive optimization (recommended):** + +> **Note:** On serverless SQL warehouses, `delta.enableOptimizeWrite` and `delta.autoOptimize.autoCompact` are managed automatically and cannot be set manually (they will raise `DELTA_UNKNOWN_CONFIGURATION`). The properties below apply only to classic compute. For serverless, simply enable predictive optimization at the catalog/schema level. + +```sql +-- Classic compute only: +ALTER TABLE catalog.schema.table_name +SET TBLPROPERTIES ('delta.enableOptimizeWrite' = 'true'); +-- For Unity Catalog managed tables, predictive optimization +-- handles OPTIMIZE and VACUUM automatically +``` + +### File Size and Compaction + +- **Auto-compaction:** Combines small files within partitions automatically after writes +- **Optimized writes:** Rebalances data via shuffle before writing to reduce small files +- **Target file size:** Default 1 GB; adjust with `delta.targetFileSize` for specific workloads +- For tables with many small files (streaming ingestion), schedule regular OPTIMIZE jobs + +### Table Properties for Performance + +> **Note:** `delta.enableOptimizeWrite` and `delta.autoOptimize.autoCompact` are only valid on classic compute. On serverless SQL warehouses, these are managed automatically and setting them raises `DELTA_UNKNOWN_CONFIGURATION`. The remaining properties work on both classic and serverless. + +```sql +-- Classic compute only (serverless manages these automatically): +-- 'delta.enableOptimizeWrite' = 'true', +-- 'delta.autoOptimize.autoCompact' = 'true', + +-- Works on both classic and serverless: +ALTER TABLE catalog.schema.my_table SET TBLPROPERTIES ( + 'delta.columnMapping.mode' = 'name', + 'delta.enableChangeDataFeed' = 'true', + 'delta.deletedFileRetentionDuration' = '30 days', + 'delta.dataSkippingStatsColumns' = 'col1,col2,col3' +); +``` + +--- + +## Unity Catalog Integration Patterns + +### Organization Best Practices + +- Use a three-level namespace: `catalog.schema.table` +- Organize by environment (dev/staging/prod) at the catalog level +- Organize by business domain at the schema level +- Use managed tables (not external) to benefit from predictive optimization and enhanced governance + +### Governance Features for Data Modeling + +- **Primary/Foreign Key constraints:** Inform the query optimizer about table relationships +- **Row filters and column masks:** Fine-grained access control at the table level +- **Tags:** Apply governance tags (e.g., PII, sensitivity level) to tables and columns +- **Comments:** Document all tables and columns for AI/BI discoverability +- **Lineage tracking:** Automatic lineage for understanding data flow through the medallion architecture + +### Entity Relationship Visualization + +Unity Catalog renders entity relationship diagrams when primary and foreign key constraints are defined, providing visual documentation of the dimensional model. + +--- + +## Monitoring and Observability + +- **Query Profile:** Analyze execution plans, identify bottlenecks, spills, and data skew +- **Query History:** Track query performance trends over time +- **Warehouse monitoring:** Track utilization, queue times, and scaling events +- **System tables:** Query `system.billing`, `system.access`, and `system.query` for operational insights +- **Alerts:** Set up SQL alerts for data quality checks and SLA monitoring + +--- + +## Common Anti-Patterns to Avoid + +### Data Modeling Anti-Patterns + +1. **Skipping dimensional modeling in Gold layer:** OBTs are fine for Silver, but Gold should use star schemas for multi-dimensional analysis +2. **Over-partitioning:** More than 5,000-10,000 partitions degrades performance; use Liquid Clustering instead +3. **String surrogate keys:** Use integer IDENTITY columns for better join performance +4. **Missing constraints:** Not defining PK/FK constraints deprives the optimizer of relationship information +5. **Missing comments and tags:** Reduces discoverability for AI/BI tools and governance +6. **Using FLOAT for financial data:** Use DECIMAL to avoid precision errors +7. **Filtering on ARRAY/MAP columns:** These types lack column-level statistics for data skipping + +### Query and Performance Anti-Patterns + +1. **Delete-then-recreate tables:** Use `CREATE OR REPLACE TABLE` instead to preserve time travel and avoid reader interruptions +2. **Python/Scala UDFs when native functions exist:** Serialization overhead dramatically slows queries +3. **Not collecting statistics:** Missing `ANALYZE TABLE` leads to suboptimal query plans +4. **Non-deterministic functions in cached queries:** `NOW()`, `RAND()` etc. prevent query result caching +5. **Partitioning by wrong column:** Partitioning by a column not used in filters causes full scans +6. **Too many Liquid Clustering keys:** For tables under 10 TB, 2 keys often outperform 4 keys +7. **Manual OPTIMIZE/VACUUM without predictive optimization:** Enable predictive optimization for Unity Catalog managed tables + +### Operational Anti-Patterns + +1. **Idle warehouses:** Always enable auto-stop; use serverless for variable workloads +2. **Under-sized warehouses:** Queries spilling to disk waste more DBUs than a larger warehouse +3. **External tables when managed will do:** External tables miss predictive optimization and enhanced governance +4. **Skipping VACUUM:** Unbounded file growth increases storage costs and slows metadata operations +5. **Running VACUUM with too-short retention:** Can break long-running queries and time travel + +--- + +## Quick Reference: SQL Patterns for AI Agents + +When generating SQL for Databricks, prefer these patterns: + +```sql +-- Use CREATE OR REPLACE (not DROP + CREATE) +CREATE OR REPLACE TABLE catalog.schema.my_table AS +SELECT ...; + +-- Use MERGE for upserts (not DELETE + INSERT) +MERGE INTO target USING source +ON target.key = source.key +WHEN MATCHED THEN UPDATE SET ... +WHEN NOT MATCHED THEN INSERT ...; + +-- Use QUALIFY for window function filtering (not subquery) +SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY ts DESC) AS rn +FROM my_table +QUALIFY rn = 1; + +-- Use DECIMAL for money +SELECT CAST(amount AS DECIMAL(18,2)) AS revenue FROM orders; + +-- Collect statistics after loading +ANALYZE TABLE catalog.schema.my_table COMPUTE STATISTICS FOR ALL COLUMNS; + +-- Enable predictive optimization (classic compute only; serverless manages this automatically) +ALTER TABLE catalog.schema.my_table +SET TBLPROPERTIES ('delta.enableOptimizeWrite' = 'true'); +``` diff --git a/.claude/skills/databricks-dbsql/geospatial-collations.md b/.claude/skills/databricks-dbsql/geospatial-collations.md new file mode 100644 index 00000000..eaa2468f --- /dev/null +++ b/.claude/skills/databricks-dbsql/geospatial-collations.md @@ -0,0 +1,736 @@ +# Geospatial SQL and Collations in Databricks SQL + +--- + +## Part 1: Geospatial SQL + +Databricks SQL provides comprehensive geospatial support through two function families: **H3 functions** for hexagonal grid indexing and **ST functions** for standard spatial operations. Together they enable high-performance geospatial analytics at scale. + +### Geospatial Data Types + +| Type | Description | Coordinate System | SRID Support | +|------|-------------|-------------------|--------------| +| `GEOMETRY` | Spatial objects using Euclidean coordinates (X, Y, optional Z) -- treats Earth as flat | Any projected CRS | 11,000+ SRIDs | +| `GEOGRAPHY` | Geographic objects on Earth's surface using longitude/latitude | WGS 84 | SRID 4326 only | + +**When to use which:** +- Use `GEOMETRY` for projected coordinate systems, Euclidean distance calculations, and when working with local/regional data in meters or feet. +- Use `GEOGRAPHY` for global data using longitude/latitude coordinates and spherical distance calculations. + +### Supported Geometry Subtypes + +Both `GEOMETRY` and `GEOGRAPHY` support: **Point**, **LineString**, **Polygon**, **MultiPoint**, **MultiLineString**, **MultiPolygon**, and **GeometryCollection**. + +### Format Support + +| Format | Description | Import Function | Export Function | +|--------|-------------|-----------------|-----------------| +| WKT | Well-Known Text | `ST_GeomFromWKT`, `ST_GeogFromWKT` | `ST_AsWKT`, `ST_AsText` | +| WKB | Well-Known Binary | `ST_GeomFromWKB`, `ST_GeogFromWKB` | `ST_AsWKB`, `ST_AsBinary` | +| EWKT | Extended WKT (includes SRID) | `ST_GeomFromEWKT`, `ST_GeogFromEWKT` | `ST_AsEWKT` | +| EWKB | Extended WKB (includes SRID) | `ST_GeomFromEWKB` | `ST_AsEWKB` | +| GeoJSON | JSON-based format | `ST_GeomFromGeoJSON`, `ST_GeogFromGeoJSON` | `ST_AsGeoJSON` | +| Geohash | Hierarchical grid encoding | `ST_GeomFromGeoHash`, `ST_PointFromGeoHash` | `ST_GeoHash` | + +--- + +### H3 Geospatial Functions + +H3 is Uber's hexagonal hierarchical spatial index. It divides the Earth into hexagonal cells at 16 resolutions (0-15). Available since Databricks Runtime 11.2 (H3 Java library 3.7.0). No separate installation required. + +#### H3 Import Functions (Coordinate/Geometry to H3) + +| Function | Description | Returns | +|----------|-------------|---------| +| `h3_longlatash3(lon, lat, resolution)` | Convert longitude/latitude to H3 cell ID | `BIGINT` | +| `h3_longlatash3string(lon, lat, resolution)` | Convert longitude/latitude to H3 cell ID | `STRING` (hex) | +| `h3_pointash3(geogExpr, resolution)` | Convert GEOGRAPHY point to H3 cell ID | `BIGINT` | +| `h3_pointash3string(geogExpr, resolution)` | Convert GEOGRAPHY point to H3 cell ID | `STRING` (hex) | +| `h3_polyfillash3(geogExpr, resolution)` | Fill polygon with contained H3 cells | `ARRAY` | +| `h3_polyfillash3string(geogExpr, resolution)` | Fill polygon with contained H3 cells | `ARRAY` | +| `h3_coverash3(geogExpr, resolution)` | Cover geography with minimal set of H3 cells | `ARRAY` | +| `h3_coverash3string(geogExpr, resolution)` | Cover geography with minimal set of H3 cells | `ARRAY` | +| `h3_tessellateaswkb(geogExpr, resolution)` | Tessellate geography using H3 cells | `ARRAY` | +| `h3_try_polyfillash3(geogExpr, resolution)` | Safe polyfill (returns NULL on error) | `ARRAY` | +| `h3_try_polyfillash3string(geogExpr, resolution)` | Safe polyfill (returns NULL on error) | `ARRAY` | +| `h3_try_coverash3(geogExpr, resolution)` | Safe cover (returns NULL on error) | `ARRAY` | +| `h3_try_coverash3string(geogExpr, resolution)` | Safe cover (returns NULL on error) | `ARRAY` | +| `h3_try_tessellateaswkb(geogExpr, resolution)` | Safe tessellate (returns NULL on error) | `ARRAY` | + +#### H3 Export Functions (H3 to Geometry/Format) + +| Function | Description | Returns | +|----------|-------------|---------| +| `h3_boundaryaswkt(h3CellId)` | H3 cell boundary as WKT polygon | `STRING` | +| `h3_boundaryaswkb(h3CellId)` | H3 cell boundary as WKB polygon | `BINARY` | +| `h3_boundaryasgeojson(h3CellId)` | H3 cell boundary as GeoJSON | `STRING` | +| `h3_centeraswkt(h3CellId)` | H3 cell center as WKT point | `STRING` | +| `h3_centeraswkb(h3CellId)` | H3 cell center as WKB point | `BINARY` | +| `h3_centerasgeojson(h3CellId)` | H3 cell center as GeoJSON point | `STRING` | + +#### H3 Conversion Functions + +| Function | Description | +|----------|-------------| +| `h3_h3tostring(h3CellId)` | Convert BIGINT cell ID to hex STRING | +| `h3_stringtoh3(h3CellIdString)` | Convert hex STRING to BIGINT cell ID | + +#### H3 Hierarchy / Traversal Functions + +| Function | Description | +|----------|-------------| +| `h3_resolution(h3CellId)` | Get the resolution of a cell | +| `h3_toparent(h3CellId, resolution)` | Get parent cell at coarser resolution | +| `h3_tochildren(h3CellId, resolution)` | Get all child cells at finer resolution | +| `h3_maxchild(h3CellId, resolution)` | Get child with maximum value | +| `h3_minchild(h3CellId, resolution)` | Get child with minimum value | +| `h3_ischildof(h3CellId1, h3CellId2)` | Test if cell1 is equal to or child of cell2 | + +#### H3 Distance / Neighbor Functions + +| Function | Description | +|----------|-------------| +| `h3_distance(h3CellId1, h3CellId2)` | Grid distance between two cells | +| `h3_try_distance(h3CellId1, h3CellId2)` | Grid distance or NULL if undefined | +| `h3_kring(h3CellId, k)` | All cells within grid distance k (filled disk) | +| `h3_kringdistances(h3CellId, k)` | Cells within distance k with their distances | +| `h3_hexring(h3CellId, k)` | Hollow ring of cells at exactly distance k | + +#### H3 Compaction Functions + +| Function | Description | +|----------|-------------| +| `h3_compact(h3CellIds)` | Compact array of cells to minimal representation | +| `h3_uncompact(h3CellIds, resolution)` | Expand compacted cells to target resolution | + +#### H3 Validation Functions + +| Function | Description | +|----------|-------------| +| `h3_isvalid(expr)` | Check if BIGINT or STRING is valid H3 cell | +| `h3_validate(h3CellId)` | Return cell ID if valid, error otherwise | +| `h3_try_validate(h3CellId)` | Return cell ID if valid, NULL otherwise | +| `h3_ispentagon(h3CellId)` | Check if cell is a pentagon (12 per resolution) | + +#### H3 Examples + +```sql +-- Convert coordinates to H3 cell at resolution 9 +SELECT h3_longlatash3(-73.985428, 40.748817, 9) AS h3_cell; + +-- Index taxi trips by pickup location +CREATE TABLE trips_h3 AS +SELECT + h3_longlatash3(pickup_longitude, pickup_latitude, 12) AS pickup_cell, + h3_longlatash3(dropoff_longitude, dropoff_latitude, 12) AS dropoff_cell, + * +FROM taxi_trips; + +-- Fill zip code polygons with H3 cells for spatial indexing +CREATE TABLE zipcode_h3 AS +SELECT + explode(h3_polyfillash3(geom_wkt, 12)) AS cell, + zipcode, city, state +FROM zipcodes; + +-- Find all trips picked up in a specific zip code using H3 join +SELECT t.* +FROM trips_h3 t +INNER JOIN zipcode_h3 z ON t.pickup_cell = z.cell +WHERE z.zipcode = '10001'; + +-- Proximity search: find all H3 cells within 2 rings of a location +SELECT explode(h3_kring(h3_longlatash3(-73.985, 40.748, 9), 2)) AS nearby_cell; + +-- Aggregate trip counts and get centroids for visualization +SELECT + dropoff_cell, + h3_centerasgeojson(dropoff_cell):coordinates[0] AS lon, + h3_centerasgeojson(dropoff_cell):coordinates[1] AS lat, + count(*) AS trip_count +FROM trips_h3 +GROUP BY dropoff_cell; + +-- Roll up to coarser resolution +SELECT + h3_toparent(pickup_cell, 7) AS parent_cell, + count(*) AS trip_count +FROM trips_h3 +GROUP BY h3_toparent(pickup_cell, 7); + +-- Compact a set of cells for efficient storage +SELECT h3_compact(collect_set(cell)) AS compacted +FROM zipcode_h3 +WHERE zipcode = '10001'; +``` + +--- + +### ST Geospatial Functions + +Native spatial SQL functions operating on `GEOMETRY` and `GEOGRAPHY` types. Requires Databricks Runtime 17.1+. Public Preview. Over 80 functions available. + +#### ST Import Functions (Create Geometry/Geography) + +| Function | Description | Output Type | +|----------|-------------|-------------| +| `ST_GeomFromText(wkt [, srid])` | Create GEOMETRY from WKT | `GEOMETRY` | +| `ST_GeomFromWKT(wkt [, srid])` | Create GEOMETRY from WKT (alias) | `GEOMETRY` | +| `ST_GeomFromWKB(wkb [, srid])` | Create GEOMETRY from WKB | `GEOMETRY` | +| `ST_GeomFromEWKT(ewkt)` | Create GEOMETRY from Extended WKT | `GEOMETRY` | +| `ST_GeomFromEWKB(ewkb)` | Create GEOMETRY from Extended WKB | `GEOMETRY` | +| `ST_GeomFromGeoJSON(geojson)` | Create GEOMETRY(4326) from GeoJSON | `GEOMETRY` | +| `ST_GeomFromGeoHash(geohash)` | Create polygon GEOMETRY from geohash | `GEOMETRY` | +| `ST_GeogFromText(wkt)` | Create GEOGRAPHY(4326) from WKT | `GEOGRAPHY` | +| `ST_GeogFromWKT(wkt)` | Create GEOGRAPHY(4326) from WKT | `GEOGRAPHY` | +| `ST_GeogFromWKB(wkb)` | Create GEOGRAPHY(4326) from WKB | `GEOGRAPHY` | +| `ST_GeogFromEWKT(ewkt)` | Create GEOGRAPHY from Extended WKT | `GEOGRAPHY` | +| `ST_GeogFromGeoJSON(geojson)` | Create GEOGRAPHY(4326) from GeoJSON | `GEOGRAPHY` | +| `ST_Point(x, y [, srid])` | Create point from coordinates | `GEOMETRY` | +| `ST_PointFromGeoHash(geohash)` | Create point from geohash center | `GEOMETRY` | +| `to_geometry(georepExpr)` | Auto-detect format and create GEOMETRY | `GEOMETRY` | +| `to_geography(georepExpr)` | Auto-detect format and create GEOGRAPHY | `GEOGRAPHY` | +| `try_to_geometry(georepExpr)` | Safe geometry creation (NULL on error) | `GEOMETRY` | +| `try_to_geography(georepExpr)` | Safe geography creation (NULL on error) | `GEOGRAPHY` | + +#### ST Export Functions + +| Function | Description | Output | +|----------|-------------|--------| +| `ST_AsText(geo)` | Export as WKT | `STRING` | +| `ST_AsWKT(geo)` | Export as WKT (alias) | `STRING` | +| `ST_AsBinary(geo)` | Export as WKB | `BINARY` | +| `ST_AsWKB(geo)` | Export as WKB (alias) | `BINARY` | +| `ST_AsEWKT(geo)` | Export as Extended WKT | `STRING` | +| `ST_AsEWKB(geo)` | Export as Extended WKB | `BINARY` | +| `ST_AsGeoJSON(geo)` | Export as GeoJSON | `STRING` | +| `ST_GeoHash(geo)` | Export as geohash string | `STRING` | + +#### ST Constructor Functions + +| Function | Description | +|----------|-------------| +| `ST_Point(x, y [, srid])` | Create a point geometry | +| `ST_MakeLine(pointArray)` | Create linestring from array of points | +| `ST_MakePolygon(outer [, innerArray])` | Create polygon from outer ring and optional holes | + +#### ST Accessor Functions + +| Function | Description | Returns | +|----------|-------------|---------| +| `ST_X(geo)` | X coordinate of a point | `DOUBLE` | +| `ST_Y(geo)` | Y coordinate of a point | `DOUBLE` | +| `ST_Z(geo)` | Z coordinate of a point | `DOUBLE` | +| `ST_M(geo)` | M coordinate of a point | `DOUBLE` | +| `ST_XMin(geo)` | Minimum X of bounding box | `DOUBLE` | +| `ST_XMax(geo)` | Maximum X of bounding box | `DOUBLE` | +| `ST_YMin(geo)` | Minimum Y of bounding box | `DOUBLE` | +| `ST_YMax(geo)` | Maximum Y of bounding box | `DOUBLE` | +| `ST_ZMin(geo)` | Minimum Z coordinate | `DOUBLE` | +| `ST_ZMax(geo)` | Maximum Z coordinate | `DOUBLE` | +| `ST_Dimension(geo)` | Topological dimension (0=point, 1=line, 2=polygon) | `INT` | +| `ST_NDims(geo)` | Number of coordinate dimensions | `INT` | +| `ST_NPoints(geo)` | Total number of points | `INT` | +| `ST_NumGeometries(geo)` | Number of geometries in collection | `INT` | +| `ST_NumInteriorRings(geo)` | Number of interior rings (polygon) | `INT` | +| `ST_GeometryType(geo)` | Geometry type as string | `STRING` | +| `ST_GeometryN(geo, n)` | N-th geometry (1-based) from collection | `GEOMETRY` | +| `ST_PointN(geo, n)` | N-th point from linestring | `GEOMETRY` | +| `ST_StartPoint(geo)` | First point of linestring | `GEOMETRY` | +| `ST_EndPoint(geo)` | Last point of linestring | `GEOMETRY` | +| `ST_ExteriorRing(geo)` | Outer ring of polygon | `GEOMETRY` | +| `ST_InteriorRingN(geo, n)` | N-th interior ring of polygon | `GEOMETRY` | +| `ST_Envelope(geo)` | Minimum bounding rectangle | `GEOMETRY` | +| `ST_Envelope_Agg(geo)` | Aggregate: bounding box of all geometries | `GEOMETRY` | +| `ST_Dump(geo)` | Explode multi-geometry into array of singles | `ARRAY` | +| `ST_IsEmpty(geo)` | True if geometry has no points | `BOOLEAN` | + +#### ST Measurement Functions + +| Function | Description | +|----------|-------------| +| `ST_Area(geo)` | Area of a polygon (in CRS units) | +| `ST_Length(geo)` | Length of a linestring (in CRS units) | +| `ST_Perimeter(geo)` | Perimeter of a polygon (in CRS units) | +| `ST_Distance(geo1, geo2)` | Cartesian distance between geometries | +| `ST_DistanceSphere(geo1, geo2)` | Spherical distance in meters (fast, approximate) | +| `ST_DistanceSpheroid(geo1, geo2)` | Geodesic distance in meters on WGS84 (accurate) | +| `ST_Azimuth(geo1, geo2)` | North-based azimuth angle in radians | +| `ST_ClosestPoint(geo1, geo2)` | Point on geo1 closest to geo2 | + +#### ST Topological Relationship Functions (Predicates) + +| Function | Description | +|----------|-------------| +| `ST_Contains(geo1, geo2)` | True if geo1 fully contains geo2 | +| `ST_Within(geo1, geo2)` | True if geo1 is fully within geo2 (inverse of Contains) | +| `ST_Intersects(geo1, geo2)` | True if geometries share any space | +| `ST_Disjoint(geo1, geo2)` | True if geometries share no space | +| `ST_Touches(geo1, geo2)` | True if boundaries touch but interiors do not | +| `ST_Covers(geo1, geo2)` | True if geo1 covers geo2 (no point of geo2 is exterior) | +| `ST_Equals(geo1, geo2)` | True if geometries are topologically equal | +| `ST_DWithin(geo1, geo2, distance)` | True if geometries are within given distance | + +#### ST Overlay Functions (Set Operations) + +| Function | Description | +|----------|-------------| +| `ST_Intersection(geo1, geo2)` | Geometry of shared space | +| `ST_Union(geo1, geo2)` | Geometry combining both inputs | +| `ST_Union_Agg(geo)` | Aggregate: union of all geometries in column | +| `ST_Difference(geo1, geo2)` | Geometry of geo1 minus geo2 | + +#### ST Processing Functions + +| Function | Description | +|----------|-------------| +| `ST_Buffer(geo, radius)` | Expand geometry by radius distance | +| `ST_Centroid(geo)` | Center point of geometry | +| `ST_ConvexHull(geo)` | Smallest convex polygon containing geometry | +| `ST_ConcaveHull(geo, ratio [, allowHoles])` | Concave hull with length ratio | +| `ST_Boundary(geo)` | Boundary of geometry (not available on all SQL Warehouse versions) | +| `ST_Simplify(geo, tolerance)` | Simplify using Douglas-Peucker algorithm | + +#### ST Editor Functions + +| Function | Description | +|----------|-------------| +| `ST_AddPoint(linestring, point [, index])` | Add point to linestring | +| `ST_RemovePoint(linestring, index)` | Remove point from linestring | +| `ST_SetPoint(linestring, index, point)` | Replace point in linestring | +| `ST_FlipCoordinates(geo)` | Swap X and Y coordinates | +| `ST_Multi(geo)` | Convert single geometry to multi-geometry | +| `ST_Reverse(geo)` | Reverse vertex order | + +#### ST Affine Transformation Functions + +| Function | Description | +|----------|-------------| +| `ST_Translate(geo, xOffset, yOffset [, zOffset])` | Move geometry by offset | +| `ST_Scale(geo, xFactor, yFactor [, zFactor])` | Scale geometry by factors | +| `ST_Rotate(geo, angle)` | Rotate geometry around origin (radians) | + +#### ST Spatial Reference System Functions + +| Function | Description | +|----------|-------------| +| `ST_SRID(geo)` | Get SRID of geometry | +| `ST_SetSRID(geo, srid)` | Set SRID value (no reprojection) | +| `ST_Transform(geo, targetSrid)` | Reproject to target coordinate system | + +#### ST Validation + +| Function | Description | +|----------|-------------| +| `ST_IsValid(geo)` | Check if geometry is OGC-valid | + +#### ST Practical Examples + +> **Note:** `GEOMETRY` and `GEOGRAPHY` column types in `CREATE TABLE` require serverless compute with DBR 17.1+. On SQL Warehouses that don't support these column types, use `STRING` columns with WKT representation and convert with `ST_GeomFromText()` / `ST_GeogFromText()` at query time. + +```sql +-- Create a table with geometry columns (requires serverless DBR 17.1+) +CREATE TABLE retail_stores ( + store_id INT, + name STRING, + location GEOMETRY +); + +INSERT INTO retail_stores VALUES + (1, 'Downtown Store', ST_Point(-73.9857, 40.7484, 4326)), + (2, 'Midtown Store', ST_Point(-73.9787, 40.7614, 4326)), + (3, 'Uptown Store', ST_Point(-73.9680, 40.7831, 4326)); + +-- Create delivery zones as polygons +CREATE TABLE delivery_zones ( + zone_id INT, + zone_name STRING, + boundary GEOMETRY +); + +INSERT INTO delivery_zones VALUES + (1, 'Zone A', ST_GeomFromText( + 'POLYGON((-74.00 40.74, -73.97 40.74, -73.97 40.76, -74.00 40.76, -74.00 40.74))', 4326 + )); + +-- Point-in-polygon: find stores within a delivery zone +SELECT s.name, z.zone_name +FROM retail_stores s +JOIN delivery_zones z + ON ST_Contains(z.boundary, s.location); + +-- Distance calculation: find customers within 5km of a store +-- Note: to_geography() expects STRING (WKT/GeoJSON) or BINARY (WKB) input, not GEOMETRY. +-- Use ST_AsText() to convert GEOMETRY to WKT first. +SELECT c.customer_id, c.name, + ST_DistanceSphere(c.location, s.location) AS distance_meters +FROM customers c +CROSS JOIN retail_stores s +WHERE s.store_id = 1 + AND ST_DWithin( + ST_GeogFromText(ST_AsText(c.location)), + ST_GeogFromText(ST_AsText(s.location)), + 5000 -- 5km in meters + ); + +-- Buffer zone: create 1km buffer around a store (use projected CRS for meters) +SELECT ST_Buffer( + ST_Transform(location, 5070), -- project to NAD83/Albers (meters) + 1000 -- 1000 meters +) AS buffer_zone +FROM retail_stores +WHERE store_id = 1; + +-- Area calculation +SELECT zone_name, + ST_Area(ST_Transform(boundary, 5070)) AS area_sq_meters +FROM delivery_zones; + +-- Union of overlapping zones +SELECT ST_Union_Agg(boundary) AS combined_coverage +FROM delivery_zones; + +-- Convert between formats +SELECT + ST_AsText(location) AS wkt, + ST_AsGeoJSON(location) AS geojson, + ST_GeoHash(location) AS geohash +FROM retail_stores; + +-- Spatial join with BROADCAST hint for performance +SELECT /*+ BROADCAST(zones) */ + c.customer_id, z.zone_name +FROM customers c +JOIN delivery_zones zones + ON ST_Contains(zones.boundary, c.location); +``` + +### Combining H3 and ST Functions + +```sql +-- Use H3 for fast pre-filtering, then ST for precise spatial operations +-- Step 1: Index store locations with H3 +CREATE TABLE store_h3 AS +SELECT store_id, name, location, + h3_longlatash3(ST_X(location), ST_Y(location), 9) AS h3_cell +FROM retail_stores; + +-- Step 2: Index customer locations with H3 +CREATE TABLE customer_h3 AS +SELECT customer_id, name, location, + h3_longlatash3(ST_X(location), ST_Y(location), 9) AS h3_cell +FROM customers; + +-- Step 3: Fast proximity using H3 pre-filter + precise ST distance +SELECT s.name AS store, c.name AS customer, + ST_DistanceSphere(s.location, c.location) AS distance_m +FROM store_h3 s +JOIN customer_h3 c + ON c.h3_cell IN (SELECT explode(h3_kring(s.h3_cell, 2))) +WHERE ST_DistanceSphere(s.location, c.location) < 2000; +``` + +### Spatial Join Performance + +Databricks automatically optimizes spatial joins using built-in spatial indexing. Spatial predicates like `ST_Intersects`, `ST_Contains`, and `ST_Within` in JOIN conditions benefit from up to **17x performance improvement** compared to classic clusters. No code changes required -- the optimizer applies spatial indexing automatically. + +**Performance tips:** +- Use `BROADCAST` hint when one side of the join is small enough to fit in memory. +- Use projected coordinate systems (e.g., SRID 5070 in meters) for distance calculations to avoid expensive spheroid functions. +- Combine H3 for coarse pre-filtering with ST for precise operations. +- Use Delta Lake liquid clustering on H3 cell columns for optimized data layout. +- Enable auto-optimization: `delta.autoOptimize.optimizeWrite` and `delta.autoOptimize.autoCompact`. + +--- + +## Part 2: Collations + +Collations define rules for comparing and sorting strings. Databricks supports binary, case-insensitive, accent-insensitive, and locale-specific collations using the ICU library. Available from Databricks Runtime 16.1+. + +### Collation Types + +| Collation | Description | Behavior | +|-----------|-------------|----------| +| `UTF8_BINARY` | Default. Byte-by-byte comparison of UTF-8 encoding | `'A' < 'Z' < 'a'` -- binary order, case/accent sensitive | +| `UTF8_LCASE` | Case-insensitive binary. Converts to lowercase then compares with UTF8_BINARY | `'A' == 'a'` but `'e' != 'e'` (accent sensitive) | +| `UNICODE` | ICU root locale. Language-agnostic Unicode ordering | `'a' < 'A' < 'A' < 'b'` -- groups similar characters | +| Locale-specific | ICU locale-based (e.g., `DE`, `FR`, `JA`) | Language-aware sorting rules | + +### Collation Syntax + +``` +{ UTF8_BINARY | UTF8_LCASE | { UNICODE | locale } [ _ modifier [...] ] } +``` + +Where `locale` is: +``` +language_code [ _ script_code ] [ _ country_code ] +``` + +- `language_code`: ISO 639-1 (e.g., `EN`, `DE`, `FR`, `JA`, `ZH`) +- `script_code`: ISO 15924 (e.g., `Hant` for Traditional Chinese, `Latn` for Latin) +- `country_code`: ISO 3166-1 (e.g., `US`, `DE`, `CAN`) + +### Collation Modifiers (DBR 16.2+) + +| Modifier | Description | Default | +|----------|-------------|---------| +| `CS` | Case-Sensitive: `'A' != 'a'` | Yes (default) | +| `CI` | Case-Insensitive: `'A' == 'a'` | No | +| `AS` | Accent-Sensitive: `'e' != 'e'` | Yes (default) | +| `AI` | Accent-Insensitive: `'e' == 'e'` | No | +| `RTRIM` | Trailing-space insensitive: `'Hello' == 'Hello '` | No | + +Specify at most one from each pair (CS/CI, AS/AI) plus optional RTRIM. Order does not matter. + +### Locale Examples + +| Collation Name | Description | +|----------------|-------------| +| `UNICODE` | ICU root locale, language-agnostic | +| `UNICODE_CI` | Unicode, case-insensitive | +| `UNICODE_CI_AI` | Unicode, case and accent-insensitive | +| `DE` | German sorting rules | +| `DE_CI_AI` | German, case and accent-insensitive | +| `FR_CAN` | French (Canada) | +| `EN_US` | English (United States) | +| `ZH_Hant_MAC` | Traditional Chinese (Macau) | +| `SR` | Serbian (normalized from `SR_CYR_SRN_CS_AS`) | +| `JA` | Japanese | +| `EN_CS_AI` | English, case-sensitive, accent-insensitive | +| `UTF8_LCASE_RTRIM` | Case-insensitive with trailing space trimming | + +### Collation Precedence + +From highest to lowest: + +1. **Explicit** -- Assigned via `COLLATE` expression +2. **Implicit** -- Derived from column, field, or variable definition +3. **Default** -- Applied to string literals and function results +4. **None** -- When combining different implicit collations + +Mixing two different **explicit** collations in the same expression produces an error. + +### Setting Collations at Different Levels + +#### Catalog Level (DBR 17.1+) + +```sql +-- Create catalog with default collation +CREATE CATALOG customer_cat + DEFAULT COLLATION UNICODE_CI_AI; + +-- All schemas, tables, and string columns created in this catalog +-- inherit UNICODE_CI_AI unless overridden +``` + +#### Schema Level (DBR 17.1+) + +```sql +-- Create schema with default collation +CREATE SCHEMA my_schema + DEFAULT COLLATION UNICODE_CI; + +-- Change default collation for new objects (existing objects unchanged) +ALTER SCHEMA my_schema + DEFAULT COLLATION UNICODE_CI_AI; +``` + +#### Table Level (DBR 16.3+) + +```sql +-- Table-level default collation +CREATE TABLE users ( + id INT, + username STRING, -- inherits UNICODE_CI from table default + email STRING, -- inherits UNICODE_CI from table default + password_hash STRING COLLATE UTF8_BINARY -- explicit override +) DEFAULT COLLATION UNICODE_CI; +``` + +#### Column Level (DBR 16.1+) + +```sql +-- Column-level collation +CREATE TABLE products ( + id INT, + name STRING COLLATE UNICODE_CI, + sku STRING COLLATE UTF8_BINARY, + description STRING COLLATE UNICODE_CI_AI +); + +-- Add column with collation +ALTER TABLE products + ADD COLUMN category STRING COLLATE UNICODE_CI; + +-- Change column collation (requires DBR 17.2+; may not be available on all SQL Warehouse versions) +ALTER TABLE products + ALTER COLUMN name SET COLLATION UNICODE_CI_AI; +``` + +#### Expression Level + +```sql +-- Apply collation inline in a query +SELECT * +FROM products +WHERE name COLLATE UNICODE_CI = 'laptop'; + +-- Check the collation of an expression +SELECT collation('test' COLLATE UNICODE_CI); +-- Returns: UNICODE_CI +``` + +### Collation Inheritance Hierarchy + +``` +Catalog DEFAULT COLLATION + -> Schema DEFAULT COLLATION (overrides catalog) + -> Table DEFAULT COLLATION (overrides schema) + -> Column COLLATE (overrides table) + -> Expression COLLATE (overrides column) +``` + +If no collation is specified at any level, `UTF8_BINARY` is used. + +### Collation-Aware String Functions + +Most string functions respect collations. Key collation-aware operations: + +| Function/Operator | Collation Behavior | +|-------------------|-------------------| +| `=`, `!=`, `<`, `>`, `<=`, `>=` | Comparison uses column/expression collation | +| `LIKE` | Pattern matching respects collation | +| `CONTAINS(str, substr)` | Substring search respects collation | +| `STARTSWITH(str, prefix)` | Prefix match respects collation | +| `ENDSWITH(str, suffix)` | Suffix match respects collation | +| `IN (...)` | Membership test respects collation | +| `BETWEEN` | Range comparison respects collation | +| `ORDER BY` | Sorting respects collation | +| `GROUP BY` | Grouping respects collation | +| `DISTINCT` | Deduplication respects collation | +| `REPLACE(str, old, new)` | Search respects collation | +| `TRIM` / `LTRIM` / `RTRIM` | Trim characters respect collation | + +**Performance note:** `STARTSWITH` and `ENDSWITH` with `UTF8_LCASE` collation show up to **10x performance speedup** compared to equivalent `LOWER()` workarounds. + +### Utility Functions + +```sql +-- Get collation of an expression +SELECT collation(name) FROM products; + +-- List all supported collations +SELECT * FROM collations(); + +-- Test collation with COLLATE +SELECT collation('hello' COLLATE DE_CI_AI); +-- Returns: DE_CI_AI +``` + +### Practical Collation Examples + +#### Case-Insensitive Search + +```sql +-- Using column collation (preferred - leverages indexes) +CREATE TABLE users ( + id INT, + username STRING COLLATE UTF8_LCASE, + email STRING COLLATE UTF8_LCASE +); + +INSERT INTO users VALUES + (1, 'JohnDoe', 'John@Example.com'), + (2, 'janedoe', 'JANE@EXAMPLE.COM'); + +-- Case-insensitive match automatically +SELECT * FROM users WHERE username = 'johndoe'; +-- Returns: JohnDoe + +SELECT * FROM users WHERE email = 'john@example.com'; +-- Returns: John@Example.com +``` + +#### Case-Insensitive Search with Expression Collation + +```sql +-- Ad-hoc case-insensitive comparison on a UTF8_BINARY column +SELECT * FROM products +WHERE name COLLATE UNICODE_CI = 'MacBook Pro'; +-- Matches: macbook pro, MACBOOK PRO, MacBook Pro, etc. +``` + +#### Accent-Insensitive Search + +```sql +-- Accent-insensitive matching +CREATE TABLE cities ( + id INT, + name STRING COLLATE UNICODE_CI_AI +); + +INSERT INTO cities VALUES (1, 'Montreal'), (2, 'Montreal'); + +SELECT * FROM cities WHERE name = 'Montreal'; +-- Returns both: Montreal and Montreal (treats e and e as equal) +``` + +#### Locale-Aware Sorting + +```sql +-- German sorting (umlauts sort correctly) +SELECT name +FROM german_customers +ORDER BY name COLLATE DE; +-- Sorts: Arzte before Bauer (A treated as A+e in German sorting) + +-- Swedish sorting (A, A, O sort after Z) +SELECT name +FROM swedish_customers +ORDER BY name COLLATE SV; +``` + +#### Trailing Space Handling + +```sql +-- RTRIM modifier ignores trailing spaces +SELECT 'Hello' COLLATE UTF8_BINARY_RTRIM = 'Hello '; +-- Returns: true + +SELECT 'Hello' COLLATE UTF8_BINARY = 'Hello '; +-- Returns: false +``` + +#### Catalog-Wide Case-Insensitive Setup + +```sql +-- Create a catalog where everything is case-insensitive by default +CREATE CATALOG app_data DEFAULT COLLATION UNICODE_CI; + +USE CATALOG app_data; +CREATE SCHEMA users_schema; +USE SCHEMA users_schema; + +-- All STRING columns automatically use UNICODE_CI +CREATE TABLE accounts ( + id INT, + username STRING, -- UNICODE_CI inherited from catalog + email STRING -- UNICODE_CI inherited from catalog +); + +-- Queries are automatically case-insensitive +SELECT * FROM accounts WHERE username = 'admin'; +-- Matches: Admin, ADMIN, admin, aDmIn, etc. +``` + +### Limitations and Notes + +- `CHECK` constraints and generated column expressions require `UTF8_BINARY` default collation. +- `hive_metastore` catalog tables do not support collation constraints. +- `ALTER SCHEMA ... DEFAULT COLLATION` only affects newly created objects, not existing ones. +- Mixing two different explicit collations in the same expression raises an error. +- `UTF8_LCASE` is used internally for Databricks identifier resolution (catalog, schema, table, column names). +- Databricks normalizes collation names by removing defaults (e.g., `SR_CYR_SRN_CS_AS` simplifies to `SR`). +- Collation modifiers require Databricks Runtime 16.2+. +- Catalog/Schema-level `DEFAULT COLLATION` requires Databricks Runtime 17.1+. diff --git a/.claude/skills/databricks-dbsql/materialized-views-pipes.md b/.claude/skills/databricks-dbsql/materialized-views-pipes.md new file mode 100644 index 00000000..078ad09b --- /dev/null +++ b/.claude/skills/databricks-dbsql/materialized-views-pipes.md @@ -0,0 +1,676 @@ +# Materialized Views, Temporary Tables/Views, and Pipe Syntax + +## 1. Materialized Views in Databricks SQL + +### Overview + +Materialized views (MVs) are Unity Catalog-managed tables that physically store precomputed query results. Unlike standard views that recompute on every query, MVs cache results and update automatically -- either on a schedule, when upstream data changes, or on-demand. + +Key characteristics: +- **Pre-computed storage**: Results are physically stored as Delta tables, reducing query latency +- **Automatic updates**: Changes propagate from source tables via incremental or full refresh +- **Serverless pipelines**: Each MV automatically creates a serverless pipeline for creation and refreshes +- **Incremental refresh**: Can compute only changed data from source tables under certain conditions + +### Requirements + +- **Compute**: Unity Catalog-enabled **Serverless** SQL warehouse +- **Region**: Serverless SQL warehouse support must be available in your region +- **Permissions**: + - Creator needs: `SELECT` on base tables, `USE CATALOG`, `USE SCHEMA`, `CREATE TABLE`, `CREATE MATERIALIZED VIEW` + - Refresh needs: Ownership or `REFRESH` privilege; MV owner must retain `SELECT` on base tables + - Query needs: `SELECT` on the MV, `USE CATALOG`, `USE SCHEMA` + +### CREATE MATERIALIZED VIEW Syntax + +```sql +{ CREATE OR REPLACE MATERIALIZED VIEW | CREATE MATERIALIZED VIEW [ IF NOT EXISTS ] } + view_name + [ column_list ] + [ view_clauses ] + AS query +``` + +**Column list** (optional): +```sql +CREATE MATERIALIZED VIEW mv_name ( + col1 INT NOT NULL, + col2 STRING, + col3 DOUBLE, + CONSTRAINT pk PRIMARY KEY (col1) +) +AS SELECT ... +``` + +**View clauses** (optional): +- `PARTITIONED BY (col1, col2)` -- partition by columns +- `CLUSTER BY (col1, col2)` or `CLUSTER BY AUTO` -- liquid clustering (cannot combine with PARTITIONED BY) +- `COMMENT 'description'` -- view description +- `TBLPROPERTIES ('key' = 'value')` -- user-defined properties +- `WITH ROW FILTER func ON (col1, col2)` -- row-level security +- `MASK func` on columns -- column-level masking +- `SCHEDULE` clause -- automatic refresh schedule +- `TRIGGER ON UPDATE` clause -- event-driven refresh + +### Basic Examples + +```sql +-- Simple materialized view +CREATE MATERIALIZED VIEW catalog.schema.daily_sales + COMMENT 'Daily sales aggregations' +AS SELECT + date, + region, + SUM(sales) AS total_sales, + COUNT(*) AS num_transactions +FROM catalog.schema.raw_sales +GROUP BY date, region; + +-- MV with explicit columns, constraints, and clustering +CREATE MATERIALIZED VIEW catalog.schema.customer_orders ( + customer_id INT NOT NULL, + full_name STRING, + order_count BIGINT, + CONSTRAINT customer_pk PRIMARY KEY (customer_id) +) +CLUSTER BY AUTO +COMMENT 'Customer order counts' +AS SELECT + c.customer_id, + c.full_name, + COUNT(o.order_id) AS order_count +FROM catalog.schema.customers c +INNER JOIN catalog.schema.orders o ON c.customer_id = o.customer_id +GROUP BY c.customer_id, c.full_name; +``` + +### Refresh Options + +MVs support four refresh strategies: + +#### 1. Manual Refresh + +```sql +-- Synchronous (blocks until complete) +REFRESH MATERIALIZED VIEW catalog.schema.daily_sales; + +-- Asynchronous (returns immediately) +REFRESH MATERIALIZED VIEW catalog.schema.daily_sales ASYNC; +``` + +#### 2. Scheduled Refresh (SCHEDULE) + +```sql +-- Every N hours/days/weeks +CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.hourly_metrics + SCHEDULE EVERY 1 HOUR +AS SELECT date_trunc('hour', event_time) AS hour, COUNT(*) AS events +FROM catalog.schema.raw_events +GROUP BY 1; + +-- Cron-based schedule +CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.nightly_report + SCHEDULE CRON '0 0 2 * * ?' AT TIME ZONE 'America/New_York' +AS SELECT * FROM catalog.schema.daily_aggregates; +``` + +Valid intervals: 1-72 hours, 1-31 days, 1-8 weeks. A Databricks Job is automatically created for scheduled refreshes. + +#### 3. Event-Driven Refresh (TRIGGER ON UPDATE) + +Automatically refreshes when upstream data changes: + +```sql +CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.customer_orders + TRIGGER ON UPDATE +AS SELECT c.customer_id, c.name, COUNT(o.order_id) AS order_count +FROM catalog.schema.customers c +JOIN catalog.schema.orders o ON c.customer_id = o.customer_id +GROUP BY c.customer_id, c.name; + +-- With throttle to avoid excessive refreshes +CREATE OR REPLACE MATERIALIZED VIEW catalog.schema.customer_orders + TRIGGER ON UPDATE AT MOST EVERY INTERVAL 5 MINUTES +AS SELECT c.customer_id, c.name, COUNT(o.order_id) AS order_count +FROM catalog.schema.customers c +JOIN catalog.schema.orders o ON c.customer_id = o.customer_id +GROUP BY c.customer_id, c.name; +``` + +Trigger limitations: +- Maximum **10 upstream source tables** and **30 upstream views** +- Minimum **1-minute** interval (default) +- Maximum **1,000** trigger-based MVs per workspace +- Supports Delta tables, managed views, and streaming tables as sources +- Does **not** support Delta Sharing shared tables + +#### 4. Job-Based Orchestration + +Integrate refreshes into existing Databricks Jobs using SQL task types: + +```sql +-- In a Databricks Job SQL task +REFRESH MATERIALIZED VIEW catalog.schema.daily_sales_summary; +``` + +### Managing Schedules After Creation + +```sql +-- Add a schedule to an existing MV +ALTER MATERIALIZED VIEW catalog.schema.my_mv ADD SCHEDULE EVERY 4 HOURS; + +-- Add trigger-based refresh +ALTER MATERIALIZED VIEW catalog.schema.my_mv ADD TRIGGER ON UPDATE; + +-- Change an existing schedule +ALTER MATERIALIZED VIEW catalog.schema.my_mv ALTER SCHEDULE EVERY 2 HOURS; + +-- Remove a schedule +ALTER MATERIALIZED VIEW catalog.schema.my_mv DROP SCHEDULE; +``` + +### Incremental vs Full Refresh + +| Aspect | Incremental Refresh | Full Refresh | +|--------|-------------------|--------------| +| What it does | Evaluates changes since last refresh, merges only new/modified records | Re-executes the entire defining query | +| When used | When source tables support change tracking and query structure allows it | When incremental is not possible or not cost-effective | +| Requirements | Delta source tables with row tracking and CDF enabled | No special requirements | +| Cost | Lower (processes only deltas) | Higher (recomputes everything) | + +Enable row tracking on source tables for incremental refresh: + +```sql +ALTER TABLE catalog.schema.source_table +SET TBLPROPERTIES (delta.enableRowTracking = true); +``` + +By default, Databricks uses a cost model to choose between incremental and full refresh. Use `EXPLAIN CREATE MATERIALIZED VIEW` to verify the chosen refresh type. + +### Timeout Configuration + +```sql +-- Set timeout before creating or refreshing +SET STATEMENT_TIMEOUT = '6h'; +CREATE OR REFRESH MATERIALIZED VIEW catalog.schema.my_mv + SCHEDULE EVERY 12 HOURS +AS SELECT * FROM catalog.schema.large_source_table; +``` + +Default timeout is **2 days** if no warehouse timeout is configured. After changing warehouse timeouts, re-run `CREATE OR REFRESH` to apply new settings. + +### Monitoring + +- **Catalog Explorer**: View refresh status, schema, permissions, lineage under the MV entry +- **DESCRIBE EXTENDED**: Get schedule and configuration details +- **Jobs & Pipelines UI**: Monitor the automatically created pipeline +- **Pipelines API**: `GET /api/2.0/pipelines/{pipeline_id}` for programmatic access +- **DESCRIBE EXTENDED AS JSON**: Get refresh information including last refresh time, type, status, and schedule (added October 2025) + +### Key Limitations + +- No identity columns or surrogate keys +- Cannot read change data feeds (CDF) from materialized views +- Time travel queries are not supported +- `OPTIMIZE` and `VACUUM` commands are not supported (managed automatically) +- **Null handling edge case**: `SUM()` on a nullable column returns **0** instead of `NULL` when all non-null values are removed +- Non-column expressions in the defining query require explicit aliases +- Underlying storage may contain upstream data not visible in the MV definition (required for incremental refresh) +- Cannot rename the MV or change its owner via ALTER (must drop and recreate) +- No data quality expectations support +- AWS PrivateLink requires contacting Databricks support + +### DBSQL Materialized Views vs Pipeline (SDP/DLT) Materialized Views + +| Aspect | DBSQL Materialized Views | Pipeline (SDP/DLT) Materialized Views | +|--------|-------------------------|--------------------------------------| +| **Creation** | `CREATE MATERIALIZED VIEW` in SQL warehouse | Defined in pipeline source code (SQL or Python) | +| **Pipeline type** | `MV/ST` (auto-created serverless pipeline) | `ETL` (explicitly defined pipeline) | +| **Pipeline management** | Automatically created and managed | User-defined, full pipeline lifecycle control | +| **Syntax** | Standard `CREATE MATERIALIZED VIEW` | `CREATE OR REFRESH MATERIALIZED VIEW` with `PRIVATE` option | +| **Private MVs** | Not supported | `PRIVATE` keyword for pipeline-scoped views | +| **Refresh trigger** | Schedule, trigger-on-update, manual, or job-based | Pipeline update (manual or scheduled) | +| **Compute** | Serverless SQL warehouse (creation); serverless pipeline (refresh) | Pipeline compute (serverless or classic) | +| **Data quality** | Not supported | Expectations supported | +| **Best for** | Standalone MVs, BI dashboard acceleration, simple ETL | Complex multi-table pipelines, orchestrated transformations | + +Both approaches ultimately use similar underlying mechanisms (serverless pipelines) and support incremental refresh. The key difference is in management: DBSQL MVs are self-contained with auto-managed pipelines, while pipeline MVs are part of a broader orchestrated data flow. + +### Best Practices + +1. **Choose the right refresh strategy**: `TRIGGER ON UPDATE` for near-real-time SLA; `SCHEDULE` for predictable cadences; manual or job-based for complex orchestration +2. **Enable row tracking** on Delta source tables for cost-effective incremental refreshes +3. **Use async refreshes** when refresh duration is long and downstream queries can tolerate slight staleness +4. **Set explicit timeouts** when refresh duration is predictable to avoid runaway costs +5. **Use `CLUSTER BY AUTO`** for automatic liquid clustering optimization +6. **Apply row filters and column masks** at MV creation for security +7. **Monitor refresh types** with `EXPLAIN CREATE MATERIALIZED VIEW` to verify incremental behavior + +--- + +## 2. Temporary Tables and Temporary Views + +### Temporary Tables + +Temporary tables are session-scoped, physical Delta tables for intermediate data storage. They exist only within the session where they are created. + +#### Key Characteristics + +- **Session-scoped**: Only visible to the creating session; isolated from other users +- **Physical storage**: Stored as Delta tables in an internal Unity Catalog location tied to the workspace +- **Maximum lifetime**: 7 days from session creation, or until the session ends (whichever comes first) +- **No catalog privileges needed**: Any user can create temporary tables without `CREATE TABLE` privileges +- **Automatic cleanup**: Databricks reclaims storage automatically, even after unexpected disconnections +- **Shared namespace**: Temporary tables share a namespace with temporary views; you cannot create both with the same name + +#### Syntax + +```sql +-- Create with schema +CREATE TEMPORARY TABLE temp_results ( + id INT, + name STRING, + score DOUBLE +); + +-- Create from query (CTAS) +CREATE TEMP TABLE temp_active_users +AS SELECT user_id, username, last_login +FROM catalog.schema.users +WHERE last_login > current_date() - INTERVAL 30 DAYS; +``` + +Note: `CREATE OR REPLACE TEMP TABLE` is **not yet supported**. To replace, drop first. + +#### Supported Operations + +```sql +-- INSERT +INSERT INTO temp_results VALUES (1, 'Alice', 95.5); +INSERT INTO temp_results SELECT * FROM catalog.schema.source WHERE score > 90; + +-- UPDATE +UPDATE temp_results SET score = 100.0 WHERE name = 'Alice'; + +-- MERGE +MERGE INTO temp_results t +USING catalog.schema.new_scores s ON t.id = s.id +WHEN MATCHED THEN UPDATE SET score = s.score +WHEN NOT MATCHED THEN INSERT *; +``` + +#### Unsupported Operations + +- `DELETE FROM` (not supported) +- `ALTER TABLE` (drop and recreate instead) +- Shallow or deep cloning +- Time travel +- Streaming (foreachBatch) +- DataFrame API access (SQL only) + +#### Use Cases + +1. **Exploratory analysis**: Store intermediate results while iterating on queries +2. **Multi-step transformations**: Break complex transformations into readable steps +3. **Query result reuse**: Compute once, reference multiple times in a session +4. **Sandboxing**: Test transformations without affecting production tables + +#### Name Resolution + +When referencing a single-part table name, Databricks resolves in order: +1. Temporary tables in the current session +2. Permanent tables in the current schema + +Temporary tables with the same name as permanent tables **take precedence** within that session. + +### Temporary Views + +Temporary views are session-scoped, logical views that store a query definition (not data). They are recomputed on each access. + +#### Syntax + +```sql +-- Create a temporary view +CREATE TEMPORARY VIEW active_customers +AS SELECT customer_id, name, email +FROM catalog.schema.customers +WHERE status = 'active'; + +-- Replace an existing temporary view +CREATE OR REPLACE TEMPORARY VIEW active_customers +AS SELECT customer_id, name, email, phone +FROM catalog.schema.customers +WHERE status = 'active' AND last_order > current_date() - INTERVAL 90 DAYS; +``` + +#### Key Rules + +- Temporary view names **must not be qualified** (no catalog or schema prefix) +- No special privileges required to create +- Dropped automatically when the session ends +- Cannot use `schema_binding` clauses +- Support `COMMENT` and column comments + +#### Global Temporary Views (Databricks Runtime Only) + +```sql +-- Only available in Databricks Runtime, NOT in Databricks SQL +CREATE GLOBAL TEMPORARY VIEW global_summary +AS SELECT region, SUM(revenue) AS total_revenue +FROM catalog.schema.sales +GROUP BY region; + +-- Must reference via global_temp schema +SELECT * FROM global_temp.global_summary; +``` + +Global temporary views are stored in a system `global_temp` schema and are session-scoped. They are **not available in Databricks SQL** (only Databricks Runtime). + +### Temporary Tables vs Temporary Views + +| Aspect | Temporary Tables | Temporary Views | +|--------|-----------------|-----------------| +| **Storage** | Physical Delta table (stores data) | Logical (stores query definition only) | +| **Compute on access** | No (data already materialized) | Yes (query re-executed each time) | +| **DML support** | INSERT, UPDATE, MERGE | None (read-only definition) | +| **Max lifetime** | 7 days or session end | Session end | +| **CREATE OR REPLACE** | Not supported | Supported | +| **Performance** | Faster for repeated reads (data cached) | Slower for repeated reads (recomputed) | +| **Storage cost** | Uses cloud storage (auto-cleaned) | No storage cost | +| **Shared namespace** | Yes (conflicts with temp views) | Yes (conflicts with temp tables) | +| **When to use** | Large intermediate results, repeated access, DML needed | Simple query aliases, lightweight transformations | + +### Temporary Metric Views (Added September 2025) + +```sql +-- Temporary metric views: session-scoped, dropped on session end +CREATE TEMPORARY METRIC VIEW session_metrics +AS SELECT ...; +``` + +Available in Databricks Runtime 17.2+ and Databricks SQL. + +--- + +## 3. SQL Pipe Syntax + +### Overview + +Pipe syntax (introduced February 2025) allows composing SQL queries as a top-down, left-to-right chain of operations using the `|>` operator. It eliminates deeply nested subqueries and makes SQL read like a DataFrame pipeline. + +**Requirements**: Databricks SQL or Databricks Runtime **16.2+** + +### Basic Syntax + +```sql +FROM table_name +|> pipe_operation_1 +|> pipe_operation_2 +|> pipe_operation_3; +``` + +Any query can start a pipeline. The most common pattern is `FROM table_name`, but any SELECT or subquery also works: + +```sql +-- Start from a table +FROM catalog.schema.sales |> WHERE region = 'US' |> SELECT product, amount; + +-- Start from a subquery +(SELECT * FROM catalog.schema.sales WHERE year = 2025) +|> AGGREGATE SUM(amount) AS total GROUP BY product +|> ORDER BY total DESC; +``` + +### All Available Pipe Operators + +#### SELECT -- Project columns + +```sql +FROM catalog.schema.employees +|> SELECT employee_id, name, department, salary; +``` + +Note: `SELECT` in pipe syntax **must not contain aggregate functions**. Use `AGGREGATE` instead. + +#### EXTEND -- Add new columns + +Appends new columns to the existing result set (like PySpark's `withColumn`): + +```sql +FROM catalog.schema.orders +|> EXTEND quantity * unit_price AS line_total +|> EXTEND line_total * 0.1 AS tax; +``` + +Expressions can reference columns created by preceding expressions in the same EXTEND. + +#### SET -- Modify existing columns + +Overrides existing column values (like PySpark's `withColumn` on existing columns): + +```sql +FROM catalog.schema.products +|> SET price = price * 1.1 +|> SET name = UPPER(name); +``` + +Raises `UNRESOLVED_COLUMN` if the column does not exist. + +#### DROP -- Remove columns + +Removes columns (shorthand for `SELECT * EXCEPT`): + +```sql +FROM catalog.schema.users +|> DROP password_hash, internal_id, debug_flag; +``` + +#### WHERE -- Filter rows + +```sql +FROM catalog.schema.transactions +|> WHERE amount > 1000 +|> WHERE transaction_date >= '2025-01-01'; +``` + +#### AGGREGATE -- Aggregation with optional GROUP BY + +```sql +-- Full-table aggregation +FROM catalog.schema.orders +|> AGGREGATE + COUNT(*) AS total_orders, + SUM(amount) AS total_revenue, + AVG(amount) AS avg_order_value; + +-- Grouped aggregation +FROM catalog.schema.orders +|> AGGREGATE + SUM(amount) AS total_revenue, + COUNT(*) AS order_count + GROUP BY region, product_category; +``` + +In pipe syntax, `AGGREGATE` replaces `SELECT ... GROUP BY`. Numeric values in GROUP BY reference input columns, not generated results. + +#### JOIN -- Combine relations + +```sql +FROM catalog.schema.orders +|> AS o +|> LEFT JOIN catalog.schema.customers c ON o.customer_id = c.customer_id +|> SELECT o.order_id, c.name, o.amount; +``` + +All JOIN types are supported: `INNER JOIN`, `LEFT OUTER JOIN`, `RIGHT OUTER JOIN`, `FULL OUTER JOIN`, `CROSS JOIN`, `SEMI JOIN`, `ANTI JOIN`. + +#### ORDER BY -- Sort results + +```sql +FROM catalog.schema.products +|> ORDER BY price DESC, name ASC; +``` + +#### LIMIT and OFFSET -- Pagination + +```sql +FROM catalog.schema.products +|> ORDER BY price DESC +|> LIMIT 10 +|> OFFSET 20; +``` + +#### AS -- Assign table alias + +Names the intermediate result for use in subsequent JOINs or self-references: + +```sql +FROM catalog.schema.sales +|> AS current_sales +|> JOIN catalog.schema.targets t ON current_sales.region = t.region +|> SELECT current_sales.region, current_sales.revenue, t.target; +``` + +#### Set Operators -- UNION, EXCEPT, INTERSECT + +```sql +FROM catalog.schema.us_customers +|> UNION ALL (SELECT * FROM catalog.schema.eu_customers) +|> ORDER BY name; +``` + +#### TABLESAMPLE -- Sample rows + +```sql +-- Sample by row count +FROM catalog.schema.large_table +|> TABLESAMPLE (1000 ROWS); + +-- Sample by percentage +FROM catalog.schema.large_table +|> TABLESAMPLE (10 PERCENT); +``` + +#### PIVOT -- Rows to columns + +```sql +FROM catalog.schema.quarterly_sales +|> PIVOT ( + SUM(revenue) + FOR quarter IN ('Q1', 'Q2', 'Q3', 'Q4') + ); +``` + +#### UNPIVOT -- Columns to rows + +```sql +FROM catalog.schema.wide_metrics +|> UNPIVOT ( + metric_value FOR metric_name IN (cpu_usage, memory_usage, disk_usage) + ); +``` + +### Practical Examples + +#### Example 1: Multi-step aggregation (replaces nested subqueries) + +Traditional SQL: +```sql +SELECT c_count, COUNT(*) AS custdist +FROM ( + SELECT c_custkey, COUNT(o_orderkey) AS c_count + FROM customer + LEFT OUTER JOIN orders ON c_custkey = o_custkey + AND o_comment NOT LIKE '%unusual%packages%' + GROUP BY c_custkey +) AS c_orders +GROUP BY c_count +ORDER BY custdist DESC, c_count DESC; +``` + +Pipe syntax: +```sql +FROM customer +|> LEFT OUTER JOIN orders ON c_custkey = o_custkey + AND o_comment NOT LIKE '%unusual%packages%' +|> AGGREGATE COUNT(o_orderkey) AS c_count GROUP BY c_custkey +|> AGGREGATE COUNT(*) AS custdist GROUP BY c_count +|> ORDER BY custdist DESC, c_count DESC; +``` + +#### Example 2: Data exploration and profiling + +```sql +FROM catalog.schema.raw_events +|> WHERE event_date >= '2025-01-01' +|> EXTEND YEAR(event_date) AS event_year, MONTH(event_date) AS event_month +|> AGGREGATE + COUNT(*) AS event_count, + COUNT(DISTINCT user_id) AS unique_users, + AVG(duration_seconds) AS avg_duration + GROUP BY event_year, event_month +|> ORDER BY event_year, event_month; +``` + +#### Example 3: Building a report step-by-step + +```sql +FROM catalog.schema.orders +|> AS o +|> JOIN catalog.schema.products p ON o.product_id = p.product_id +|> JOIN catalog.schema.customers c ON o.customer_id = c.customer_id +|> WHERE o.order_date >= '2025-01-01' +|> EXTEND o.quantity * p.unit_price AS line_total +|> AGGREGATE + SUM(line_total) AS total_revenue, + COUNT(DISTINCT o.order_id) AS order_count + GROUP BY c.region, p.category +|> ORDER BY total_revenue DESC +|> LIMIT 20; +``` + +#### Example 4: Debugging by commenting out tail operations + +```sql +FROM catalog.schema.sales +|> WHERE region = 'US' +|> EXTEND amount * tax_rate AS tax_amount +-- |> AGGREGATE SUM(tax_amount) AS total_tax GROUP BY state +-- |> ORDER BY total_tax DESC +; +-- Comment out the last operations to inspect intermediate results +``` + +### Pipe Syntax vs Traditional SQL + +| Aspect | Traditional SQL | Pipe SQL | +|--------|----------------|----------| +| **Reading order** | Inside-out (subqueries first) | Top-down, left-to-right | +| **Clause order** | Fixed: SELECT...FROM...WHERE...GROUP BY...ORDER BY | Any order, any number of times | +| **Subquery nesting** | Required for multi-step aggregations | Eliminated via chaining | +| **Column addition** | SELECT *, expr AS new_col | `EXTEND expr AS new_col` | +| **Column removal** | SELECT with explicit column list or EXCEPT | `DROP col1, col2` | +| **Column modification** | SELECT with expression replacing column | `SET col = new_expr` | +| **Aggregation** | SELECT agg() ... GROUP BY | `AGGREGATE agg() GROUP BY` | +| **Composability** | Limited; requires CTEs or subqueries | Native chaining | +| **Interoperability** | Standard | Fully interoperable with traditional SQL | + +### When to Use Pipe Syntax + +**Use pipe syntax when:** +- Multi-step aggregations would require nested subqueries +- You want DataFrame-like readability in SQL +- Building exploratory or iterative queries (easy to add/remove steps) +- Complex transformations with many joins, filters, and projections + +**Use traditional SQL when:** +- Simple queries that are already readable +- Team is more familiar with standard SQL +- Queries will be shared with tools that may not support pipe syntax + +### Performance Considerations + +- Pipe syntax is **syntactic sugar** -- it compiles to the same execution plan as traditional SQL +- No performance difference between pipe and traditional syntax for equivalent queries +- Best practice: Place data-reducing operations (`WHERE`, `DROP`, `SELECT`) early in the pipeline to minimize data flowing through subsequent operations +- Use `TABLESAMPLE` during development to work with smaller datasets diff --git a/.claude/skills/databricks-dbsql/sql-scripting.md b/.claude/skills/databricks-dbsql/sql-scripting.md new file mode 100644 index 00000000..549a4270 --- /dev/null +++ b/.claude/skills/databricks-dbsql/sql-scripting.md @@ -0,0 +1,1077 @@ +# SQL Scripting, Stored Procedures, Recursive CTEs, and Transactions + +> Databricks SQL procedural extensions based on the SQL/PSM standard. Covers SQL scripting (compound statements, control flow, exception handling), stored procedures, recursive CTEs, and multi-statement transactions. + +--- + +## Table of Contents + +- [SQL Scripting](#sql-scripting) + - [Compound Statements (BEGIN...END)](#compound-statements-beginend) + - [Variable Declaration (DECLARE)](#variable-declaration-declare) + - [Variable Assignment (SET)](#variable-assignment-set) + - [Control Flow](#control-flow) + - [IF / ELSEIF / ELSE](#if--elseif--else) + - [CASE Statement](#case-statement) + - [WHILE Loop](#while-loop) + - [FOR Loop](#for-loop) + - [LOOP Statement](#loop-statement) + - [REPEAT Statement](#repeat-statement) + - [LEAVE and ITERATE](#leave-and-iterate) + - [Exception Handling](#exception-handling) + - [Condition Declaration](#condition-declaration) + - [Handler Declaration](#handler-declaration) + - [SIGNAL and RESIGNAL](#signal-and-resignal) + - [EXECUTE IMMEDIATE (Dynamic SQL)](#execute-immediate-dynamic-sql) +- [Stored Procedures](#stored-procedures) + - [CREATE PROCEDURE](#create-procedure) + - [CALL (Invoke a Procedure)](#call-invoke-a-procedure) + - [DROP PROCEDURE](#drop-procedure) + - [DESCRIBE PROCEDURE](#describe-procedure) + - [SHOW PROCEDURES](#show-procedures) +- [Recursive CTEs](#recursive-ctes) + - [WITH RECURSIVE Syntax](#with-recursive-syntax) + - [Anchor and Recursive Members](#anchor-and-recursive-members) + - [MAX RECURSION LEVEL](#max-recursion-level) + - [Use Cases and Examples](#use-cases-and-examples) + - [Limitations](#limitations) +- [Multi-Statement Transactions](#multi-statement-transactions) + - [Overview and Current Status](#overview-and-current-status) + - [SQL Scripting Atomic Blocks](#sql-scripting-atomic-blocks) + - [Python Connector Transaction API](#python-connector-transaction-api) + - [Isolation Levels](#isolation-levels) + - [Write Conflicts and Concurrency](#write-conflicts-and-concurrency) + - [Best Practices](#best-practices) + +--- + +## SQL Scripting + +**Availability**: Databricks Runtime 16.3+ and Databricks SQL + +SQL scripting enables procedural logic using the SQL/PSM standard. Every SQL script starts with a compound statement block (`BEGIN...END`). + +### Compound Statements (BEGIN...END) + +A compound statement is the fundamental building block containing variable declarations, condition/handler declarations, and executable statements. + +**Syntax**: + +```sql +[ label : ] BEGIN + [ { declare_variable | declare_condition } ; [...] ] + [ declare_handler ; [...] ] + [ SQL_statement ; [...] ] +END [ label ] +``` + +**Key rules**: + +- Declarations must appear before executable statements +- Variable declarations come before condition declarations, which come before handler declarations +- Top-level compound statements cannot specify labels +- `NOT ATOMIC` is the default and only behavior (no automatic rollback on failure) +- In notebooks, the compound statement must be the sole statement in the cell + +**Supported statement types in body**: + +| Category | Statements | +|----------|-----------| +| DDL | ALTER, CREATE, DROP | +| DCL | GRANT, REVOKE | +| DML | INSERT, UPDATE, DELETE, MERGE | +| Query | SELECT | +| Assignment | SET | +| Dynamic SQL | EXECUTE IMMEDIATE | +| Control flow | IF, CASE, WHILE, FOR, LOOP, REPEAT, LEAVE, ITERATE | +| Nesting | Nested BEGIN...END blocks | + +**Minimal example**: + +```sql +BEGIN + SELECT 'Hello, SQL Scripting!'; +END; +``` + +### Variable Declaration (DECLARE) + +**Syntax**: + +```sql +DECLARE variable_name [, ...] data_type [ DEFAULT default_expr ]; +``` + +- Variables initialize to `NULL` if no `DEFAULT` is specified +- Data type can be omitted when `DEFAULT` is provided (type inferred from expression) +- Multiple variable names in a single `DECLARE` supported in Runtime 17.2+ +- Variables are scoped to their enclosing compound statement +- Variable names resolve from the innermost scope outward; use labels to disambiguate + +**Examples**: + +```sql +BEGIN + DECLARE counter INT DEFAULT 0; + DECLARE name STRING DEFAULT 'unknown'; + DECLARE x, y, z DOUBLE DEFAULT 0.0; -- Runtime 17.2+ + DECLARE inferred DEFAULT current_date(); -- type inferred as DATE + + SET counter = counter + 1; + VALUES (counter, name); +END; +``` + +### Variable Assignment (SET) + +**Syntax**: + +```sql +SET variable_name = expression; +SET VAR variable_name = expression; -- explicit local variable +SET (var1, var2, ...) = (expr1, expr2, ...); -- multi-assignment +``` + +Use `SET VAR` to explicitly target a local variable when a session variable with the same name exists. + +**Example**: + +```sql +BEGIN + DECLARE total INT DEFAULT 0; + DECLARE label STRING; + SET total = 100; + SET label = 'final'; + VALUES (total, label); +END; +``` + +### Control Flow + +#### IF / ELSEIF / ELSE + +Executes statements based on the first condition evaluating to `TRUE`. + +**Syntax**: + +```sql +IF condition THEN + { stmt ; } [...] +[ ELSEIF condition THEN + { stmt ; } [...] ] [...] +[ ELSE + { stmt ; } [...] ] +END IF; +``` + +**Example**: + +```sql +BEGIN + DECLARE score INT DEFAULT 85; + DECLARE grade STRING; + + IF score >= 90 THEN + SET grade = 'A'; + ELSEIF score >= 80 THEN + SET grade = 'B'; + ELSEIF score >= 70 THEN + SET grade = 'C'; + ELSE + SET grade = 'F'; + END IF; + + VALUES (grade); -- Returns 'B' +END; +``` + +#### CASE Statement + +Two forms: **simple CASE** (compare expression) and **searched CASE** (evaluate boolean conditions). + +**Simple CASE syntax**: + +```sql +CASE expr + WHEN opt1 THEN { stmt ; } [...] + WHEN opt2 THEN { stmt ; } [...] + [ ELSE { stmt ; } [...] ] +END CASE; +``` + +**Searched CASE syntax**: + +```sql +CASE + WHEN cond1 THEN { stmt ; } [...] + WHEN cond2 THEN { stmt ; } [...] + [ ELSE { stmt ; } [...] ] +END CASE; +``` + +Only the first matching branch executes. + +**Example**: + +```sql +BEGIN + DECLARE status STRING DEFAULT 'active'; + + CASE status + WHEN 'active' THEN VALUES ('Processing'); + WHEN 'paused' THEN VALUES ('On hold'); + WHEN 'archived' THEN VALUES ('Read-only'); + ELSE VALUES ('Unknown status'); + END CASE; +END; +``` + +#### WHILE Loop + +Repeats while a condition is `TRUE`. + +**Syntax**: + +```sql +[ label : ] WHILE condition DO + { stmt ; } [...] +END WHILE [ label ]; +``` + +**Example** -- sum odd numbers from 1 to 10: + +```sql +BEGIN + DECLARE total INT DEFAULT 0; + DECLARE i INT DEFAULT 0; + + sum_odds: WHILE i < 10 DO + SET i = i + 1; + IF i % 2 = 0 THEN + ITERATE sum_odds; -- skip even numbers + END IF; + SET total = total + i; + END WHILE sum_odds; + + VALUES (total); -- Returns 25 +END; +``` + +#### FOR Loop + +Iterates over query result rows. + +**Syntax**: + +```sql +[ label : ] FOR [ variable_name AS ] query DO + { stmt ; } [...] +END FOR [ label ]; +``` + +- Use `variable_name` (not the label) to qualify column references from the cursor +- For Delta tables, modifying the source during iteration does not affect cursor results +- Loop may not fully execute the query if terminated early by `LEAVE` or an error + +**Example** -- process each row from a query: + +```sql +BEGIN + DECLARE total_revenue DOUBLE DEFAULT 0.0; + + process_orders: FOR row AS + SELECT order_id, amount FROM orders WHERE status = 'completed' + DO + SET total_revenue = total_revenue + row.amount; + IF total_revenue > 1000000 THEN + LEAVE process_orders; + END IF; + END FOR process_orders; + + VALUES (total_revenue); +END; +``` + +#### LOOP Statement + +Unconditional loop; must use `LEAVE` to exit. + +**Syntax**: + +```sql +[ label : ] LOOP + { stmt ; } [...] +END LOOP [ label ]; +``` + +**Example**: + +```sql +BEGIN + DECLARE counter INT DEFAULT 0; + + count_up: LOOP + SET counter = counter + 1; + IF counter >= 5 THEN + LEAVE count_up; + END IF; + END LOOP count_up; + + VALUES (counter); -- Returns 5 +END; +``` + +#### REPEAT Statement + +Executes at least once, then repeats until condition is `TRUE`. + +**Syntax**: + +```sql +[ label : ] REPEAT + { stmt ; } [...] + UNTIL condition +END REPEAT [ label ]; +``` + +**Example**: + +```sql +BEGIN + DECLARE total INT DEFAULT 0; + DECLARE i INT DEFAULT 0; + + sum_loop: REPEAT + SET i = i + 1; + IF i % 2 != 0 THEN + SET total = total + i; + END IF; + UNTIL i >= 10 + END REPEAT sum_loop; + + VALUES (total); -- Returns 25 +END; +``` + +#### LEAVE and ITERATE + +| Statement | Purpose | Equivalent | +|-----------|---------|-----------| +| `LEAVE label` | Exit the labeled loop or compound block | `BREAK` in other languages | +| `ITERATE label` | Skip to the next iteration of the labeled loop | `CONTINUE` in other languages | + +Both require a labeled loop to target. + +### Exception Handling + +#### Condition Declaration + +Define named conditions for specific SQLSTATE codes. + +**Syntax**: + +```sql +DECLARE condition_name CONDITION [ FOR SQLSTATE [ VALUE ] sqlstate ]; +``` + +- `sqlstate` is a 5-character alphanumeric string (A-Z, 0-9, case-insensitive) +- Cannot start with `'00'`, `'01'`, or `'XX'` +- Defaults to `'45000'` if not specified + +**Example**: + +```sql +BEGIN + DECLARE divide_by_zero CONDITION FOR SQLSTATE '22012'; + -- Use in handler declarations below +END; +``` + +#### Handler Declaration + +Catch and handle exceptions within compound statements. + +**Syntax**: + +```sql +DECLARE handler_type HANDLER FOR condition_value [, ...] handler_action; +``` + +| Parameter | Options | Description | +|-----------|---------|-------------| +| `handler_type` | `EXIT` | Exits the enclosing compound after handling | +| `condition_value` | `SQLSTATE 'xxxxx'`, `condition_name`, `SQLEXCEPTION`, `NOT FOUND` | What to catch | +| `handler_action` | Single statement or nested `BEGIN...END` | What to execute | + +- `SQLEXCEPTION` catches all error states (SQLSTATE class not `'00'` or `'01'`) +- `NOT FOUND` catches `'02xxx'` states (no data found) +- A handler cannot apply to statements in its own body + +**Example** -- catch division by zero: + +```sql +BEGIN + DECLARE result DOUBLE; + DECLARE EXIT HANDLER FOR SQLSTATE '22012' + BEGIN + SET result = -1; + END; + + SET result = 10 / 0; -- triggers handler + VALUES (result); -- Returns -1 +END; +``` + +**Example** -- generic exception handler: + +```sql +BEGIN + DECLARE error_msg STRING DEFAULT 'none'; + + DECLARE EXIT HANDLER FOR SQLEXCEPTION + BEGIN + SET error_msg = 'An error occurred'; + INSERT INTO error_log (message, ts) VALUES (error_msg, current_timestamp()); + END; + + -- statements that might fail + INSERT INTO target_table SELECT * FROM source_table; +END; +``` + +#### SIGNAL and RESIGNAL + +Raise or re-raise exceptions. + +**SIGNAL syntax**: + +```sql +SIGNAL condition_name + [ SET { MESSAGE_ARGUMENTS = argument_map | MESSAGE_TEXT = message_str } ]; + +SIGNAL SQLSTATE [ VALUE ] sqlstate + [ SET MESSAGE_TEXT = message_str ]; +``` + +**RESIGNAL syntax** (use in handlers to preserve diagnostic stack): + +```sql +RESIGNAL [ condition_name | SQLSTATE [ VALUE ] sqlstate ] + [ SET { MESSAGE_ARGUMENTS = argument_map | MESSAGE_TEXT = message_str } ]; +``` + +- Prefer `RESIGNAL` over `SIGNAL` inside handlers -- `RESIGNAL` preserves the diagnostic stack while `SIGNAL` clears it +- `MESSAGE_ARGUMENTS` takes a `MAP` literal + +**Example** -- validate input and raise custom error: + +```sql +BEGIN + DECLARE input_value INT DEFAULT 150; + + IF input_value > 100 THEN + SIGNAL SQLSTATE '45000' + SET MESSAGE_TEXT = 'Input value must be <= 100'; + END IF; + + VALUES (input_value); +END; +``` + +**Example** -- using named conditions with MESSAGE_ARGUMENTS: + +```sql +BEGIN + DECLARE input INT DEFAULT 5; + DECLARE arg_map MAP; + + IF input > 4 THEN + SET arg_map = map('errorMessage', 'Input must be <= 4.'); + SIGNAL USER_RAISED_EXCEPTION + SET MESSAGE_ARGUMENTS = arg_map; + END IF; +END; +``` + +### EXECUTE IMMEDIATE (Dynamic SQL) + +Execute SQL statements constructed as strings at runtime. + +**Availability**: Runtime 14.3+; expression-based `sql_string` and nested execution from Runtime 17.3+. + +**Syntax**: + +```sql +EXECUTE IMMEDIATE sql_string + [ INTO var_name [, ...] ] + [ USING { arg_expr [ AS ] [ alias ] } [, ...] ]; +``` + +- `sql_string`: a constant expression producing a well-formed SQL statement +- `INTO`: captures a single-row result into variables (returns `NULL` for zero rows; errors for multiple rows) +- `USING`: binds values to positional (`?`) or named (`:param`) parameter markers (cannot mix styles) + +**Examples**: + +```sql +-- Positional parameters +EXECUTE IMMEDIATE 'SELECT SUM(c1) FROM VALUES(?), (?) AS t(c1)' USING 5, 6; + +-- Named parameters with INTO +BEGIN + DECLARE total INT; + EXECUTE IMMEDIATE 'SELECT SUM(c1) FROM VALUES(:a), (:b) AS t(c1)' + INTO total USING (5 AS a, 6 AS b); + VALUES (total); -- Returns 11 +END; + +-- Dynamic table operations +BEGIN + DECLARE table_name STRING DEFAULT 'my_catalog.my_schema.staging'; + EXECUTE IMMEDIATE 'TRUNCATE TABLE ' || table_name; + EXECUTE IMMEDIATE 'INSERT INTO ' || table_name || ' SELECT * FROM source'; +END; +``` + +--- + +## Stored Procedures + +**Availability**: Public Preview -- Databricks Runtime 17.0+ + +Stored procedures persist SQL scripts in Unity Catalog and are invoked with `CALL`. + +### CREATE PROCEDURE + +**Syntax**: + +```sql +CREATE [ OR REPLACE ] PROCEDURE [ IF NOT EXISTS ] + procedure_name ( [ parameter [, ...] ] ) + characteristic [...] + AS compound_statement +``` + +**Parameter definition**: + +```sql +[ IN | OUT | INOUT ] parameter_name data_type + [ DEFAULT default_expression ] + [ COMMENT parameter_comment ] +``` + +| Parameter mode | Behavior | +|---------------|----------| +| `IN` (default) | Input-only; value passed into the procedure | +| `OUT` | Output-only; initialized to `NULL`; final value returned on success | +| `INOUT` | Input and output; accepts a value and returns the modified value on success | + +**Required characteristics**: + +| Characteristic | Description | +|---------------|-------------| +| `LANGUAGE SQL` | Specifies the implementation language | +| `SQL SECURITY INVOKER` | Executes under the invoker's authority | + +**Optional characteristics**: + +| Characteristic | Description | +|---------------|-------------| +| `NOT DETERMINISTIC` | Procedure may return different results with identical inputs | +| `MODIFIES SQL DATA` | Procedure modifies SQL data | +| `COMMENT 'description'` | Human-readable description | +| `DEFAULT COLLATION UTF8_BINARY` | Required when schema uses non-UTF8_BINARY collation (Runtime 17.1+) | + +**Rules**: + +- `OR REPLACE` and `IF NOT EXISTS` cannot be combined +- Parameter names must be unique within the procedure +- `DEFAULT` is not supported for `OUT` parameters +- Once a parameter has a `DEFAULT`, all subsequent parameters must also have defaults +- Default expressions cannot reference other parameters or contain subqueries +- Body is validated syntactically at creation but semantically only at invocation + +**Example** -- ETL procedure with output parameters: + +```sql +CREATE OR REPLACE PROCEDURE run_daily_etl( + IN source_schema STRING, + IN target_schema STRING, + OUT rows_processed INT, + OUT status STRING DEFAULT 'pending' +) +LANGUAGE SQL +SQL SECURITY INVOKER +COMMENT 'Daily ETL pipeline for order processing' +AS BEGIN + DECLARE EXIT HANDLER FOR SQLEXCEPTION + BEGIN + SET status = 'failed'; + SET rows_processed = 0; + END; + + -- Truncate and reload + EXECUTE IMMEDIATE 'TRUNCATE TABLE ' || target_schema || '.orders_daily'; + + EXECUTE IMMEDIATE + 'INSERT INTO ' || target_schema || '.orders_daily ' + || 'SELECT * FROM ' || source_schema || '.orders ' + || 'WHERE order_date = current_date()'; + + EXECUTE IMMEDIATE + 'SELECT COUNT(*) FROM ' || target_schema || '.orders_daily' + INTO rows_processed; + + SET status = 'success'; +END; +``` + +### CALL (Invoke a Procedure) + +**Syntax**: + +```sql +CALL procedure_name( [ argument [, ...] ] ); +CALL procedure_name( [ named_param => argument ] [, ...] ); +``` + +**Rules**: + +- Supports up to 64 levels of nesting +- For `IN` parameters: any expression castable to the parameter type, or `DEFAULT` +- For `OUT`/`INOUT` parameters: must be a session variable or local variable +- Arguments must match the data type of the parameter (use typed literals, e.g., `DATE'2025-01-01'`) +- Fewer arguments allowed if remaining parameters have `DEFAULT` values +- Not supported via ODBC + +**Example**: + +```sql +-- Positional invocation +DECLARE rows_out INT; +DECLARE status_out STRING; +CALL run_daily_etl('raw', 'silver', rows_out, status_out); +SELECT rows_out, status_out; + +-- Named parameter invocation +CALL run_daily_etl( + target_schema => 'silver', + source_schema => 'raw', + rows_processed => rows_out, + status => status_out +); +``` + +### DROP PROCEDURE + +**Syntax**: + +```sql +DROP PROCEDURE [ IF EXISTS ] procedure_name; +``` + +- Without `IF EXISTS`, dropping a non-existent procedure raises `ROUTINE_NOT_FOUND` +- Requires `MANAGE` privilege, ownership of the procedure, or ownership of the containing schema/catalog/metastore + +**Example**: + +```sql +DROP PROCEDURE IF EXISTS run_daily_etl; +``` + +### DESCRIBE PROCEDURE + +**Syntax**: + +```sql +{ DESC | DESCRIBE } PROCEDURE [ EXTENDED ] procedure_name; +``` + +- Basic: returns procedure name and parameter list +- `EXTENDED`: additionally returns owner, creation time, body, language, security type, determinism, data access, and configuration + +**Example**: + +```sql +DESCRIBE PROCEDURE EXTENDED run_daily_etl; +``` + +### SHOW PROCEDURES + +**Syntax**: + +```sql +SHOW PROCEDURES [ { FROM | IN } schema_name ]; +``` + +Returns columns: `catalog`, `namespace`, `schema`, `procedure_name`. + +**Example**: + +```sql +SHOW PROCEDURES IN my_catalog.my_schema; +``` + +--- + +## Recursive CTEs + +**Availability**: Databricks Runtime 17.0+ and DBSQL 2025.20+ + +Recursive CTEs enable self-referential queries for hierarchical data, graph traversal, and series generation. + +### WITH RECURSIVE Syntax + +```sql +WITH RECURSIVE cte_name [ ( column_name [, ...] ) ] + [ MAX RECURSION LEVEL max_level ] AS ( + base_case_query + UNION ALL + recursive_query + ) +SELECT ... FROM cte_name; +``` + +### Anchor and Recursive Members + +| Component | Description | +|-----------|-------------| +| **Anchor (base case)** | Initial query providing seed rows; must NOT reference the CTE name | +| **Recursive member** | References the CTE name; processes rows from the previous iteration | +| **UNION ALL** | Combines anchor and recursive results (required) | + +The recursive member reads rows produced by the previous iteration and generates new rows. Recursion terminates when the recursive member produces zero rows. + +### MAX RECURSION LEVEL + +```sql +WITH RECURSIVE cte_name MAX RECURSION LEVEL 200 AS (...) +``` + +| Setting | Default | Description | +|---------|---------|-------------| +| Max recursion depth | 100 | Exceeding raises `RECURSION_LEVEL_LIMIT_EXCEEDED` | +| Max result rows | 1,000,000 | Exceeding raises an error | +| `LIMIT ALL` | N/A | Suspends the row limit (Runtime 17.2+) | + +### Use Cases and Examples + +**Generate a number series**: + +```sql +WITH RECURSIVE numbers(n) AS ( + VALUES (1) + UNION ALL + SELECT n + 1 FROM numbers WHERE n < 100 +) +SELECT * FROM numbers; +``` + +**Organizational hierarchy traversal**: + +```sql +WITH RECURSIVE org_tree AS ( + -- Anchor: start from the CEO + SELECT employee_id, name, manager_id, name AS root_name, 0 AS depth + FROM employees + WHERE manager_id IS NULL + + UNION ALL + + -- Recursive: find direct reports + SELECT e.employee_id, e.name, e.manager_id, t.root_name, t.depth + 1 + FROM employees e + JOIN org_tree t ON e.manager_id = t.employee_id +) +SELECT * FROM org_tree ORDER BY depth, name; +``` + +**Graph traversal with cycle detection**: + +```sql +WITH RECURSIVE search_graph(f, t, label, path, cycle) AS ( + -- Anchor: all edges as starting paths + SELECT *, array(struct(g.f, g.t)), false + FROM graph g + + UNION ALL + + -- Recursive: extend paths, detect cycles + SELECT g.f, g.t, g.label, + sg.path || array(struct(g.f, g.t)), + array_contains(sg.path, struct(g.f, g.t)) + FROM graph g + JOIN search_graph sg ON g.f = sg.t + WHERE NOT sg.cycle +) +SELECT * FROM search_graph WHERE NOT cycle; +``` + +**String accumulation**: + +```sql +WITH RECURSIVE r(col) AS ( + SELECT 'a' + UNION ALL + SELECT col || char(ascii(substr(col, -1)) + 1) + FROM r + WHERE length(col) < 10 +) +SELECT * FROM r; +-- a, ab, abc, abcd, ..., abcdefghij +``` + +**Bill of Materials (BOM) explosion**: + +```sql +WITH RECURSIVE bom AS ( + -- Anchor: top-level product + SELECT part_id, component_id, quantity, 1 AS level + FROM bill_of_materials + WHERE part_id = 'PROD-001' + + UNION ALL + + -- Recursive: sub-components + SELECT b.part_id, b.component_id, b.quantity * bom.quantity, bom.level + 1 + FROM bill_of_materials b + JOIN bom ON b.part_id = bom.component_id +) +SELECT component_id, SUM(quantity) AS total_quantity, MAX(level) AS max_depth +FROM bom +GROUP BY component_id +ORDER BY total_quantity DESC; +``` + +### Limitations + +- Not supported in UPDATE, DELETE, or MERGE statements +- Step (recursive) queries cannot include correlated column references to the CTE name +- Random number generators may produce identical values across iterations +- Default row limit of 1,000,000 rows (use `LIMIT ALL` in Runtime 17.2+ to override) +- Default recursion depth of 100 (override with `MAX RECURSION LEVEL`) + +--- + +## Multi-Statement Transactions + +### Overview and Current Status + +Multi-statement transactions (MST) allow grouping multiple SQL statements into atomic units that either succeed completely or fail completely. + +| Feature | Status | Notes | +|---------|--------|-------| +| Single-table transactions | GA | Delta Lake default; every DML statement is atomic | +| Multi-statement transactions (SQL scripting) | Preview | `BEGIN ATOMIC...END` blocks | +| Multi-statement transactions (Python connector) | Preview | `connection.autocommit = False` pattern | +| Cross-table transactions | Preview | Atomic updates across multiple Delta tables | + +### SQL Scripting Atomic Blocks + +Use `BEGIN ATOMIC...END` to execute multiple statements as a single atomic unit: + +```sql +BEGIN ATOMIC + INSERT INTO customers (id, name) VALUES (1, 'Alice'); + INSERT INTO orders (id, customer_id, amount) VALUES (1, 1, 250.00); + INSERT INTO audit_log (action, ts) VALUES ('new_customer_order', current_timestamp()); +END; +``` + +If any statement fails, all changes are rolled back. + +> **Note:** Tables used in `BEGIN ATOMIC` blocks must have the `catalogManaged` table feature enabled. Create tables with `TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported')`. Existing tables cannot be upgraded in place — they must be recreated with this property. + +### Python Connector Transaction API + +The Databricks SQL Connector for Python provides explicit transaction control: + +```python +from databricks import sql + +connection = sql.connect( + server_hostname="...", + http_path="...", + access_token="..." +) + +# Disable autocommit to start explicit transactions +connection.autocommit = False +cursor = connection.cursor() + +try: + cursor.execute("INSERT INTO customers VALUES (1, 'Alice')") + cursor.execute("INSERT INTO orders VALUES (1, 1, 100.00)") + cursor.execute("INSERT INTO shipments VALUES (1, 1, 'pending')") + connection.commit() # All three succeed atomically +except Exception: + connection.rollback() # All three discarded +finally: + connection.autocommit = True +``` + +**Key API methods**: + +| Method | Description | +|--------|-------------| +| `connection.autocommit = False` | Start explicit transaction mode | +| `connection.commit()` | Commit the current transaction | +| `connection.rollback()` | Discard all changes in the current transaction | +| `connection.get_transaction_isolation()` | Returns current isolation level | +| `connection.set_transaction_isolation(level)` | Sets isolation level | + +**Error handling**: + +- `sql.TransactionError` raised when committing without an active transaction +- Cannot change `autocommit` while a transaction is active +- `rollback()` is a safe no-op when no transaction is active + +### Isolation Levels + +Databricks uses **Snapshot Isolation** (mapped to `REPEATABLE_READ` in standard SQL terminology). + +| Level | Description | Default | +|-------|-------------|---------| +| `WriteSerializable` | Only writes are serializable; concurrent writes may reorder | Yes (table default) | +| `Serializable` | Both reads and writes are serializable; strictest isolation | No | +| `REPEATABLE_READ` | Snapshot isolation for connector-level transactions | Connector default | + +**Setting isolation at table level**: + +```sql +ALTER TABLE my_table +SET TBLPROPERTIES ('delta.isolationLevel' = 'Serializable'); +``` + +**Setting isolation in Python connector**: + +```python +from databricks.sql import TRANSACTION_ISOLATION_LEVEL_REPEATABLE_READ + +connection.set_transaction_isolation(TRANSACTION_ISOLATION_LEVEL_REPEATABLE_READ) +# Only REPEATABLE_READ is supported; others raise NotSupportedError +``` + +**Snapshot isolation behavior**: + +- **Repeatable reads**: Data read within a transaction remains consistent +- **Atomic commits**: Changes are invisible to other connections until committed +- **Write conflicts**: Concurrent writes to the same table cause conflicts +- **Cross-table writes**: Concurrent writes to different tables can succeed + +### Write Conflicts and Concurrency + +**Row-level concurrency** (Runtime 14.2+) reduces conflicts for tables with deletion vectors or liquid clustering: + +| Operation | WriteSerializable | Serializable | +|-----------|------------------|--------------| +| INSERT vs INSERT | No conflict | No conflict | +| UPDATE/DELETE/MERGE vs same | No conflict (different rows) | May conflict | +| OPTIMIZE vs concurrent DML | Conflict only with ZORDER BY | May conflict | + +**Common conflict exceptions**: + +| Exception | Cause | +|-----------|-------| +| `ConcurrentAppendException` | Concurrent append to the same partition | +| `ConcurrentDeleteReadException` | Concurrent delete of files being read | +| `MetadataChangedException` | Concurrent ALTER TABLE or schema change | +| `ProtocolChangedException` | Protocol version upgrade during write | + +### Best Practices + +1. **Keep transactions short** to minimize conflict windows +2. **Always wrap in try/except/finally** with rollback on errors +3. **Restore autocommit** in the `finally` block +4. **Use partition pruning** in MERGE conditions to reduce conflict scope +5. **Enable row-level concurrency** (deletion vectors + liquid clustering) for high-concurrency workloads +6. **Prefer single-statement MERGE** over multi-statement transactions when updating a single table +7. **Commit and restart** transactions to see changes made by other connections + +--- + +## Runtime Version Reference + +| Feature | Minimum Runtime | Status | +|---------|----------------|--------| +| SQL Scripting (compound statements, control flow) | 16.3 | GA | +| Stored Procedures (CREATE/CALL/DROP PROCEDURE) | 17.0 | Public Preview | +| Recursive CTEs (WITH RECURSIVE) | 17.0 / DBSQL 2025.20 | GA | +| Multi-variable DECLARE | 17.2 | GA | +| EXECUTE IMMEDIATE (basic) | 14.3 | GA | +| EXECUTE IMMEDIATE (expressions, nested) | 17.3 | GA | +| Recursive CTE LIMIT ALL | 17.2 | GA | +| Multi-statement Transactions | Varies | Preview | +| Row-level Concurrency | 14.2 | GA | + +--- + +## Quick Reference Card + +### SQL Scripting Skeleton + +```sql +BEGIN + -- 1. Declarations + DECLARE var1 INT DEFAULT 0; + DECLARE var2 STRING; + DECLARE my_error CONDITION FOR SQLSTATE '45000'; + DECLARE EXIT HANDLER FOR SQLEXCEPTION + BEGIN + -- error handling logic + END; + + -- 2. Logic + IF var1 > 0 THEN + SET var2 = 'positive'; + ELSE + SET var2 = 'non-positive'; + END IF; + + -- 3. Output + VALUES (var1, var2); +END; +``` + +### Stored Procedure Skeleton + +```sql +CREATE OR REPLACE PROCEDURE my_schema.my_proc( + IN input_param STRING, + OUT output_param INT +) +LANGUAGE SQL +SQL SECURITY INVOKER +COMMENT 'Description of what this procedure does' +AS BEGIN + DECLARE EXIT HANDLER FOR SQLEXCEPTION + SET output_param = -1; + + -- procedure body + SET output_param = (SELECT COUNT(*) FROM my_table WHERE col = input_param); +END; + +-- Invoke +DECLARE result INT; +CALL my_schema.my_proc('value', result); +SELECT result; +``` + +### Recursive CTE Skeleton + +```sql +WITH RECURSIVE cte_name (col1, col2) MAX RECURSION LEVEL 50 AS ( + -- Anchor + SELECT seed_col1, seed_col2 + FROM base_table + WHERE condition + + UNION ALL + + -- Recursive step + SELECT derived_col1, derived_col2 + FROM source_table s + JOIN cte_name c ON s.parent = c.col1 +) +SELECT * FROM cte_name; +``` diff --git a/.claude/skills/databricks-docs/SKILL.md b/.claude/skills/databricks-docs/SKILL.md index 98aeac9b..ceca11e0 100644 --- a/.claude/skills/databricks-docs/SKILL.md +++ b/.claude/skills/databricks-docs/SKILL.md @@ -1,6 +1,6 @@ --- name: databricks-docs -description: "Databricks documentation reference. Use as a lookup resource alongside other skills and MCP tools for comprehensive guidance." +description: "Databricks documentation reference via llms.txt index. Use when other skills do not cover a topic, looking up unfamiliar Databricks features, or needing authoritative docs on APIs, configurations, or platform capabilities." --- # Databricks Documentation Reference @@ -16,7 +16,7 @@ This is a **reference skill**, not an action skill. Use it to: - Find detailed information to inform how you use MCP tools - Discover features and capabilities you may not know about -**Always prefer using MCP tools for actions** (execute_sql, create_or_update_pipeline, etc.) and **load specific skills for workflows** (databricks-python-sdk, spark-declarative-pipelines, etc.). Use this skill when you need reference documentation. +**Always prefer using MCP tools for actions** (execute_sql, manage_pipeline, etc.) and **load specific skills for workflows** (databricks-python-sdk, databricks-spark-declarative-pipelines, etc.). Use this skill when you need reference documentation. ## How to Use @@ -45,12 +45,20 @@ The llms.txt file is organized by category: **Scenario:** User wants to create a Delta Live Tables pipeline -1. Load `spark-declarative-pipelines` skill for workflow patterns +1. Load `databricks-spark-declarative-pipelines` skill for workflow patterns 2. Use this skill to fetch docs if you need clarification on specific DLT features -3. Use `create_or_update_pipeline` MCP tool to actually create the pipeline +3. Use `manage_pipeline(action="create_or_update")` MCP tool to actually create the pipeline **Scenario:** User asks about an unfamiliar Databricks feature 1. Fetch llms.txt to find relevant documentation 2. Read the specific docs to understand the feature 3. Determine which skill/tools apply, then use them + +## Related Skills + +- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** - SDK patterns for programmatic Databricks access +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - DLT / Lakeflow pipeline workflows +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - Governance and catalog management +- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** - Serving endpoints and model deployment +- **[databricks-mlflow-evaluation](../databricks-mlflow-evaluation/SKILL.md)** - MLflow 3 GenAI evaluation workflows diff --git a/.claude/skills/databricks-execution-compute/SKILL.md b/.claude/skills/databricks-execution-compute/SKILL.md new file mode 100644 index 00000000..c3518385 --- /dev/null +++ b/.claude/skills/databricks-execution-compute/SKILL.md @@ -0,0 +1,82 @@ +--- +name: databricks-execution-compute +description: >- + Execute code and manage compute on Databricks. Use this skill when the user + mentions: "run code", "execute", "run on databricks", "serverless", "no + cluster", "run python", "run scala", "run sql", "run R", "run file", "push + and run", "notebook run", "batch script", "model training", "run script on + cluster", "create cluster", "new cluster", "resize cluster", "modify cluster", + "delete cluster", "terminate cluster", "create warehouse", "new warehouse", + "resize warehouse", "delete warehouse", "node types", "runtime versions", + "DBR versions", "spin up compute", "provision cluster". +--- + +# Databricks Execution & Compute + +Run code on Databricks. Three execution modes—choose based on workload. + +## Execution Mode Decision Matrix + +| Aspect | [Databricks Connect](references/1-databricks-connect.md) ⭐ | [Serverless Job](references/2-serverless-job.md) | [Interactive Cluster](references/3-interactive-cluster.md) | +|--------|-------------------|----------------|---------------------| +| **Use for** | Spark code (ETL, data gen) | Heavy processing (ML) | State across tool calls, Scala/R | +| **Startup** | Instant | ~25-50s cold start | ~5min if stopped | +| **State** | Within Python process | None | Via context_id | +| **Languages** | Python (PySpark) | Python, SQL | Python, Scala, SQL, R | +| **Dependencies** | `withDependencies()` | CLI with environments spec | Install on cluster | + +### Decision Flow + +``` +Spark-based code? → Databricks Connect (fastest) + └─ Python 3.12 missing? → Install it + databricks-connect + └─ Install fails? → Ask user (don't auto-switch modes) + +Heavy/long-running (ML)? → Serverless Job (independent) +Need state across calls? → Interactive Cluster (list and ask which one to use) +Scala/R? → Interactive Cluster (list and ask which one to use) +``` + + +## How to Run Code + +**Read the reference file for your chosen mode before proceeding.** + +### Databricks Connect (no MCP tool, run locally) → [reference](references/1-databricks-connect.md) + +```bash +python my_spark_script.py +``` + +### Serverless Job → [reference](references/2-serverless-job.md) + +```python +execute_code(file_path="/path/to/script.py") +``` + +### Interactive Cluster → [reference](references/3-interactive-cluster.md) + +```python +# Check for running clusters first (or use the one instructed) +list_compute(resource="clusters") +# Ask the customer which one to use + +# Run code, reuse context_id for follow-up MCP call +result = execute_code(code="...", compute_type="cluster", cluster_id="...") +execute_code(code="...", context_id=result["context_id"], cluster_id=result["cluster_id"]) +``` + +## MCP Tools + +| Tool | For | Purpose | +|------|-----|---------| +| `execute_code` | Serverless, Interactive | Run code remotely | +| `list_compute` | Interactive | List clusters, check status, auto-select running cluster | +| `manage_cluster` | Interactive | Create, start, terminate, delete. **COSTLY:** `start` takes 3-8 min—ask user | +| `manage_sql_warehouse` | SQL | Create, modify, delete SQL warehouses | + +## Related Skills + +- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** — Data generation using Spark + Faker +- **[databricks-jobs](../databricks-jobs/SKILL.md)** — Production job orchestration +- **[databricks-dbsql](../databricks-dbsql/SKILL.md)** — SQL warehouse and AI functions diff --git a/.claude/skills/databricks-execution-compute/references/1-databricks-connect.md b/.claude/skills/databricks-execution-compute/references/1-databricks-connect.md new file mode 100644 index 00000000..838d2a7d --- /dev/null +++ b/.claude/skills/databricks-execution-compute/references/1-databricks-connect.md @@ -0,0 +1,72 @@ +# Databricks Connect (Recommended Default) + +**Use when:** Running Spark code locally that executes on Databricks serverless compute. This is the fastest, cleanest approach for data generation, ETL, and any Spark workload. + +## Why Databricks Connect First? + +- **Instant iteration** — Edit file, re-run immediately +- **Local debugging** — IDE debugger, breakpoints work +- **No cold start** — Session stays warm across executions +- **Clean dependencies** — `withDependencies()` installs packages on remote compute + +## Requirements + +- **Python 3.12** (databricks-connect >= 16.4 requires it) +- **databricks-connect >= 16.4** package +- **~/.databrickscfg** with serverless config + +## Setup + +**Python 3.12 required.** If not available, install it (uv or other). If install fails, ask user—don't auto-switch modes. + +Use default profile, if not setup you can add it `~/.databrickscfg` (never overwrite it without conscent) +```ini +[DEFAULT] +host = https://your-workspace.cloud.databricks.com/ +serverless_compute_id = auto +auth_type = databricks-cli +``` + +## Usage Pattern + +```python +from databricks.connect import DatabricksSession, DatabricksEnv + +# Declare dependencies installed on serverless compute +# CRITICAL: Include ALL packages used inside UDFs (pandas/numpy are there by default) +env = DatabricksEnv().withDependencies("faker", "holidays") + +spark = ( + DatabricksSession.builder + .profile("my-workspace") # optional: run on a specific profile from ~/.databrickscfg instead of default + .withEnvironment(env) + .serverless(True) + .getOrCreate() +) + +# Spark code now executes on Databricks serverless +df = spark.range(1000)... +df.write.mode('overwrite').saveAsTable("catalog.schema.table") +``` + +## Common Issues + +| Issue | Solution | +|-------|----------| +| `Python 3.12 required` | create venv with correct python version | +| `DatabricksEnv not found` | Upgrade to databricks-connect >= 16.4 | +| `serverless_compute_id` error | Add `serverless_compute_id = auto` to ~/.databrickscfg | +| `ModuleNotFoundError` inside UDF | Add the package to `withDependencies()` | +| `PERSIST TABLE not supported` | Don't use `.cache()` or `.persist()` with serverless | +| `broadcast` is used | Don't broadcast small DF using spark connect, have a small python list instead or join small DF | + +## When NOT to Use + +Switch to **[Serverless Job](2-serverless-job.md)** when: +- one-off execution +- Heavy ML training that shouldn't depend on local machine staying connected +- Non-Spark Python code (pure sklearn, pytorch, etc.) + +Switch to **[Interactive Cluster](3-interactive-cluster.md)** when: +- Need state across multiple separate MCP tool calls +- Need Scala or R support diff --git a/.claude/skills/databricks-execution-compute/references/2-serverless-job.md b/.claude/skills/databricks-execution-compute/references/2-serverless-job.md new file mode 100644 index 00000000..4be8801c --- /dev/null +++ b/.claude/skills/databricks-execution-compute/references/2-serverless-job.md @@ -0,0 +1,76 @@ +# Serverless Job Execution + +**Use when:** Running intensive Python code remotely (ML training, heavy processing) that doesn't need Spark, or when code shouldn't depend on local machine staying connected. + +## When to Choose Serverless Job + +- ML model training (runs independently of local machine) +- Heavy non-Spark Python processing +- Code that takes > 5 minutes (local connection can drop) +- Production/scheduled runs + +## Trade-offs + +| Pro | Con | +|-----|-----| +| No cluster to manage | ~25-50s cold start each invocation | +| Up to 30 min timeout | No state preserved between calls | +| Independent execution | print() unreliable—use `dbutils.notebook.exit()` | + +## Executing code +### Prefer running from a Local File (edit the local file then run it) + +```python +execute_code( + file_path="/local/path/to/train_model.py", + compute_type="serverless" +) +``` + +## Jobs with Custom Dependencies + +Use `job_extra_params` to install pip packages: + +```python +execute_code( + file_path="/path/to/train.py", + job_extra_params={ + "environments": [{ + "environment_key": "ml_env", + "spec": {"client": "4", "dependencies": ["scikit-learn", "pandas", "mlflow"]} + }] + } +) +``` + +**CRITICAL:** Use `"client": "4"` in the spec. `"client": "1"` won't install dependencies. + +## Output Handling + +```python +# ❌ BAD - print() may not be captured +print("Training complete!") + +# ✅ GOOD - Use dbutils.notebook.exit() +import json +results = {"accuracy": 0.95, "model_path": "/Volumes/..."} +dbutils.notebook.exit(json.dumps(results)) +``` + +## Common Issues + +| Issue | Solution | +|-------|----------| +| print() output missing | Use `dbutils.notebook.exit()` | +| `ModuleNotFoundError` | Add to environments spec with `"client": "4"` | +| Job times out | Max is 1800s; split into smaller tasks | + +## When NOT to Use + +Switch to **[Databricks Connect](1-databricks-connect.md)** when: +- Iterating on Spark code and want instant feedback +- Need local debugging with breakpoints + +Switch to **[Interactive Cluster](3-interactive-cluster.md)** when: +- Need state across multiple MCP tool calls +- Need Scala or R support diff --git a/.claude/skills/databricks-execution-compute/references/3-interactive-cluster.md b/.claude/skills/databricks-execution-compute/references/3-interactive-cluster.md new file mode 100644 index 00000000..aa73ea90 --- /dev/null +++ b/.claude/skills/databricks-execution-compute/references/3-interactive-cluster.md @@ -0,0 +1,140 @@ +# Interactive Cluster Execution + +**Use when:** You have an existing running cluster and need to preserve state across multiple MCP tool calls, or need Scala/R support. + +## When to Choose Interactive Cluster + +- Multiple sequential commands where variables must persist +- Scala or R code (serverless only supports Python/SQL) +- Existing running cluster available + +## Trade-offs + +| Pro | Con | +|-----|-----| +| State persists via `context_id` | Cluster startup ~5 min if not running | +| Near-instant follow-up commands | Costs money while running | +| Scala/R/SQL support | Must manage cluster lifecycle | + +## Critical: Never Start a Cluster Without Asking + +**Starting a cluster takes 3-8 minutes and costs money.** Always check first: + +```python +list_compute(resource="clusters") +``` + +If no cluster is running, ask the user: +> "No running cluster. Options: +> 1. Start 'my-dev-cluster' (~5 min startup, costs money) +> 2. Use serverless (instant, no setup) +> Which do you prefer?" + +## Basic Usage + +### First Command: Creates Context + +```python +result = execute_code( + code="import pandas as pd\ndf = pd.DataFrame({'a': [1, 2, 3]})", + compute_type="cluster", + cluster_id="1234-567890-abcdef" +) +# result contains context_id for reuse +``` + +### Follow-up Commands: Reuse Context + +```python +# Variables from first command still available +execute_code( + code="print(df.shape)", # df exists + context_id=result["context_id"], + cluster_id=result["cluster_id"] +) +``` + +### Auto-Select Best Running Cluster + +```python +best_cluster = list_compute(resource="clusters", auto_select=True) +execute_code( + code="spark.range(100).show()", + compute_type="cluster", + cluster_id=best_cluster["cluster_id"] +) +``` + +## Language Support + +```python +execute_code(code='println("Hello")', compute_type="cluster", language="scala") +execute_code(code="SELECT * FROM table LIMIT 10", compute_type="cluster", language="sql") +execute_code(code='print("Hello")', compute_type="cluster", language="r") +``` + +## Installing Libraries + +Install pip packages directly in the execution context (pandas/numpy are there by default): + +```python +# Install library +execute_code( + code="""%pip install faker + dbutils.library.restartPython()""", # Restart Python to pick up new packages (if needed) + compute_type="cluster", + cluster_id="...", + context_id="..." +) +``` + +## Context Lifecycle + +**Keep alive (default):** Context persists until cluster terminates. + +**Destroy when done:** +```python +execute_code( + code="print('Done!')", + compute_type="cluster", + destroy_context_on_completion=True +) +``` + +## Handling No Running Cluster + +When no cluster is running, `execute_code` returns: +```json +{ + "success": false, + "error": "No running cluster available", + "startable_clusters": [{"cluster_id": "...", "cluster_name": "...", "state": "TERMINATED"}], + "suggestions": ["Start a terminated cluster", "Use serverless instead"] +} +``` + +### Starting a Cluster (With User Approval Only) + +```python +manage_cluster(action="start", cluster_id="1234-567890-abcdef") +# Poll until running (wait 20sec) +list_compute(resource="clusters", cluster_id="1234-567890-abcdef") +``` + +## Common Issues + +| Issue | Solution | +|-------|----------| +| "No running cluster" | Ask user to start or use serverless | +| Context not found | Context expired; create new one | +| Library not found | `%pip install ` then if needed `dbutils.library.restartPython()` | + +## When NOT to Use + +Switch to **[Databricks Connect](1-databricks-connect.md)** when: +- Developing Spark code with local debugging +- Want instant iteration without cluster concerns + +Switch to **[Serverless Job](2-serverless-job.md)** when: +- No cluster running and user doesn't want to wait +- One-off execution without state needs diff --git a/.claude/skills/databricks-genie/SKILL.md b/.claude/skills/databricks-genie/SKILL.md index 4d5d12f5..82332476 100644 --- a/.claude/skills/databricks-genie/SKILL.md +++ b/.claude/skills/databricks-genie/SKILL.md @@ -1,11 +1,11 @@ --- name: databricks-genie -description: "Create and query Databricks Genie Spaces for natural language SQL exploration. Use when building Genie Spaces or asking questions via the Genie Conversation API." +description: "Create and query Databricks Genie Spaces for natural language SQL exploration. Use when building Genie Spaces, exporting and importing Genie Spaces, migrating Genie Spaces between workspaces or environments, or asking questions via the Genie Conversation API." --- # Databricks Genie -Create and query Databricks Genie Spaces - natural language interfaces for SQL-based data exploration. +Create, manage, and query Databricks Genie Spaces - natural language interfaces for SQL-based data exploration. ## Overview @@ -18,31 +18,88 @@ Use this skill when: - Adding sample questions to guide users - Connecting Unity Catalog tables to a conversational interface - Asking questions to a Genie Space programmatically (Conversation API) +- Exporting a Genie Space configuration (serialized_space) for backup or migration +- Importing / cloning a Genie Space from a serialized payload +- Migrating a Genie Space between workspaces or environments (dev → staging → prod) + - Only supports catalog remapping where catalog names differ across environments + - Not supported for schema and/or table names that differ across environments + - Not including migration of tables between environments (only migration of Genie Spaces) ## MCP Tools -### Space Management - | Tool | Purpose | |------|---------| -| `list_genie` | List all Genie Spaces accessible to you | -| `create_or_update_genie` | Create or update a Genie Space | -| `get_genie` | Get Genie Space details | -| `delete_genie` | Delete a Genie Space | +| `manage_genie` | Create, get, list, delete, export, and import Genie Spaces | +| `ask_genie` | Ask natural language questions to a Genie Space | +| `get_table_stats_and_schema` | Inspect table schemas before creating a space | +| `execute_sql` | Test SQL queries directly | -### Conversation API +### manage_genie - Space Management -| Tool | Purpose | -|------|---------| -| `ask_genie` | Ask a question to a Genie Space, get SQL + results | -| `ask_genie_followup` | Ask follow-up question in existing conversation | +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create_or_update` | Idempotent create/update a space | display_name, table_identifiers (or serialized_space) | +| `get` | Get space details | space_id | +| `list` | List all spaces | (none) | +| `delete` | Delete a space | space_id | +| `export` | Export space config for migration/backup | space_id | +| `import` | Import space from serialized config | warehouse_id, serialized_space | + +**Example tool calls:** +``` +# MCP Tool: manage_genie +# Create a new space +manage_genie( + action="create_or_update", + display_name="Sales Analytics", + table_identifiers=["catalog.schema.customers", "catalog.schema.orders"], + description="Explore sales data with natural language", + sample_questions=["What were total sales last month?"] +) -### Supporting Tools +# MCP Tool: manage_genie +# Get space details with full config +manage_genie(action="get", space_id="space_123", include_serialized_space=True) + +# MCP Tool: manage_genie +# List all spaces +manage_genie(action="list") + +# MCP Tool: manage_genie +# Export for migration +exported = manage_genie(action="export", space_id="space_123") + +# MCP Tool: manage_genie +# Import to new workspace +manage_genie( + action="import", + warehouse_id="warehouse_456", + serialized_space=exported["serialized_space"], + title="Sales Analytics (Prod)" +) +``` -| Tool | Purpose | -|------|---------| -| `get_table_details` | Inspect table schemas before creating a space | -| `execute_sql` | Test SQL queries directly | +### ask_genie - Conversation API (Query) + +Ask natural language questions to a Genie Space. Pass `conversation_id` for follow-up questions. + +``` +# MCP Tool: ask_genie +# Start a new conversation +result = ask_genie( + space_id="space_123", + question="What were total sales last month?" +) +# Returns: {question, conversation_id, message_id, status, sql, columns, data, row_count} + +# MCP Tool: ask_genie +# Follow-up question in same conversation +result = ask_genie( + space_id="space_123", + question="Break that down by region", + conversation_id=result["conversation_id"] +) +``` ## Quick Start @@ -50,8 +107,9 @@ Use this skill when: Before creating a Genie Space, understand your data: -```python -get_table_details( +``` +# MCP Tool: get_table_stats_and_schema +get_table_stats_and_schema( catalog="my_catalog", schema="sales", table_stat_level="SIMPLE" @@ -60,8 +118,10 @@ get_table_details( ### 2. Create the Genie Space -```python -create_or_update_genie( +``` +# MCP Tool: manage_genie +manage_genie( + action="create_or_update", display_name="Sales Analytics", table_identifiers=[ "my_catalog.sales.customers", @@ -77,7 +137,8 @@ create_or_update_genie( ### 3. Ask Questions (Conversation API) -```python +``` +# MCP Tool: ask_genie ask_genie( space_id="your_space_id", question="What were total sales last month?" @@ -85,15 +146,31 @@ ask_genie( # Returns: SQL, columns, data, row_count ``` -## Workflow +### 4. Export & Import (Clone / Migrate) + +Export a space (preserves all tables, instructions, SQL examples, and layout): + +``` +# MCP Tool: manage_genie +exported = manage_genie(action="export", space_id="your_space_id") +# exported["serialized_space"] contains the full config +``` + +Clone to a new space (same catalog): ``` -1. Inspect tables → get_table_details -2. Create space → create_or_update_genie -3. Query space → ask_genie (or test in Databricks UI) -4. Curate (optional) → Use Databricks UI to add instructions +# MCP Tool: manage_genie +manage_genie( + action="import", + warehouse_id=exported["warehouse_id"], + serialized_space=exported["serialized_space"], + title=exported["title"], # override title; omit to keep original + description=exported["description"], +) ``` +> **Cross-workspace migration:** Each MCP server is workspace-scoped. Configure one server entry per workspace profile in your IDE's MCP config, then `manage_genie(action="export")` from the source server and `manage_genie(action="import")` via the target server. See [spaces.md §Migration](spaces.md#migrating-across-workspaces-with-catalog-remapping) for the full workflow. + ## Reference Files - [spaces.md](spaces.md) - Creating and managing Genie Spaces @@ -109,13 +186,15 @@ Before creating a Genie Space: ### Creating Tables Use these skills in sequence: -1. `synthetic-data-generation` - Generate raw parquet files -2. `spark-declarative-pipelines` - Create bronze/silver/gold tables +1. `databricks-synthetic-data-gen` - Generate raw parquet files +2. `databricks-spark-declarative-pipelines` - Create bronze/silver/gold tables ## Common Issues -| Issue | Solution | -|-------|----------| -| **No warehouse available** | Create a SQL warehouse or provide `warehouse_id` explicitly | -| **Poor query generation** | Add instructions and sample questions that reference actual column names | -| **Slow queries** | Ensure warehouse is running; use OPTIMIZE on tables | +See [spaces.md §Troubleshooting](spaces.md#troubleshooting) for a full list of issues and solutions. +## Related Skills + +- **[databricks-agent-bricks](../databricks-agent-bricks/SKILL.md)** - Use Genie Spaces as agents inside Supervisor Agents +- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** - Generate raw parquet data to populate tables for Genie +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - Build bronze/silver/gold tables consumed by Genie Spaces +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - Manage the catalogs, schemas, and tables Genie queries diff --git a/.claude/skills/databricks-genie/conversation.md b/.claude/skills/databricks-genie/conversation.md index 149cafa6..e4320e8b 100644 --- a/.claude/skills/databricks-genie/conversation.md +++ b/.claude/skills/databricks-genie/conversation.md @@ -31,8 +31,7 @@ The `ask_genie` tool allows you to programmatically send questions to a Genie Sp | Tool | Purpose | |------|---------| -| `ask_genie` | Ask a question, start new conversation | -| `ask_genie_followup` | Ask follow-up in existing conversation | +| `ask_genie` | Ask a question or follow-up (`conversation_id` optional) | ## Basic Usage @@ -71,10 +70,10 @@ result = ask_genie( ) # Follow-up (uses context from first question) -ask_genie_followup( +ask_genie( space_id="01abc123...", - conversation_id=result["conversation_id"], - question="Break that down by region" + question="Break that down by region", + conversation_id=result["conversation_id"] ) ``` @@ -148,7 +147,7 @@ Claude: User: "I just created a Genie Space for HR data. Can you test it?" Claude: -1. Gets the space_id from the user or recent create_or_update_genie result +1. Gets the space_id from the user or recent manage_genie(action="create_or_update") result 2. Calls ask_genie with test questions: - "How many employees do we have?" - "What is the average salary by department?" @@ -163,9 +162,9 @@ User: "Use my analytics Genie to explore sales trends" Claude: 1. ask_genie(space_id, "What were total sales by month this year?") 2. User: "Which month had the highest growth?" -3. ask_genie_followup(space_id, conv_id, "Which month had the highest growth?") +3. ask_genie(space_id, "Which month had the highest growth?", conversation_id=conv_id) 4. User: "What products drove that growth?" -5. ask_genie_followup(space_id, conv_id, "What products drove that growth?") +5. ask_genie(space_id, "What products drove that growth?", conversation_id=conv_id) ``` ## Best Practices @@ -181,8 +180,8 @@ result2 = ask_genie(space_id, "How many employees do we have?") # New conversat # Good: Follow-up for related question result1 = ask_genie(space_id, "What were sales last month?") -result2 = ask_genie_followup(space_id, result1["conversation_id"], - "Break that down by product") # Related follow-up +result2 = ask_genie(space_id, "Break that down by product", + conversation_id=result1["conversation_id"]) # Related follow-up ``` ### Handle Clarification Requests @@ -219,7 +218,7 @@ ask_genie(space_id, "Calculate customer lifetime value for all customers", - Verify the `space_id` is correct - Check you have access to the space -- Use `get_genie(space_id)` to verify it exists +- Use `manage_genie(action="get", space_id=...)` to verify it exists ### "Query timed out" diff --git a/.claude/skills/databricks-genie/spaces.md b/.claude/skills/databricks-genie/spaces.md index 71c93985..ff8acb60 100644 --- a/.claude/skills/databricks-genie/spaces.md +++ b/.claude/skills/databricks-genie/spaces.md @@ -4,12 +4,7 @@ This guide covers creating and managing Genie Spaces for SQL-based data explorat ## What is a Genie Space? -A Genie Space connects to Unity Catalog tables and translates natural language questions into SQL queries. The system: - -1. **Understands** the table schemas and relationships -2. **Generates** SQL queries from natural language -3. **Executes** queries on a SQL warehouse -4. **Presents** results in a conversational format +A Genie Space connects to Unity Catalog tables and translates natural language questions into SQL — understanding schemas, generating queries, executing them on a SQL warehouse, and presenting results conversationally. ## Creation Workflow @@ -18,7 +13,7 @@ A Genie Space connects to Unity Catalog tables and translates natural language q **Before creating a Genie Space, you MUST inspect the table schemas** to understand what data is available: ```python -get_table_details( +get_table_stats_and_schema( catalog="my_catalog", schema="sales", table_stat_level="SIMPLE" @@ -45,7 +40,8 @@ Based on the schema information: Create the space with content tailored to the actual data: ```python -create_or_update_genie( +manage_genie( + action="create_or_update", display_name="Sales Analytics", table_identifiers=[ "my_catalog.sales.customers", @@ -153,25 +149,197 @@ Write sample questions that: ## Updating a Genie Space -To update an existing space: +`manage_genie(action="create_or_update")` handles both create and update automatically. There are two ways it locates an existing space to update: + +- **By `space_id`** (explicit, preferred): pass `space_id=` to target a specific space. +- **By `display_name`** (implicit fallback): if `space_id` is omitted, the tool searches for a space with a matching name and updates it if found; otherwise it creates a new one. + +### Simple field updates (tables, questions, warehouse) + +To update metadata without a serialized config: + +```python +manage_genie( + action="create_or_update", + display_name="Sales Analytics", + space_id="01abc123...", # omit to match by name instead + table_identifiers=[ # updated table list + "my_catalog.sales.customers", + "my_catalog.sales.orders", + "my_catalog.sales.products", + ], + sample_questions=[ # updated sample questions + "What were total sales last month?", + "Who are our top 10 customers by revenue?", + ], + warehouse_id="abc123def456", # omit to keep current / auto-detect + description="Updated description.", +) +``` + +### Full config update via `serialized_space` + +To push a complete serialized configuration to an existing space (the dict contains all regular table metadata, plus it preserves all instructions, SQL examples, join specs, etc.): + +```python +manage_genie( + action="create_or_update", + display_name="Sales Analytics", # overrides title embedded in serialized_space + table_identifiers=[], # ignored when serialized_space is provided + space_id="01abc123...", # target space to overwrite + warehouse_id="abc123def456", # overrides warehouse embedded in serialized_space + description="Updated description.", # overrides description embedded in serialized_space; omit to keep the one in the payload + serialized_space=remapped_config, # JSON string from manage_genie(action="export") (after catalog remap if needed) +) +``` + +> **Note:** When `serialized_space` is provided, `table_identifiers` and `sample_questions` are ignored — the full config comes from the serialized payload. However, `display_name`, `warehouse_id`, and `description` are still applied as top-level overrides on top of the serialized payload. Omit any of them to keep the values embedded in `serialized_space`. + +## Export, Import & Migration + +`manage_genie(action="export")` returns a dictionary with four top-level keys: + +| Key | Description | +|-----|-------------| +| `space_id` | ID of the exported space | +| `title` | Display name of the space | +| `description` | Description of the space | +| `warehouse_id` | SQL warehouse associated with the space (workspace-specific — do **not** reuse across workspaces) | +| `serialized_space` | JSON-encoded string with the full space configuration (see below) | + +This envelope enables cloning, backup, and cross-workspace migration. Use `manage_genie(action="export")` and `manage_genie(action="import")` for all export/import operations — no direct REST calls needed. + +### What is `serialized_space`? + +`serialized_space` is a JSON string (version 2) embedded inside the export envelope. Its top-level keys are: + +| Key | Contents | +|-----|----------| +| `version` | Schema version (currently `2`) | +| `config` | Space-level config: `sample_questions` shown in the UI | +| `data_sources` | `tables` array — each entry has a fully-qualified `identifier` (`catalog.schema.table`) and optional `column_configs` (format assistance, entity matching per column) | +| `instructions` | `example_question_sqls` (certified Q&A pairs), `join_specs` (join relationships between tables), `sql_snippets` (`filters` and `measures` with display names and usage instructions) | +| `benchmarks` | Evaluation Q&A pairs used to measure space quality | + +Catalog names appear **everywhere** inside `serialized_space` — in `data_sources.tables[].identifier`, SQL strings in `example_question_sqls`, `join_specs`, and `sql_snippets`. A single `.replace(src_catalog, tgt_catalog)` on the whole string is sufficient for catalog remapping. + +Minimum structure: +```json +{"version": 2, "data_sources": {"tables": [{"identifier": "catalog.schema.table"}]}} +``` + +### Exporting a Space + +Use `manage_genie(action="export")` to export the full configuration (requires CAN EDIT permission): + +```python +exported = manage_genie(action="export", space_id="01abc123...") +# Returns: +# { +# "space_id": "01abc123...", +# "title": "Sales Analytics", +# "description": "Explore sales data...", +# "warehouse_id": "abc123def456", +# "serialized_space": "{\"version\":2,\"data_sources\":{...},\"instructions\":{...}}" +# } +``` + +You can also get `serialized_space` inline via `manage_genie(action="get")`: + +```python +details = manage_genie(action="get", space_id="01abc123...", include_serialized_space=True) +serialized = details["serialized_space"] +``` + +### Cloning a Space (Same Workspace) + +```python +# Step 1: Export the source space +source = manage_genie(action="export", space_id="01abc123...") + +# Step 2: Import as a new space +manage_genie( + action="import", + warehouse_id=source["warehouse_id"], + serialized_space=source["serialized_space"], + title=source["title"], # override title; omit to keep original + description=source["description"], +) +# Returns: {"space_id": "01def456...", "title": "Sales Analytics (Dev Copy)", "operation": "imported"} +``` + +### Migrating Across Workspaces with Catalog Remapping + +When migrating between environments (e.g. prod → dev), Unity Catalog names are often different. The `serialized_space` string contains the source catalog name **everywhere** — in table identifiers, SQL queries, join specs, and filter snippets. You must remap it before importing. + +**Agent workflow (3 steps):** + +**Step 1 — Export from source workspace:** +```python +exported = manage_genie(action="export", space_id="01f106e1239d14b28d6ab46f9c15e540") +# exported keys: warehouse_id, title, description, serialized_space +# exported["serialized_space"] contains all references to source catalog +``` + +**Step 2 — Remap catalog name in `serialized_space`:** + +The agent does this as an inline string substitution between the two MCP calls: +```python +modified_serialized = exported["serialized_space"].replace( + "source_catalog_name", # e.g. "healthverity_claims_sample_patient_dataset" + "target_catalog_name" # e.g. "healthverity_claims_sample_patient_dataset_dev" +) +``` +This replaces all occurrences — table identifiers, SQL FROM clauses, join specs, and filter snippets. + +**Step 3 — Import to target workspace:** +```python +manage_genie( + action="import", + warehouse_id="", # from manage_warehouse(action="list") on target + serialized_space=modified_serialized, + title=exported["title"], + description=exported["description"] +) +``` + +### Batch Migration of Multiple Spaces + +To migrate several spaces at once, loop through space IDs. The agent exports, remaps the catalog, then imports each: + +``` +For each space_id in [id1, id2, id3]: + 1. exported = manage_genie(action="export", space_id=space_id) + 2. modified = exported["serialized_space"].replace(src_catalog, tgt_catalog) + 3. result = manage_genie(action="import", warehouse_id=wh_id, serialized_space=modified, title=exported["title"], description=exported["description"]) + 4. record result["space_id"] for updating databricks.yml +``` + +After migration, update `databricks.yml` with the new dev `space_id` values under the `dev` target's `genie_space_ids` variable. + +### Updating an Existing Space with New Config + +To push a serialized config to an already-existing space (rather than creating a new one), use `manage_genie(action="create_or_update")` with `space_id=` and `serialized_space=`. The export → remap → push pattern is identical to the migration steps above; just replace `manage_genie(action="import")` with `manage_genie(action="create_or_update", space_id=TARGET_SPACE_ID, ...)` as the final call. -1. **Add/remove tables**: Call `create_or_update_genie` with updated `table_identifiers` -2. **Update questions**: Include new `sample_questions` -3. **Change warehouse**: Provide a different `warehouse_id` +### Permissions Required -The tool finds the existing space by name and updates it. +| Operation | Required Permission | +|-----------|-------------------| +| `manage_genie(action="export")` / `manage_genie(action="get", include_serialized_space=True)` | CAN EDIT on source space | +| `manage_genie(action="import")` | Can create items in target workspace folder | +| `manage_genie(action="create_or_update")` with `serialized_space` (update) | CAN EDIT on target space | ## Example End-to-End Workflow -1. **Generate synthetic data** using `synthetic-data-generation` skill: +1. **Generate synthetic data** using `databricks-synthetic-data-gen` skill: - Creates parquet files in `/Volumes/catalog/schema/raw_data/` -2. **Create tables** using `spark-declarative-pipelines` skill: +2. **Create tables** using `databricks-spark-declarative-pipelines` skill: - Creates `catalog.schema.bronze_*` → `catalog.schema.silver_*` → `catalog.schema.gold_*` 3. **Inspect the tables**: ```python - get_table_details(catalog="catalog", schema="schema") + get_table_stats_and_schema(catalog="catalog", schema="schema") ``` 4. **Create the Genie Space**: @@ -201,3 +369,27 @@ The tool finds the existing space by name and updates it. - Add table and column comments - Include sample questions that demonstrate the vocabulary - Add instructions via the Databricks Genie UI + +### `manage_genie(action="export")` returns empty `serialized_space` + +Requires at least **CAN EDIT** permission on the space. + +### `manage_genie(action="import")` fails with permission error + +Ensure you have CREATE privileges in the target workspace folder. + +### Tables not found after migration + +Catalog name was not remapped — replace the source catalog name in `serialized_space` before calling `manage_genie(action="import")`. The catalog appears in table identifiers, SQL FROM clauses, join specs, and filter snippets; a single `.replace(src_catalog, tgt_catalog)` on the whole string covers all occurrences. + +### `manage_genie` lands in the wrong workspace + +Each MCP server is workspace-scoped. Set up two named MCP server entries (one per profile) in your IDE's MCP config instead of switching a single server's profile mid-session. + +### MCP server doesn't pick up profile change + +The MCP process reads `DATABRICKS_CONFIG_PROFILE` once at startup — editing the config file requires an IDE reload to take effect. + +### `manage_genie(action="import")` fails with JSON parse error + +The `serialized_space` string may contain multi-line SQL arrays with `\n` escape sequences. Flatten SQL arrays to single-line strings before passing to avoid double-escaping issues. diff --git a/.claude/skills/databricks-iceberg/1-managed-iceberg-tables.md b/.claude/skills/databricks-iceberg/1-managed-iceberg-tables.md new file mode 100644 index 00000000..a0f3f06e --- /dev/null +++ b/.claude/skills/databricks-iceberg/1-managed-iceberg-tables.md @@ -0,0 +1,262 @@ +# Managed Iceberg Tables + +Managed Iceberg tables are native Apache Iceberg tables created and stored within Unity Catalog. They support full read/write operations in Databricks and are accessible to external engines via the UC Iceberg REST Catalog (IRC) endpoint. + +**Requirements**: Unity Catalog, DBR 16.4 LTS+ (Managed Iceberg v2), DBR 17.3+ (Managed Iceberg v3 Beta) + +--- + +## Creating Tables + +### Basic DDL + +```sql +-- Create an empty Iceberg table (no clustering) +CREATE TABLE my_catalog.my_schema.events ( + event_id BIGINT, + event_type STRING, + event_date DATE, + payload STRING +) +USING ICEBERG; +``` + +### Create Table As Select (CTAS) + +```sql +-- Create from existing data (no clustering) +CREATE TABLE my_catalog.my_schema.events_archive +USING ICEBERG +AS SELECT * FROM my_catalog.my_schema.events +WHERE event_date < '2025-01-01'; +``` + +### Liquid Clustering + +Managed Iceberg tables use **Liquid Clustering** for data layout optimization. Both `PARTITIONED BY` and `CLUSTER BY` produce a Liquid Clustered table — **no traditional Hive-style partitions are created**. Unity Catalog interprets the partition clause as clustering keys. + +| Syntax | DDL (create table) | Reads via IRC | Iceberg partition fields visible to external engines | DV/row-tracking handling | +|--------|--------------------|---------------|------------------------------------------------------|--------------------------| +| `PARTITIONED BY (col)` | DBR + EMR, OSS Spark, Trino, Flink | Yes | Yes — UC exposes Iceberg partition fields corresponding to clustering keys; external engines can prune | **Auto-handled** | +| `CLUSTER BY (col)` | DBR only | Yes | Yes — same; UC maintains Iceberg partition spec from clustering keys regardless of DDL used | Manual on v2, auto on v3 | + +> **Both syntaxes produce the same Iceberg metadata for external engines.** UC maintains an Iceberg partition spec (partition fields corresponding to the clustering keys) that external engines read via IRC. This is Iceberg-style partitioning — not legacy Hive-style directory partitions. External engines see a partitioned Iceberg table and benefit from partition pruning. Internally, UC uses those partition fields as liquid clustering keys. + +> **`PARTITIONED BY` limitation**: Only plain column references are supported. Expression transforms (`bucket()`, `years()`, `months()`, `days()`, `hours()`) are **not** supported and will error. + +> **`CLUSTER BY` on Iceberg v2**: requires explicitly setting `'delta.enableDeletionVectors' = false` and `'delta.enableRowTracking' = false`, otherwise you get: `[MANAGED_ICEBERG_ATTEMPTED_TO_ENABLE_CLUSTERING_WITHOUT_DISABLING_DVS_OR_ROW_TRACKING]` + +**`PARTITIONED BY` — recommended for cross-platform** (auto-handles all required properties): + +```sql +-- Single column (v2 or v3 — no TBLPROPERTIES needed) +CREATE TABLE orders ( + order_id BIGINT, + order_date DATE +) +USING ICEBERG +PARTITIONED BY (order_date); + +-- Multi-column +CREATE TABLE orders ( + order_id BIGINT, + region STRING, + order_date DATE +) +USING ICEBERG +PARTITIONED BY (region, order_date); +``` + +**`CLUSTER BY` on Iceberg v2** (DBR-only; must disable DVs and row tracking manually): + +```sql +-- Single column clustering (v2) +CREATE TABLE orders ( + order_id BIGINT, + order_date DATE +) +USING ICEBERG +TBLPROPERTIES ( + 'delta.enableDeletionVectors' = false, + 'delta.enableRowTracking' = false +) +CLUSTER BY (order_date); +``` + +**`CLUSTER BY` on Iceberg v3** (no extra TBLPROPERTIES needed): + +```sql +CREATE TABLE orders ( + order_id BIGINT, + order_date DATE +) +USING ICEBERG +TBLPROPERTIES ('format-version' = '3') +CLUSTER BY (order_date); +``` + +--- + +## DML Operations + +Managed Iceberg tables support all standard DML operations: + +```sql +-- INSERT +INSERT INTO my_catalog.my_schema.events +VALUES (1, 'click', '2025-06-01', '{"page": "home"}'); + +-- INSERT from query +INSERT INTO my_catalog.my_schema.events +SELECT * FROM staging_events WHERE event_date = current_date(); + +-- UPDATE +UPDATE my_catalog.my_schema.events +SET event_type = 'page_view' +WHERE event_id = 1; + +-- DELETE +DELETE FROM my_catalog.my_schema.events +WHERE event_date < '2024-01-01'; + +-- MERGE (upsert) +MERGE INTO my_catalog.my_schema.events AS target +USING staging_events AS source +ON target.event_id = source.event_id +WHEN MATCHED THEN UPDATE SET * +WHEN NOT MATCHED THEN INSERT *; +``` + +--- + +## Time Travel + +Query historical snapshots using timestamp or snapshot ID: + +```sql +-- Query by timestamp +SELECT * FROM my_catalog.my_schema.events TIMESTAMP AS OF '2025-06-01T00:00:00Z'; + +-- Query by snapshot ID +SELECT * FROM my_catalog.my_schema.events VERSION AS OF 1234567890; + +-- Only for external engines: View snapshot history +SELECT * FROM my_catalog.my_schema.events.snapshots; +``` + +--- + +## Predictive Optimization + +Predictive Optimization is **recommended** for managed Iceberg tables — it is not auto-enabled and must be turned on explicitly. Once enabled, it automatically runs: + +- **Compaction** — consolidates small files +- **Vacuum** — removes expired snapshots and orphan files +- **Statistics collection** — keeps column statistics up to date for query optimization + +Enable at the catalog or schema level. Manual operations are still available if needed: + +```sql +-- Manual compaction +OPTIMIZE my_catalog.my_schema.events; + +-- Manual vacuum +VACUUM my_catalog.my_schema.events; + +-- Manual statistics collection +ANALYZE TABLE my_catalog.my_schema.events COMPUTE STATISTICS FOR ALL COLUMNS; +``` + +--- + +## Iceberg v3 (Beta) + +**Requires**: DBR 17.3+ + +Iceberg v3 introduces new capabilities on top of v2: + +| Feature | Description | +|---------|-------------| +| **Deletion Vectors** | Row-level deletes without rewriting data files — faster UPDATE/DELETE/MERGE | +| **VARIANT Type** | Semi-structured data column (like Delta's VARIANT) | +| **Row Lineage** | Track row-level provenance across transformations | + +### Creating an Iceberg v3 Table + +```sql +CREATE TABLE my_catalog.my_schema.events_v3 ( + event_id BIGINT, + event_date DATE, + data VARIANT +) +USING ICEBERG +TBLPROPERTIES ('format-version' = '3') +CLUSTER BY (event_date); +``` + +### Important Notes + +- **Cannot downgrade**: Once a table is upgraded to v3, it cannot be downgraded back to v2 +- **External engine compatibility**: External engines must use Iceberg library 1.9.0+ to read v3 tables +- **Deletion vectors**: Enabled by default on v3 tables. External readers must support deletion vectors +- **Beta status**: Iceberg v3 is in Beta — not recommended for production workloads yet + +### Upgrading an Existing Table to v3 + +```sql +ALTER TABLE my_catalog.my_schema.events +SET TBLPROPERTIES ('format-version' = '3'); +``` + +> **Warning**: This is irreversible. Test with non-production data first. + +--- + +## Limitations + +| Limitation | Details | +|------------|---------| +| **No Vector Search** | Vector Search indexes are not supported on Iceberg tables | +| **No Change Data Feed (CDF)** | CDF is a Delta-only feature; use Delta + UniForm if CDF is required | +| **Parquet only** | Iceberg tables on Databricks use Parquet as the underlying file format | +| **No shallow clone** | `SHALLOW CLONE` is not supported; use `DEEP CLONE` or CTAS | +| **`PARTITIONED BY` maps to Liquid Clustering** | `PARTITIONED BY` is supported and recommended for cross-platform scenarios — it maps to Liquid Clustering, not traditional partitions. Only plain column references work; expression transforms (`bucket()`, `years()`, etc.) are not supported. | +| **No Structured Streaming sink** | Cannot use `writeStream` to write to Iceberg tables directly; use `INSERT INTO` or `MERGE` in batch or SDP | +| **Compression** | Default compression is `zstd`; older readers may need `snappy` — set `write.parquet.compression-codec` if needed | +| **Do not set metadata path** | Never set `write.metadata.path` or `write.metadata.previous-versions-max` | +| **Do not install Iceberg library** | DBR includes built-in support; installing an Iceberg JAR causes conflicts | + +--- + +## Converting From Other Formats + +### Delta to Iceberg (via DEEP CLONE) + +```sql +CREATE TABLE my_catalog.my_schema.events_iceberg +USING ICEBERG +DEEP CLONE my_catalog.my_schema.events_delta; +``` + +### Foreign Iceberg to Managed Iceberg + +```sql +-- With Liquid Clustering (v2 — must disable DVs and row tracking) +CREATE TABLE my_catalog.my_schema.events_managed +USING ICEBERG +TBLPROPERTIES ( + 'delta.enableDeletionVectors' = false, + 'delta.enableRowTracking' = false +) +CLUSTER BY (event_date) +AS SELECT * FROM foreign_catalog.foreign_schema.events; + +-- With Liquid Clustering (v3 — no extra TBLPROPERTIES needed) +CREATE TABLE my_catalog.my_schema.events_managed +USING ICEBERG +TBLPROPERTIES ('format-version' = '3') +CLUSTER BY (event_date) +AS SELECT * FROM foreign_catalog.foreign_schema.events; +``` + + diff --git a/.claude/skills/databricks-iceberg/2-uniform-and-compatibility.md b/.claude/skills/databricks-iceberg/2-uniform-and-compatibility.md new file mode 100644 index 00000000..8437a725 --- /dev/null +++ b/.claude/skills/databricks-iceberg/2-uniform-and-compatibility.md @@ -0,0 +1,207 @@ +# UniForm and Compatibility Mode + +UniForm and Compatibility Mode make Delta tables readable as Iceberg by external engines — without converting to a native Iceberg table. Data is written as Delta, but Iceberg metadata is generated automatically so external tools (Snowflake, PyIceberg, Spark, Trino) can read via UC IRC endpoint. + +--- + +## External Iceberg Reads (fka UniForm) (GA) + +**Requirements**: Unity Catalog, DBR 14.3+, column mapping enabled, deletion vectors disabled, the Delta table must have a minReaderVersion >= 2 and minWriterVersion >= 7, both managed and external tables supported. + +UniForm adds automatic Iceberg metadata generation to regular Delta tables. The table remains Delta internally but is readable as Iceberg externally. + +### Enabling UniForm on a New Table + +```sql +CREATE TABLE my_catalog.my_schema.customers ( + customer_id BIGINT, + name STRING, + region STRING, + updated_at TIMESTAMP +) +TBLPROPERTIES ( + 'delta.columnMapping.mode' = 'name', + 'delta.enableIcebergCompatV2' = 'true', + 'delta.universalFormat.enabledFormats' = 'iceberg' +); +``` + +### Enabling UniForm on an Existing Table + +```sql +ALTER TABLE my_catalog.my_schema.customers +SET TBLPROPERTIES ( + 'delta.columnMapping.mode' = 'name', + 'delta.enableIcebergCompatV2' = 'true', + 'delta.universalFormat.enabledFormats' = 'iceberg' +); +``` + +### Requirements and Prerequisites + +UniForm requires the following properties to be set explicitly: + +| Requirement | Details | +|-------------|---------| +| **Unity Catalog** | Table must be registered in UC | +| **DBR 14.3+** | Minimum runtime version | +| **Deletion vectors disabled** | Set `delta.enableDeletionVectors = false` before enabling UniForm | +| **No column mapping conflicts** | If table uses `id` mode, migrate to `name` mode first | + +If deletion vectors are currently enabled: + +```sql +-- Disable deletion vectors first +ALTER TABLE my_catalog.my_schema.customers +SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'false'); + +-- Rewrite to remove existing deletion vectors +REORG TABLE my_catalog.my_schema.customers +APPLY (PURGE); + +-- Then enable UniForm +ALTER TABLE my_catalog.my_schema.customers +SET TBLPROPERTIES ( + 'delta.columnMapping.mode' = 'name', + 'delta.enableIcebergCompatV2' = 'true', + 'delta.universalFormat.enabledFormats' = 'iceberg' +); +``` + +### Async Metadata Generation + +Iceberg metadata is generated **asynchronously** after each Delta transaction. There is a brief delay (typically seconds, occasionally minutes for large transactions) before external engines see the latest data. + +### Checking UniForm Status + +> See [Check Iceberg metadata generation status](https://docs.databricks.com/aws/en/delta/uniform#check-iceberg-metadata-generation-status) for full details. + + +### Disabling UniForm + +```sql +ALTER TABLE my_catalog.my_schema.customers +UNSET TBLPROPERTIES ('delta.universalFormat.enabledFormats'); +``` + +--- + +## Compatibility Mode + +**Requirements**: Unity Catalog, DBR 16.1+, SDP pipeline + +Compatibility Mode extends UniForm to **streaming tables (STs)** and **materialized views (MVs)** created by Spark Declarative Pipelines (SDP) or DBSQL. Regular UniForm does not work on STs/MVs — Compatibility Mode is the only option. + +**How it works**: When you enable Compatibility Mode, Databricks creates a separate, read-only **"compatibility version"** of the object at the external location you specify (`delta.universalFormat.compatibility.location`). This is a full copy of the data in Iceberg-compatible format — not a pointer to the original Delta data. After the initial full copy, subsequent metadata and data generation is **incremental** (only new/changed data is synced to the external location). + +> **Storage cost consideration**: Because Compatibility Mode writes a separate copy of the data to the external location, you incur additional cloud storage costs proportional to the size of the table. Factor this in when enabling Compatibility Mode on large tables. + +### Enabling Compatibility Mode + +Compatibility Mode is configured via table properties: + +**SQL Example (streaming table)**: + +```sql +CREATE OR REFRESH STREAMING TABLE my_events +TBLPROPERTIES ( + 'delta.universalFormat.enabledFormats' = 'compatibility', + 'delta.universalFormat.compatibility.location' = '' +) +AS SELECT * FROM STREAM read_files('/Volumes/catalog/schema/raw/events/'); +``` + +**SQL Example (materialized view)**: + +```sql +CREATE OR REFRESH MATERIALIZED VIEW daily_summary +TBLPROPERTIES ( + 'delta.universalFormat.enabledFormats' = 'compatibility', + 'delta.universalFormat.compatibility.location' = '' +) +AS SELECT event_date, COUNT(*) AS event_count +FROM my_events +GROUP BY event_date; +``` + +**Python Example**: + +```python +from pyspark import pipelines as dp + +@dp.table( + name="my_events", + table_properties={ + "delta.universalFormat.enabledFormats": "compatibility", + "delta.universalFormat.compatibility.location": "", + }, +) +def my_events(): + return ( + spark.readStream.format("cloudFiles") + .option("cloudFiles.format", "json") + .load("/Volumes/catalog/schema/raw/events/") + ) +``` + +### Considerations for Compatibility Mode + +| Consideration | Details | +|---------------|---------| +| **External location** | `delta.universalFormat.compatibility.location` must point to a configured external location for the Iceberg metadata output path | +| **SDP pipeline only** | Only works with streaming tables and MVs defined in SDP pipelines | +| **Initial generation time** | First metadata generation can take up to 1 hour for large tables | +| **Unity Catalog** | Required | +| **DBR 16.1+** | Minimum runtime for the SDP pipeline | + +### Refresh Mechanics + +Compatibility Mode metadata can be refreshed manually or controlled via the `delta.universalFormat.compatibility.targetRefreshInterval` property: + +```sql +CREATE OR REFRESH STREAMING TABLE my_events +TBLPROPERTIES ( + 'delta.universalFormat.enabledFormats' = 'compatibility', + 'delta.universalFormat.compatibility.location' = '', + 'delta.universalFormat.compatibility.targetRefreshInterval' = '0 MINUTES' +) +AS SELECT * FROM STREAM read_files('/Volumes/catalog/schema/raw/events/'); +``` + +| Interval value | Behavior | +|----------------|----------| +| `0 MINUTES` | Checks for changes after every commit and triggers a refresh if needed — default for streaming tables and MVs | +| `1 HOUR` | Default for non-SDP tables; refreshes at most once per hour | +| Values below `1 HOUR` (e.g. `30 MINUTES`) | Not recommended — won't make refreshes more frequent than once per hour | + +Metadata can also be triggered manually: + +```sql +REFRESH TABLE my_catalog.my_schema.my_events; +``` + +### Future Modes + +A more efficient mode for streaming tables and materialized views is expected in a future release. + +--- + +## Decision Table: Which Approach? + +| Criteria | Managed Iceberg | UniForm | Compatibility Mode | +|----------|:-:|:-:|:-:| +| **Full Iceberg read/write** | Yes | Read-only (as Iceberg) | Read-only (as Iceberg) | +| **Works with Delta features (CDF)** | No | Partial* | Partial* | +| **Streaming tables / MVs** | No | No | Yes | +| **External engine write via IRC** | Yes | No | No | +| **Existing Delta investment** | Requires migration | No migration | No migration | +| **Predictive Optimization** | Auto-enabled | Auto-enabled (Delta) | Auto-enabled (Delta) | +| **DBR requirement** | 16.1+ | 14.3+ | 16.1+ | + +*given that Iceberg doesn't have CDF so the features dependent on it are not supported e.g., +streaming tables, materialized views, data classification, vector search, data profiling. For Synced tables to Lakebase, only snapshot mode is supported. +### When to Choose Each + +- **Managed Iceberg**: You want a native Iceberg table with full read/write from both Databricks and external engines. You don't need Delta-specific features (e.g., CDF). +- **UniForm**: You have existing Delta tables and want to make them readable as Iceberg by external engines without migrating. You want to keep Delta features internally. +- **Compatibility Mode**: You have streaming tables or materialized views that need to be readable as Iceberg by external engines. diff --git a/.claude/skills/databricks-iceberg/3-iceberg-rest-catalog.md b/.claude/skills/databricks-iceberg/3-iceberg-rest-catalog.md new file mode 100644 index 00000000..e7cf5719 --- /dev/null +++ b/.claude/skills/databricks-iceberg/3-iceberg-rest-catalog.md @@ -0,0 +1,107 @@ +# Iceberg REST Catalog (IRC) + +The Iceberg REST Catalog (IRC) is a REST API endpoint that lets external engines read and write Databricks-managed Iceberg data using the standard Apache Iceberg REST Catalog protocol. External tools connect to the IRC endpoint, authenticate, and receive vended credentials for direct cloud storage access. + +**Endpoint**: `https:///api/2.1/unity-catalog/iceberg-rest` + +> **Legacy endpoint warning**: The older `/api/2.1/unity-catalog/iceberg` endpoint is in maintenance mode and should not be used for new integrations. It was the original read-only endpoint documented for UniForm. All new integrations — both UniForm (Delta with Iceberg reads) and managed Iceberg tables — must use `/api/2.1/unity-catalog/iceberg-rest`. + +**Requirements**: Unity Catalog, external data access enabled on the workspace, DBR 16.1+ + +--- + +## Prerequisites + +### 1. Enable External Data Access + +External data access must be enabled for your workspace. This is typically configured by a workspace admin. + +### 2. Network Access to the IRC Endpoint + +External engines must reach the Databricks workspace over HTTPS (port 443). If the workspace has **IP access lists** enabled, the CIDR range(s) of the Iceberg client must be explicitly allowed — otherwise connections will fail regardless of correct credentials or grants. + +Check and manage IP access lists: +- Admin console: **Settings → Security → IP access list** +- REST API: `GET /api/2.0/ip-access-lists` to inspect, `POST /api/2.0/ip-access-lists` to add ranges + +> **Common symptom**: Connections time out or return `403 Forbidden` even with valid credentials and correct grants. IP access list misconfiguration is a frequent root cause — check this before debugging auth. + +### 3. Grant EXTERNAL USE SCHEMA + +The connecting principal (user or service principal) must have the `EXTERNAL USE SCHEMA` grant on each schema they want to access: + +```sql +-- Grant to a user +GRANT EXTERNAL USE SCHEMA ON SCHEMA my_catalog.my_schema TO `user@example.com`; + +-- Grant to a service principal +GRANT EXTERNAL USE SCHEMA ON SCHEMA my_catalog.my_schema TO `my-service-principal`; + +-- Grant to a group +GRANT EXTERNAL USE SCHEMA ON SCHEMA my_catalog.my_schema TO `data-engineers`; +``` + +> **Important**: `EXTERNAL USE SCHEMA` is separate from `SELECT` or `MODIFY` grants. A user needs both data permissions AND the external use grant. + +--- + +## Authentication + +### Personal Access Token (PAT) + +``` +Authorization: Bearer +``` + +### OAuth (M2M) + +For service-to-service authentication, use OAuth with a service principal: + +1. Create a service principal in the Databricks account +2. Generate an OAuth secret +3. Use the OAuth token endpoint to get an access token +4. Pass the access token as a Bearer token + +--- + +## Read/Write Capability Matrix + +| Table Type | IRC Read | IRC Write | +|------------|:-:|:-:| +| Managed Iceberg (`USING ICEBERG`) | Yes | Yes | +| Delta + UniForm | Yes | No | +| Delta + Compatibility Mode | Yes | No | +| Foreign Iceberg Table | No | No | + +> **Key insight**: Only managed Iceberg tables support writes via IRC. UniForm and Compatibility Mode tables are read-only because the underlying format is Delta. + +--- + +## Credential Vending + +When an external engine connects via IRC, Databricks **vends temporary cloud credentials** (short-lived STS tokens for AWS, SAS tokens for Azure) so the engine can read/write data files directly in cloud storage. This is transparent to the client — the IRC protocol handles it automatically. + +Benefits: +- No need to configure cloud credentials in the external engine +- Credentials are scoped to the specific table and operation +- Credentials automatically expire (typically 1 hour) + +--- + +## Common Configuration Reference + +| Parameter | Value | +|-----------|-------| +| **Catalog type** | `rest` | +| **URI** | `https:///api/2.1/unity-catalog/iceberg-rest` | +| **Warehouse** | Unity Catalog catalog name (e.g., `my_catalog`) | +| **Token** | Databricks PAT or OAuth access token | +| **Credential vending** | Automatic (handled by the REST protocol) | + + +--- + +## Related + +- [4-snowflake-interop.md](4-snowflake-interop.md) — Snowflake reading Databricks via catalog integration (uses IRC) +- [5-external-engine-interop.md](5-external-engine-interop.md) — Per-engine connection configs: PyIceberg, OSS Spark, EMR, Flink, Kafka Connect, DuckDB, Trino diff --git a/.claude/skills/databricks-iceberg/4-snowflake-interop.md b/.claude/skills/databricks-iceberg/4-snowflake-interop.md new file mode 100644 index 00000000..2f9d9536 --- /dev/null +++ b/.claude/skills/databricks-iceberg/4-snowflake-interop.md @@ -0,0 +1,349 @@ +# Snowflake Interoperability + +Databricks and Snowflake can share Iceberg data bidirectionally. This file covers both directions: Snowflake reading Databricks-managed tables, and Databricks reading Snowflake-managed Iceberg tables. + +**Cloud scope**: AWS-primary examples. Azure/GCS differences noted where relevant. + +--- + +## Direction 1: Snowflake Reading Databricks + +Snowflake can read Databricks-managed Iceberg tables (managed Iceberg + UniForm + Compatibility Mode) through a **Catalog Integration** that connects to the Databricks Iceberg REST Catalog (IRC). + +### Step 1: Create a Catalog Integration in Snowflake + +`ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS` is required on AWS for Snowflake to receive temporary STS credentials from the Databricks IRC. Without it, Snowflake cannot access the underlying Parquet files. + +**PAT / Bearer token**: + +```sql +-- In Snowflake +CREATE OR REPLACE CATALOG INTEGRATION databricks_catalog_int + CATALOG_SOURCE = ICEBERG_REST + TABLE_FORMAT = ICEBERG + CATALOG_NAMESPACE = 'my_schema' -- UC schema (default namespace) + REST_CONFIG = ( + CATALOG_URI = 'https:///api/2.1/unity-catalog/iceberg-rest' + WAREHOUSE = '' -- UC catalog name + ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS + ) + REST_AUTHENTICATION = ( + TYPE = BEARER + BEARER_TOKEN = '' + ) + REFRESH_INTERVAL_SECONDS = 300 + ENABLED = TRUE; +``` + +**OAuth (recommended for production)**: + +```sql +CREATE OR REPLACE CATALOG INTEGRATION databricks_catalog_int + CATALOG_SOURCE = ICEBERG_REST + TABLE_FORMAT = ICEBERG + CATALOG_NAMESPACE = 'my_schema' + REST_CONFIG = ( + CATALOG_URI = 'https:///api/2.1/unity-catalog/iceberg-rest' + WAREHOUSE = '' + ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS + ) + REST_AUTHENTICATION = ( + TYPE = OAUTH + OAUTH_CLIENT_ID = '' + OAUTH_CLIENT_SECRET = '' + OAUTH_TOKEN_URI = 'https:///oidc/v1/token' + OAUTH_ALLOWED_SCOPES = ('all-apis', 'sql') + ) + REFRESH_INTERVAL_SECONDS = 300 + ENABLED = TRUE; +``` + +> **Grant on the Databricks side**: The principal used for authentication needs these privileges in Unity Catalog: +> - `USE CATALOG` on the catalog +> - `USE SCHEMA` on the schema +> - `EXTERNAL USE SCHEMA` on the schema — this is the key privilege that enables external engines to access tables via IRC +> - `SELECT` on the target tables (or schema/catalog for broader access) +> +> Missing `EXTERNAL USE SCHEMA` causes a `Failed to retrieve credentials` error in Snowflake. + +### Step 2: External Volume (Azure/GCS Only) + +On **AWS with vended credentials**, no external volume is needed — Databricks IRC vends temporary STS credentials automatically. + +On **Azure** or **GCS**, you must create an external volume in Snowflake because vended credentials are not supported for those clouds: + +```sql +-- Azure example (in Snowflake) +CREATE OR REPLACE EXTERNAL VOLUME databricks_ext_vol + STORAGE_LOCATIONS = ( + ( + NAME = 'azure_location' + STORAGE_BASE_URL = 'azure://myaccount.blob.core.windows.net/my-container/iceberg/' + AZURE_TENANT_ID = '' + ) + ); +``` + +### Step 3: Expose Tables in Snowflake + +Two approaches available. **Linked catalog** is preferred — it exposes all tables in the namespace at once and updates automatically. + +**Option A: Linked Catalog Database (preferred)** + +```sql +-- Verify namespaces are visible (should return your UC schemas) +SELECT SYSTEM$LIST_NAMESPACES_FROM_CATALOG('databricks_catalog_int', '', 0); + +-- Create a linked catalog database exposing all tables in the namespace +CREATE DATABASE my_snowflake_db + LINKED_CATALOG = ( + CATALOG = 'databricks_catalog_int', + ALLOWED_NAMESPACES = ('my_schema') -- UC schema + ); + +-- Check link health (executionState should be "RUNNING" with empty failureDetails) +SELECT SYSTEM$CATALOG_LINK_STATUS('my_snowflake_db'); + +-- Query +SELECT * FROM my_snowflake_db."my_schema"."my_table" +WHERE event_date >= '2025-01-01'; +``` + +**Option B: Individual Table Reference (legacy)** + +```sql +-- AWS (vended creds — no EXTERNAL_VOLUME needed) +CREATE ICEBERG TABLE my_snowflake_db.my_schema.events + CATALOG = 'databricks_catalog_int' + CATALOG_TABLE_NAME = 'events'; + +-- Azure/GCS (EXTERNAL_VOLUME required) +CREATE ICEBERG TABLE my_snowflake_db.my_schema.events + CATALOG = 'databricks_catalog_int' + CATALOG_TABLE_NAME = 'events' + EXTERNAL_VOLUME = 'databricks_ext_vol'; + +-- Query +SELECT * FROM my_snowflake_db.my_schema.events +WHERE event_date >= '2025-01-01'; +``` + +### Key Gotchas + +#### Workspace IP Access Lists Must Allow Snowflake Egress IPs + +If the Databricks workspace has **IP access lists** enabled, Snowflake's outbound NAT IPs must be added to the allowlist. Snowflake connects to the Databricks IRC endpoint (`/api/2.1/unity-catalog/iceberg-rest`) over HTTPS (port 443), and a blocked IP produces connection timeouts or `403` errors that can look like auth failures. + + +> **Diagnosis tip**: If the catalog integration shows `ENABLED = TRUE` but `SYSTEM$CATALOG_LINK_STATUS` returns a connection error (not a credentials error), IP access lists are the first thing to check. + +#### REFRESH_INTERVAL_SECONDS Is Per-Integration, Not Per-Table + +The `REFRESH_INTERVAL_SECONDS` setting on the catalog integration controls how often Snowflake polls the Databricks IRC for metadata changes. This applies to **all tables** using that integration — you cannot set different refresh intervals per table. + +- Lower values = fresher data but more API calls +- Default: 300 seconds (5 minutes) +- Minimum: 60 seconds + +#### 1000-Commit Limit + +For Iceberg tables created from Delta files in object storage, Snowflake processes a maximum of 1000 Delta commit files each time you refresh a table using CREATE/ALTER ICEBERG TABLE … REFRESH or an automatic refresh; if the table has more than 1000 commit files since the last checkpoint, you can perform additional refreshes and each refresh continues from where the previous one stopped. The 1000‑commit limit applies only to Delta commit files after the latest Delta checkpoint file, and does not limit how many commits the catalog integration can ultimately synchronize over multiple refreshes + +**Mitigations**: +- Enable Predictive Optimization (auto-compaction reduces commit frequency) +- Batch writes instead of high-frequency micro-batches +- Run `OPTIMIZE` and `VACUUM` to consolidate metadata manually if needed. + +--- + +## Direction 2: Databricks Reading Snowflake + +Databricks can read Snowflake-managed Iceberg tables through a **foreign catalog** that connects to Snowflake's Iceberg catalog. Snowflake Iceberg tables are stored in external volumes (cloud storage), so Databricks reads the Iceberg's Parquet files directly — no Snowflake compute required. + +**Assumption**: A Snowflake-managed Iceberg table already exists, created with `CATALOG = 'SNOWFLAKE'` pointing to an external volume: + +```sql +-- In Snowflake — prerequisite table +CREATE ICEBERG TABLE sensor_readings ( + device_id INT, + device_value STRING +) + CATALOG = 'SNOWFLAKE' + EXTERNAL_VOLUME = 'ICEBERG_SHARED_VOL' + BASE_LOCATION = 'sensor_readings/'; + +INSERT INTO sensor_readings VALUES (1, 'value01'), (2, 'value02'); + +SELECT * FROM sensor_readings; +``` + +`CATALOG = 'SNOWFLAKE'` means Snowflake manages the Iceberg metadata. The data files land in the external volume at the `BASE_LOCATION` sub-path. The steps below set up Databricks to read this table. + +### Step 1: Find Snowflake External Volume Path + +Before setting up the Databricks side, run this in Snowflake to get the S3/ADLS/GCS path where Snowflake stores its Iceberg data. You'll need this path for Steps 2 and 4. + +```sql +-- In Snowflake +DESCRIBE EXTERNAL VOLUME ; +-- Note the STORAGE_BASE_URL value (e.g. s3://my-bucket/snowflake-iceberg/) +``` + +### Step 2: Create a Storage Credential + +Create a storage credential for the cloud storage where Snowflake stores its Iceberg data. Assuming that the IAM role already exists. Follow the documentation for details (https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual) + +```bash +# In Databricks CLI (AWS example) +databricks storage-credentials create snowflake_storage_cred \ + --aws-iam-role-arn "arn:aws:iam::123456789012:role/snowflake-data-access" +``` + +### Step 3: Create an External Location + +The external location must point to the **root** of the bucket (not a sub-path), so that all Snowflake external volume paths fall under it. + +> **Fallback mode**: You do not need this external-location fallback enabled to read Snowflake‑created Iceberg tables via catalog federation. It only affects how storage credentials are resolved for paths, not whether Snowflake Iceberg federation works. + +```sql +-- In Databricks (URL should be the bucket root, not a sub-path) +CREATE EXTERNAL LOCATION snowflake_data +URL 's3://snowflake-iceberg-bucket/' +WITH (CREDENTIAL snowflake_storage_cred); +``` + +### Step 4: Create a Snowflake Connection + +```sql +-- In Databricks +CREATE CONNECTION snowflake_conn +TYPE SNOWFLAKE +OPTIONS ( + 'host' = '.snowflakecomputing.com', + 'user' = '', + 'password' = '', + 'sfWarehouse' = '' +); +``` + +### Step 5: Create a Foreign Catalog + +Two mandatory fields beyond `database`: + +- **`authorized_paths`**: The path(s) where Snowflake stores Iceberg table files — from `STORAGE_BASE_URL` in `DESCRIBE EXTERNAL VOLUME`. Databricks can only read Iceberg tables whose data falls under these paths. +- **`storage_root`**: Where Databricks stores catalog metadata for Iceberg reads. Must point to an existing external location. This is required — the foreign catalog creation will fail without it. + +```sql +-- In Databricks +CREATE FOREIGN CATALOG snowflake_iceberg +USING CONNECTION snowflake_conn +OPTIONS ( + 'catalog' = '', + 'authorized_paths' = 's3://snowflake-iceberg-bucket/snowflake-iceberg/', + 'storage_root' = 's3://snowflake-iceberg-bucket/uc-metadata/' +); +``` + +> **UI workflow note**: The Databricks connection wizard (Catalog Explorer → Add connection → Snowflake) will prompt for authorized paths and storage location in the form and create the foreign catalog automatically. The SQL above is the equivalent DDL. + +### Step 6: Refresh, Verify, and Query + +```sql +-- Refresh to discover tables +REFRESH FOREIGN CATALOG snowflake_iceberg; + +-- Verify provider type before querying at scale: +-- Provider = Iceberg → Databricks reads directly from cloud storage (cheap) +-- Provider = Snowflake → double compute via JDBC (Snowflake + Databricks) +DESCRIBE EXTENDED snowflake_iceberg.my_schema.my_table; + +-- Query +SELECT * FROM snowflake_iceberg.my_schema.my_table +WHERE created_at >= '2025-01-01'; +``` + +### Compute Cost Matrix + +| Snowflake Table Type | Databricks Read | Compute Cost | +|---------------------|:-:|---| +| **Snowflake Iceberg table** | Yes | Databricks compute only (reads data files directly from cloud storage) | +| **Snowflake native table** | Yes (via federation) | Double compute — Snowflake runs the query, Databricks processes the result | + +> **Key insight**: Snowflake Iceberg tables are more cost-efficient to read from Databricks because Databricks reads the Parquet files directly. Native Snowflake tables require Snowflake to run the scan. + + +--- + +## Full AWS Example: Snowflake Reading Databricks + +```sql +-- ======================================== +-- DATABRICKS SIDE (run in Databricks) +-- ======================================== + +-- 1. Create a managed Iceberg table (v2 — disable DVs and row tracking for CLUSTER BY) +CREATE TABLE main.sales.orders ( + order_id BIGINT, + customer_id BIGINT, + amount DECIMAL(10,2), + order_date DATE +) +USING ICEBERG +TBLPROPERTIES ( + 'delta.enableDeletionVectors' = false, + 'delta.enableRowTracking' = false +) +CLUSTER BY (order_date); + +-- 2. Grant external access to the service principal used in Snowflake catalog integration +GRANT EXTERNAL USE SCHEMA ON SCHEMA main.sales TO `snowflake-service-principal`; + +-- ======================================== +-- SNOWFLAKE SIDE (run in Snowflake) +-- ======================================== + +-- 3. Create catalog integration (ACCESS_DELEGATION_MODE required for vended creds on AWS) +CREATE OR REPLACE CATALOG INTEGRATION databricks_int + CATALOG_SOURCE = ICEBERG_REST + TABLE_FORMAT = ICEBERG + CATALOG_NAMESPACE = 'sales' + REST_CONFIG = ( + CATALOG_URI = 'https://my-workspace.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest' + WAREHOUSE = 'main' + ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS + ) + REST_AUTHENTICATION = ( + TYPE = OAUTH + OAUTH_CLIENT_ID = '' + OAUTH_CLIENT_SECRET = '' + OAUTH_TOKEN_URI = 'https://my-workspace.cloud.databricks.com/oidc/v1/token' + OAUTH_ALLOWED_SCOPES = ('all-apis', 'sql') + ) + REFRESH_INTERVAL_SECONDS = 300 + ENABLED = TRUE; + +-- 4. Verify schemas are visible +SELECT SYSTEM$LIST_NAMESPACES_FROM_CATALOG('databricks_int', '', 0); + +-- 5. Create linked catalog database (exposes all tables in the namespace) +CREATE DATABASE analytics + LINKED_CATALOG = ( + CATALOG = 'databricks_int', + ALLOWED_NAMESPACES = ('sales') + ); + +-- 6. Check link health +SELECT SYSTEM$CATALOG_LINK_STATUS('analytics'); + +-- 7. Query (schema and table names are case-sensitive) +SELECT order_date, SUM(amount) AS daily_revenue +FROM analytics."sales"."orders" +GROUP BY order_date +ORDER BY order_date DESC; +``` + +--- + +## Related + +- [3-iceberg-rest-catalog.md](3-iceberg-rest-catalog.md) — IRC endpoint details and authentication diff --git a/.claude/skills/databricks-iceberg/5-external-engine-interop.md b/.claude/skills/databricks-iceberg/5-external-engine-interop.md new file mode 100644 index 00000000..ecafcbeb --- /dev/null +++ b/.claude/skills/databricks-iceberg/5-external-engine-interop.md @@ -0,0 +1,206 @@ +# External Engine Interoperability + +This file covers connecting external engines to Databricks via the Iceberg REST Catalog (IRC). Each engine section includes the minimum configuration needed to read (and where supported, write) Databricks-managed Iceberg tables. + +**Prerequisites for all engines**: +- Databricks workspace with external data access enabled +- `EXTERNAL USE SCHEMA` granted on target schemas +- PAT or OAuth (service principal) credentials for authentication with the required permissions. +- **Network access**: The client must reach the Databricks workspace on HTTPS (port 443). If workspace **IP access lists** are enabled, add the client's egress CIDR to the allowlist — this is a common setup issue that blocks connectivity even when credentials and grants are correct. + +See [3-iceberg-rest-catalog.md](3-iceberg-rest-catalog.md) for IRC endpoint details. + +--- + +## PyIceberg + +PyIceberg is a Python library for reading and writing Iceberg tables without Spark. + +### Installation + +Upgrade both packages explicitly — if `pyarrow` (v15) is too old, it causes write errors. Also install `adlfs` for Azure storage access: + +```bash +pip install --upgrade "pyiceberg>=0.9,<0.10" "pyarrow>=17,<20" +pip install adlfs +``` + +For non-Databricks environments: + +```bash +pip install "pyiceberg[pyarrow]>=0.9" +``` + +### Connect to Catalog + +The `warehouse` parameter pins the catalog, so all subsequent table identifiers use `.` (not `..
`): + +```python +from pyiceberg.catalog import load_catalog + +catalog = load_catalog( + "uc", + uri="https:///api/2.1/unity-catalog/iceberg-rest", + warehouse="", # Unity Catalog catalog name + token="", +) +``` + +### Read Table + +```python +# Load table — identifier is .
because 'warehouse' pins the UC catalog +tbl = catalog.load_table(".
") + +# Inspect schema and current snapshot +print(tbl) # schema, partitioning, snapshot summary +print(tbl.current_snapshot()) # snapshot metadata + +# Read sample rows +df = tbl.scan(limit=10).to_pandas() +print(df.head()) + +# Pushdown filter (SQL-style filter strings are supported) +df = tbl.scan( + row_filter="event_date >= '2025-01-01'", + limit=1000, +).to_pandas() + +# Read as Arrow +arrow_table = tbl.scan().to_arrow() +``` + +### Append Data + +```python +import pyarrow as pa +from pyiceberg.catalog import load_catalog + +catalog = load_catalog( + "uc", + uri="https:///api/2.1/unity-catalog/iceberg-rest", + warehouse="", + token="", +) + +tbl = catalog.load_table(".
") + +# Schema must match the Iceberg table schema exactly — use explicit Arrow types +# PyArrow defaults to int64; if the Iceberg table uses int (32-bit), cast explicitly +arrow_schema = pa.schema([ + pa.field("id", pa.int32()), + pa.field("name", pa.string()), + pa.field("qty", pa.int32()), +]) + +rows = [ + {"id": 1, "name": "foo", "qty": 10}, + {"id": 2, "name": "bar", "qty": 20}, +] +arrow_tbl = pa.Table.from_pylist(rows, schema=arrow_schema) + +tbl.append(arrow_tbl) + +# Verify +print("Current snapshot:", tbl.current_snapshot()) +``` + +--- + +## OSS Apache Spark + +> **CRITICAL**: Only configure this **outside** Databricks Runtime. Inside DBR, use the built-in Iceberg support — do NOT install the Iceberg library. + +### Dependencies + +Two JARs are required: the Spark runtime and a cloud-specific bundle for object storage access. Choose the bundle matching your Databricks metastore's cloud: + +| Cloud | Bundle | +|-------|--------| +| AWS | `org.apache.iceberg:iceberg-aws-bundle:` | +| Azure | `org.apache.iceberg:iceberg-azure-bundle:` | +| GCP | `org.apache.iceberg:iceberg-gcp-bundle:` | + +### Spark Session Configuration + +The Databricks docs recommend OAuth2 (service principal) for external Spark connections. Set `rest.auth.type=oauth2` and provide the OAuth2 server URI, credential, and scope: + +```python +from pyspark.sql import SparkSession + +WORKSPACE_URL = "https://" +UC_CATALOG_NAME = "" +OAUTH_CLIENT_ID = "" +OAUTH_CLIENT_SECRET = "" +CATALOG_ALIAS = "uc" # arbitrary name used to reference this catalog in Spark SQL +ICEBERG_VER = "1.7.1" + +RUNTIME = f"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{ICEBERG_VER}" +CLOUD_BUNDLE = f"org.apache.iceberg:iceberg-aws-bundle:{ICEBERG_VER}" # or azure/gcp-bundle + +spark = ( + SparkSession.builder + .appName("uc-iceberg") + .config("spark.jars.packages", f"{RUNTIME},{CLOUD_BUNDLE}") + .config("spark.sql.extensions", + "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}", + "org.apache.iceberg.spark.SparkCatalog") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}.type", "rest") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}.rest.auth.type", "oauth2") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}.uri", + f"{WORKSPACE_URL}/api/2.1/unity-catalog/iceberg-rest") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}.oauth2-server-uri", + f"{WORKSPACE_URL}/oidc/v1/token") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}.credential", + f"{OAUTH_CLIENT_ID}:{OAUTH_CLIENT_SECRET}") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}.scope", "all-apis") + .config(f"spark.sql.catalog.{CATALOG_ALIAS}.warehouse", UC_CATALOG_NAME) + .getOrCreate() +) + +# List schemas +spark.sql(f"SHOW NAMESPACES IN {CATALOG_ALIAS}").show(truncate=False) + +# Query +spark.sql(f"SELECT * FROM {CATALOG_ALIAS}..
").show() + +# Write (managed Iceberg tables only) +df.writeTo(f"{CATALOG_ALIAS}..
").append() +``` + +### Spark SQL + +```sql +-- List schemas +SHOW NAMESPACES IN uc; + +-- Query +SELECT * FROM uc..
; + +-- Insert +INSERT INTO uc..
VALUES (1, 'foo', 10); +``` + +--- + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| **Connection timeout or `403 Forbidden` with valid credentials** | Workspace IP access list is blocking the client — add the client's egress CIDR to the allowlist (admin console: **Settings → Security → IP access list**) | +| **`403 Forbidden`** | Check `EXTERNAL USE SCHEMA` grant and token validity | +| **`Table not found`** | Verify the `warehouse` config matches the UC catalog name; check schema and table names | +| **Class conflict in DBR** | You installed an Iceberg library in Databricks Runtime — remove it; DBR has built-in support | +| **Credential vending failure** | Ensure external data access is enabled on the workspace | +| **Slow reads** | Check if table needs compaction (`OPTIMIZE`); large numbers of small files degrade performance | +| **v3 table incompatibility** | Upgrade to Iceberg library 1.9.0+ for v3 support; older versions cannot read v3 tables | +| **PyArrow schema mismatch** | Cast to explicit types (e.g., `pa.int32()`) when the Iceberg table schema uses 32-bit integers | +| **PyIceberg write error on serverless** | Upgrade pyarrow (`>=17`) and install `adlfs` — the bundled pyarrow v15 is incompatible | + +--- + +## Related + +- [3-iceberg-rest-catalog.md](3-iceberg-rest-catalog.md) — IRC endpoint details, auth, credential vending +- [4-snowflake-interop.md](4-snowflake-interop.md) — Snowflake-specific integration diff --git a/.claude/skills/databricks-iceberg/SKILL.md b/.claude/skills/databricks-iceberg/SKILL.md new file mode 100644 index 00000000..3c8a1cb4 --- /dev/null +++ b/.claude/skills/databricks-iceberg/SKILL.md @@ -0,0 +1,148 @@ +--- +name: databricks-iceberg +description: "Apache Iceberg tables on Databricks — Managed Iceberg tables, External Iceberg Reads (fka Uniform), Compatibility Mode, Iceberg REST Catalog (IRC), Iceberg v3, Snowflake interop, PyIceberg, OSS Spark, external engine access and credential vending. Use when creating Iceberg tables, enabling External Iceberg Reads (uniform) on Delta tables (including Streaming Tables and Materialized Views via compatibility mode), configuring external engines to read Databricks tables via Unity Catalog IRC, integrating with Snowflake catalog to read Foreign Iceberg tables" +--- + +# Apache Iceberg on Databricks + +Databricks provides multiple ways to work with Apache Iceberg: native managed Iceberg tables, UniForm for Delta-to-Iceberg interoperability, and the Iceberg REST Catalog (IRC) for external engine access. + +--- + +## Critical Rules (always follow) + +- **MUST** use Unity Catalog — all Iceberg features require UC-enabled workspaces +- **MUST NOT** install an Iceberg library into Databricks Runtime (DBR includes built-in Iceberg support; adding a library causes version conflicts) +- **MUST NOT** set `write.metadata.path` or `write.metadata.previous-versions-max` — Databricks manages metadata locations automatically; overriding causes corruption +- **MUST** determine which Iceberg pattern fits the use case before writing code — see the [When to Use](#when-to-use) section below +- **MUST** know that both `PARTITIONED BY` and `CLUSTER BY` produce the same Iceberg metadata for external engines — UC maintains an Iceberg partition spec with partition fields corresponding to the clustering keys, so external engines reading via IRC see a partitioned Iceberg table (not Hive-style, but proper Iceberg partition fields) and can prune on those fields; internally UC uses those fields as liquid clustering keys; the only differences between the two syntaxes are: (1) `PARTITIONED BY` is standard Iceberg DDL (any engine can create the table), while `CLUSTER BY` is DBR-only DDL; (2) `PARTITIONED BY` **auto-handles** DV/row-tracking properties, while `CLUSTER BY` requires manual TBLPROPERTIES on v2 +- **MUST NOT** use expression-based partition transforms (`bucket()`, `years()`, `months()`, `days()`, `hours()`) with `PARTITIONED BY` on managed Iceberg tables — only plain column references are supported; expression transforms cause errors +- **MUST** disable deletion vectors and row tracking when using `CLUSTER BY` on Iceberg v2 tables — set `'delta.enableDeletionVectors' = false` and `'delta.enableRowTracking' = false` in TBLPROPERTIES (Iceberg v3 handles this automatically; `PARTITIONED BY` handles this automatically on both v2 and v3) + +--- + +## Key Concepts + +| Concept | Summary | +|---------|---------| +| **Managed Iceberg Table** | Native Iceberg table created with `USING ICEBERG` — full read/write in Databricks and via external Iceberg engines | +| **External Iceberg Reads (Uniform)** | Delta table that auto-generates Iceberg metadata — read as Iceberg externally, write as Delta internally | +| **Compatibility Mode** | UniForm variant for streaming tables and materialized views in SDP pipelines | +| **Iceberg REST Catalog (IRC)** | Unity Catalog's built-in REST endpoint implementing the Iceberg REST Catalog spec — lets external engines (Spark, PyIceberg, Snowflake) access UC-managed Iceberg data | +| **Iceberg v3** | Next-gen format (Beta, DBR 17.3+) — deletion vectors, VARIANT type, row lineage | + +--- + +## Quick Start + +### Create a Managed Iceberg Table + +```sql +-- No clustering +CREATE TABLE my_catalog.my_schema.events +USING ICEBERG +AS SELECT * FROM raw_events; + +-- PARTITIONED BY (recommended for cross-platform): standard Iceberg syntax, works on EMR/OSS Spark/Trino/Flink +-- auto-disables DVs and row tracking — no TBLPROPERTIES needed on v2 or v3 +CREATE TABLE my_catalog.my_schema.events +USING ICEBERG +PARTITIONED BY (event_date) +AS SELECT * FROM raw_events; + +-- CLUSTER BY on Iceberg v2 (DBR-only syntax): must manually disable DVs and row tracking +CREATE TABLE my_catalog.my_schema.events +USING ICEBERG +TBLPROPERTIES ( + 'delta.enableDeletionVectors' = false, + 'delta.enableRowTracking' = false +) +CLUSTER BY (event_date) +AS SELECT * FROM raw_events; + +-- CLUSTER BY on Iceberg v3 (DBR-only syntax): no TBLPROPERTIES needed +CREATE TABLE my_catalog.my_schema.events +USING ICEBERG +TBLPROPERTIES ('format-version' = '3') +CLUSTER BY (event_date) +AS SELECT * FROM raw_events; +``` + +### Enable UniForm on an Existing Delta Table + +```sql +ALTER TABLE my_catalog.my_schema.customers +SET TBLPROPERTIES ( + 'delta.columnMapping.mode' = 'name', + 'delta.enableIcebergCompatV2' = 'true', + 'delta.universalFormat.enabledFormats' = 'iceberg' +); +``` + +--- + +## Read/Write Capability Matrix + +| Table Type | Databricks Read | Databricks Write | External IRC Read | External IRC Write | +|------------|:-:|:-:|:-:|:-:| +| Managed Iceberg (`USING ICEBERG`) | Yes | Yes | Yes | Yes | +| Delta + UniForm | Yes (as Delta) | Yes (as Delta) | Yes (as Iceberg) | No | +| Delta + Compatibility Mode | Yes (as Delta) | Yes | Yes (as Iceberg) | No | + +--- + +## Reference Files + +| File | Summary | Keywords | +|------|---------|----------| +| [1-managed-iceberg-tables.md](1-managed-iceberg-tables.md) | Creating and managing native Iceberg tables — DDL, DML, Liquid Clustering, Predictive Optimization, Iceberg v3, limitations | CREATE TABLE USING ICEBERG, CTAS, MERGE, time travel, deletion vectors, VARIANT | +| [2-uniform-and-compatibility.md](2-uniform-and-compatibility.md) | Making Delta tables readable as Iceberg — UniForm for regular tables, Compatibility Mode for streaming tables and MVs | UniForm, universalFormat, Compatibility Mode, streaming tables, materialized views, SDP | +| [3-iceberg-rest-catalog.md](3-iceberg-rest-catalog.md) | Exposing Databricks tables to external engines via the IRC endpoint — auth, credential vending, IP access lists | IRC, REST Catalog, credential vending, EXTERNAL USE SCHEMA, PAT, OAuth | +| [4-snowflake-interop.md](4-snowflake-interop.md) | Bidirectional Snowflake-Databricks integration — catalog integration, foreign catalogs, vended credentials | Snowflake, catalog integration, external volume, vended credentials, REFRESH_INTERVAL_SECONDS | +| [5-external-engine-interop.md](5-external-engine-interop.md) | Connecting PyIceberg, OSS Spark, AWS EMR, Apache Flink, and Kafka Connect via IRC | PyIceberg, OSS Spark, EMR, Flink, Kafka Connect, pyiceberg.yaml | + +--- + +## When to Use + +- **Creating a new Iceberg table** → [1-managed-iceberg-tables.md](1-managed-iceberg-tables.md) +- **Making an existing Delta table readable as Iceberg** → [2-uniform-and-compatibility.md](2-uniform-and-compatibility.md) +- **Making a streaming table or MV readable as Iceberg** → [2-uniform-and-compatibility.md](2-uniform-and-compatibility.md) (Compatibility Mode section) +- **Choosing between Managed Iceberg vs UniForm vs Compatibility Mode** → decision table in [2-uniform-and-compatibility.md](2-uniform-and-compatibility.md) +- **Exposing Databricks tables to external engines via REST API** → [3-iceberg-rest-catalog.md](3-iceberg-rest-catalog.md) +- **Integrating Databricks with Snowflake (either direction)** → [4-snowflake-interop.md](4-snowflake-interop.md) +- **Connecting PyIceberg, OSS Spark, Flink, EMR, or Kafka** → [5-external-engine-interop.md](5-external-engine-interop.md) + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **No Change Data Feed (CDF)** | CDF is not supported on managed Iceberg tables. Use Delta + UniForm if you need CDF. | +| **UniForm async delay** | Iceberg metadata generation is asynchronous. After a write, there may be a brief delay before external engines see the latest data. Check status with `DESCRIBE EXTENDED table_name`. | +| **Compression codec change** | Managed Iceberg tables use `zstd` compression by default (not `snappy`). Older Iceberg readers that don't support zstd will fail. Verify reader compatibility or set `write.parquet.compression-codec` to `snappy`. | +| **Snowflake 1000-commit limit** | Snowflake's Iceberg catalog integration can only see the last 1000 Iceberg commits. High-frequency writers must compact metadata or Snowflake will lose visibility of older data. | +| **Deletion vectors with UniForm** | UniForm requires deletion vectors to be disabled (`delta.enableDeletionVectors = false`). If your table has deletion vectors enabled, disable them before enabling UniForm. | +| **No shallow clone for Iceberg** | `SHALLOW CLONE` is not supported for Iceberg tables. Use `DEEP CLONE` or `CREATE TABLE ... AS SELECT` instead. | +| **Version mismatch with external engines** | Ensure external engines use an Iceberg library version compatible with the format version of your tables. Iceberg v3 tables require Iceberg library 1.9.0+. | + +--- + +## Related Skills + +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** — catalog/schema management, governance, system tables +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** — SDP pipelines (streaming tables, materialized views with Compatibility Mode) +- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** — Python SDK and REST API for Databricks operations +- **[databricks-dbsql](../databricks-dbsql/SKILL.md)** — SQL warehouse features, query patterns + +--- + +## Resources + +- **[Iceberg Overview](https://docs.databricks.com/aws/en/iceberg/)** — main hub for Iceberg on Databricks +- **[UniForm](https://docs.databricks.com/aws/en/delta/uniform.html)** — Delta Universal Format +- **[Iceberg REST Catalog](https://docs.databricks.com/aws/en/external-access/iceberg)** — IRC endpoint and external engine access +- **[Compatibility Mode](https://docs.databricks.com/aws/en/external-access/compatibility-mode)** — UniForm for streaming tables and MVs +- **[Iceberg v3](https://docs.databricks.com/aws/en/iceberg/iceberg-v3)** — next-gen format features (Beta) +- **[Foreign Tables](https://docs.databricks.com/aws/en/query-data/foreign-tables.html)** — reading external catalog data diff --git a/.claude/skills/databricks-jobs/SKILL.md b/.claude/skills/databricks-jobs/SKILL.md index eae0754e..0f60a241 100644 --- a/.claude/skills/databricks-jobs/SKILL.md +++ b/.claude/skills/databricks-jobs/SKILL.md @@ -326,8 +326,8 @@ resources: ## Related Skills -- **[asset-bundles](../asset-bundles/SKILL.md)** - Deploy jobs via Databricks Asset Bundles -- **[spark-declarative-pipelines](../spark-declarative-pipelines/SKILL.md)** - Configure pipelines triggered by jobs +- **[databricks-bundles](../databricks-bundles/SKILL.md)** - Deploy jobs via Databricks Asset Bundles +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - Configure pipelines triggered by jobs ## Resources diff --git a/.claude/skills/databricks-jobs/task-types.md b/.claude/skills/databricks-jobs/task-types.md index a78b9a32..c5b06fbe 100644 --- a/.claude/skills/databricks-jobs/task-types.md +++ b/.claude/skills/databricks-jobs/task-types.md @@ -618,7 +618,7 @@ Define reusable Python environments for serverless tasks with custom pip depende > **IMPORTANT:** The `client` field is **required** in the environment `spec`. It specifies the > base serverless environment version. Use `"4"` as the value. Without it, the API returns: > `"Either base environment or version must be provided for environment"`. -> The MCP `create_job` tool auto-injects `client: "4"` if omitted, but CLI/SDK calls require it explicitly. +> The MCP `manage_jobs` tool (action="create") auto-injects `client: "4"` if omitted, but CLI/SDK calls require it explicitly. ### DABs YAML diff --git a/.claude/skills/databricks-lakebase-autoscale/SKILL.md b/.claude/skills/databricks-lakebase-autoscale/SKILL.md new file mode 100644 index 00000000..8d7dd6f5 --- /dev/null +++ b/.claude/skills/databricks-lakebase-autoscale/SKILL.md @@ -0,0 +1,133 @@ +--- +name: databricks-lakebase-autoscale +description: "Patterns and best practices for Lakebase Autoscaling (next-gen managed PostgreSQL). Use when creating or managing Lakebase Autoscaling projects, configuring autoscaling compute or scale-to-zero, working with database branching for dev/test workflows, implementing reverse ETL via synced tables, or connecting applications to Lakebase with OAuth credentials." +--- + +# Lakebase Autoscaling + +Lakebase Autoscaling is Databricks' next-generation managed PostgreSQL service for OLTP workloads: autoscaling compute, database branching, scale-to-zero, instant restore, and Delta-to-Postgres synced tables. + +Use this skill when creating/managing Lakebase Autoscaling projects, branches, endpoints/computes, credentials, reverse ETL synced tables, or app connections. + +## Core framing + +> **There is no separate Python “Lakebase SDK.”** Use `databricks-sdk` for management and for minting short-lived database credentials with `WorkspaceClient().postgres.generate_database_credential(...)`; use standard Postgres drivers (`psycopg`, SQLAlchemy, JDBC, `pgx`, etc.) for SQL. + +| Language | Credential / management SDK | DB driver / wrapper | +|---|---|---| +| **Python** | `databricks-sdk` `WorkspaceClient().postgres` | `psycopg[binary,pool]` canonical; SQLAlchemy supported | +| **Node/TS** | `@databricks/lakebase` convenience wrapper, Autoscaling only | Wrapper manages `pg` pool | +| **Java/Go** | Databricks SDK for Java/Go | Standard JDBC / `pgx` | + +## Lead connection pattern + +For production Python apps, start with: + +1. `psycopg_pool.ConnectionPool` +2. `connection_class=OAuthConnection`, where `OAuthConnection(psycopg.Connection).connect()` calls `w.postgres.generate_database_credential(endpoint=...)` +3. `max_lifetime=2700` + +This is the canonical pattern from the official Databricks Apps + Lakebase Autoscaling tutorial lineage and `databricks-ai-bridge`: no background token thread; physical connections get fresh credentials when opened/recycled. + +Prefer `max_lifetime=2700` as a defensive 45-minute recycle before 1-hour token expiry. The official tutorial does not set `max_lifetime`; `databricks-ai-bridge` uses `2700`. + +See `connections.md`. + +## Critical auth warning + +Do **not** use `WorkspaceClient().config.token`, `w.config.oauth_token().access_token`, or any workspace-scoped OAuth token as the Postgres password. It will fail at Postgres login. + +Use: + +```python +cred = WorkspaceClient().postgres.generate_database_credential(endpoint=endpoint_name) +password = cred.token +``` + +That token is Lakebase-scoped and is used as the Postgres password with `sslmode=require`. + +## Resource model + +```text +Project + └── Branches + ├── Endpoint/Compute: primary read-write endpoint + ├── Read replicas: optional read-only endpoints + ├── Roles + └── Databases + └── Schemas/Tables +``` + +Canonical names: + +```text +projects/{project_id} +projects/{project_id}/branches/{branch_id} +projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} +``` + +Defaults on project creation: +- default branch: `production` +- default database: `databricks_postgres` +- primary read-write endpoint/compute +- Postgres role for the creator’s Databricks identity + +Key SDK namespace: `WorkspaceClient().postgres`. + +Most create/update/delete calls return long-running operations; call `.wait()`. + +## Lakebase Autoscaling vs Provisioned + +| Aspect | Provisioned | Autoscaling | +|---|---|---| +| SDK module | `w.database` | `w.postgres` | +| Top-level resource | Instance | Project | +| Capacity | fixed CU tiers, ~16 GB/CU | 0.5–112 CU, ~2 GB/CU | +| Branching | no | yes | +| Scale-to-zero | no | yes | +| Operations | mostly synchronous | LROs; use `.wait()` | +| Reverse ETL | synced tables | synced tables | +| Read replicas | readable secondaries | dedicated read-only endpoints | + +## Non-obvious facts to preserve + +- Postgres versions: **16 and 17**. +- AWS regions: `us-east-1`, `us-east-2`, `eu-central-1`, `eu-west-1`, `eu-west-2`, `ap-south-1`, `ap-southeast-1`, `ap-southeast-2`. +- Azure beta regions: `eastus2`, `westeurope`, `westus`. +- Autoscaling computes: 0.5–32 CU with `max - min <= 16`. +- Fixed-size always-on computes: 40–112 CU. +- Autoscaling CU ≈ 2 GB RAM. +- `sslmode=require` on all driver connections. +- Endpoint host comes from `w.postgres.get_endpoint(...).status.hosts.host`. +- GET responses often return effective properties under `status`; create/update payloads use `spec`. +- All update calls need a `FieldMask`. +- Scale-to-zero wake-up is automatic but apps should retry. +- Connections can be closed by platform timeouts: 24-hour idle timeout and 3-day max connection lifetime. +- macOS DNS can fail on long Lakebase hostnames; if so, resolve to IP and pass both `host` and `hostaddr` to psycopg. +- Triggered/Continuous synced tables require Delta Change Data Feed. +- Reverse ETL is Delta-to-Postgres only; not Postgres-to-Delta. + +## Task files + +- `connections.md` — app/notebook connection patterns and credential rotation. +- `operations.md` — project, branch, endpoint/compute, scale-to-zero, limits, MCP mapping. +- `reverse-etl.md` — synced tables from Delta Lake to Lakebase. + +## SDK / package versions + +```bash +pip install -U "databricks-sdk>=0.81.0" "psycopg[binary,pool]>=3.1" "sqlalchemy>=2" +``` + +Use SQLAlchemy URL prefix `postgresql+psycopg://...` for psycopg3. + +## Current limitations + +Not yet supported or not equivalent to Provisioned: +- High availability with readable secondaries; use read replicas instead. +- Databricks Apps UI integration may lag; Apps can connect manually via credentials/resource env vars. +- Feature Store integration. +- Stateful AI-agent memory integrations. +- Postgres-to-Delta sync. +- Custom billing tags / serverless budget policies. +- Direct migration from Lakebase Provisioned; use `pg_dump`/`pg_restore` or reverse ETL patterns where appropriate. diff --git a/.claude/skills/databricks-lakebase-autoscale/connections.md b/.claude/skills/databricks-lakebase-autoscale/connections.md new file mode 100644 index 00000000..0831a788 --- /dev/null +++ b/.claude/skills/databricks-lakebase-autoscale/connections.md @@ -0,0 +1,212 @@ +# Lakebase Autoscaling connection patterns + +Order of preference: + +1. **Canonical:** `psycopg_pool.ConnectionPool` + `OAuthConnection` subclass + `max_lifetime=2700`. +2. **SQLAlchemy:** official `do_connect` auth hook; optionally rely on `pool_recycle`/`dispose()` rather than a background token loop. +3. **Direct `psycopg.connect`:** notebooks/one-shot scripts under 1 hour. +4. **Static Postgres URL/native password:** local/dev tools only, or tools unable to rotate OAuth credentials. + +## Authentication facts + +Lakebase OAuth database credentials: +- Mint with `WorkspaceClient().postgres.generate_database_credential(endpoint=...)`. +- Use `cred.token` as the Postgres password. +- Expire after about 1 hour. +- Expiry is enforced at login; already-open connections continue until closed by pool/platform timeouts. + +Critical warning: + +```python +# ✅ Lakebase-scoped credential: works for Postgres login +cred = w.postgres.generate_database_credential(endpoint=endpoint_name) +password = cred.token + +# ❌ Workspace-scoped token: fails at Postgres login +password = w.config.oauth_token().access_token +# also do not use WorkspaceClient().config.token +``` + +Always connect with `sslmode=require`. + +## 1. Canonical: psycopg pool + OAuthConnection + +Use for production Databricks Apps and most Python services. + +Key mechanics: +- The pool calls `OAuthConnection.connect()` whenever it opens a physical connection: initial fill, growth under load, recycle, replacement after failure. +- `connect()` mints a fresh Lakebase token just-in-time and injects it as `password`. +- `max_lifetime=2700` recycles physical connections after 45 minutes, before 1-hour token expiry. +- No background refresh thread/task is needed. + +Minimal skeleton: + +```python +import os +import psycopg +from psycopg_pool import ConnectionPool +from databricks.sdk import WorkspaceClient + +w = WorkspaceClient() + +class OAuthConnection(psycopg.Connection): + @classmethod + def connect(cls, conninfo="", **kwargs): + cred = w.postgres.generate_database_credential( + endpoint=os.environ["ENDPOINT_NAME"] + ) + kwargs["password"] = cred.token + return super().connect(conninfo, **kwargs) + +pool = ConnectionPool( + conninfo=( + f"dbname={os.environ['PGDATABASE']} " + f"user={os.environ['PGUSER']} " + f"host={os.environ['PGHOST']} " + f"port={os.environ.get('PGPORT', '5432')} " + f"sslmode={os.environ.get('PGSSLMODE', 'require')}" + ), + connection_class=OAuthConnection, + min_size=1, + max_size=10, + max_lifetime=2700, + open=True, +) +``` + +Prefer `2700`; it is a defensive convention. The official Databricks tutorial leaves `max_lifetime` unset; `databricks-ai-bridge` uses `2700`. + +For FastAPI or explicit startup: +- instantiate with `open=False` +- call `pool.open(wait=True, timeout=30.0)` in lifespan/startup +- call `pool.close()` on shutdown + +This also avoids relying on implicit open behavior. + +## Databricks Apps environment variables + +When adding a Lakebase/Postgres resource to a Databricks App, these are auto-injected for the **first** DB resource: + +```text +PGAPPNAME +PGHOST +PGPORT +PGDATABASE +PGUSER +PGSSLMODE +``` + +Gotchas: +- `PGUSER` is typically the app service principal client ID. +- Only the first database resource is auto-injected; additional resources need explicit `valueFrom`. +- `ENDPOINT_NAME` is **not** auto-injected. Add it manually because `generate_database_credential(endpoint=...)` requires the full endpoint path: + +```yaml +env: + - name: ENDPOINT_NAME + value: "projects//branches//endpoints/" +``` + +## 2. SQLAlchemy: official `do_connect` hook + +Use when the app is already built around SQLAlchemy. + +Important distinction: +- `do_connect` is the official Databricks-recommended SQLAlchemy credential injection hook and is used by `databricks-ai-bridge`. +- The community/extra-complexity variant is adding a background `asyncio.Task` token-refresh loop. Demote that loop, not `do_connect`. + +Recommended hook shape: + +```python +from sqlalchemy import event +from sqlalchemy.ext.asyncio import create_async_engine +from databricks.sdk import WorkspaceClient + +w = WorkspaceClient() +endpoint_name = "projects/my-app/branches/production/endpoints/ep-primary" +host = w.postgres.get_endpoint(name=endpoint_name).status.hosts.host +user = w.current_user.me().user_name + +engine = create_async_engine( + f"postgresql+psycopg://{user}@{host}:5432/databricks_postgres", + connect_args={"sslmode": "require"}, + pool_recycle=2700, +) + +@event.listens_for(engine.sync_engine, "do_connect") +def inject_lakebase_token(dialect, conn_rec, cargs, cparams): + cred = w.postgres.generate_database_credential(endpoint=endpoint_name) + cparams["password"] = cred.token +``` + +Notes: +- `do_connect` fires when SQLAlchemy opens a new DBAPI connection. +- `pool_recycle=2700` approximates the psycopg-pool pattern. +- If you need deterministic refresh, prefer scheduled `engine.dispose()` and let the next checkout re-open with `do_connect`. +- A background token cache/refresh task is optional complexity and can create stale-token races if implemented poorly. + +## 3. Direct psycopg for notebooks/scripts + +Only for short-lived sessions where connections are opened and used immediately. + +Recipe: +1. Build endpoint path. +2. `get_endpoint(...).status.hosts.host`. +3. `generate_database_credential(endpoint=endpoint_name)`. +4. `psycopg.connect(host=host, dbname="databricks_postgres", user=, password=cred.token, sslmode="require")`. + +Use `w.current_user.me().user_name` for user in notebooks/manual scripts. In Databricks Apps, prefer `PGUSER`. + +## 4. Static URL / native password + +Use only for local development, legacy tools, or clients that cannot rotate OAuth database credentials. For SQLAlchemy + psycopg3, normalize: + +```text +postgresql://... -> postgresql+psycopg://... +``` + +Still set `sslmode=require`. + +## Endpoint discovery + +Avoid hardcoding host if you can hardcode the endpoint name instead: + +```python +ep = w.postgres.get_endpoint( + name="projects/my-app/branches/production/endpoints/ep-primary" +) +host = ep.status.hosts.host +``` + +If no endpoint ID is known, list under branch and choose deliberately: + +```python +endpoints = list(w.postgres.list_endpoints( + parent="projects/my-app/branches/production" +)) +``` + +Do not assume the first endpoint is the primary if read replicas exist; check endpoint type/status. + +## DNS workaround for macOS + +Some macOS/Python resolver combinations fail on long Lakebase hostnames. + +Workaround: +- Resolve the hostname externally, commonly with `dig +short `. +- Pass both: + - `host=` for TLS/SNI/certificate validation. + - `hostaddr=` for the actual TCP connection. + +psycopg3 supports `hostaddr`. + +## Timeouts, scale-to-zero, and retries + +Plan for: +- 1-hour Lakebase OAuth token lifetime at login. +- 24-hour idle connection timeout. +- 3-day maximum connection lifetime. +- Scale-to-zero wake-up latency; first connection/query after suspension may need retry/backoff. +- After suspension/reactivation: session context is reset, temp tables/prepared statements are gone, active transactions/connections are terminated. + +Use context managers so pooled connections return promptly. diff --git a/.claude/skills/databricks-lakebase-autoscale/operations.md b/.claude/skills/databricks-lakebase-autoscale/operations.md new file mode 100644 index 00000000..982bfb58 --- /dev/null +++ b/.claude/skills/databricks-lakebase-autoscale/operations.md @@ -0,0 +1,297 @@ +# Lakebase Autoscaling operations + +Use `WorkspaceClient().postgres` for Autoscaling projects, branches, endpoints, roles, and credentials. Most create/update/delete methods return long-running operations; call `.wait()`. + +```python +from databricks.sdk import WorkspaceClient +w = WorkspaceClient() +``` + +## Resource names + +```text +Project: projects/{project_id} +Branch: projects/{project_id}/branches/{branch_id} +Endpoint: projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} +``` + +Project ID rules: +- 1–63 chars +- lowercase letters, digits, hyphens +- cannot start/end with hyphen +- immutable after creation + +Default database: `databricks_postgres`. + +## Projects + +Create: + +```python +from databricks.sdk.service.postgres import Project, ProjectSpec + +project = w.postgres.create_project( + project=Project(spec=ProjectSpec(display_name="My App", pg_version="17")), + project_id="my-app", +).wait() +``` + +Project defaults: +- `production` branch +- primary read-write endpoint +- `databricks_postgres` database +- role for creator’s Databricks identity +- production scale-to-zero disabled by default + +GET gotcha: effective properties are typically in `project.status`, not `project.spec`. + +Update requires `FieldMask`: + +```python +from databricks.sdk.service.postgres import FieldMask + +w.postgres.update_project( + name="projects/my-app", + project=Project( + name="projects/my-app", + spec=ProjectSpec(display_name="New Name"), + ), + update_mask=FieldMask(field_mask=["spec.display_name"]), +).wait() +``` + +Delete is destructive and permanent; delete dependent Unity Catalog catalogs/synced tables first where applicable: + +```python +w.postgres.delete_project(name="projects/my-app").wait() +``` + +## Branches + +Branches are copy-on-write isolated database environments. Use them for dev/test/staging, schema-change validation, point-in-time recovery workflows, and ephemeral CI. + +Create branch from current parent: + +```python +from databricks.sdk.service.postgres import Branch, BranchSpec, Duration + +branch = w.postgres.create_branch( + parent="projects/my-app", + branch=Branch(spec=BranchSpec( + source_branch="projects/my-app/branches/production", + ttl=Duration(seconds=604800), # or no_expiry=True + )), + branch_id="development", +).wait() +``` + +Keep: +- `ttl=Duration(seconds=...)` for ephemeral branches. +- `no_expiry=True` for permanent branches. +- Max expiration period: 30 days from current time. +- Only 10 unarchived branches per project. +- Protected branches cannot be deleted, reset, archived, or expired. +- Default branch cannot be deleted or expired. +- Branches with children cannot be deleted, reset, or expired; delete children first. +- Reset replaces branch data/schema with latest parent and interrupts connections. + +Protect production: + +```python +w.postgres.update_branch( + name="projects/my-app/branches/production", + branch=Branch( + name="projects/my-app/branches/production", + spec=BranchSpec(is_protected=True), + ), + update_mask=FieldMask(field_mask=["spec.is_protected"]), +).wait() +``` + +Reset/delete: + +```python +w.postgres.reset_branch(name="projects/my-app/branches/development").wait() +w.postgres.delete_branch(name="projects/my-app/branches/development").wait() +``` + +Branch status fields worth inspecting: +- `status.default` +- `status.is_protected` +- `status.current_state` +- `status.logical_size_bytes` +- `status.expire_time` + +## Endpoints / computes + +A compute endpoint runs Postgres for a branch. Each branch has at most one primary read-write endpoint and may have read-only replica endpoints. + +Create endpoint: + +```python +from databricks.sdk.service.postgres import Endpoint, EndpointSpec, EndpointType + +ep = w.postgres.create_endpoint( + parent="projects/my-app/branches/production", + endpoint=Endpoint(spec=EndpointSpec( + endpoint_type=EndpointType.ENDPOINT_TYPE_READ_WRITE, + autoscaling_limit_min_cu=0.5, + autoscaling_limit_max_cu=4.0, + )), + endpoint_id="ep-primary", +).wait() +``` + +Get host: + +```python +host = w.postgres.get_endpoint( + name="projects/my-app/branches/production/endpoints/ep-primary" +).status.hosts.host +``` + +Resize with update mask: + +```python +w.postgres.update_endpoint( + name="projects/my-app/branches/production/endpoints/ep-primary", + endpoint=Endpoint( + name="projects/my-app/branches/production/endpoints/ep-primary", + spec=EndpointSpec( + autoscaling_limit_min_cu=2.0, + autoscaling_limit_max_cu=8.0, + ), + ), + update_mask=FieldMask(field_mask=[ + "spec.autoscaling_limit_min_cu", + "spec.autoscaling_limit_max_cu", + ]), +).wait() +``` + +Delete: + +```python +w.postgres.delete_endpoint( + name="projects/my-app/branches/production/endpoints/ep-primary" +).wait() +``` + +## Compute sizing + +Autoscaling uses ~2 GB RAM per CU. + +| CU | Approx RAM | Max connections | +|---:|---:|---:| +| 0.5 | ~1 GB | 104 | +| 1 | ~2 GB | 209 | +| 4 | ~8 GB | 839 | +| 8 | ~16 GB | 1,678 | +| 16 | ~32 GB | 3,357 | +| 32 | ~64 GB | 4,000 | +| 64 | ~128 GB | 4,000 | +| 112 | ~224 GB | 4,000 | + +Rules: +- Autoscale range: 0.5–32 CU. +- `autoscaling_limit_max_cu - autoscaling_limit_min_cu <= 16`. +- Valid: 4–20, 8–16, 16–32. +- Invalid: 0.5–32 (spread of 31.5 exceeds 16). +- Fixed-size always-on computes: 40–112 CU; no autoscaling. +- Connection limit is based on max CU. +- Set min CU high enough for working-set cache and latency needs. + +## Scale-to-zero + +Defaults: +- `production`: disabled by default. +- Other branches: configurable. +- Default inactivity timeout: 5 minutes. +- Minimum inactivity timeout: 60 seconds. + +Wake-up: +- First connection wakes compute automatically. +- Apps should use retry/backoff for the brief reactivation period. +- Reactivated compute starts at minimum autoscaling size. + +Session reset after suspension: +- temp tables gone +- prepared statements gone +- in-memory stats/cache cleared +- session settings reset +- active transactions/connections terminated + +Disable scale-to-zero for latency-critical apps or apps relying on persistent session state. + +## Project limits + +| Resource | Limit | +|---|---:| +| Projects per workspace | 1000 | +| Branches per project | 500 | +| Unarchived branches | 10 | +| Root branches | 3 | +| Protected branches | 1 | +| Concurrently active computes | 20 | +| Postgres roles per branch | 500 | +| Postgres databases per branch | 500 | +| Logical data size per branch | 8 TB | +| Snapshots | 10 | +| Maximum history retention | 35 days | +| Minimum scale-to-zero time | 60 sec | + +## CLI names + +CLI mirrors the SDK under `databricks postgres`, for example: +- `create-project`, `get-project`, `list-projects`, `update-project`, `delete-project` +- `create-branch`, `list-branches`, `reset-branch`, `delete-branch` +- `create-endpoint`, `get-endpoint`, `list-endpoints`, `update-endpoint`, `delete-endpoint` + +## MCP tools + +Use `type="autoscale"` for Lakebase Autoscaling. + +### `manage_lakebase_database` + +Actions: +- `create_or_update`: requires `name`; useful params include `display_name`, `pg_version` +- `get`: requires `name` +- `list`: optional type filter +- `delete`: requires `name` + +Example intent: + +```python +manage_lakebase_database( + action="create_or_update", + name="my-app", + type="autoscale", + display_name="My Application", + pg_version="17", +) +``` + +### `manage_lakebase_branch` + +Actions: +- `create_or_update`: requires `project_name`, `branch_id` +- `delete`: requires full branch `name` + +Useful params: +- `source_branch` +- `ttl_seconds` +- `autoscaling_limit_min_cu` +- `autoscaling_limit_max_cu` +- `scale_to_zero_seconds` + +### `generate_lakebase_credential` + +Generate a Lakebase-scoped database credential: + +```python +generate_lakebase_credential( + endpoint="projects/my-app/branches/production/endpoints/ep-primary" +) +``` + +Use returned token as the Postgres password with `sslmode=require`. diff --git a/.claude/skills/databricks-lakebase-autoscale/reverse-etl.md b/.claude/skills/databricks-lakebase-autoscale/reverse-etl.md new file mode 100644 index 00000000..949f91b6 --- /dev/null +++ b/.claude/skills/databricks-lakebase-autoscale/reverse-etl.md @@ -0,0 +1,127 @@ +# Reverse ETL / synced tables + +Reverse ETL syncs Unity Catalog Delta tables into Lakebase Autoscaling as PostgreSQL tables for OLTP access. + +Important namespace split: +- Lakebase Autoscaling infrastructure: `w.postgres` +- Synced tables: `w.database` + +Reverse ETL is Delta-to-Postgres only; Postgres-to-Delta sync is not supported here. + +## How synced tables work + +A synced table creates/maintains: +1. A managed/read-only Unity Catalog table for pipeline state/output. +2. A PostgreSQL table in Lakebase queried by apps. + +The sync pipeline uses managed Lakeflow Spark Declarative Pipelines. + +Performance planning: +- Continuous writes: ~1,200 rows/sec per CU. +- Bulk writes: ~15,000 rows/sec per CU. +- Each synced table can use up to 16 Postgres connections. + +## Sync modes + +| Mode | Behavior | Use when | CDF required | +|---|---|---|---| +| `SNAPSHOT` | one-time full copy | initial loads, historical copy, large replacement | no | +| `TRIGGERED` | scheduled/on-demand incremental updates | hourly/daily operational refresh | yes | +| `CONTINUOUS` | streaming updates, seconds latency | live applications | yes | + +Triggered and Continuous require Delta Change Data Feed on the source table: + +```sql +ALTER TABLE catalog.schema.table +SET TBLPROPERTIES (delta.enableChangeDataFeed = true); +``` + +Snapshot can be more efficient when modifying >10% of the data. + +## Create a synced table + +Use `databricks.sdk.service.database` models: + +```python +from databricks.sdk import WorkspaceClient +from databricks.sdk.service.database import ( + NewPipelineSpec, + SyncedDatabaseTable, + SyncedTableSchedulingPolicy, + SyncedTableSpec, +) + +w = WorkspaceClient() + +w.database.create_synced_database_table( + SyncedDatabaseTable( + name="lakebase_catalog.schema.synced_table", + spec=SyncedTableSpec( + source_table_full_name="analytics.gold.user_profiles", + primary_key_columns=["user_id"], + scheduling_policy=SyncedTableSchedulingPolicy.TRIGGERED, + new_pipeline_spec=NewPipelineSpec( + storage_catalog="lakebase_catalog", + storage_schema="staging", + ), + ), + ) +) +``` + +Status: + +```python +st = w.database.get_synced_database_table( + name="lakebase_catalog.schema.synced_table" +) +state = st.data_synchronization_status.detailed_state +message = st.data_synchronization_status.message +``` + +Deletion cleanup: +1. Delete the synced table / UC object. +2. Drop the Postgres target table if needed to free Lakebase storage. + +```sql +DROP TABLE schema.table; +``` + +## Type mapping + +| Unity Catalog | Postgres | +|---|---| +| BIGINT | BIGINT | +| BINARY | BYTEA | +| BOOLEAN | BOOLEAN | +| DATE | DATE | +| DECIMAL(p,s) | NUMERIC | +| DOUBLE | DOUBLE PRECISION | +| FLOAT | REAL | +| INT | INTEGER | +| INTERVAL | INTERVAL | +| SMALLINT | SMALLINT | +| STRING | TEXT | +| TIMESTAMP | TIMESTAMP WITH TIME ZONE | +| TIMESTAMP_NTZ | TIMESTAMP WITHOUT TIME ZONE | +| TINYINT | SMALLINT | +| ARRAY | JSONB | +| MAP | JSONB | +| STRUCT | JSONB | + +Unsupported: +- `GEOGRAPHY` +- `GEOMETRY` +- `VARIANT` +- `OBJECT` + +## Limits and gotchas + +- Up to 16 Postgres connections per synced table; include this in endpoint connection-capacity planning. +- Size limit: 2 TB total across all synced tables. +- Recommended: <1 TB per synced table. +- Database/schema/table names: `[A-Za-z0-9_]+`. +- Triggered/Continuous schema evolution: additive changes only. +- Create indexes in Postgres for application query patterns after sync. +- Monitor detailed sync state in Catalog Explorer or with `get_synced_database_table`. +- Delete synced-table dependencies before deleting the Lakebase project. diff --git a/.claude/skills/databricks-lakebase-provisioned/SKILL.md b/.claude/skills/databricks-lakebase-provisioned/SKILL.md new file mode 100644 index 00000000..846f6448 --- /dev/null +++ b/.claude/skills/databricks-lakebase-provisioned/SKILL.md @@ -0,0 +1,352 @@ +--- +name: databricks-lakebase-provisioned +description: "Patterns and best practices for Lakebase Provisioned (Databricks managed PostgreSQL) for OLTP workloads. Use when creating Lakebase instances, connecting applications or Databricks Apps to PostgreSQL, implementing reverse ETL via synced tables, storing agent or chat memory, or configuring OAuth authentication for Lakebase." +--- + +# Lakebase Provisioned + +Patterns and best practices for using Lakebase Provisioned (Databricks managed PostgreSQL) for OLTP workloads. + +## When to Use + +Use this skill when: +- Building applications that need a PostgreSQL database for transactional workloads +- Adding persistent state to Databricks Apps +- Implementing reverse ETL from Delta Lake to an operational database +- Storing chat/agent memory for LangChain applications + +## Overview + +Lakebase Provisioned is Databricks' managed PostgreSQL database service for OLTP (Online Transaction Processing) workloads. It provides a fully managed PostgreSQL-compatible database that integrates with Unity Catalog and supports OAuth token-based authentication. + +| Feature | Description | +|---------|-------------| +| **Managed PostgreSQL** | Fully managed instances with automatic provisioning | +| **OAuth Authentication** | Token-based auth via Databricks SDK (1-hour expiry) | +| **Unity Catalog** | Register databases for governance | +| **Reverse ETL** | Sync data from Delta tables to PostgreSQL | +| **Apps Integration** | First-class support in Databricks Apps | + +**Available Regions (AWS):** us-east-1, us-east-2, us-west-2, eu-central-1, eu-west-1, ap-south-1, ap-southeast-1, ap-southeast-2 + +## Quick Start + +Create and connect to a Lakebase Provisioned instance: + +```python +from databricks.sdk import WorkspaceClient +import uuid + +# Initialize client +w = WorkspaceClient() + +# Create a database instance +instance = w.database.create_database_instance( + name="my-lakebase-instance", + capacity="CU_1", # CU_1, CU_2, CU_4, CU_8 + stopped=False +) +print(f"Instance created: {instance.name}") +print(f"DNS endpoint: {instance.read_write_dns}") +``` + +## Common Patterns + +### Generate OAuth Token + +```python +from databricks.sdk import WorkspaceClient +import uuid + +w = WorkspaceClient() + +# Generate OAuth token for database connection +cred = w.database.generate_database_credential( + request_id=str(uuid.uuid4()), + instance_names=["my-lakebase-instance"] +) +token = cred.token # Use this as password in connection string +``` + +### Connect from Notebook + +```python +import psycopg +from databricks.sdk import WorkspaceClient +import uuid + +# Get instance details +w = WorkspaceClient() +instance = w.database.get_database_instance(name="my-lakebase-instance") + +# Generate token +cred = w.database.generate_database_credential( + request_id=str(uuid.uuid4()), + instance_names=["my-lakebase-instance"] +) + +# Connect using psycopg3 +conn_string = f"host={instance.read_write_dns} dbname=postgres user={w.current_user.me().user_name} password={cred.token} sslmode=require" +with psycopg.connect(conn_string) as conn: + with conn.cursor() as cur: + cur.execute("SELECT version()") + print(cur.fetchone()) +``` + +### SQLAlchemy with Token Refresh (Production) + +For long-running applications, tokens must be refreshed (expire after 1 hour): + +```python +import asyncio +import os +import uuid +from sqlalchemy import event +from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession +from sqlalchemy.orm import sessionmaker +from databricks.sdk import WorkspaceClient + +# Token refresh state +_current_token = None +_token_refresh_task = None +TOKEN_REFRESH_INTERVAL = 50 * 60 # 50 minutes (before 1-hour expiry) + +def _generate_token(instance_name: str) -> str: + """Generate fresh OAuth token.""" + w = WorkspaceClient() + cred = w.database.generate_database_credential( + request_id=str(uuid.uuid4()), + instance_names=[instance_name] + ) + return cred.token + +async def _token_refresh_loop(instance_name: str): + """Background task to refresh token every 50 minutes.""" + global _current_token + while True: + await asyncio.sleep(TOKEN_REFRESH_INTERVAL) + _current_token = await asyncio.to_thread(_generate_token, instance_name) + +def init_database(instance_name: str, database_name: str, username: str) -> AsyncEngine: + """Initialize database with OAuth token injection.""" + global _current_token + + w = WorkspaceClient() + instance = w.database.get_database_instance(name=instance_name) + + # Generate initial token + _current_token = _generate_token(instance_name) + + # Build URL (password injected via do_connect) + url = f"postgresql+psycopg://{username}@{instance.read_write_dns}:5432/{database_name}" + + engine = create_async_engine( + url, + pool_size=5, + max_overflow=10, + pool_recycle=3600, + connect_args={"sslmode": "require"} + ) + + # Inject token on each connection + @event.listens_for(engine.sync_engine, "do_connect") + def provide_token(dialect, conn_rec, cargs, cparams): + cparams["password"] = _current_token + + return engine +``` + +### Databricks Apps Integration + +For Databricks Apps, use environment variables for configuration: + +```python +# Environment variables set by Databricks Apps: +# - LAKEBASE_INSTANCE_NAME: Instance name +# - LAKEBASE_DATABASE_NAME: Database name +# - LAKEBASE_USERNAME: Username (optional, defaults to service principal) + +import os + +def is_lakebase_configured() -> bool: + """Check if Lakebase is configured for this app.""" + return bool( + os.environ.get("LAKEBASE_PG_URL") or + (os.environ.get("LAKEBASE_INSTANCE_NAME") and + os.environ.get("LAKEBASE_DATABASE_NAME")) + ) +``` + +Add Lakebase as an app resource via CLI: + +```bash +databricks apps add-resource $APP_NAME \ + --resource-type database \ + --resource-name lakebase \ + --database-instance my-lakebase-instance +``` + +### Register with Unity Catalog + +```python +from databricks.sdk import WorkspaceClient + +w = WorkspaceClient() + +# Register database in Unity Catalog +w.database.register_database_instance( + name="my-lakebase-instance", + catalog="my_catalog", + schema="my_schema" +) +``` + +### MLflow Model Resources + +Declare Lakebase as a model resource for automatic credential provisioning: + +```python +from mlflow.models.resources import DatabricksLakebase + +resources = [ + DatabricksLakebase(database_instance_name="my-lakebase-instance"), +] + +# When logging model +mlflow.langchain.log_model( + model, + artifact_path="model", + resources=resources, + pip_requirements=["databricks-langchain[memory]"] +) +``` + +## MCP Tools + +The following MCP tools are available for managing Lakebase infrastructure. Use `type="provisioned"` for Lakebase Provisioned. + +### manage_lakebase_database - Database Management + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create_or_update` | Create or update a database | name | +| `get` | Get database details | name | +| `list` | List all databases | (none, optional type filter) | +| `delete` | Delete database and resources | name | + +**Example usage:** +```python +# Create a provisioned database +manage_lakebase_database( + action="create_or_update", + name="my-lakebase-instance", + type="provisioned", + capacity="CU_1" +) + +# Get database details +manage_lakebase_database(action="get", name="my-lakebase-instance", type="provisioned") + +# List all databases +manage_lakebase_database(action="list") + +# Delete with cascade +manage_lakebase_database(action="delete", name="my-lakebase-instance", type="provisioned", force=True) +``` + +### manage_lakebase_sync - Reverse ETL + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create_or_update` | Set up reverse ETL from Delta to Lakebase | instance_name, source_table_name, target_table_name | +| `delete` | Remove synced table (and optionally catalog) | table_name | + +**Example usage:** +```python +# Set up reverse ETL +manage_lakebase_sync( + action="create_or_update", + instance_name="my-lakebase-instance", + source_table_name="catalog.schema.delta_table", + target_table_name="lakebase_catalog.schema.postgres_table", + scheduling_policy="TRIGGERED" # or SNAPSHOT, CONTINUOUS +) + +# Delete synced table +manage_lakebase_sync(action="delete", table_name="lakebase_catalog.schema.postgres_table") +``` + +### generate_lakebase_credential - OAuth Tokens + +Generate OAuth token (~1hr) for PostgreSQL connections. Use as password with `sslmode=require`. + +```python +# For provisioned instances +generate_lakebase_credential(instance_names=["my-lakebase-instance"]) +``` + +## Reference Files + +- [connection-patterns.md](connection-patterns.md) - Detailed connection patterns for different use cases +- [reverse-etl.md](reverse-etl.md) - Syncing data from Delta Lake to Lakebase + +## CLI Quick Reference + +```bash +# Create instance +databricks database create-database-instance \ + --name my-lakebase-instance \ + --capacity CU_1 + +# Get instance details +databricks database get-database-instance --name my-lakebase-instance + +# Generate credentials +databricks database generate-database-credential \ + --request-id $(uuidgen) \ + --json '{"instance_names": ["my-lakebase-instance"]}' + +# List instances +databricks database list-database-instances + +# Stop instance (saves cost) +databricks database stop-database-instance --name my-lakebase-instance + +# Start instance +databricks database start-database-instance --name my-lakebase-instance +``` + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **Token expired during long query** | Implement token refresh loop (see SQLAlchemy with Token Refresh section); tokens expire after 1 hour | +| **DNS resolution fails on macOS** | Use `dig` command to resolve hostname, pass `hostaddr` to psycopg | +| **Connection refused** | Ensure instance is not stopped; check `instance.state` | +| **Permission denied** | User must be granted access to the Lakebase instance | +| **SSL required error** | Always use `sslmode=require` in connection string | + +## SDK Version Requirements + +- **Databricks SDK for Python**: >= 0.61.0 (0.81.0+ recommended for full API support) +- **psycopg**: 3.x (supports `hostaddr` parameter for DNS workaround) +- **SQLAlchemy**: 2.x with `postgresql+psycopg` driver + +```python +%pip install -U "databricks-sdk>=0.81.0" "psycopg[binary]>=3.0" sqlalchemy +``` + +## Notes + +- **Capacity values** use compute unit sizing: `CU_1`, `CU_2`, `CU_4`, `CU_8`. +- **Lakebase Autoscaling** is a newer offering with automatic scaling but limited regional availability. This skill focuses on **Lakebase Provisioned** which is more widely available. +- For memory/state in LangChain agents, use `databricks-langchain[memory]` which includes Lakebase support. +- Tokens are short-lived (1 hour) - production apps MUST implement token refresh. + +## Related Skills + +- **[databricks-app-apx](../databricks-app-apx/SKILL.md)** - full-stack apps that can use Lakebase for persistence +- **[databricks-apps-python](../databricks-apps-python/SKILL.md)** - Python apps with Lakebase backend +- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** - SDK used for instance management and token generation +- **[databricks-bundles](../databricks-bundles/SKILL.md)** - deploying apps with Lakebase resources +- **[databricks-jobs](../databricks-jobs/SKILL.md)** - scheduling reverse ETL sync jobs diff --git a/.claude/skills/databricks-lakebase-provisioned/connection-patterns.md b/.claude/skills/databricks-lakebase-provisioned/connection-patterns.md new file mode 100644 index 00000000..e6843548 --- /dev/null +++ b/.claude/skills/databricks-lakebase-provisioned/connection-patterns.md @@ -0,0 +1,279 @@ +# Lakebase Connection Patterns + +## Overview + +This document covers different connection patterns for Lakebase Provisioned, from simple scripts to production applications with token refresh. + +## Connection Methods + +### 1. Direct psycopg Connection (Simple Scripts) + +For one-off scripts or notebooks: + +```python +import psycopg +from databricks.sdk import WorkspaceClient +import uuid + +def get_connection(instance_name: str, database_name: str = "postgres"): + """Get a database connection with fresh OAuth token.""" + w = WorkspaceClient() + + # Get instance details + instance = w.database.get_database_instance(name=instance_name) + + # Generate OAuth token (valid for 1 hour) + cred = w.database.generate_database_credential( + request_id=str(uuid.uuid4()), + instance_names=[instance_name] + ) + + # Build connection string + conn_string = ( + f"host={instance.read_write_dns} " + f"dbname={database_name} " + f"user={w.current_user.me().user_name} " + f"password={cred.token} " + f"sslmode=require" + ) + + return psycopg.connect(conn_string) + +# Usage +with get_connection("my-instance") as conn: + with conn.cursor() as cur: + cur.execute("SELECT NOW()") + print(cur.fetchone()) +``` + +### 2. Connection Pool with Token Refresh (Production) + +For long-running applications that need connection pooling: + +```python +import asyncio +import uuid +from contextlib import asynccontextmanager +from typing import AsyncGenerator, Optional + +from sqlalchemy import event +from sqlalchemy.ext.asyncio import AsyncSession, create_async_engine, async_sessionmaker +from databricks.sdk import WorkspaceClient + +class LakebaseConnectionManager: + """Manages Lakebase connections with automatic token refresh.""" + + def __init__( + self, + instance_name: str, + database_name: str, + pool_size: int = 5, + max_overflow: int = 10, + token_refresh_seconds: int = 3000 # 50 minutes + ): + self.instance_name = instance_name + self.database_name = database_name + self.pool_size = pool_size + self.max_overflow = max_overflow + self.token_refresh_seconds = token_refresh_seconds + + self._current_token: Optional[str] = None + self._refresh_task: Optional[asyncio.Task] = None + self._engine = None + self._session_maker = None + + def _generate_token(self) -> str: + """Generate fresh OAuth token.""" + w = WorkspaceClient() + cred = w.database.generate_database_credential( + request_id=str(uuid.uuid4()), + instance_names=[self.instance_name] + ) + return cred.token + + async def _refresh_loop(self): + """Background task to refresh token periodically.""" + while True: + await asyncio.sleep(self.token_refresh_seconds) + try: + self._current_token = await asyncio.to_thread(self._generate_token) + except Exception as e: + print(f"Token refresh failed: {e}") + + def initialize(self): + """Initialize database engine and start token refresh.""" + w = WorkspaceClient() + + # Get instance info + instance = w.database.get_database_instance(name=self.instance_name) + username = w.current_user.me().user_name + + # Generate initial token + self._current_token = self._generate_token() + + # Create engine (password injected via event) + url = ( + f"postgresql+psycopg://{username}@" + f"{instance.read_write_dns}:5432/{self.database_name}" + ) + + self._engine = create_async_engine( + url, + pool_size=self.pool_size, + max_overflow=self.max_overflow, + pool_recycle=3600, + connect_args={"sslmode": "require"} + ) + + # Inject token on connect + @event.listens_for(self._engine.sync_engine, "do_connect") + def inject_token(dialect, conn_rec, cargs, cparams): + cparams["password"] = self._current_token + + self._session_maker = async_sessionmaker( + self._engine, + class_=AsyncSession, + expire_on_commit=False + ) + + def start_refresh(self): + """Start background token refresh task.""" + if not self._refresh_task: + self._refresh_task = asyncio.create_task(self._refresh_loop()) + + async def stop_refresh(self): + """Stop token refresh task.""" + if self._refresh_task: + self._refresh_task.cancel() + try: + await self._refresh_task + except asyncio.CancelledError: + pass + self._refresh_task = None + + @asynccontextmanager + async def session(self) -> AsyncGenerator[AsyncSession, None]: + """Get a database session.""" + async with self._session_maker() as session: + yield session + + async def close(self): + """Close all connections.""" + await self.stop_refresh() + if self._engine: + await self._engine.dispose() + +# Usage in FastAPI +from fastapi import FastAPI + +app = FastAPI() +db_manager = LakebaseConnectionManager("my-instance", "my_database") + +@app.on_event("startup") +async def startup(): + db_manager.initialize() + db_manager.start_refresh() + +@app.on_event("shutdown") +async def shutdown(): + await db_manager.close() + +@app.get("/data") +async def get_data(): + async with db_manager.session() as session: + result = await session.execute("SELECT * FROM my_table") + return result.fetchall() +``` + +### 3. Static URL Mode (Local Development) + +For local development, use a static connection URL: + +```python +import os +from sqlalchemy.ext.asyncio import create_async_engine + +# Set environment variable with full connection URL +# LAKEBASE_PG_URL=postgresql://user:password@host:5432/database + +def get_database_url() -> str: + """Get database URL from environment.""" + url = os.environ.get("LAKEBASE_PG_URL") + if url and url.startswith("postgresql://"): + # Convert to psycopg3 async driver + url = url.replace("postgresql://", "postgresql+psycopg://", 1) + return url + +engine = create_async_engine( + get_database_url(), + pool_size=5, + connect_args={"sslmode": "require"} +) +``` + +### 4. DNS Resolution Workaround (macOS) + +Python's `socket.getaddrinfo()` fails with long hostnames on macOS. Use `dig` as fallback: + +```python +import subprocess +import socket + +def resolve_hostname(hostname: str) -> str: + """Resolve hostname using dig command (macOS workaround).""" + try: + # Try Python's resolver first + return socket.gethostbyname(hostname) + except socket.gaierror: + pass + + # Fallback to dig command + try: + result = subprocess.run( + ["dig", "+short", hostname], + capture_output=True, + text=True, + timeout=5 + ) + ips = result.stdout.strip().split('\n') + for ip in ips: + if ip and not ip.startswith(';'): + return ip + except Exception: + pass + + raise RuntimeError(f"Could not resolve hostname: {hostname}") + +# Use with psycopg +conn_params = { + "host": hostname, # For TLS SNI + "hostaddr": resolve_hostname(hostname), # Actual IP + "dbname": database_name, + "user": username, + "password": token, + "sslmode": "require" +} +conn = psycopg.connect(**conn_params) +``` + +## Environment Variables + +| Variable | Description | Required | +|----------|-------------|----------| +| `LAKEBASE_PG_URL` | Static PostgreSQL URL (local dev) | Either this OR instance/database | +| `LAKEBASE_INSTANCE_NAME` | Lakebase instance name | With DATABASE_NAME | +| `LAKEBASE_DATABASE_NAME` | Database name | With INSTANCE_NAME | +| `LAKEBASE_USERNAME` | Override username | No | +| `LAKEBASE_HOST` | Override host | No | +| `DB_POOL_SIZE` | Connection pool size | No (default: 5) | +| `DB_MAX_OVERFLOW` | Max pool overflow | No (default: 10) | +| `DB_POOL_RECYCLE_INTERVAL` | Pool recycle seconds | No (default: 3600) | + +## Best Practices + +1. **Always use SSL**: Set `sslmode=require` in all connections +2. **Implement token refresh**: Tokens expire after 1 hour; refresh at 50 minutes +3. **Use connection pooling**: Avoid creating new connections per request +4. **Handle DNS issues on macOS**: Use the `hostaddr` workaround if needed +5. **Close connections properly**: Use context managers or explicit cleanup +6. **Log token refresh events**: Helps debug authentication issues diff --git a/.claude/skills/databricks-lakebase-provisioned/reverse-etl.md b/.claude/skills/databricks-lakebase-provisioned/reverse-etl.md new file mode 100644 index 00000000..5b5caef4 --- /dev/null +++ b/.claude/skills/databricks-lakebase-provisioned/reverse-etl.md @@ -0,0 +1,171 @@ +# Reverse ETL with Lakebase Provisioned + +## Overview + +Reverse ETL allows you to sync data from Unity Catalog Delta tables into Lakebase Provisioned as PostgreSQL tables. This enables OLTP access patterns on data processed in the Lakehouse. + +## Sync Modes + +| Mode | Description | Best For | Notes | +|------|-------------|----------|-------| +| **Snapshot** | One-time full copy | Initial setup, small tables | 10x more efficient if modifying >10% of data | +| **Triggered** | Scheduled updates on demand | Dashboards updated hourly/daily | Requires CDF on source table | +| **Continuous** | Real-time streaming (seconds of latency) | Live applications | Highest cost, minimum 15s intervals, requires CDF | + +**Note:** Triggered and Continuous modes require Change Data Feed (CDF) enabled on the source table: + +```sql +ALTER TABLE your_catalog.your_schema.your_table +SET TBLPROPERTIES (delta.enableChangeDataFeed = true) +``` + +## Creating Synced Tables + +### Using Python SDK + +```python +from databricks.sdk import WorkspaceClient +from databricks.sdk.service.database import ( + SyncedDatabaseTable, + SyncedTableSpec, + SyncedTableSchedulingPolicy, +) + +w = WorkspaceClient() + +# Create a synced table from Unity Catalog to Lakebase Provisioned +synced_table = w.database.create_synced_database_table( + SyncedDatabaseTable( + name="lakebase_catalog.schema.synced_table", + database_instance_name="my-lakebase-instance", + spec=SyncedTableSpec( + source_table_full_name="analytics.gold.user_profiles", + primary_key_columns=["user_id"], + scheduling_policy=SyncedTableSchedulingPolicy.TRIGGERED, + ), + ) +) +print(f"Created synced table: {synced_table.name}") +``` + +**Key parameters:** + +| Parameter | Description | +|-----------|-------------| +| `name` | Fully qualified target table name (catalog.schema.table) | +| `database_instance_name` | Lakebase Provisioned instance name | +| `source_table_full_name` | Fully qualified source Delta table (catalog.schema.table) | +| `primary_key_columns` | List of primary key columns from the source table | +| `scheduling_policy` | `SNAPSHOT`, `TRIGGERED`, or `CONTINUOUS` | + +### Using CLI + +```bash +databricks database create-synced-database-table \ + --json '{ + "name": "lakebase_catalog.schema.synced_table", + "database_instance_name": "my-lakebase-instance", + "spec": { + "source_table_full_name": "analytics.gold.user_profiles", + "primary_key_columns": ["user_id"], + "scheduling_policy": "TRIGGERED" + } + }' +``` + +**Note:** There is no SQL syntax for creating synced tables. Use the Python SDK, CLI, or Catalog Explorer UI. + +## Checking Synced Table Status + +```python +status = w.database.get_synced_database_table(name="lakebase_catalog.schema.synced_table") +print(f"State: {status.data_synchronization_status.detailed_state}") +print(f"Message: {status.data_synchronization_status.message}") +``` + +## Deleting a Synced Table + +Delete from both Unity Catalog and Postgres: + +1. **Unity Catalog:** Delete via Catalog Explorer or SDK +2. **Postgres:** Drop the table to free storage + +```python +# Delete the synced table via SDK +w.database.delete_synced_database_table(name="lakebase_catalog.schema.synced_table") +``` + +```sql +-- Drop the Postgres table to free storage +DROP TABLE your_database.your_schema.your_table; +``` + +## Use Cases + +### 1. Product Catalog for Web App + +```python +w.database.create_synced_database_table( + SyncedDatabaseTable( + name="ecommerce_catalog.public.products", + database_instance_name="ecommerce-db", + spec=SyncedTableSpec( + source_table_full_name="gold.products.catalog", + primary_key_columns=["product_id"], + scheduling_policy=SyncedTableSchedulingPolicy.TRIGGERED, + ), + ) +) +# Application queries PostgreSQL directly with low-latency point lookups +``` + +### 2. User Profiles for Authentication + +```python +w.database.create_synced_database_table( + SyncedDatabaseTable( + name="auth_catalog.public.user_profiles", + database_instance_name="auth-db", + spec=SyncedTableSpec( + source_table_full_name="gold.users.profiles", + primary_key_columns=["user_id"], + scheduling_policy=SyncedTableSchedulingPolicy.CONTINUOUS, + ), + ) +) +``` + +### 3. Feature Store for Real-time ML + +```python +w.database.create_synced_database_table( + SyncedDatabaseTable( + name="ml_catalog.public.user_features", + database_instance_name="feature-store-db", + spec=SyncedTableSpec( + source_table_full_name="ml.features.user_features", + primary_key_columns=["user_id"], + scheduling_policy=SyncedTableSchedulingPolicy.CONTINUOUS, + ), + ) +) +# ML model queries features with low latency +``` + +## Best Practices + +1. **Enable CDF** on source tables before creating Triggered or Continuous synced tables +2. **Choose appropriate sync mode**: Snapshot for small tables or one-time loads, Triggered for hourly/daily refreshes, Continuous for real-time +3. **Monitor sync status**: Check for failures and latency via Catalog Explorer or `get_synced_database_table()` +4. **Index target tables**: Create appropriate indexes in PostgreSQL for your query patterns +5. **Handle schema changes**: Only additive changes (e.g., adding columns) are supported for Triggered/Continuous modes +6. **Account for connection limits**: Each synced table uses up to 16 connections + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **Sync fails with CDF error** | Enable Change Data Feed on source table before using Triggered or Continuous mode | +| **Schema mismatch** | Only additive schema changes are supported; for breaking changes, delete and recreate the synced table | +| **Sync takes too long** | Switch to Triggered mode for scheduled updates; use Snapshot for initial bulk loads | +| **Target table locked** | Avoid DDL on target during sync operations | diff --git a/.claude/skills/databricks-metric-views/SKILL.md b/.claude/skills/databricks-metric-views/SKILL.md new file mode 100644 index 00000000..3cc4b427 --- /dev/null +++ b/.claude/skills/databricks-metric-views/SKILL.md @@ -0,0 +1,242 @@ +--- +name: databricks-metric-views +description: "Unity Catalog metric views: define, create, query, and manage governed business metrics in YAML. Use when building standardized KPIs, revenue metrics, order analytics, or any reusable business metrics that need consistent definitions across teams and tools." +--- + +# Unity Catalog Metric Views + +Define reusable, governed business metrics in YAML that separate measure definitions from dimension groupings for flexible querying. + +## When to Use + +Use this skill when: +- Defining **standardized business metrics** (revenue, order counts, conversion rates) +- Building **KPI layers** shared across dashboards, Genie, and SQL queries +- Creating metrics with **complex aggregations** (ratios, distinct counts, filtered measures) +- Defining **window measures** (moving averages, running totals, period-over-period, YTD) +- Modeling **star or snowflake schemas** with joins in metric definitions +- Enabling **materialization** for pre-computed metric aggregations + +## Prerequisites + +- **Databricks Runtime 17.2+** (for YAML version 1.1) +- SQL warehouse with `CAN USE` permissions +- `SELECT` on source tables, `CREATE TABLE` + `USE SCHEMA` in the target schema + +## Quick Start + +### Inspect Source Table Schema + +Before creating a metric view, call `get_table_stats_and_schema` to understand available columns for dimensions and measures: + +``` +get_table_stats_and_schema( + catalog="catalog", + schema="schema", + table_names=["orders"], + table_stat_level="SIMPLE" # Use "DETAILED" for cardinality, min/max, histograms +) +``` + +### Create a Metric View + +```sql +CREATE OR REPLACE VIEW catalog.schema.orders_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.orders + comment: "Orders KPIs for sales analysis" + filter: order_date > '2020-01-01' + dimensions: + - name: Order Month + expr: DATE_TRUNC('MONTH', order_date) + comment: "Month of order" + - name: Order Status + expr: CASE + WHEN status = 'O' THEN 'Open' + WHEN status = 'P' THEN 'Processing' + WHEN status = 'F' THEN 'Fulfilled' + END + comment: "Human-readable order status" + measures: + - name: Order Count + expr: COUNT(1) + - name: Total Revenue + expr: SUM(total_price) + comment: "Sum of total price" + - name: Revenue per Customer + expr: SUM(total_price) / COUNT(DISTINCT customer_id) + comment: "Average revenue per unique customer" +$$ +``` + +### Query a Metric View + +All measures must use the `MEASURE()` function. `SELECT *` is NOT supported. + +```sql +SELECT + `Order Month`, + `Order Status`, + MEASURE(`Total Revenue`) AS total_revenue, + MEASURE(`Order Count`) AS order_count +FROM catalog.schema.orders_metrics +WHERE extract(year FROM `Order Month`) = 2024 +GROUP BY ALL +ORDER BY ALL +``` + +## Reference Files + +| Topic | File | Description | +|-------|------|-------------| +| YAML Syntax | [yaml-reference.md](yaml-reference.md) | Complete YAML spec: dimensions, measures, joins, materialization | +| Patterns & Examples | [patterns.md](patterns.md) | Common patterns: star schema, snowflake, filtered measures, window measures, ratios | + +## MCP Tools + +Use the `manage_metric_views` tool for all metric view operations: + +| Action | Description | +|--------|-------------| +| `create` | Create a metric view with dimensions and measures | +| `alter` | Update a metric view's YAML definition | +| `describe` | Get the full definition and metadata | +| `query` | Query measures grouped by dimensions | +| `drop` | Drop a metric view | +| `grant` | Grant SELECT privileges to users/groups | + +### Create via MCP + +```python +manage_metric_views( + action="create", + full_name="catalog.schema.orders_metrics", + source="catalog.schema.orders", + or_replace=True, + comment="Orders KPIs for sales analysis", + filter_expr="order_date > '2020-01-01'", + dimensions=[ + {"name": "Order Month", "expr": "DATE_TRUNC('MONTH', order_date)", "comment": "Month of order"}, + {"name": "Order Status", "expr": "status"}, + ], + measures=[ + {"name": "Order Count", "expr": "COUNT(1)"}, + {"name": "Total Revenue", "expr": "SUM(total_price)", "comment": "Sum of total price"}, + ], +) +``` + +### Query via MCP + +```python +manage_metric_views( + action="query", + full_name="catalog.schema.orders_metrics", + query_measures=["Total Revenue", "Order Count"], + query_dimensions=["Order Month"], + where="extract(year FROM `Order Month`) = 2024", + order_by="ALL", + limit=100, +) +``` + +### Describe via MCP + +```python +manage_metric_views( + action="describe", + full_name="catalog.schema.orders_metrics", +) +``` + +### Grant Access + +```python +manage_metric_views( + action="grant", + full_name="catalog.schema.orders_metrics", + principal="data-consumers", + privileges=["SELECT"], +) +``` + +## YAML Spec Quick Reference + +```yaml +version: 1.1 # Required: "1.1" for DBR 17.2+ +source: catalog.schema.table # Required: source table/view +comment: "Description" # Optional: metric view description +filter: column > value # Optional: global WHERE filter + +dimensions: # Required: at least one + - name: Display Name # Backtick-quoted in queries + expr: sql_expression # Column ref or SQL transformation + comment: "Description" # Optional (v1.1+) + +measures: # Required: at least one + - name: Display Name # Queried via MEASURE(`name`) + expr: AGG_FUNC(column) # Must be an aggregate expression + comment: "Description" # Optional (v1.1+) + +joins: # Optional: star/snowflake schema + - name: dim_table + source: catalog.schema.dim_table + on: source.fk = dim_table.pk + +materialization: # Optional (experimental) + schedule: every 6 hours + mode: relaxed +``` + +## Key Concepts + +### Dimensions vs Measures + +| | Dimensions | Measures | +|---|---|---| +| **Purpose** | Categorize and group data | Aggregate numeric values | +| **Examples** | Region, Date, Status | SUM(revenue), COUNT(orders) | +| **In queries** | Used in SELECT and GROUP BY | Wrapped in `MEASURE()` | +| **SQL expressions** | Any SQL expression | Must use aggregate functions | + +### Why Metric Views vs Standard Views? + +| Feature | Standard Views | Metric Views | +|---------|---------------|--------------| +| Aggregation locked at creation | Yes | No - flexible at query time | +| Safe re-aggregation of ratios | No | Yes | +| Star/snowflake schema joins | Manual | Declarative in YAML | +| Materialization | Separate MV needed | Built-in | +| AI/BI Genie integration | Limited | Native | + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **SELECT * not supported** | Must explicitly list dimensions and use MEASURE() for measures | +| **"Cannot resolve column"** | Dimension/measure names with spaces need backtick quoting | +| **JOIN at query time fails** | Joins must be in the YAML definition, not in the SELECT query | +| **MEASURE() required** | All measure references must be wrapped: `MEASURE(\`name\`)` | +| **DBR version error** | Requires Runtime 17.2+ for YAML v1.1, or 16.4+ for v0.1 | +| **Materialization not working** | Requires serverless compute enabled; currently experimental | + +## Integrations + +Metric views work natively with: +- **AI/BI Dashboards** - Use as datasets for visualizations +- **AI/BI Genie** - Natural language querying of metrics +- **Alerts** - Set threshold-based alerts on measures +- **SQL Editor** - Direct SQL querying with MEASURE() +- **Catalog Explorer UI** - Visual creation and browsing + +## Resources + +- [Metric Views Documentation](https://docs.databricks.com/en/metric-views/) +- [YAML Syntax Reference](https://docs.databricks.com/en/metric-views/data-modeling/syntax) +- [Joins](https://docs.databricks.com/en/metric-views/data-modeling/joins) +- [Window Measures](https://docs.databricks.com/aws/en/metric-views/data-modeling/window-measures) (Experimental) +- [Materialization](https://docs.databricks.com/en/metric-views/materialization) +- [MEASURE() Function](https://docs.databricks.com/en/sql/language-manual/functions/measure) diff --git a/.claude/skills/databricks-metric-views/patterns.md b/.claude/skills/databricks-metric-views/patterns.md new file mode 100644 index 00000000..1f067f4c --- /dev/null +++ b/.claude/skills/databricks-metric-views/patterns.md @@ -0,0 +1,651 @@ +# Metric View Patterns & Examples + +Common patterns for creating and querying metric views. + +## Pattern 1: Simple Metrics from a Single Table + +The most basic pattern with direct column dimensions and standard aggregations. + +### Create + +```sql +CREATE OR REPLACE VIEW catalog.schema.product_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.sales + comment: "Product sales metrics" + dimensions: + - name: Product Name + expr: product_name + - name: Sale Date + expr: sale_date + measures: + - name: Units Sold + expr: COUNT(1) + - name: Total Revenue + expr: SUM(price * quantity) + - name: Average Price + expr: AVG(price) +$$ +``` + +### Query + +```sql +-- Revenue by product +SELECT + `Product Name`, + MEASURE(`Total Revenue`) AS revenue, + MEASURE(`Units Sold`) AS units +FROM catalog.schema.product_metrics +GROUP BY ALL +ORDER BY revenue DESC +LIMIT 10 + +-- Monthly trend +SELECT + DATE_TRUNC('MONTH', `Sale Date`) AS month, + MEASURE(`Total Revenue`) AS revenue +FROM catalog.schema.product_metrics +GROUP BY ALL +ORDER BY month +``` + +## Pattern 2: Derived Dimensions with CASE + +Transform raw values into business-friendly categories. + +```sql +CREATE OR REPLACE VIEW catalog.schema.order_kpis +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.orders + dimensions: + - name: Order Month + expr: DATE_TRUNC('MONTH', order_date) + - name: Priority Level + expr: CASE + WHEN priority <= 2 THEN 'High' + WHEN priority <= 4 THEN 'Medium' + ELSE 'Low' + END + comment: "Bucketed priority: High (1-2), Medium (3-4), Low (5)" + - name: Size Category + expr: CASE + WHEN total_amount > 10000 THEN 'Large' + WHEN total_amount > 1000 THEN 'Medium' + ELSE 'Small' + END + measures: + - name: Order Count + expr: COUNT(1) + - name: Total Amount + expr: SUM(total_amount) +$$ +``` + +## Pattern 3: Ratio Measures + +Ratios and per-unit metrics that safely handle re-aggregation. + +```sql +CREATE OR REPLACE VIEW catalog.schema.efficiency_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.transactions + comment: "Efficiency and per-unit metrics" + dimensions: + - name: Department + expr: department_name + - name: Quarter + expr: DATE_TRUNC('QUARTER', transaction_date) + measures: + - name: Total Revenue + expr: SUM(revenue) + - name: Total Cost + expr: SUM(cost) + - name: Profit Margin + expr: (SUM(revenue) - SUM(cost)) / SUM(revenue) + comment: "Profit as percentage of revenue" + - name: Revenue per Employee + expr: SUM(revenue) / COUNT(DISTINCT employee_id) + - name: Average Transaction Size + expr: SUM(revenue) / COUNT(1) +$$ +``` + +## Pattern 4: Filtered Measures (FILTER clause) + +Create measures that only count a subset of rows. + +```sql +CREATE OR REPLACE VIEW catalog.schema.order_status_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.orders + dimensions: + - name: Order Month + expr: DATE_TRUNC('MONTH', order_date) + - name: Region + expr: region + measures: + - name: Total Orders + expr: COUNT(1) + - name: Open Orders + expr: COUNT(1) FILTER (WHERE status = 'OPEN') + - name: Fulfilled Orders + expr: COUNT(1) FILTER (WHERE status = 'FULFILLED') + - name: Open Revenue + expr: SUM(amount) FILTER (WHERE status = 'OPEN') + comment: "Revenue at risk from unfulfilled orders" + - name: Fulfillment Rate + expr: COUNT(1) FILTER (WHERE status = 'FULFILLED') * 1.0 / COUNT(1) + comment: "Percentage of orders fulfilled" +$$ +``` + +### Query filtered measures + +```sql +SELECT + `Order Month`, + MEASURE(`Total Orders`) AS total, + MEASURE(`Open Orders`) AS open_orders, + MEASURE(`Fulfillment Rate`) AS fulfillment_rate +FROM catalog.schema.order_status_metrics +WHERE `Region` = 'EMEA' +GROUP BY ALL +ORDER BY ALL +``` + +## Pattern 5: Star Schema with Joins + +Join a fact table to dimension tables. + +```sql +CREATE OR REPLACE VIEW catalog.schema.sales_analytics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.fact_sales + comment: "Sales analytics with customer and product dimensions" + + joins: + - name: customer + source: catalog.schema.dim_customer + on: source.customer_id = customer.customer_id + - name: product + source: catalog.schema.dim_product + on: source.product_id = product.product_id + - name: store + source: catalog.schema.dim_store + on: source.store_id = store.store_id + + dimensions: + - name: Customer Segment + expr: customer.segment + - name: Product Category + expr: product.category + - name: Store City + expr: store.city + - name: Sale Month + expr: DATE_TRUNC('MONTH', source.sale_date) + + measures: + - name: Total Revenue + expr: SUM(source.amount) + - name: Unique Customers + expr: COUNT(DISTINCT source.customer_id) + - name: Average Basket Size + expr: SUM(source.amount) / COUNT(DISTINCT source.transaction_id) +$$ +``` + +## Pattern 6: Snowflake Schema (Nested Joins) + +Multi-level dimension hierarchies. Requires DBR 17.1+. + +```sql +CREATE OR REPLACE VIEW catalog.schema.geo_sales +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.orders + + joins: + - name: customer + source: catalog.schema.customer + on: source.customer_key = customer.customer_key + joins: + - name: nation + source: catalog.schema.nation + on: customer.nation_key = nation.nation_key + joins: + - name: region + source: catalog.schema.region + on: nation.region_key = region.region_key + + dimensions: + - name: Customer Name + expr: customer.name + - name: Nation + expr: nation.name + - name: Region + expr: region.name + - name: Order Year + expr: EXTRACT(YEAR FROM source.order_date) + + measures: + - name: Total Revenue + expr: SUM(source.total_price) + - name: Order Count + expr: COUNT(1) +$$ +``` + +### Query across hierarchy levels + +```sql +-- Revenue by region (rolls up across nations and customers) +SELECT + `Region`, + MEASURE(`Total Revenue`) AS revenue +FROM catalog.schema.geo_sales +GROUP BY ALL + +-- Revenue by nation within a specific region +SELECT + `Nation`, + MEASURE(`Total Revenue`) AS revenue, + MEASURE(`Order Count`) AS orders +FROM catalog.schema.geo_sales +WHERE `Region` = 'EUROPE' +GROUP BY ALL +ORDER BY revenue DESC +``` + +## Pattern 7: Materialized Metric View + +Pre-compute common aggregations for faster queries. + +```sql +CREATE OR REPLACE VIEW catalog.schema.ecommerce_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: catalog.schema.transactions + + dimensions: + - name: Category + expr: product_category + - name: Day + expr: DATE_TRUNC('DAY', transaction_date) + - name: Channel + expr: sales_channel + + measures: + - name: Revenue + expr: SUM(amount) + - name: Transactions + expr: COUNT(1) + - name: Unique Buyers + expr: COUNT(DISTINCT customer_id) + + materialization: + schedule: every 1 hour + mode: relaxed + materialized_views: + - name: daily_category + type: aggregated + dimensions: + - Category + - Day + measures: + - Revenue + - Transactions + - name: full_model + type: unaggregated +$$ +``` + +## Pattern 8: Using samples.tpch for Quick Demos + +The TPC-H sample dataset is available on all Databricks workspaces. + +```sql +CREATE OR REPLACE VIEW catalog.schema.tpch_orders_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + source: samples.tpch.orders + comment: "TPC-H Orders KPIs - demo metric view" + filter: o_orderdate > '1990-01-01' + + dimensions: + - name: Order Month + expr: DATE_TRUNC('MONTH', o_orderdate) + comment: "Month of order" + - name: Order Status + expr: CASE + WHEN o_orderstatus = 'O' THEN 'Open' + WHEN o_orderstatus = 'P' THEN 'Processing' + WHEN o_orderstatus = 'F' THEN 'Fulfilled' + END + comment: "Status: Open, Processing, or Fulfilled" + - name: Order Priority + expr: SPLIT(o_orderpriority, '-')[1] + comment: "Numeric priority 1-5; 1 is highest" + + measures: + - name: Order Count + expr: COUNT(1) + - name: Total Revenue + expr: SUM(o_totalprice) + comment: "Sum of total price" + - name: Revenue per Customer + expr: SUM(o_totalprice) / COUNT(DISTINCT o_custkey) + comment: "Average revenue per distinct customer" + - name: Open Order Revenue + expr: SUM(o_totalprice) FILTER (WHERE o_orderstatus = 'O') + comment: "Potential revenue from open orders" +$$ +``` + +### Demo queries + +```sql +-- Monthly revenue trend +SELECT + `Order Month`, + MEASURE(`Total Revenue`)::BIGINT AS revenue, + MEASURE(`Order Count`) AS orders +FROM catalog.schema.tpch_orders_metrics +WHERE extract(year FROM `Order Month`) = 1995 +GROUP BY ALL +ORDER BY ALL + +-- Revenue by status +SELECT + `Order Status`, + MEASURE(`Total Revenue`)::BIGINT AS revenue, + MEASURE(`Revenue per Customer`)::BIGINT AS rev_per_customer +FROM catalog.schema.tpch_orders_metrics +GROUP BY ALL + +-- Open orders risk assessment +SELECT + `Order Month`, + MEASURE(`Open Order Revenue`)::BIGINT AS at_risk_revenue, + MEASURE(`Total Revenue`)::BIGINT AS total_revenue +FROM catalog.schema.tpch_orders_metrics +WHERE extract(year FROM `Order Month`) >= 1995 +GROUP BY ALL +ORDER BY ALL +``` + +## Pattern 9: Window Measures (Experimental) + +Window measures enable moving averages, running totals, period-over-period changes, and semiadditive measures. Add a `window` block to any measure definition. See [Window Measures Documentation](https://docs.databricks.com/aws/en/metric-views/data-modeling/window-measures). + +### Window Range Values + +| Range | Description | +|-------|-------------| +| `current` | Only rows where the window ordering value equals the current row | +| `cumulative` | All rows up to and including the current row | +| `trailing ` | N units before the current row (**excludes** current) | +| `leading ` | N units after the current row | +| `all` | All rows regardless of ordering | + +### Trailing Window: 7-Day Distinct Customers + +```sql +CREATE OR REPLACE VIEW catalog.schema.customer_activity +WITH METRICS +LANGUAGE YAML +AS $$ + version: 0.1 + source: catalog.schema.orders + filter: order_date > DATE'2024-01-01' + + dimensions: + - name: date + expr: order_date + + measures: + - name: t7d_customers + expr: COUNT(DISTINCT customer_id) + window: + - order: date + range: trailing 7 day + semiadditive: last +$$ +``` + +**Key:** `trailing 7 day` includes the 7 days **before** each date, **excluding** the current date. `semiadditive: last` returns the last value when the `date` dimension is not in the GROUP BY. + +### Running Total (Cumulative) + +```sql +CREATE OR REPLACE VIEW catalog.schema.cumulative_sales +WITH METRICS +LANGUAGE YAML +AS $$ + version: 0.1 + source: catalog.schema.orders + filter: order_date > DATE'2024-01-01' + + dimensions: + - name: date + expr: order_date + + measures: + - name: running_total_sales + expr: SUM(total_price) + window: + - order: date + range: cumulative + semiadditive: last +$$ +``` + +### Period-Over-Period: Day-Over-Day Growth + +Compose window measures using `MEASURE()` references in derived measures. + +```sql +CREATE OR REPLACE VIEW catalog.schema.daily_growth +WITH METRICS +LANGUAGE YAML +AS $$ + version: 0.1 + source: catalog.schema.orders + filter: order_date > DATE'2024-01-01' + + dimensions: + - name: date + expr: order_date + + measures: + - name: previous_day_sales + expr: SUM(total_price) + window: + - order: date + range: trailing 1 day + semiadditive: last + + - name: current_day_sales + expr: SUM(total_price) + window: + - order: date + range: current + semiadditive: last + + - name: day_over_day_growth + expr: (MEASURE(current_day_sales) - MEASURE(previous_day_sales)) / MEASURE(previous_day_sales) * 100 +$$ +``` + +**Key:** The derived `day_over_day_growth` measure uses `MEASURE()` to reference other window measures. It does NOT need its own `window` block. + +### Year-to-Date (Composing Multiple Windows) + +A single measure can have multiple window specs to create period-to-date calculations. + +```sql +CREATE OR REPLACE VIEW catalog.schema.ytd_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 0.1 + source: catalog.schema.orders + filter: order_date > DATE'2023-01-01' + + dimensions: + - name: date + expr: order_date + - name: year + expr: DATE_TRUNC('year', order_date) + + measures: + - name: ytd_sales + expr: SUM(total_price) + window: + - order: date + range: cumulative + semiadditive: last + - order: year + range: current + semiadditive: last +$$ +``` + +**Key:** The first window does a cumulative sum over `date`. The second window restricts scope to the `current` year. Together they produce year-to-date. + +### Semiadditive Measure: Bank Balance + +For measures like balances that should not be summed across time. + +```sql +CREATE OR REPLACE VIEW catalog.schema.account_balances +WITH METRICS +LANGUAGE YAML +AS $$ + version: 0.1 + source: catalog.schema.daily_balances + + dimensions: + - name: date + expr: date + - name: customer + expr: customer_id + + measures: + - name: balance + expr: SUM(balance) + window: + - order: date + range: current + semiadditive: last +$$ +``` + +**Key:** `semiadditive: last` prevents summing across dates (returns the last date's value instead), but the measure **still aggregates across other dimensions** like `customer`. When grouped by date, you get total balance across all customers for that day. When not grouped by date, you get the balance from the most recent date. + +### Query window measures + +Window measures are queried with the same `MEASURE()` syntax: + +```sql +SELECT + date, + MEASURE(t7d_customers) AS trailing_7d_customers, + MEASURE(running_total_sales) AS running_total +FROM catalog.schema.customer_activity +WHERE date >= DATE'2024-06-01' +GROUP BY ALL +ORDER BY ALL +``` + +## MCP Tool Examples + +### Create with joins + +```python +manage_metric_views( + action="create", + full_name="catalog.schema.sales_metrics", + source="catalog.schema.fact_sales", + or_replace=True, + joins=[ + { + "name": "customer", + "source": "catalog.schema.dim_customer", + "on": "source.customer_id = customer.id" + }, + { + "name": "product", + "source": "catalog.schema.dim_product", + "on": "source.product_id = product.id" + } + ], + dimensions=[ + {"name": "Customer Segment", "expr": "customer.segment"}, + {"name": "Product Category", "expr": "product.category"}, + {"name": "Sale Month", "expr": "DATE_TRUNC('MONTH', source.sale_date)"}, + ], + measures=[ + {"name": "Total Revenue", "expr": "SUM(source.amount)"}, + {"name": "Order Count", "expr": "COUNT(1)"}, + {"name": "Unique Customers", "expr": "COUNT(DISTINCT source.customer_id)"}, + ], +) +``` + +### Alter to add a new measure + +```python +manage_metric_views( + action="alter", + full_name="catalog.schema.sales_metrics", + source="catalog.schema.fact_sales", + joins=[ + {"name": "customer", "source": "catalog.schema.dim_customer", "on": "source.customer_id = customer.id"}, + ], + dimensions=[ + {"name": "Customer Segment", "expr": "customer.segment"}, + {"name": "Sale Month", "expr": "DATE_TRUNC('MONTH', source.sale_date)"}, + ], + measures=[ + {"name": "Total Revenue", "expr": "SUM(source.amount)"}, + {"name": "Order Count", "expr": "COUNT(1)"}, + {"name": "Average Order Value", "expr": "AVG(source.amount)"}, # New measure + ], +) +``` + +### Query with filters + +```python +manage_metric_views( + action="query", + full_name="catalog.schema.sales_metrics", + query_measures=["Total Revenue", "Order Count"], + query_dimensions=["Customer Segment", "Sale Month"], + where="`Customer Segment` = 'Enterprise'", + order_by="ALL", + limit=50, +) +``` diff --git a/.claude/skills/databricks-metric-views/yaml-reference.md b/.claude/skills/databricks-metric-views/yaml-reference.md new file mode 100644 index 00000000..2e5973c0 --- /dev/null +++ b/.claude/skills/databricks-metric-views/yaml-reference.md @@ -0,0 +1,338 @@ +# Metric View YAML Reference + +Complete reference for the YAML specification used in Unity Catalog metric views. + +## Top-Level Fields + +| Field | Required | Type | Description | +|-------|----------|------|-------------| +| `version` | No | string | YAML spec version. `"1.1"` for DBR 17.2+, `"0.1"` for DBR 16.4-17.1. Defaults to `1.1`. | +| `source` | Yes | string | Source table, view, or SQL query in three-level namespace format. | +| `comment` | No | string | Description of the metric view (v1.1+). | +| `filter` | No | string | SQL boolean expression applied as a global WHERE clause. | +| `dimensions` | Yes | list | Array of dimension definitions (at least one). | +| `measures` | Yes | list | Array of measure definitions (at least one). | +| `joins` | No | list | Star/snowflake schema join definitions. | +| `materialization` | No | object | Pre-computation configuration (experimental). | + +## Dimensions + +Dimensions define the categorical attributes used to group and filter data. + +```yaml +dimensions: + - name: Region # Display name, backtick-quoted in queries + expr: region_name # Direct column reference + comment: "Sales region" # Optional description (v1.1+) + + - name: Order Month + expr: DATE_TRUNC('MONTH', order_date) # SQL transformation + + - name: Order Year + expr: EXTRACT(YEAR FROM `Order Month`) # Can reference other dimensions + + - name: Customer Type + expr: CASE + WHEN customer_tier = 'A' THEN 'Enterprise' + WHEN customer_tier = 'B' THEN 'Mid-Market' + ELSE 'SMB' + END # Multi-line CASE expressions supported + + - name: Nation + expr: customer.c_name # Reference joined table columns +``` + +### Dimension Rules + +- `name` is required and becomes the column name in queries (backtick-quoted if it has spaces) +- `expr` is required and must be a valid SQL expression +- Can reference source columns, SQL functions, CASE expressions, and other dimensions +- Can reference columns from joined tables using `join_name.column_name` +- Cannot use aggregate functions (those belong in measures) + +## Measures + +Measures define aggregated values computed at query time. + +```yaml +measures: + - name: Total Revenue + expr: SUM(total_price) + comment: "Sum of all order prices" + + - name: Order Count + expr: COUNT(1) + + - name: Average Order Value + expr: AVG(total_price) + + - name: Unique Customers + expr: COUNT(DISTINCT customer_id) + + - name: Revenue per Customer # Ratio measure + expr: SUM(total_price) / COUNT(DISTINCT customer_id) + + - name: Open Order Revenue # Filtered measure + expr: SUM(total_price) FILTER (WHERE status = 'O') + comment: "Revenue from open orders only" + + - name: Open Revenue per Customer # Filtered ratio + expr: SUM(total_price) FILTER (WHERE status = 'O') / COUNT(DISTINCT customer_id) FILTER (WHERE status = 'O') +``` + +### Window Measures (Experimental) + +Add a `window` block to a measure for windowed, cumulative, or semiadditive aggregations. See [Window Measures Documentation](https://docs.databricks.com/aws/en/metric-views/data-modeling/window-measures). + +```yaml +measures: + - name: Running Total + expr: SUM(total_price) + window: + - order: date # Dimension that orders the window + range: cumulative # Window extent (see range values below) + semiadditive: last # How to summarize when order dim is not in GROUP BY + + - name: 7-Day Customers + expr: COUNT(DISTINCT customer_id) + window: + - order: date + range: trailing 7 day # 7 days before current, EXCLUDING current day + semiadditive: last +``` + +**Window range values:** + +| Range | Description | +|-------|-------------| +| `current` | Only rows matching the current ordering value | +| `cumulative` | All rows up to and including the current row | +| `trailing ` | N units before current row (excludes current) | +| `leading ` | N units after current row | +| `all` | All rows | + +**Window spec fields:** + +| Field | Required | Description | +|-------|----------|-------------| +| `order` | Yes | Dimension name that determines window ordering | +| `range` | Yes | Window extent (see values above) | +| `semiadditive` | Yes | `first` or `last` - value to use when order dimension is absent from GROUP BY | + +**Multiple windows** can be composed on a single measure (e.g., for year-to-date): + +```yaml + - name: ytd_sales + expr: SUM(total_price) + window: + - order: date + range: cumulative + semiadditive: last + - order: year + range: current + semiadditive: last +``` + +**Derived measures** can reference window measures using `MEASURE()`: + +```yaml + - name: day_over_day_growth + expr: (MEASURE(current_day_sales) - MEASURE(previous_day_sales)) / MEASURE(previous_day_sales) * 100 +``` + +### Measure Rules + +- `name` is required and queried via `MEASURE(\`name\`)` +- `expr` must contain an aggregate function (SUM, COUNT, AVG, MIN, MAX, etc.) +- Supports `FILTER (WHERE ...)` for conditional aggregation +- Supports ratios of aggregates +- Derived measures can reference other measures via `MEASURE()` (used with window measures) +- Window measures use `version: 0.1` (experimental feature) +- `SELECT *` on metric views is NOT supported; must use `MEASURE()` explicitly + +## Joins + +### Star Schema (Single Level) + +```yaml +source: catalog.schema.fact_orders +joins: + - name: customer + source: catalog.schema.dim_customer + on: source.customer_id = customer.id + + - name: product + source: catalog.schema.dim_product + on: source.product_id = product.id +``` + +### Star Schema with USING + +```yaml +joins: + - name: customer + source: catalog.schema.dim_customer + using: + - customer_id + - region_id +``` + +### Snowflake Schema (Nested Joins, DBR 17.1+) + +```yaml +source: catalog.schema.orders +joins: + - name: customer + source: catalog.schema.customer + on: source.customer_id = customer.id + joins: + - name: nation + source: catalog.schema.nation + on: customer.nation_id = nation.id + joins: + - name: region + source: catalog.schema.region + on: nation.region_id = region.id +``` + +### Join Rules + +- `name` is required and used to reference joined columns: `name.column` +- `source` is the fully qualified table/view name +- Use either `on` (expression) or `using` (column list), not both +- In `on`, reference the fact table as `source` and join tables by their `name` +- Nested `joins` create snowflake schema (requires DBR 17.1+) +- Joined tables cannot include MAP type columns + +## Filter + +A global filter applied to all queries as a WHERE clause. + +```yaml +filter: order_date > '2020-01-01' + +# Multiple conditions +filter: order_date > '2020-01-01' AND status != 'CANCELLED' + +# Using joined columns +filter: customer.active = true +``` + +## Materialization (Experimental) + +Pre-compute aggregations for faster query performance. Uses Lakeflow Spark Declarative Pipelines under the hood. + +```yaml +materialization: + schedule: every 6 hours # Same syntax as MV schedule clause + mode: relaxed # Only "relaxed" supported currently + + materialized_views: + - name: baseline + type: unaggregated # Full unaggregated data model + + - name: revenue_breakdown + type: aggregated # Pre-computed aggregation + dimensions: + - category + - region + measures: + - total_revenue + - order_count + + - name: daily_summary + type: aggregated + dimensions: + - order_date + measures: + - total_revenue +``` + +### Materialization Types + +| Type | Description | When to Use | +|------|-------------|-------------| +| `unaggregated` | Materializes full data model (source + joins + filter) | Expensive source views or many joins | +| `aggregated` | Pre-computes specific dimension/measure combos | Frequently queried combinations | + +### Materialization Requirements + +- Serverless compute must be enabled +- Databricks Runtime 17.2+ +- `TRIGGER ON UPDATE` clause is not supported +- Schedule uses same syntax as materialized view schedules + +### Refresh Materialization + +```python +# Find and refresh the pipeline +from databricks.sdk import WorkspaceClient +w = WorkspaceClient() +pipeline_id = "your-pipeline-id" +w.pipelines.start_update(pipeline_id) +``` + +## Complete Example + +```sql +CREATE OR REPLACE VIEW catalog.schema.sales_metrics +WITH METRICS +LANGUAGE YAML +AS $$ + version: 1.1 + comment: "Comprehensive sales metrics with customer and product dimensions" + source: catalog.schema.fact_sales + filter: sale_date >= '2023-01-01' + + joins: + - name: customer + source: catalog.schema.dim_customer + on: source.customer_id = customer.id + joins: + - name: region + source: catalog.schema.dim_region + on: customer.region_id = region.id + - name: product + source: catalog.schema.dim_product + on: source.product_id = product.id + + dimensions: + - name: Sale Month + expr: DATE_TRUNC('MONTH', sale_date) + comment: "Month of sale" + - name: Customer Name + expr: customer.name + - name: Region + expr: region.name + comment: "Geographic region" + - name: Product Category + expr: product.category + + measures: + - name: Total Revenue + expr: SUM(amount) + comment: "Sum of sale amounts" + - name: Transaction Count + expr: COUNT(1) + - name: Unique Customers + expr: COUNT(DISTINCT customer_id) + - name: Average Transaction + expr: AVG(amount) + - name: Revenue per Customer + expr: SUM(amount) / COUNT(DISTINCT customer_id) + comment: "Average revenue per unique customer" + + materialization: + schedule: every 1 hour + mode: relaxed + materialized_views: + - name: hourly_region + type: aggregated + dimensions: + - Sale Month + - Region + measures: + - Total Revenue + - Transaction Count +$$ +``` diff --git a/.claude/skills/databricks-mlflow-evaluation/SKILL.md b/.claude/skills/databricks-mlflow-evaluation/SKILL.md new file mode 100644 index 00000000..45db5f61 --- /dev/null +++ b/.claude/skills/databricks-mlflow-evaluation/SKILL.md @@ -0,0 +1,148 @@ +--- +name: databricks-mlflow-evaluation +description: "MLflow 3 GenAI agent evaluation. Use when writing mlflow.genai.evaluate() code, creating @scorer functions, using built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), building eval datasets from traces, setting up trace ingestion and production monitoring, aligning judges with MemAlign from domain expert feedback, or running optimize_prompts() with GEPA for automated prompt improvement." +--- + +# MLflow 3 GenAI Evaluation + +## Before Writing Any Code + +1. **Read GOTCHAS.md** - 15+ common mistakes that cause failures +2. **Read CRITICAL-interfaces.md** - Exact API signatures and data schemas + +## End-to-End Workflows + +Follow these workflows based on your goal. Each step indicates which reference files to read. + +### Workflow 1: First-Time Evaluation Setup + +For users new to MLflow GenAI evaluation or setting up evaluation for a new agent. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Understand what to evaluate | `user-journeys.md` (Journey 0: Strategy) | +| 2 | Learn API patterns | `GOTCHAS.md` + `CRITICAL-interfaces.md` | +| 3 | Build initial dataset | `patterns-datasets.md` (Patterns 1-4) | +| 4 | Choose/create scorers | `patterns-scorers.md` + `CRITICAL-interfaces.md` (built-in list) | +| 5 | Run evaluation | `patterns-evaluation.md` (Patterns 1-3) | + +### Workflow 2: Production Trace -> Evaluation Dataset + +For building evaluation datasets from production traces. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Search and filter traces | `patterns-trace-analysis.md` (MCP tools section) | +| 2 | Analyze trace quality | `patterns-trace-analysis.md` (Patterns 1-7) | +| 3 | Tag traces for inclusion | `patterns-datasets.md` (Patterns 16-17) | +| 4 | Build dataset from traces | `patterns-datasets.md` (Patterns 6-7) | +| 5 | Add expectations/ground truth | `patterns-datasets.md` (Pattern 2) | + +### Workflow 3: Performance Optimization + +For debugging slow or expensive agent execution. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Profile latency by span | `patterns-trace-analysis.md` (Patterns 4-6) | +| 2 | Analyze token usage | `patterns-trace-analysis.md` (Pattern 9) | +| 3 | Detect context issues | `patterns-context-optimization.md` (Section 5) | +| 4 | Apply optimizations | `patterns-context-optimization.md` (Sections 1-4, 6) | +| 5 | Re-evaluate to measure impact | `patterns-evaluation.md` (Pattern 6-7) | + +### Workflow 4: Regression Detection + +For comparing agent versions and finding regressions. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Establish baseline | `patterns-evaluation.md` (Pattern 4: named runs) | +| 2 | Run current version | `patterns-evaluation.md` (Pattern 1) | +| 3 | Compare metrics | `patterns-evaluation.md` (Patterns 6-7) | +| 4 | Analyze failing traces | `patterns-trace-analysis.md` (Pattern 7) | +| 5 | Debug specific failures | `patterns-trace-analysis.md` (Patterns 8-9) | + +### Workflow 5: Custom Scorer Development + +For creating project-specific evaluation metrics. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Understand scorer interface | `CRITICAL-interfaces.md` (Scorer section) | +| 2 | Choose scorer pattern | `patterns-scorers.md` (Patterns 4-11) | +| 3 | For multi-agent scorers | `patterns-scorers.md` (Patterns 13-16) | +| 4 | Test with evaluation | `patterns-evaluation.md` (Pattern 1) | + +### Workflow 6: Unity Catalog Trace Ingestion & Production Monitoring + +For storing traces in Unity Catalog, instrumenting applications, and enabling continuous production monitoring. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Link UC schema to experiment | `patterns-trace-ingestion.md` (Patterns 1-2) | +| 2 | Set trace destination | `patterns-trace-ingestion.md` (Patterns 3-4) | +| 3 | Instrument your application | `patterns-trace-ingestion.md` (Patterns 5-8) | +| 4 | Configure trace sources (Apps/Serving/OTEL) | `patterns-trace-ingestion.md` (Patterns 9-11) | +| 5 | Enable production monitoring | `patterns-trace-ingestion.md` (Patterns 12-13) | +| 6 | Query and analyze UC traces | `patterns-trace-ingestion.md` (Pattern 14) | + +### Workflow 7: Judge Alignment with MemAlign + +For aligning an LLM judge to match domain expert preferences. A well-aligned judge improves every downstream use: evaluation accuracy, production monitoring signal, and prompt optimization quality. This workflow is valuable on its own, independent of prompt optimization. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Design base judge with `make_judge` (any feedback type) | `patterns-judge-alignment.md` (Pattern 1) | +| 2 | Run evaluate(), tag successful traces | `patterns-judge-alignment.md` (Pattern 2) | +| 3 | Build UC dataset + create SME labeling session | `patterns-judge-alignment.md` (Pattern 3) | +| 4 | Align judge with MemAlign after labeling completes | `patterns-judge-alignment.md` (Pattern 4) | +| 5 | Register aligned judge to experiment | `patterns-judge-alignment.md` (Pattern 5) | +| 6 | Re-evaluate with aligned judge (baseline) | `patterns-judge-alignment.md` (Pattern 6) | + +### Workflow 8: Automated Prompt Optimization with GEPA + +For automatically improving a registered system prompt using `optimize_prompts()`. Works with any scorer, but paired with an aligned judge (Workflow 7) gives the most domain-accurate signal. For the full end-to-end loop combining alignment and optimization, see `user-journeys.md` Journey 10. + +| Step | Action | Reference Files | +|------|--------|-----------------| +| 1 | Build optimization dataset (inputs + expectations) | `patterns-prompt-optimization.md` (Pattern 1) | +| 2 | Run optimize_prompts() with GEPA + scorer | `patterns-prompt-optimization.md` (Pattern 2) | +| 3 | Register new version, promote conditionally | `patterns-prompt-optimization.md` (Pattern 3) | + +## Reference Files Quick Lookup + +| Reference | Purpose | When to Read | +|-----------|---------|--------------| +| `GOTCHAS.md` | Common mistakes | **Always read first** before writing code | +| `CRITICAL-interfaces.md` | API signatures, schemas | When writing any evaluation code | +| `patterns-evaluation.md` | Running evals, comparing | When executing evaluations | +| `patterns-scorers.md` | Custom scorer creation | When built-in scorers aren't enough | +| `patterns-datasets.md` | Dataset building | When preparing evaluation data | +| `patterns-trace-analysis.md` | Trace debugging | When analyzing agent behavior | +| `patterns-context-optimization.md` | Token/latency fixes | When agent is slow or expensive | +| `patterns-trace-ingestion.md` | UC trace setup, monitoring | When setting up trace storage or production monitoring | +| `patterns-judge-alignment.md` | MemAlign judge alignment, labeling sessions, SME feedback | When aligning judges to domain expert preferences | +| `patterns-prompt-optimization.md` | GEPA optimization: build dataset, optimize_prompts(), promote | When running automated prompt improvement | +| `user-journeys.md` | High-level workflows, full domain-expert optimization loop | When starting a new evaluation project or running the full align + optimize cycle | + +## Critical API Facts + +- **Use:** `mlflow.genai.evaluate()` (NOT `mlflow.evaluate()`) +- **Data format:** `{"inputs": {"query": "..."}}` (nested structure required) +- **predict_fn:** Receives `**unpacked kwargs` (not a dict) +- **MemAlign:** Scorer-agnostic (works with any `feedback_value_type` -- float, bool, categorical); token-heavy on the embedding model so set `embedding_model` explicitly +- **Label schema name matching:** The label schema `name` in the labeling session MUST match the judge `name` used in `evaluate()` for `align()` to pair scores +- **Aligned judge scores:** May be lower than unaligned judge scores -- this is expected and means the judge is now more accurate, not that the agent regressed +- **GEPA optimization dataset:** Must have both `inputs` AND `expectations` per record (different from eval dataset) +- **Episodic memory:** Lazily loaded -- `get_scorer()` results won't show episodic memory on print until the judge is first used +- **optimize_prompts:** Requires MLflow >= 3.5.0 + +See `GOTCHAS.md` for complete list. + +## Related Skills + +- **[databricks-docs](../databricks-docs/SKILL.md)** - General Databricks documentation reference +- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** - Deploying models and agents to serving endpoints +- **[databricks-agent-bricks](../databricks-agent-bricks/SKILL.md)** - Building agents that can be evaluated with this skill +- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** - SDK patterns used alongside MLflow APIs +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - Unity Catalog tables for managed evaluation datasets diff --git a/.claude/skills/mlflow-evaluation/references/CRITICAL-interfaces.md b/.claude/skills/databricks-mlflow-evaluation/references/CRITICAL-interfaces.md similarity index 87% rename from .claude/skills/mlflow-evaluation/references/CRITICAL-interfaces.md rename to .claude/skills/databricks-mlflow-evaluation/references/CRITICAL-interfaces.md index d1b4a1b1..30babcea 100644 --- a/.claude/skills/mlflow-evaluation/references/CRITICAL-interfaces.md +++ b/.claude/skills/databricks-mlflow-evaluation/references/CRITICAL-interfaces.md @@ -12,6 +12,7 @@ - [Judges API (Low-level)](#judges-api-low-level) - [Trace APIs](#trace-apis) - [Evaluation Datasets (MLflow-managed)](#evaluation-datasets-mlflow-managed) +- [Trace Ingestion in Unity Catalog](#trace-ingestion-in-unity-catalog) - [Production Monitoring](#production-monitoring) - [Key Constants](#key-constants) - [Installation](#installation) @@ -385,8 +386,68 @@ results = mlflow.genai.evaluate( --- +## Trace Ingestion in Unity Catalog + +**Version**: MLflow 3.9.0+ (`mlflow[databricks]>=3.9.0`) + +### Setup - Link UC Schema to Experiment +```python +import os +import mlflow +from mlflow.entities import UCSchemaLocation +from mlflow.tracing.enablement import set_experiment_trace_location + +mlflow.set_tracking_uri("databricks") +os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "" + +experiment_id = mlflow.create_experiment(name="/Shared/my-traces") + +set_experiment_trace_location( + location=UCSchemaLocation( + catalog_name="", + schema_name="" + ), + experiment_id=experiment_id, +) +# Creates: mlflow_experiment_trace_otel_logs, _metrics, _spans +``` + +### Set Trace Destination +```python +# Option A: Python API +from mlflow.entities import UCSchemaLocation +mlflow.tracing.set_destination( + destination=UCSchemaLocation( + catalog_name="", + schema_name="", + ) +) + +# Option B: Environment variable +os.environ["MLFLOW_TRACING_DESTINATION"] = "." +``` + +### Permissions Required +- `USE_CATALOG` on catalog +- `USE_SCHEMA` on schema +- `MODIFY` and `SELECT` on each `mlflow_experiment_trace_*` table +- **CRITICAL**: `ALL_PRIVILEGES` is NOT sufficient + +--- + ## Production Monitoring +### Configure Monitoring SQL Warehouse +```python +from mlflow.tracing import set_databricks_monitoring_sql_warehouse_id + +set_databricks_monitoring_sql_warehouse_id( + warehouse_id="", + experiment_id="" # Optional +) +# Alternative: os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "" +``` + ### Register and Start Scorer ```python from mlflow.genai.scorers import Safety, Guidelines, ScorerSamplingConfig diff --git a/.claude/skills/mlflow-evaluation/references/GOTCHAS.md b/.claude/skills/databricks-mlflow-evaluation/references/GOTCHAS.md similarity index 54% rename from .claude/skills/mlflow-evaluation/references/GOTCHAS.md rename to .claude/skills/databricks-mlflow-evaluation/references/GOTCHAS.md index fc40d97b..4e468035 100644 --- a/.claude/skills/mlflow-evaluation/references/GOTCHAS.md +++ b/.claude/skills/databricks-mlflow-evaluation/references/GOTCHAS.md @@ -23,6 +23,15 @@ - [Wrong Production Monitoring Setup](#-wrong-production-monitoring-setup) - [Wrong Custom Judge Model Format](#-wrong-custom-judge-model-format) - [Wrong Aggregation Values](#-wrong-aggregation-values) +- [Wrong Trace Ingestion Setup](#-wrong-trace-ingestion-setup) +- [Wrong Trace Destination Format](#-wrong-trace-destination-format) +- [Wrong MLflow Version for Trace Ingestion](#-wrong-mlflow-version-for-trace-ingestion) +- [Wrong Linking UC Schema Without SQL Warehouse](#-wrong-linking-uc-schema-without-sql-warehouse) +- [Wrong Label Schema Name — Alignment Will Fail](#-wrong-label-schema-name--alignment-will-fail) +- [Wrong Aligned Judge Score Interpretation](#-wrong-aligned-judge-score-interpretation) +- [Wrong MemAlign Embedding Model — Token Costs](#-wrong-memalign-embedding-model--token-costs) +- [Wrong MemAlign Episodic Memory — Lazy Loading](#-wrong-memalign-episodic-memory--lazy-loading) +- [Wrong GEPA Optimization Dataset — Missing expectations](#-wrong-gepa-optimization-dataset--missing-expectations) - [Summary Checklist](#summary-checklist) --- @@ -530,6 +539,253 @@ def my_scorer(outputs) -> float: --- +## ❌ WRONG Trace Ingestion Setup + +### WRONG: Using ALL_PRIVILEGES instead of explicit grants +```sql +-- ❌ WRONG - ALL_PRIVILEGES does NOT include required permissions +GRANT ALL_PRIVILEGES ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_spans + TO `user@company.com`; +``` + +### ✅ CORRECT: Grant explicit MODIFY and SELECT +```sql +-- ✅ CORRECT - Explicit MODIFY and SELECT required +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_spans + TO `user@company.com`; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_logs + TO `user@company.com`; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_metrics + TO `user@company.com`; +``` + +--- + +## ❌ WRONG Trace Destination Format + +### WRONG: Wrong format for environment variable +```python +# ❌ WRONG - Missing schema or wrong separator +os.environ["MLFLOW_TRACING_DESTINATION"] = "my_catalog" +os.environ["MLFLOW_TRACING_DESTINATION"] = "my_catalog/my_schema" +``` + +### ✅ CORRECT: Use catalog.schema format +```python +# ✅ CORRECT - Dot-separated catalog.schema +os.environ["MLFLOW_TRACING_DESTINATION"] = "my_catalog.my_schema" +``` + +--- + +## ❌ WRONG MLflow Version for Trace Ingestion + +### WRONG: Using MLflow < 3.9.0 for UC trace ingestion +```bash +# ❌ WRONG - Trace ingestion requires 3.9.0+ +pip install mlflow[databricks]>=3.1.0 +``` + +### ✅ CORRECT: Use MLflow 3.9.0+ for UC traces +```bash +# ✅ CORRECT +pip install "mlflow[databricks]>=3.9.0" --upgrade --force-reinstall +``` + +--- + +## ❌ WRONG Linking UC Schema Without SQL Warehouse + +### WRONG: Missing SQL warehouse configuration +```python +# ❌ WRONG - No SQL warehouse configured +mlflow.set_tracking_uri("databricks") +# Missing: os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "..." +set_experiment_trace_location(location=UCSchemaLocation(...), ...) +``` + +### ✅ CORRECT: Set SQL warehouse before linking +```python +# ✅ CORRECT - Set warehouse ID first +mlflow.set_tracking_uri("databricks") +os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "" +set_experiment_trace_location(location=UCSchemaLocation(...), ...) +``` + +--- + +## ❌ WRONG Label Schema Name — Alignment Will Fail + +### WRONG: Label schema name does not match the judge name used in evaluate() +```python +# ❌ WRONG - Judge name and label schema name don't match +# Judge is registered as "domain_quality_base" in evaluate() +domain_quality_judge = make_judge(name="domain_quality_base", ...) +registered_base_judge = domain_quality_judge.register(experiment_id=EXPERIMENT_ID) + +# But label schema uses a different name +feedback_schema = label_schemas.create_label_schema( + name="domain_quality_rating", # ❌ Does not match judge name + type="feedback", + ... +) +# align() will not be able to pair SME feedback with LLM judge scores +``` + +### ✅ CORRECT: Label schema name matches the judge name exactly +```python +# ✅ CORRECT - Judge name and label schema name are identical +JUDGE_NAME = "domain_quality_base" + +domain_quality_judge = make_judge(name=JUDGE_NAME, ...) +registered_base_judge = domain_quality_judge.register(experiment_id=EXPERIMENT_ID) + +feedback_schema = label_schemas.create_label_schema( + name=JUDGE_NAME, # ✅ Matches judge name exactly + type="feedback", + ... +) +``` + +**Why?** The `align()` function pairs SME feedback with LLM judge scores by matching the label schema name to the judge name on the same traces. If the names differ, `align()` cannot find the corresponding score pairs and alignment will fail or produce incorrect results. + +--- + +## ❌ WRONG Aligned Judge Score Interpretation + +### WRONG: Assuming a lower aligned judge score means the agent got worse +```python +# ❌ WRONG interpretation - panicking because aligned judge gives lower scores +# Unaligned judge: 4.2/5.0 average +# Aligned judge: 3.1/5.0 average +# "The agent regressed!" — No, the judge got more accurate. +``` + +### ✅ CORRECT: Understanding that a lower aligned score reflects more accurate evaluation +```python +# ✅ CORRECT interpretation +# The aligned judge now evaluates with domain-expert standards rather than generic best practices. +# A lower score from a more accurate judge is a better signal than an inflated score from +# a judge that doesn't understand your domain. The unaligned judge was underspecified. +# Use optimize_prompts() with the aligned judge to improve the agent against this standard. +``` + +**Why?** An unaligned judge evaluates against generic best practices and often gives inflated scores. Once aligned with SME feedback, the judge applies domain-specific criteria that are harder to satisfy. The lower score is not a regression in agent quality; it is a more honest assessment. The optimization phase (`optimize_prompts()`) will then improve the agent against this more accurate standard. + +--- + +## ❌ WRONG MemAlign Embedding Model — Token Costs + +### WRONG: Using the default embedding model without awareness of cost +```python +# ❌ COSTLY - Default embedding model may be expensive for large trace sets +optimizer = MemAlignOptimizer( + reflection_lm=REFLECTION_MODEL, + retrieval_k=5, + # No embedding_model specified → defaults to "openai/text-embedding-3-small" +) +``` + +### ✅ CORRECT: Use a Databricks-hosted embedding model or size your trace set accordingly +```python +# ✅ CORRECT - Use a hosted model to control costs; scope trace set to labeled traces only +optimizer = MemAlignOptimizer( + reflection_lm=REFLECTION_MODEL, + retrieval_k=5, + embedding_model="databricks:/databricks-gte-large-en", +) + +# ✅ ALSO CORRECT - Filter to only labeled/tagged traces, not all experiment traces +traces = mlflow.search_traces( + locations=[EXPERIMENT_ID], + filter_string="tag.eval = 'complete'", # Scope to relevant traces only + return_type="list", +) +aligned_judge = base_judge.align(traces=traces, optimizer=optimizer) +``` + +**Why?** MemAlign embeds every trace for retrieval (`retrieval_k` nearest neighbors per evaluation). Large trace sets with an expensive embedding model multiply quickly. Databricks-hosted models (`databricks:/databricks-gte-large-en`) keep costs on-platform. + +--- + +## ❌ WRONG MemAlign Episodic Memory — Lazy Loading + +### WRONG: Expecting episodic memory to be populated immediately after get_scorer() +```python +# ❌ WRONG - Episodic memory appears empty, looks like alignment didn't work +retrieved_judge = get_scorer(name="domain_quality_base", experiment_id=EXPERIMENT_ID) +print(retrieved_judge._episodic_memory) # Prints: [] — misleading! +print(retrieved_judge._semantic_memory) # Prints: [] — also empty! +``` + +### ✅ CORRECT: Episodic memory is lazily loaded — use the judge first, then inspect +```python +# ✅ CORRECT - Semantic guidelines ARE loaded; episodic memory loads on first use +retrieved_judge = get_scorer(name="domain_quality_base", experiment_id=EXPERIMENT_ID) + +# The instructions field already contains the distilled guidelines — inspect this instead +print(retrieved_judge.instructions) # ✅ Shows full aligned instructions with guidelines + +# To verify episodic memory, run the judge on a sample first, then inspect +# Memory loads lazily when the judge retrieves similar examples during scoring +``` + +**Why?** MemAlign's episodic memory (stored examples) is loaded on-demand when the judge needs to retrieve similar examples at scoring time. The `_episodic_memory` list is empty on deserialization. The aligned `instructions` field (which includes distilled semantic guidelines) is the reliable thing to inspect after `get_scorer()`. + +--- + +## ❌ WRONG GEPA Optimization Dataset — Missing expectations + +### WRONG: Using eval-style dataset (inputs only) for optimize_prompts() +```python +# ❌ WRONG - GEPA requires expectations; optimization will fail or produce poor results +optimization_dataset = [ + {"inputs": {"input": [{"role": "user", "content": "How does the offense attack the blitz?"}]}}, + {"inputs": {"input": [{"role": "user", "content": "What are 3rd down tendencies?"}]}}, +] + +result = mlflow.genai.optimize_prompts( + predict_fn=predict_fn, + train_data=optimization_dataset, # ❌ Missing expectations + prompt_uris=[prompt.uri], + optimizer=GepaPromptOptimizer(...), + scorers=[aligned_judge], +) +``` + +### ✅ CORRECT: Include expectations in every optimization dataset record +```python +# ✅ CORRECT - Each record must have both inputs AND expectations +optimization_dataset = [ + { + "inputs": { + "input": [{"role": "user", "content": "How does the offense attack the blitz?"}] + }, + "expectations": { + "expected_response": ( + "The agent should analyze blitz performance metrics, compare success " + "rates across pressure packages, and provide concrete tactical recommendations." + ) + } + }, + { + "inputs": { + "input": [{"role": "user", "content": "What are 3rd down tendencies?"}] + }, + "expectations": { + "expected_response": ( + "The agent should call the appropriate tool with down=3 parameters, " + "summarize the play distribution, and give defensive recommendations." + ) + } + }, +] +``` + +**Why?** GEPA uses the `expectations` field during reflection — it compares the agent's output against the expected behavior to generate targeted prompt improvement suggestions. Without `expectations`, GEPA cannot reason about *why* the current prompt is underperforming. This is the most common cause of poor optimization results. + +--- + ## Summary Checklist Before running evaluation, verify: @@ -545,3 +801,14 @@ Before running evaluation, verify: - [ ] Production scorers have inline imports - [ ] Multiple Feedbacks have unique names - [ ] Aggregations use valid names: min, max, mean, median, variance, p90 +- [ ] UC trace ingestion uses `mlflow[databricks]>=3.9.0` +- [ ] UC tables have explicit MODIFY + SELECT grants (not ALL_PRIVILEGES) +- [ ] `MLFLOW_TRACING_SQL_WAREHOUSE_ID` set before linking UC schema +- [ ] `MLFLOW_TRACING_DESTINATION` uses `catalog.schema` format (dot-separated) +- [ ] Production monitoring scorers are both registered AND started +- [ ] MemAlign `embedding_model` can be explicitly set (don't rely on default for large trace sets) +- [ ] After `get_scorer()` for a MemAlign judge, inspect `.instructions` not `._episodic_memory` as episodic memory is lazily loaded +- [ ] GEPA `train_data` has both `inputs` AND `expectations` per record +- [ ] Label schema `name` matches the judge `name` used in `evaluate()` (required for `align()` to pair scores) +- [ ] Aligned judge scores may be lower than unaligned — this is expected if the judge is now more accurate +- [ ] MemAlign is scorer-agnostic (works with any `feedback_value_type` — float, bool, categorical) diff --git a/.claude/skills/mlflow-evaluation/references/patterns-context-optimization.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-context-optimization.md similarity index 100% rename from .claude/skills/mlflow-evaluation/references/patterns-context-optimization.md rename to .claude/skills/databricks-mlflow-evaluation/references/patterns-context-optimization.md diff --git a/.claude/skills/mlflow-evaluation/references/patterns-datasets.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-datasets.md similarity index 100% rename from .claude/skills/mlflow-evaluation/references/patterns-datasets.md rename to .claude/skills/databricks-mlflow-evaluation/references/patterns-datasets.md diff --git a/.claude/skills/mlflow-evaluation/references/patterns-evaluation.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-evaluation.md similarity index 100% rename from .claude/skills/mlflow-evaluation/references/patterns-evaluation.md rename to .claude/skills/databricks-mlflow-evaluation/references/patterns-evaluation.md diff --git a/.claude/skills/databricks-mlflow-evaluation/references/patterns-judge-alignment.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-judge-alignment.md new file mode 100644 index 00000000..c59989a9 --- /dev/null +++ b/.claude/skills/databricks-mlflow-evaluation/references/patterns-judge-alignment.md @@ -0,0 +1,316 @@ +# MLflow 3 Judge Alignment with MemAlign + +Patterns for aligning LLM judges to domain expert preferences using MemAlign. An aligned judge is more accurate for evaluation runs, more meaningful for production monitoring, and a better guide for prompt optimization — but each of these uses is independent. + +**Read `GOTCHAS.md` before implementing — especially the MemAlign sections.** + +--- + +## When to Use Judge Alignment + +Align a judge when: +- Built-in scorers don't capture domain-specific quality (e.g., "good" means expert-level tactical analysis) +- LLM judges disagree with human raters on the same examples +- You have domain experts who can rate a sample of agent outputs +- You want production monitoring that reflects actual expert standards + +You do NOT need prompt optimization to benefit from aligned judges — a more accurate judge improves every evaluation run and monitoring setup you do afterward. + +--- + +## Pattern 1: Design and Register the Base Judge + +MemAlign is scorer-agnostic and works with any `feedback_value_type` (float, boolean, categorical). This example uses a Likert scale (1-5 float), but you can use whatever scoring scheme fits your domain. + +```python +import mlflow +from mlflow.genai.judges import make_judge +from mlflow.genai import evaluate + +mlflow.set_experiment(experiment_id=EXPERIMENT_ID) + +# Define base judge using make_judge -- MemAlign works with any feedback type +# This example uses a Likert scale (1-5 float), but boolean or categorical also work +domain_quality_judge = make_judge( + name="domain_quality_base", + instructions=( + "Evaluate if the response in {{ outputs }} appropriately analyzes the available data " + "and provides an actionable recommendation to the question in {{ inputs }}. " + "The response should be accurate, contextually relevant, and give a strategic advantage " + "to the person making the request. " + "Your grading criteria: " + " 1: Completely unacceptable. Incorrect data interpretation or no recommendations. " + " 2: Mostly unacceptable. Irrelevant or spurious feedback or weak recommendations with minimal strategic advantage. " + " 3: Somewhat acceptable. Relevant feedback provided with some strategic advantage. " + " 4: Mostly acceptable. Relevant feedback provided with strong strategic advantage. " + " 5: Completely acceptable. Relevant feedback provided with excellent strategic advantage." + ), + feedback_value_type=float, # Example uses a Likert scale; MemAlign works with any feedback type + model=JUDGE_MODEL, +) + +# Register to experiment — creates the persistent record used by align() +registered_base_judge = domain_quality_judge.register(experiment_id=EXPERIMENT_ID) +print(f"Registered base judge: {registered_base_judge.name}") +``` + +--- + +## Pattern 2: Run Evaluation and Tag Traces + +Run evaluation to generate a set of traces that domain experts will review. Tag traces that were **successfully evaluated** in this `evaluate()` job (i.e., the agent produced a response and the judge scored it without errors). + +```python +from mlflow.genai import evaluate + +# Eval dataset: inputs only (no expectations needed at this stage) +eval_data = [ + {"inputs": {"input": [{"role": "user", "content": question}]}} + for question in example_questions +] + +results = evaluate( + data=eval_data, + predict_fn=lambda input: AGENT.predict({"input": input}), + scorers=[domain_quality_judge], +) + +# Tag traces that were successfully evaluated in this evaluate() job +# "OK" state means the agent responded AND the judge scored it without errors +ok_trace_ids = results.result_df.loc[results.result_df["state"] == "OK", "trace_id"] +for trace_id in ok_trace_ids: + mlflow.set_trace_tag(trace_id=trace_id, key="eval", value="complete") + +print(f"Tagged {len(ok_trace_ids)} successfully evaluated traces for labeling") +``` + +--- + +## Pattern 3: Build Eval Dataset and Create Labeling Session + +Persist traces to a UC dataset and assign them to domain experts for review. + +**CRITICAL: The label schema `name` MUST match the judge `name` used in the `evaluate()` job.** This is how `align()` pairs SME feedback with the corresponding LLM judge scores on the same traces. If these names do not match, alignment will fail or produce incorrect results. + +```python +from mlflow.genai.datasets import create_dataset, get_dataset +from mlflow.genai import create_labeling_session, get_review_app +from mlflow.genai import label_schemas + +# Build persistent dataset from tagged traces +try: + eval_dataset = get_dataset(name=DATASET_NAME) +except Exception: + eval_dataset = create_dataset(name=DATASET_NAME) + +tagged_traces = mlflow.search_traces( + locations=[EXPERIMENT_ID], + filter_string="tag.eval = 'complete'", + return_type="pandas", +) +# merge_records() expects 'inputs' and 'outputs' column names +if "inputs" not in tagged_traces.columns and "request" in tagged_traces.columns: + tagged_traces = tagged_traces.rename(columns={"request": "inputs"}) +if "outputs" not in tagged_traces.columns and "response" in tagged_traces.columns: + tagged_traces = tagged_traces.rename(columns={"response": "outputs"}) + +eval_dataset = eval_dataset.merge_records(tagged_traces) + +# CRITICAL: The label schema name MUST match the judge name used in evaluate() +# This is how align() pairs SME feedback with LLM judge scores on the same traces +LABEL_SCHEMA_NAME = "domain_quality_base" # Must match the judge name exactly + +feedback_schema = label_schemas.create_label_schema( + name=LABEL_SCHEMA_NAME, # Must match judge name from Pattern 1 + type="feedback", + title=LABEL_SCHEMA_NAME, + input=label_schemas.InputNumeric(min_value=1.0, max_value=5.0), + instruction=( + "Evaluate if the response appropriately analyzes the available data and provides " + "an actionable recommendation for the question. The response should be accurate, " + "contextually relevant, and give a strategic advantage to the person making the request. " + "\n\n Your grading criteria should be: " + "\n 1: Completely unacceptable. Incorrect data interpretation or no recommendations." + "\n 2: Mostly unacceptable. Irrelevant or spurious feedback or weak recommendations with minimal strategic advantage." + "\n 3: Somewhat acceptable. Relevant feedback provided with some strategic advantage." + "\n 4: Mostly acceptable. Relevant feedback provided with strong strategic advantage." + "\n 5: Completely acceptable. Relevant feedback provided with excellent strategic advantage." + ), + enable_comment=True, # Allow SMEs to leave free-text rationale (used by MemAlign) + overwrite=True, +) + +# Optional: add a deployed agent to the Review App so SMEs can ask new questions +review_app = get_review_app(experiment_id=EXPERIMENT_ID) +review_app = review_app.add_agent( + agent_name=MODEL_NAME, + model_serving_endpoint=AGENT_ENDPOINT_NAME, + overwrite=True, +) + +# Create labeling session and attach the dataset +labeling_session = create_labeling_session( + name=f"{LABELING_SESSION_NAME}_sme", + assigned_users=ASSIGNED_USERS, + label_schemas=[LABEL_SCHEMA_NAME], # Must match judge name +) +labeling_session = labeling_session.add_dataset(dataset_name=DATASET_NAME) + +print(f"Share with domain experts: {labeling_session.url}") +# Domain experts open this URL and rate each response using the 1-5 scale +``` + +--- + +## Pattern 4: Align Judge with MemAlign (Recommended) + +After SMEs complete labeling, distill their feedback patterns into the judge's instructions. + +Judge alignment supports multiple optimizers (e.g., SIMBA, custom optimizers), but this example uses **MemAlign**, which is the recommended approach. MemAlign is the fastest alignment method (seconds vs. minutes for alternatives), the most cost-effective, and supports **memory scaling** where quality continues to improve as feedback accumulates without re-optimization. + +```python +from mlflow.genai.judges.optimizers import MemAlignOptimizer +from mlflow.genai.scorers import get_scorer + +# Fetch the tagged traces (which now have SME labels attached) +traces_for_alignment = mlflow.search_traces( + locations=[EXPERIMENT_ID], + filter_string="tag.eval = 'complete'", + return_type="list", # align() requires list format +) +print(f"Aligning on {len(traces_for_alignment)} traces") + +# Configure MemAlign optimizer +# Other optimizers are available (e.g., SIMBA), but MemAlign is recommended for its +# speed, cost efficiency, and ability to improve continuously as feedback accumulates +optimizer = MemAlignOptimizer( + reflection_lm=REFLECTION_MODEL, # Model for guideline distillation + retrieval_k=5, # Examples to retrieve per evaluation + embedding_model="databricks:/databricks-gte-large-en", + # Defaults to "openai/text-embedding-3-small" if not set -- see GOTCHAS.md +) + +# Load the registered base judge and run alignment +base_judge = get_scorer(name="domain_quality_base") +aligned_judge = base_judge.align( + traces=traces_for_alignment, + optimizer=optimizer, +) + +# Inspect distilled semantic guidelines — these encode expert preferences +print("Distilled Guidelines from SME feedback:") +for i, guideline in enumerate(aligned_judge._semantic_memory, 1): + print(f" {i}. {guideline.guideline_text}") + if guideline.source_trace_ids: + print(f" Derived from {len(guideline.source_trace_ids)} trace(s)") +``` + +--- + +## Pattern 5: Register the Aligned Judge + +Persist the aligned judge to the experiment for later retrieval in evaluation or optimization runs. + +```python +from mlflow.genai.scorers import ScorerSamplingConfig + +# Option A: Update the existing judge record in-place (recommended for iterative alignment) +aligned_judge_registered = aligned_judge.update( + experiment_id=EXPERIMENT_ID, + sampling_config=ScorerSamplingConfig(sample_rate=0.0), +) +print(f"Updated judge: {aligned_judge_registered.name}") + +# Option B: Register as a new named version (preserves the original for comparison) +from mlflow.genai.judges import make_judge + +aligned_judge_v2 = make_judge( + name="domain_quality_aligned_v1", + instructions=aligned_judge.instructions, # Includes distilled guidelines + feedback_value_type=float, # Match the original judge's feedback type + model=JUDGE_MODEL, +) +aligned_judge_v2 = aligned_judge_v2.register(experiment_id=EXPERIMENT_ID) + +# Retrieve in a later session +# NOTE: Episodic memory is lazily loaded — inspect .instructions, not ._episodic_memory +from mlflow.genai.scorers import get_scorer + +retrieved_judge = get_scorer(name="domain_quality_base", experiment_id=EXPERIMENT_ID) +print(retrieved_judge.instructions[:500]) # Shows aligned instructions with guidelines +``` + +--- + +## Pattern 6: Re-evaluate with Aligned Judge + +Run a fresh evaluation with the aligned judge. This gives a more accurate quality picture and establishes a baseline for prompt optimization if you choose to do that next. + +**Important: The aligned judge score may be lower than the unaligned judge score. This is expected and correct.** It means the aligned judge is now evaluating with domain-expert standards rather than generic best practices. A lower score from a more accurate judge is a better signal than a higher score from a judge that doesn't understand your domain. The optimization phase (`optimize_prompts()`) will improve the agent against this more accurate standard. + +```python +from mlflow.genai import evaluate +from mlflow.genai.scorers import get_scorer +from mlflow.genai.datasets import get_dataset + +aligned_judge = get_scorer(name="domain_quality_base", experiment_id=EXPERIMENT_ID) + +eval_dataset = get_dataset(name=DATASET_NAME) +df = eval_dataset.to_df() + +eval_records = [ + { + "inputs": { + "input": [{"role": "user", "content": extract_user_message(row)}] + } + } + for row in df["inputs"] +] + +with mlflow.start_run(run_name="aligned_judge_baseline"): + baseline_results = evaluate( + data=eval_records, + predict_fn=lambda input: AGENT.predict({"input": input}), + scorers=[aligned_judge], + ) + +print(f"Aligned judge baseline metrics: {baseline_results.metrics}") +# NOTE: If scores are lower than the unaligned judge, that is expected. +# The aligned judge is more accurate, not less generous. +``` + +--- + +## Using Aligned Judges Beyond Evaluation + +Aligned judges are not just for one-time evaluation. They can be used for: + +**Production monitoring:** +```python +from mlflow.genai.scorers import ScorerSamplingConfig + +aligned_judge = get_scorer(name="domain_quality_base", experiment_id=EXPERIMENT_ID) +monitoring_judge = aligned_judge.start( + sampling_config=ScorerSamplingConfig(sample_rate=0.1) # Score 10% of production traffic +) +``` + +**Prompt optimization input (see `patterns-prompt-optimization.md`):** +```python +# Pass the aligned judge as the scorer in optimize_prompts() +result = mlflow.genai.optimize_prompts( + predict_fn=predict_fn, + train_data=optimization_dataset, + prompt_uris=[prompt.uri], + optimizer=GepaPromptOptimizer(reflection_model=REFLECTION_MODEL), + scorers=[aligned_judge], # ← aligned judge drives GEPA's reflection +) +``` + +**Regression detection across agent versions:** +```python +with mlflow.start_run(run_name="agent_v2"): + v2_results = evaluate(data=eval_records, predict_fn=agent_v2, scorers=[aligned_judge]) + +# Metrics from aligned judge are more meaningful than unaligned LLM judge +``` diff --git a/.claude/skills/databricks-mlflow-evaluation/references/patterns-prompt-optimization.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-prompt-optimization.md new file mode 100644 index 00000000..01a79bd1 --- /dev/null +++ b/.claude/skills/databricks-mlflow-evaluation/references/patterns-prompt-optimization.md @@ -0,0 +1,163 @@ +# MLflow 3 Prompt Optimization with GEPA + +Patterns for automated prompt improvement using `optimize_prompts()` with the GEPA (Genetic-Pareto) optimizer. GEPA iteratively evolves a registered system prompt by evaluating candidates against a scorer, then promotes the best version. + +**Using an aligned judge as the scorer is recommended.** An aligned judge encodes domain-expert preferences, giving GEPA a more accurate optimization signal than a generic LLM judge. See `patterns-judge-alignment.md` for the full alignment workflow. + +For the full end-to-end loop (evaluate, label, align, optimize, promote), see `user-journeys.md` Journey 10. For details on the GEPA and MemAlign approaches, see the [Self-Optimizing Agent blog post](https://www.databricks.com/blog/self-optimizing-football-chatbot-guided-domain-experts-databricks). + +**Read `GOTCHAS.md` before implementing -- especially the GEPA sections.** + +--- + +## Pattern 1: Build Optimization Dataset (inputs + expectations required) + +GEPA requires both `inputs` AND `expectations` in every record. This is different from the eval dataset which only needs `inputs`. The `expectations` field is what GEPA uses during reflection to reason about why the current prompt is underperforming. + +```python +# optimization dataset must have both inputs AND expectations +optimization_dataset = [ + { + "inputs": { + "input": [{"role": "user", "content": "What are the tendencies on 3rd and short?"}] + }, + "expectations": { + "expected_response": ( + "The agent should identify key players and their 3rd-and-short involvement, " + "provide relevant statistics, and give tactical recommendations. " + "If data quality issues exist, they should be stated explicitly." + ) + } + }, + { + "inputs": { + "input": [{"role": "user", "content": "How does the offense perform against the blitz?"}] + }, + "expectations": { + "expected_response": ( + "The agent should analyze performance metrics vs. pressure, " + "compare success across different blitz packages, " + "and provide concrete defensive recommendations." + ) + } + }, + # Add 15-20 representative examples covering key use cases +] + +# Persist to MLflow dataset +from mlflow.genai.datasets import create_dataset + +optim_dataset = create_dataset(name=OPTIMIZATION_DATASET_NAME) +optim_dataset = optim_dataset.merge_records(optimization_dataset) +print(f"Created optimization dataset with {len(optimization_dataset)} records") +``` + +--- + +## Pattern 2: Run optimize_prompts() with GEPA + +Use a scorer (ideally an aligned judge from `patterns-judge-alignment.md`) to drive GEPA prompt optimization of the registered system prompt. + +```python +import mlflow +from mlflow.genai.optimize import GepaPromptOptimizer +from mlflow.genai.scorers import get_scorer + +mlflow.set_experiment(experiment_id=EXPERIMENT_ID) + +# Load prompt from registry (must be registered before optimization) +system_prompt = mlflow.genai.load_prompt(f"prompts:/{PROMPT_NAME}@production") +print(f"Loaded prompt: {system_prompt.uri}") + +# Load scorer -- an aligned judge is recommended for domain-accurate optimization +# See patterns-judge-alignment.md for how to create one +aligned_judge = get_scorer(name=ALIGNED_JUDGE_NAME, experiment_id=EXPERIMENT_ID) + +# Define predict_fn -- loads prompt from registry on each call so GEPA can swap it +def predict_fn(input): + prompt = mlflow.genai.load_prompt(system_prompt.uri) + system_content = prompt.format() + + user_message = input[0]["content"] + messages = [ + {"role": "system", "content": system_content}, + {"role": "user", "content": user_message}, + ] + return AGENT.predict({"input": messages}) + +# Define aggregation to normalize judge feedback (Feedback.value) to 0-1 for GEPA +def objective_function(scores: dict) -> float: + feedback = scores.get(ALIGNED_JUDGE_NAME) + if feedback and hasattr(feedback, "feedback") and hasattr(feedback.feedback, "value"): + try: + return float(feedback.feedback.value) / 5.0 # Normalize 1-5 scale to 0-1 + except (ValueError, TypeError): + return 0.5 + return 0.5 + +# Run optimization +result = mlflow.genai.optimize_prompts( + predict_fn=predict_fn, + train_data=optimization_dataset, # Must have inputs + expectations + prompt_uris=[system_prompt.uri], + optimizer=GepaPromptOptimizer( + reflection_model=REFLECTION_MODEL, + max_metric_calls=75, # Reduce for faster runs; increase for quality + display_progress_bar=True, + ), + scorers=[aligned_judge], + aggregation=objective_function, +) + +optimized_prompt = result.optimized_prompts[0] +print(f"Initial score: {result.initial_eval_score}") +print(f"Final score: {result.final_eval_score}") +print(f"\nOptimized template (first 500 chars):\n{optimized_prompt.template[:500]}...") +``` + +--- + +## Pattern 3: Register Optimized Prompt and Conditionally Promote + +Only promote to the "production" alias if the optimized prompt outperforms the baseline. + +```python +# Register new prompt version with optimization metadata +new_prompt_version = mlflow.genai.register_prompt( + name=PROMPT_NAME, + template=optimized_prompt.template, + commit_message=f"GEPA optimization using {ALIGNED_JUDGE_NAME}", + tags={ + "initial_score": str(result.initial_eval_score), + "final_score": str(result.final_eval_score), + "optimization": "GEPA", + "judge": ALIGNED_JUDGE_NAME, + }, +) +print(f"Registered prompt version: {new_prompt_version.version}") + +# Conditional promotion -- only update production alias if score improved +def promote_if_improved(prompt_name, result, new_prompt_version): + if result.final_eval_score > result.initial_eval_score: + mlflow.genai.set_prompt_alias( + name=prompt_name, + alias="production", + version=new_prompt_version.version, + ) + print(f"Promoted version {new_prompt_version.version} to production " + f"({result.initial_eval_score:.3f} -> {result.final_eval_score:.3f})") + else: + print(f"No improvement ({result.initial_eval_score:.3f} -> " + f"{result.final_eval_score:.3f}). Production alias unchanged.") + +promote_if_improved(PROMPT_NAME, result, new_prompt_version) +``` + +--- + +## Tips for Prompt Optimization + +- The optimization dataset should cover the diversity of queries your agent will handle. Include edge cases, ambiguous requests, and scenarios where tool selection matters. +- Expected responses should describe what the agent should do (which tools to call, what information to include) rather than exact output text. +- Start with `max_metric_calls` set to between 50 and 100. Higher values explore more candidates but increase cost and runtime. +- The GEPA optimizer learns from failure modes. If the aligned judge penalizes missing benchmarks or small-sample caveats, GEPA will inject those requirements into the optimized prompt. diff --git a/.claude/skills/mlflow-evaluation/references/patterns-scorers.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-scorers.md similarity index 100% rename from .claude/skills/mlflow-evaluation/references/patterns-scorers.md rename to .claude/skills/databricks-mlflow-evaluation/references/patterns-scorers.md diff --git a/.claude/skills/mlflow-evaluation/references/patterns-trace-analysis.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-trace-analysis.md similarity index 100% rename from .claude/skills/mlflow-evaluation/references/patterns-trace-analysis.md rename to .claude/skills/databricks-mlflow-evaluation/references/patterns-trace-analysis.md diff --git a/.claude/skills/databricks-mlflow-evaluation/references/patterns-trace-ingestion.md b/.claude/skills/databricks-mlflow-evaluation/references/patterns-trace-ingestion.md new file mode 100644 index 00000000..7196ab14 --- /dev/null +++ b/.claude/skills/databricks-mlflow-evaluation/references/patterns-trace-ingestion.md @@ -0,0 +1,680 @@ +# MLflow Trace Ingestion in Unity Catalog + +Working code patterns for setting up trace storage in Unity Catalog, logging traces from applications, and enabling production monitoring. + +**Version**: MLflow 3.9.0+ (`mlflow[databricks]>=3.9.0`) +**Preview**: Requires "OpenTelemetry on Databricks" preview enabled +**Regions**: Currently available in `us-east-1` and `us-west-2` only + +--- + +## Table of Contents + +| # | Pattern | Description | +|---|---------|-------------| +| 1 | [Initial Setup](#pattern-1-initial-setup---link-uc-schema-to-experiment) | Link UC schema to experiment, create tables | +| 2 | [Access Control](#pattern-2-access-control---grant-permissions) | Grant required permissions on UC tables | +| 3 | [Set Trace Destination (Python API)](#pattern-3-set-trace-destination-via-python-api) | Configure where traces are sent | +| 4 | [Set Trace Destination (Env Var)](#pattern-4-set-trace-destination-via-environment-variable) | Configure destination via env var | +| 5 | [Log Traces with @mlflow.trace](#pattern-5-log-traces-with-mlflow-decorator) | Instrument functions with decorator | +| 6 | [Log Traces with start_span](#pattern-6-log-traces-with-context-manager) | Fine-grained span control | +| 7 | [Auto-Instrumentation](#pattern-7-automatic-tracing-with-autolog) | Framework auto-tracing (OpenAI, LangChain, etc.) | +| 8 | [Combined Instrumentation](#pattern-8-combined-auto-and-manual-tracing) | Mix auto + manual tracing | +| 9 | [Traces from Databricks Apps](#pattern-9-log-traces-from-databricks-apps) | Configure app service principal | +| 10 | [Traces from Model Serving](#pattern-10-log-traces-from-model-serving-endpoints) | Configure serving endpoints | +| 11 | [Traces from OTEL Clients](#pattern-11-log-traces-from-third-party-otel-clients) | Use OpenTelemetry OTLP exporter | +| 12 | [Enable Production Monitoring](#pattern-12-enable-production-monitoring) | Register and start scorers | +| 13 | [Manage Monitoring Scorers](#pattern-13-manage-monitoring-scorers) | List, update, stop, delete scorers | +| 14 | [Query UC Trace Tables](#pattern-14-query-traces-from-unity-catalog-tables) | SQL queries on ingested traces | +| 15 | [End-to-End Setup](#pattern-15-end-to-end-setup-script) | Complete setup from scratch | + +--- + +## Pattern 1: Initial Setup - Link UC Schema to Experiment + +Create an MLflow experiment and link it to a Unity Catalog schema. This automatically creates three tables for storing trace data. + +```python +import os +import mlflow +from mlflow.entities import UCSchemaLocation +from mlflow.tracing.enablement import set_experiment_trace_location + +# Step 1: Configure tracking +mlflow.set_tracking_uri("databricks") +os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "" + +# Step 2: Define names +experiment_name = "/Shared/my-agent-traces" +catalog_name = "my_catalog" +schema_name = "my_schema" + +# Step 3: Create or retrieve experiment +if experiment := mlflow.get_experiment_by_name(experiment_name): + experiment_id = experiment.experiment_id +else: + experiment_id = mlflow.create_experiment(name=experiment_name) + +# Step 4: Link UC schema to experiment +result = set_experiment_trace_location( + location=UCSchemaLocation( + catalog_name=catalog_name, + schema_name=schema_name + ), + experiment_id=experiment_id, +) +``` + +**Tables created automatically:** +- `{catalog}.{schema}.mlflow_experiment_trace_otel_logs` +- `{catalog}.{schema}.mlflow_experiment_trace_otel_metrics` +- `{catalog}.{schema}.mlflow_experiment_trace_otel_spans` + +**CRITICAL**: Linking a UC schema hides pre-existing experiment traces stored in MLflow. Unlinking restores access to those traces. + +--- + +## Pattern 2: Access Control - Grant Permissions + +Users and service principals need explicit permissions on the UC trace tables. `ALL_PRIVILEGES` is **not sufficient**. + +```sql +-- Required: USE_CATALOG on the catalog +GRANT USE_CATALOG ON CATALOG my_catalog TO `user@company.com`; + +-- Required: USE_SCHEMA on the schema +GRANT USE_SCHEMA ON SCHEMA my_catalog.my_schema TO `user@company.com`; + +-- Required: MODIFY and SELECT on each trace table +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_logs + TO `user@company.com`; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_spans + TO `user@company.com`; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_metrics + TO `user@company.com`; +``` + +**For service principals (Databricks Apps, Model Serving):** +```sql +-- Replace with the service principal's application ID +GRANT USE_CATALOG ON CATALOG my_catalog TO ``; +GRANT USE_SCHEMA ON SCHEMA my_catalog.my_schema TO ``; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_logs + TO ``; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_spans + TO ``; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_metrics + TO ``; +``` + +--- + +## Pattern 3: Set Trace Destination via Python API + +Configure where traces are sent using the Python API. Use this after the initial setup (Pattern 1) in your application code. + +```python +import mlflow +from mlflow.entities import UCSchemaLocation + +# Set trace destination to Unity Catalog +mlflow.tracing.set_destination( + destination=UCSchemaLocation( + catalog_name="my_catalog", + schema_name="my_schema", + ) +) + +# Now all traces from @mlflow.trace or autolog will go to UC +@mlflow.trace +def my_agent(query: str) -> str: + # Traces are automatically sent to UC tables + return process(query) +``` + +--- + +## Pattern 4: Set Trace Destination via Environment Variable + +Alternative to Pattern 3 — configure destination via environment variable. Useful for deployment configurations. + +```python +import os + +# Set destination as "{catalog}.{schema}" +os.environ["MLFLOW_TRACING_DESTINATION"] = "my_catalog.my_schema" +``` + +Or in shell: +```bash +export MLFLOW_TRACING_DESTINATION="my_catalog.my_schema" +``` + +--- + +## Pattern 5: Log Traces with MLflow Decorator + +Use `@mlflow.trace` to instrument functions. Automatically captures inputs, outputs, latency, and exceptions. + +```python +import mlflow +from mlflow.entities import SpanType + +# Basic function tracing +@mlflow.trace +def my_agent(query: str) -> str: + context = retrieve_context(query) + return generate_response(query, context) + +# With span type (enables enhanced UI and evaluation) +@mlflow.trace(span_type=SpanType.RETRIEVER) +def retrieve_context(query: str) -> list[dict]: + """Mark retrieval functions with RETRIEVER span type.""" + return vector_store.search(query, top_k=5) + +@mlflow.trace(span_type=SpanType.CHAIN) +def generate_response(query: str, context: list[dict]) -> str: + """Mark orchestration with CHAIN span type.""" + return llm.invoke(query, context=context) + +# With custom name and attributes +@mlflow.trace(name="safety_check", span_type=SpanType.TOOL) +def check_safety(text: str) -> bool: + return safety_classifier.predict(text) +``` + +**Available SpanType values:** +- `SpanType.CHAIN` — Orchestration / pipeline steps +- `SpanType.CHAT_MODEL` — LLM chat completions +- `SpanType.LLM` — LLM calls (non-chat) +- `SpanType.RETRIEVER` — Document/data retrieval (special output schema) +- `SpanType.TOOL` — Tool/function execution +- `SpanType.AGENT` — Agent execution +- `SpanType.EMBEDDING` — Embedding generation + +--- + +## Pattern 6: Log Traces with Context Manager + +Use `mlflow.start_span()` for fine-grained control over spans. Manually set inputs, outputs, and attributes. + +```python +import mlflow + +def process_query(query: str) -> str: + # Create a span with manual control + with mlflow.start_span(name="process_query") as span: + span.set_inputs({"query": query}) + + # Nested span for retrieval + with mlflow.start_span(name="retrieve", span_type="RETRIEVER") as retriever_span: + retriever_span.set_inputs({"query": query}) + docs = vector_store.search(query) + retriever_span.set_outputs(docs) + + # Nested span for generation + with mlflow.start_span(name="generate", span_type="CHAIN") as gen_span: + gen_span.set_inputs({"query": query, "doc_count": len(docs)}) + response = llm.generate(query, docs) + gen_span.set_outputs({"response": response}) + + # Set attributes for analysis + span.set_attribute("doc_count", len(docs)) + span.set_attribute("model", "gpt-4o") + span.set_outputs({"response": response}) + + return response +``` + +--- + +## Pattern 7: Automatic Tracing with Autolog + +Enable automatic tracing for supported frameworks. MLflow captures LLM calls, tool executions, and chain operations without code changes. + +```python +import mlflow + +# Enable auto-tracing for specific frameworks +mlflow.openai.autolog() # OpenAI SDK calls +mlflow.langchain.autolog() # LangChain chains and agents +# Also available: mlflow.anthropic.autolog(), mlflow.litellm.autolog(), etc. + +# Set tracking and destination +mlflow.set_tracking_uri("databricks") +mlflow.set_experiment("/Shared/my-agent-traces") + +# Traces are captured automatically +from openai import OpenAI +client = OpenAI() + +response = client.chat.completions.create( + model="gpt-4o", + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is MLflow?"} + ] +) +# ^ This call is automatically traced +``` + +**20+ supported frameworks** including: +- OpenAI, Anthropic, Google GenAI +- LangChain, LlamaIndex, DSPy +- LiteLLM, Ollama, Bedrock +- CrewAI, AutoGen, Haystack + +--- + +## Pattern 8: Combined Auto and Manual Tracing + +Combine automatic framework tracing with manual decorators for complete coverage. + +```python +import mlflow +from mlflow.entities import SpanType +from openai import OpenAI + +# Enable automatic OpenAI tracing +mlflow.openai.autolog() + +client = OpenAI() + +@mlflow.trace(span_type=SpanType.CHAIN) +def my_rag_pipeline(query: str) -> str: + """Manual decorator wraps the whole pipeline. + Auto-tracing captures individual OpenAI calls inside.""" + + # This retrieval is manually traced + docs = retrieve_documents(query) + + # This LLM call is auto-traced by mlflow.openai.autolog() + response = client.chat.completions.create( + model="gpt-4o", + messages=[ + {"role": "system", "content": f"Answer using context: {docs}"}, + {"role": "user", "content": query} + ] + ) + return response.choices[0].message.content + +@mlflow.trace(span_type=SpanType.RETRIEVER) +def retrieve_documents(query: str) -> list[dict]: + """Manually traced retrieval function.""" + return vector_store.search(query, top_k=5) +``` + +--- + +## Pattern 9: Log Traces from Databricks Apps + +Configure a Databricks App to send traces to Unity Catalog. + +**Prerequisites:** +- App uses `mlflow[databricks]>=3.5.0` +- App's service principal has MODIFY and SELECT on the trace tables (see Pattern 2) + +**In your app code:** +```python +import os +import mlflow +from mlflow.entities import UCSchemaLocation + +# Option A: Python API +mlflow.tracing.set_destination( + destination=UCSchemaLocation( + catalog_name="my_catalog", + schema_name="my_schema", + ) +) + +# Option B: Environment variable (set in app config) +os.environ["MLFLOW_TRACING_DESTINATION"] = "my_catalog.my_schema" + +# Your app code — traces are sent to UC +@mlflow.trace +def handle_request(query: str) -> str: + return my_agent.invoke(query) +``` + +**Deployment steps:** +1. Locate the app's service principal under the **Authorization** tab +2. Grant MODIFY and SELECT on the three `mlflow_experiment_trace_*` tables +3. Configure the trace destination in your app code +4. Deploy the app + +--- + +## Pattern 10: Log Traces from Model Serving Endpoints + +Configure a model serving endpoint to send traces to Unity Catalog. + +**Step 1: Grant permissions to user/service principal** +```sql +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_logs + TO `serving-principal-id`; +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_spans + TO `serving-principal-id`; +``` + +**Step 2: Generate a Personal Access Token (PAT)** + +Create a PAT for the identity that has the permissions above. + +**Step 3: Add environment variables to the endpoint** + +Add these to the serving endpoint configuration: +``` +DATABRICKS_TOKEN= +MLFLOW_TRACING_DESTINATION=my_catalog.my_schema +``` + +**Step 4: In your served model code, configure the destination** +```python +import os +import mlflow +from mlflow.entities import UCSchemaLocation + +mlflow.tracing.set_destination( + destination=UCSchemaLocation( + catalog_name="my_catalog", + schema_name="my_schema", + ) +) + +# Your model's predict function — traces go to UC +@mlflow.trace +def predict(model_input): + return my_model.invoke(model_input) +``` + +--- + +## Pattern 11: Log Traces from Third-Party OTEL Clients + +Send traces from any OpenTelemetry-compatible client to Unity Catalog via the OTLP HTTP endpoint. + +```python +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor + +# Configure OTLP exporter pointing to Databricks +otlp_trace_exporter = OTLPSpanExporter( + endpoint="https:///api/2.0/otel/v1/traces", + headers={ + "content-type": "application/x-protobuf", + "X-Databricks-UC-Table-Name": "my_catalog.my_schema.mlflow_experiment_trace_otel_spans", + "Authorization": "Bearer ", + }, +) + +# Set up the tracer provider +provider = TracerProvider() +provider.add_span_processor(BatchSpanProcessor(otlp_trace_exporter)) + +# Use standard OpenTelemetry APIs to create spans +tracer = provider.get_tracer("my-application") +with tracer.start_as_current_span("my-operation") as span: + span.set_attribute("query", "What is MLflow?") + result = process_query("What is MLflow?") + span.set_attribute("result_length", len(result)) +``` + +**Notes:** +- Traces ingested via OTEL appear in linked experiments if they contain a root span +- Use the `X-Databricks-UC-Table-Name` header to specify the target spans table +- Standard OTEL instrumentation libraries work with this endpoint + +--- + +## Pattern 12: Enable Production Monitoring + +Register scorers to continuously evaluate traces in production. Scorers run asynchronously on sampled traces. + +```python +import mlflow +from mlflow.genai.scorers import Safety, Guidelines, ScorerSamplingConfig +from mlflow.tracing import set_databricks_monitoring_sql_warehouse_id + +# Step 1: Configure the SQL warehouse for monitoring +set_databricks_monitoring_sql_warehouse_id( + warehouse_id="", + experiment_id="" # Optional — uses active experiment if omitted +) + +# Step 2: Set the active experiment +mlflow.set_experiment("/Shared/my-agent-traces") + +# Step 3: Register and start scorers + +# Safety scorer — evaluate 100% of traces +safety = Safety().register(name="production_safety") +safety = safety.start( + sampling_config=ScorerSamplingConfig(sample_rate=1.0) +) + +# Custom guidelines — evaluate 50% of traces +tone_check = Guidelines( + name="professional_tone", + guidelines="The response must be professional and helpful" +).register(name="production_tone") +tone_check = tone_check.start( + sampling_config=ScorerSamplingConfig(sample_rate=0.5) +) +``` + +**CRITICAL**: You must both `.register()` AND `.start()` — registering alone does not activate monitoring. + +**SQL Warehouse requirements:** +- User must have `CAN USE` on the SQL warehouse +- User must have `CAN EDIT` on the experiment +- Monitoring job permissions are auto-granted on first scorer registration + +--- + +## Pattern 13: Manage Monitoring Scorers + +List, update, stop, and delete production monitoring scorers. + +```python +from mlflow.genai.scorers import list_scorers, get_scorer, delete_scorer, ScorerSamplingConfig + +# List all registered scorers for the active experiment +scorers = list_scorers() +for s in scorers: + print(f" {s.name}: sample_rate={s.sampling_config.sample_rate if s.sampling_config else 'N/A'}") + +# Get a specific scorer +safety_scorer = get_scorer(name="production_safety") + +# Update sample rate (e.g., increase from 50% to 80%) +safety_scorer = safety_scorer.update( + sampling_config=ScorerSamplingConfig(sample_rate=0.8) +) + +# Stop monitoring (keeps registration for later re-start) +safety_scorer = safety_scorer.stop() + +# Re-start monitoring +safety_scorer = safety_scorer.start( + sampling_config=ScorerSamplingConfig(sample_rate=0.5) +) + +# Delete entirely (removes registration) +delete_scorer(name="production_safety") +``` + +--- + +## Pattern 14: Query Traces from Unity Catalog Tables + +Query ingested traces directly using SQL for custom analysis and dashboards. + +```sql +-- Count traces per day +SELECT + DATE(timestamp) as trace_date, + COUNT(DISTINCT trace_id) as trace_count +FROM my_catalog.my_schema.mlflow_experiment_trace_otel_spans +WHERE parent_span_id IS NULL -- root spans only +GROUP BY DATE(timestamp) +ORDER BY trace_date DESC; + +-- Find slow traces (root span duration > 10s) +SELECT + trace_id, + name as root_span_name, + (end_time_unix_nano - start_time_unix_nano) / 1e9 as duration_seconds +FROM my_catalog.my_schema.mlflow_experiment_trace_otel_spans +WHERE parent_span_id IS NULL + AND (end_time_unix_nano - start_time_unix_nano) / 1e9 > 10 +ORDER BY duration_seconds DESC +LIMIT 20; + +-- Error rate by span name +SELECT + name, + COUNT(*) as total, + SUM(CASE WHEN status_code = 'ERROR' THEN 1 ELSE 0 END) as errors, + ROUND(SUM(CASE WHEN status_code = 'ERROR' THEN 1 ELSE 0 END) * 100.0 / COUNT(*), 2) as error_pct +FROM my_catalog.my_schema.mlflow_experiment_trace_otel_spans +GROUP BY name +HAVING COUNT(*) > 10 +ORDER BY error_pct DESC; +``` + +**From Python (via Spark):** +```python +from databricks.connect import DatabricksSession + +spark = DatabricksSession.builder.remote(serverless=True).getOrCreate() + +# Query trace spans +spans_df = spark.sql(""" + SELECT trace_id, name, span_kind, + (end_time_unix_nano - start_time_unix_nano) / 1e6 as duration_ms + FROM my_catalog.my_schema.mlflow_experiment_trace_otel_spans + WHERE name LIKE '%retriever%' + ORDER BY duration_ms DESC + LIMIT 100 +""") +spans_df.show() +``` + +--- + +## Pattern 15: End-to-End Setup Script + +Complete setup script for a new project — from creating the UC schema link to logging the first trace and enabling monitoring. + +```python +import os +import mlflow +from mlflow.entities import UCSchemaLocation +from mlflow.tracing.enablement import set_experiment_trace_location +from mlflow.tracing import set_databricks_monitoring_sql_warehouse_id +from mlflow.genai.scorers import Safety, Guidelines, ScorerSamplingConfig + +# ============================================================ +# Configuration — UPDATE THESE VALUES +# ============================================================ +EXPERIMENT_NAME = "/Shared/my-agent-traces" +CATALOG_NAME = "my_catalog" +SCHEMA_NAME = "my_schema" +SQL_WAREHOUSE_ID = "abc123def456" # Your SQL warehouse ID + +# ============================================================ +# Step 1: Initial Setup +# ============================================================ +mlflow.set_tracking_uri("databricks") +os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = SQL_WAREHOUSE_ID + +# Create or retrieve experiment +if experiment := mlflow.get_experiment_by_name(EXPERIMENT_NAME): + experiment_id = experiment.experiment_id +else: + experiment_id = mlflow.create_experiment(name=EXPERIMENT_NAME) + +# Link UC schema (creates trace tables automatically) +set_experiment_trace_location( + location=UCSchemaLocation( + catalog_name=CATALOG_NAME, + schema_name=SCHEMA_NAME + ), + experiment_id=experiment_id, +) +print(f"Linked experiment '{EXPERIMENT_NAME}' to {CATALOG_NAME}.{SCHEMA_NAME}") + +# ============================================================ +# Step 2: Set Trace Destination +# ============================================================ +mlflow.set_experiment(EXPERIMENT_NAME) +mlflow.tracing.set_destination( + destination=UCSchemaLocation( + catalog_name=CATALOG_NAME, + schema_name=SCHEMA_NAME, + ) +) + +# ============================================================ +# Step 3: Enable Production Monitoring +# ============================================================ +set_databricks_monitoring_sql_warehouse_id( + warehouse_id=SQL_WAREHOUSE_ID, + experiment_id=experiment_id, +) + +# Register and start safety monitoring (100% of traces) +safety = Safety().register(name="safety_monitor") +safety = safety.start( + sampling_config=ScorerSamplingConfig(sample_rate=1.0) +) +print("Safety monitoring enabled (100% sample rate)") + +# Register and start custom guidelines (50% of traces) +tone = Guidelines( + name="professional_tone", + guidelines="The response must be professional, helpful, and concise" +).register(name="tone_monitor") +tone = tone.start( + sampling_config=ScorerSamplingConfig(sample_rate=0.5) +) +print("Tone monitoring enabled (50% sample rate)") + +# ============================================================ +# Step 4: Verify with a Test Trace +# ============================================================ +@mlflow.trace +def test_agent(query: str) -> str: + return f"Test response to: {query}" + +result = test_agent("Hello, is tracing working?") +print(f"Test trace logged. Check the Experiments UI at: {EXPERIMENT_NAME}") +``` + +--- + +## Limitations & Quotas + +| Limit | Value | +|-------|-------| +| Trace ingestion rate | 100 traces/second per workspace | +| Table ingestion throughput | 100 MB/second per table | +| Query throughput | 200 queries/second | +| UI performance | Degrades with >2TB of data | +| Trace deletion | Individual deletion not supported (use SQL) | +| MLflow MCP server | Does not support UC-stored traces | +| Region availability | `us-east-1` and `us-west-2` only (Beta) | + +--- + +## Viewing Traces in the UI + +1. Navigate to the **Experiments** page in your Databricks workspace +2. Select your experiment +3. Click the **Traces** tab +4. Select a **SQL warehouse** from the dropdown to query UC-stored traces +5. Browse traces, inspect spans, view inputs/outputs + +**Note:** You must select a SQL warehouse to view UC-stored traces — they are not loaded automatically. diff --git a/.claude/skills/databricks-mlflow-evaluation/references/user-journeys.md b/.claude/skills/databricks-mlflow-evaluation/references/user-journeys.md new file mode 100644 index 00000000..6ff09b28 --- /dev/null +++ b/.claude/skills/databricks-mlflow-evaluation/references/user-journeys.md @@ -0,0 +1,627 @@ +# User Journey Guides + +Step-by-step workflows for common evaluation scenarios. + +--- + +## Journey 0: Strategy Alignment (ALWAYS START HERE) + +**Starting Point**: You need to evaluate an agent +**Goal**: Align on what to evaluate before writing any code + +**PRIORITY:** Before writing evaluation code, complete strategy alignment. This ensures evaluations measure what matters and provide actionable insights. + +### Step 1: Understand the Agent + +Before evaluating, gather context about what you're evaluating: + +**Questions to ask (or investigate in the codebase):** +1. **What does this agent do?** (data analysis, RAG, multi-turn chat, task automation) +2. **What tools does it use?** (UC functions, vector search, external APIs) +3. **What is the input/output format?** (messages format, structured output) +4. **What is the current state?** (prototype, production, needs improvement) + +**Actions to take:** +- Read the agent's main code file (e.g., `agent.py`) +- Review the config file for system prompts and tool definitions +- Check existing tests or evaluation scripts +- Look at CLAUDE.md or README for project context + +### Step 2: Align on What to Evaluate + +**Evaluation dimensions to consider:** + +| Dimension | When to Use | Example Scorer | +|-----------|-------------|----------------| +| **Safety** | Always (table stakes) | `Safety()` | +| **Correctness** | When ground truth exists | `Correctness()` | +| **Relevance** | When responses should address queries | `RelevanceToQuery()` | +| **Groundedness** | RAG systems with retrieved context | `RetrievalGroundedness()` | +| **Domain Guidelines** | Domain-specific requirements | `Guidelines(name="...", guidelines="...")` | +| **Format/Structure** | Structured output requirements | Custom scorer | +| **Tool Usage** | Agents with tool calls | Custom scorer checking tool selection | + +**Questions to ask the user:** +1. What are the **must-have** quality criteria? (safety, accuracy, relevance) +2. What are the **nice-to-have** criteria? (conciseness, tone, format) +3. Are there **specific failure modes** you've seen or worry about? +4. Do you have **ground truth** or expected answers for test cases? + +### Step 3: Define User Scenarios (Evaluation Dataset) + +**Types of test cases to include:** + +| Category | Purpose | Example | +|----------|---------|---------| +| **Happy Path** | Core functionality works | Typical user questions | +| **Edge Cases** | Boundary conditions | Empty inputs, very long queries | +| **Adversarial** | Robustness testing | Prompt injection, off-topic | +| **Multi-turn** | Conversation handling | Follow-up questions, context recall | +| **Domain-specific** | Business logic | Industry terminology, specific formats | + +**Questions to ask the user:** +1. What are the **most common** questions users ask? +2. What are **challenging** questions the agent should handle? +3. Are there questions it should **refuse** to answer? +4. Do you have **existing test cases** or production traces to start from? + +### Step 4: Establish Success Criteria + +**Define quality gates before running evaluation:** + +```python +QUALITY_GATES = { + "safety": 1.0, # 100% - non-negotiable + "correctness": 0.9, # 90% - high bar for accuracy + "relevance": 0.85, # 85% - good relevance + "concise": 0.8, # 80% - nice to have +} +``` + +**Questions to ask the user:** +1. What pass rates are **acceptable** for each dimension? +2. Which metrics are **blocking** vs **informational**? +3. How will evaluation results **inform decisions**? (ship/no-ship, iterate, investigate) + +### Strategy Alignment Checklist + +Before implementing evaluation, confirm: +- [ ] Agent purpose and architecture understood +- [ ] Evaluation dimensions agreed upon +- [ ] Test case categories identified +- [ ] Success criteria defined +- [ ] Data source identified (new, traces, existing dataset) + +--- + +## Journey 3: "Something Broke" - Regression Detection + +**Starting Point**: You made changes to your agent and suspect something regressed +**Goal**: Identify what broke and verify the fix + +### Steps + +1. **Establish baseline metrics** + ```bash + # Run evaluation on the previous version (or use saved baseline) + cd agents/tool_calling_dspy + python run_quick_eval.py + ``` + Record key metrics: `classifier_accuracy`, `tool_selection_accuracy`, `follows_instructions` + +2. **Run evaluation on current version** + ```bash + python run_quick_eval.py + ``` + +3. **Compare metrics** + ```python + from evaluation.optimization_history import OptimizationHistory + + history = OptimizationHistory() + print(history.compare_iterations(-2, -1)) # Compare last two + ``` + +4. **Identify regression source** + - If `classifier_accuracy` dropped → Check ClassifierSignature changes + - If `tool_selection_accuracy` dropped → Check tool descriptions, required_tools field + - If `follows_instructions` dropped → Check ExecutorSignature output format + +5. **Analyze failing traces** + ``` + /eval:analyze-traces [experiment-id] + ``` + Look for: + - Error patterns in specific test categories + - Tool call failures + - Unexpected outputs + +6. **Fix and re-evaluate** + - Revert problematic changes or apply targeted fix + - Re-run evaluation + - Verify metrics restored + +### Commands Used +- `python run_quick_eval.py` - Run evaluation +- `/eval:analyze-traces` - Deep trace analysis +- `OptimizationHistory.compare_iterations()` - Metric comparison + +### Success Indicators +- Metrics return to baseline or improve +- No new failing test cases +- Trace analysis shows expected behavior + +--- + +## Journey 7: "My Multi-Agent is Slow" - Performance Optimization + +**Starting Point**: Your agent responses are too slow +**Goal**: Identify bottlenecks and reduce latency + +### Steps + +1. **Run evaluation with latency scoring** + ```bash + cd agents/tool_calling_dspy + python run_quick_eval.py + ``` + Note the latency metrics: + - `classifier_latency_ms` + - `rewriter_latency_ms` + - `executor_latency_ms` + - `total_latency_ms` + +2. **Identify the bottleneck stage** + | Latency | Typical Range | If High, Check | + |---------|---------------|----------------| + | classifier_latency | <5s | ClassifierSignature verbosity | + | rewriter_latency | <10s | QueryRewriterSignature complexity | + | executor_latency | <30s | Tool call count, response generation | + +3. **Analyze traces for slow stages** + ``` + /eval:analyze-traces [experiment-id] + ``` + Focus on: + - Span durations by stage + - Number of LLM calls per stage + - Tool execution times + +4. **Run signature analysis** + ```bash + python -m evaluation.analyze_signatures + ``` + Look for: + - High total description chars (>2000) + - Verbose OutputField descriptions + - Missing examples (causes more retries) + +5. **Apply optimizations** + + **For high classifier latency:** + - Simplify ClassifierSignature docstring + - Add concrete examples to reduce ambiguity + + **For high executor latency:** + - Simplify ExecutorSignature.answer format + - Reduce output format requirements + - Consider caching repeated tool calls + + **For high total latency:** + - Review if all stages are necessary + - Consider parallel execution where possible + +6. **Re-evaluate and compare** + ```bash + python run_quick_eval.py + ``` + Use `OptimizationHistory.compare_iterations()` to verify improvement + +### Commands Used +- `python run_quick_eval.py` - Run evaluation with latency scoring +- `/eval:analyze-traces` - Trace analysis with timing breakdown +- `python -m evaluation.analyze_signatures` - Signature verbosity analysis + +### Success Indicators +- Target latencies: classifier <5s, executor <30s, total <60s +- No regression in accuracy metrics +- Consistent improvement across test categories + +--- + +## Journey 8: "Improve My Prompts" - Systematic Prompt Optimization + +**Starting Point**: Your agent works but could be more accurate +**Goal**: Systematically improve prompt quality through evaluation + +### Steps + +1. **Establish baseline** + ```bash + cd agents/tool_calling_dspy + python run_quick_eval.py + ``` + Record all metrics in `optimization_history.json` + +2. **Run signature analysis** + ```bash + python -m evaluation.analyze_signatures + ``` + Review the report for: + - Metric correlations (which signatures affect which metrics) + - Specific issues flagged per signature + +3. **Prioritize fixes by metric impact** + + | Metric | Primary Signature | Common Issues | + |--------|-------------------|---------------| + | follows_instructions | ExecutorSignature | Verbose answer format, unclear structure | + | tool_selection_accuracy | ClassifierSignature | No examples, ambiguous tool descriptions | + | classifier_accuracy | ClassifierSignature | Verbose docstring, unclear query_type mapping | + +4. **Apply ONE fix at a time** + - Make a single, targeted change + - Document the change in your commit message + - Track in optimization_history.json + +5. **Re-evaluate immediately** + ```bash + python run_quick_eval.py + ``` + - If improved → Keep change, move to next fix + - If regressed → Revert and try different approach + - If unchanged → Consider if fix was necessary + +6. **Iterate until targets met** + + | Metric | Target | + |--------|--------| + | classifier_accuracy | 95%+ | + | tool_selection_accuracy | 90%+ | + | follows_instructions | 80%+ | + +7. **Document successful optimizations** + ```python + from evaluation.optimization_history import OptimizationHistory + + history = OptimizationHistory() + print(history.summary()) + ``` + +### Commands Used +- `python run_quick_eval.py` - Run evaluation +- `python -m evaluation.analyze_signatures` - Identify prompt issues +- `/optimize:context --quick` - Full optimization loop (when endpoint available) + +### Success Indicators +- All target metrics met +- No regressions from baseline +- Clear documentation of what changed and why +- Optimization history shows positive trend + +--- + +## Journey 9: "Store Traces in Unity Catalog" - Trace Ingestion & Production Monitoring + +**Starting Point**: You want to persist traces in Unity Catalog for long-term analysis, compliance, or production monitoring +**Goal**: Set up trace ingestion, instrument your app, and enable continuous monitoring + +### Prerequisites + +- Unity Catalog-enabled workspace +- "OpenTelemetry on Databricks" preview enabled +- SQL warehouse with `CAN USE` permissions +- MLflow 3.9.0+ (`pip install mlflow[databricks]>=3.9.0`) +- Workspace in `us-east-1` or `us-west-2` (Beta limitation) + +### Steps + +1. **Link UC schema to experiment** + ```python + import os + import mlflow + from mlflow.entities import UCSchemaLocation + from mlflow.tracing.enablement import set_experiment_trace_location + + mlflow.set_tracking_uri("databricks") + os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "" + + experiment_id = mlflow.create_experiment(name="/Shared/my-traces") + set_experiment_trace_location( + location=UCSchemaLocation(catalog_name="my_catalog", schema_name="my_schema"), + experiment_id=experiment_id, + ) + ``` + This creates three tables: `mlflow_experiment_trace_otel_logs`, `_metrics`, `_spans` + +2. **Grant permissions** + ```sql + GRANT USE_CATALOG ON CATALOG my_catalog TO `user@company.com`; + GRANT USE_SCHEMA ON SCHEMA my_catalog.my_schema TO `user@company.com`; + GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_logs TO `user@company.com`; + GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_spans TO `user@company.com`; + GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.mlflow_experiment_trace_otel_metrics TO `user@company.com`; + ``` + **CRITICAL**: `ALL_PRIVILEGES` is not sufficient — explicit MODIFY + SELECT required. + +3. **Set trace destination in your app** + ```python + mlflow.tracing.set_destination( + destination=UCSchemaLocation(catalog_name="my_catalog", schema_name="my_schema") + ) + # OR + os.environ["MLFLOW_TRACING_DESTINATION"] = "my_catalog.my_schema" + ``` + +4. **Instrument your application** + + Choose the appropriate approach: + - **Auto-tracing**: `mlflow.openai.autolog()` (or langchain, anthropic, etc.) + - **Manual tracing**: `@mlflow.trace` decorator on functions + - **Context manager**: `mlflow.start_span()` for fine-grained control + - **Combined**: Auto-tracing + manual decorators for full coverage + + See `patterns-trace-ingestion.md` Patterns 5-8 for detailed examples. + +5. **Configure additional trace sources** (if applicable) + + | Source | Key Configuration | + |--------|-------------------| + | Databricks Apps | Grant SP permissions, set `MLFLOW_TRACING_DESTINATION` | + | Model Serving | Add `DATABRICKS_TOKEN` + `MLFLOW_TRACING_DESTINATION` env vars | + | OTEL Clients | Use OTLP exporter with `X-Databricks-UC-Table-Name` header | + + See `patterns-trace-ingestion.md` Patterns 9-11 for detailed setup per source. + +6. **Enable production monitoring** + ```python + from mlflow.tracing import set_databricks_monitoring_sql_warehouse_id + from mlflow.genai.scorers import Safety, ScorerSamplingConfig + + set_databricks_monitoring_sql_warehouse_id(warehouse_id="") + + safety = Safety().register(name="safety_monitor") + safety = safety.start(sampling_config=ScorerSamplingConfig(sample_rate=1.0)) + ``` + +7. **Verify in the UI** + - Navigate to **Experiments** → your experiment → **Traces** tab + - Select a SQL warehouse from the dropdown to load UC traces + - Verify traces appear with correct span hierarchy + +### Reference Files +- `patterns-trace-ingestion.md` — All setup and instrumentation patterns +- `CRITICAL-interfaces.md` — Trace ingestion API signatures +- `GOTCHAS.md` — Common trace ingestion mistakes + +### Success Indicators +- Traces visible in the Experiments UI Traces tab +- Three UC tables populated with data +- Production monitoring scorers running and producing assessments +- No permission errors in trace ingestion + +--- + +## Journey 10: Domain Expert Optimization Loop + +**Starting Point**: You have an agent and want to incorporate domain expert feedback to continuously improve quality. +**Goal**: Run the full evaluate, label, align judge, optimize prompt, promote cycle. + +For the full architecture and end-to-end walkthrough, see the [Self-Optimizing Agent blog post](https://www.databricks.com/blog/self-optimizing-football-chatbot-guided-domain-experts-databricks). For details on the MemAlign alignment approach, see the [MemAlign research blog post](https://www.databricks.com/blog/memalign-building-better-llm-judges-human-feedback-scalable-memory). + +### The Loop at a Glance + +``` +1. Run evaluate() -> Generate traces, score with base judge +2. Tag traces -> Mark successfully evaluated traces for dataset +3. Build eval dataset -> Persist traces to UC for labeling +4. Labeling session -> SMEs review & score responses in Review App + (label schema name MUST match judge name) +5. Align judge (MemAlign) -> Distill SME feedback into judge guidelines +6. Re-evaluate -> Baseline with aligned judge (score may decrease, that's OK) +7. Build optim dataset -> inputs + expectations (required for GEPA) +8. optimize_prompts() -> GEPA iteratively improves system prompt +9. Conditional promote -> Update "production" alias only if score improves +``` + +### Why This Works + +Generic LLM judges and static prompts fail to capture domain-specific nuance. Determining what makes a response "good" requires domain knowledge that general-purpose evaluators miss. This loop solves the problem in two phases: + +- **Align the judge**: Domain experts review outputs and rate quality. MemAlign distills their feedback into judge guidelines, teaching the judge what "good" means for your specific domain. This is valuable on its own -- an aligned judge improves every evaluation run and monitoring setup. +- **Optimize the prompt**: The aligned judge drives GEPA prompt optimization, automatically evolving the system prompt to maximize the domain-expert-calibrated score. Only improvements get promoted to production. + +### Steps + +**Phase 1: Evaluate and Collect Feedback** + +1. **Design base judge, run evaluation, and tag traces** + + Create a domain-specific judge with `make_judge`, register it, run `evaluate()`, and tag traces that were successfully evaluated (agent responded AND judge scored without errors). + + See `patterns-judge-alignment.md` Patterns 1-2 + +2. **Build dataset and create labeling session** + + Persist tagged traces to a UC dataset and create a labeling session for domain experts. + + **CRITICAL: The label schema `name` MUST match the judge `name` used in `evaluate()`.** This is how `align()` pairs SME feedback with LLM judge scores. If they don't match, alignment will fail. + + See `patterns-judge-alignment.md` Pattern 3 + +3. **Wait for SMEs to complete labeling** (asynchronous step) + + Share `labeling_session.url` with domain experts. They review agent responses and submit ratings using the Review App. + +**Phase 2: Align the Judge** + +4. **Align judge with MemAlign (recommended)** + + MemAlign is the recommended alignment optimizer. It is the fastest (seconds vs. minutes for alternatives), most cost-effective ($0.03 vs. $1-$5), and supports memory scaling where quality continues to improve as feedback accumulates. Other optimizers (e.g., SIMBA) are also supported. + + See `patterns-judge-alignment.md` Patterns 4-5 + +5. **Re-evaluate with the aligned judge** + + The aligned judge score **may be lower** than the unaligned judge score. This is expected and correct -- it means the judge is now evaluating with domain-expert standards rather than generic best practices. A lower score from a more accurate judge is a better signal than an inflated score from a judge that doesn't understand your domain. + + See `patterns-judge-alignment.md` Pattern 6 + +6. **(Optional) Stop here** -- the aligned judge improves all future evaluations and production monitoring, independent of prompt optimization. + +**Phase 3: Optimize the Prompt** + +7. **Build optimization dataset with expectations** (required for GEPA) + + Unlike the eval dataset, the optimization dataset must have both `inputs` AND `expectations` per record. GEPA uses expectations during reflection to reason about why the current prompt is underperforming. + + See `patterns-prompt-optimization.md` Pattern 1 + +8. **Run `optimize_prompts()` with GEPA + aligned judge** + + GEPA iteratively evolves the system prompt, using the aligned judge as the scoring function. + + See `patterns-prompt-optimization.md` Pattern 2 + +9. **Conditionally promote** + + Register the new prompt version and only promote to the "production" alias if the score improved. + + See `patterns-prompt-optimization.md` Pattern 3 + +10. **Repeat from Step 1** -- each labeling session accumulates more SME signal for alignment + +### Complete Loop Summary + +```python +# -- PHASE 1: Evaluate and collect feedback ----------------------------------- + +# Step 1: Evaluate and tag successfully evaluated traces +results = evaluate(data=eval_data, predict_fn=..., scorers=[base_judge]) +ok_trace_ids = results.result_df.loc[results.result_df["state"] == "OK", "trace_id"] +for trace_id in ok_trace_ids: + mlflow.set_trace_tag(trace_id, key="eval", value="complete") + +# Step 2: Build dataset and labeling session +eval_dataset = create_dataset(name=DATASET_NAME) +eval_dataset.merge_records(tagged_traces) +# CRITICAL: label schema name must match judge name for align() to work +labeling_session = create_labeling_session( + name="sme_session", assigned_users=[...], label_schemas=[JUDGE_NAME] +) +labeling_session.add_dataset(dataset_name=DATASET_NAME) +# -> Share labeling_session.url with domain experts + +# Step 3: Wait for SMEs to complete labeling + +# -- PHASE 2: Align the judge ------------------------------------------------- + +# Step 4: Align judge (MemAlign recommended; SIMBA and others also supported) +optimizer = MemAlignOptimizer(reflection_lm=..., retrieval_k=5, embedding_model=...) +aligned_judge = base_judge.align(traces=traces, optimizer=optimizer) +aligned_judge.update(experiment_id=EXPERIMENT_ID) +# NOTE: Aligned judge scores may be lower than unaligned -- this is expected + +# Step 5: Re-evaluate with aligned judge (optional but recommended) +baseline_results = evaluate(data=eval_records, predict_fn=..., scorers=[aligned_judge]) + +# Step 6: (Optional) Stop here if you only need an aligned judge + +# -- PHASE 3: Optimize the prompt --------------------------------------------- + +# Step 7: Build optimization dataset (must have inputs + expectations) +optimization_dataset = [ + {"inputs": {...}, "expectations": {"expected_response": "..."}} +] + +# Step 8: Optimize prompt with GEPA + aligned judge +result = mlflow.genai.optimize_prompts( + predict_fn=predict_fn, + train_data=optimization_dataset, + prompt_uris=[system_prompt.uri], + optimizer=GepaPromptOptimizer(reflection_model=..., max_metric_calls=75), + scorers=[aligned_judge], + aggregation=objective_function, +) + +# Step 9: Conditional promotion +new_version = mlflow.genai.register_prompt( + name=PROMPT_NAME, template=result.optimized_prompts[0].template +) +if result.final_eval_score > result.initial_eval_score: + mlflow.genai.set_prompt_alias( + name=PROMPT_NAME, alias="production", version=new_version.version + ) + +# -- Repeat from Step 1 with new labeling session ----------------------------- +``` + +### Automation + +The loop can be orchestrated as a Databricks job using Asset Bundles: + +1. SMEs label agent outputs through the MLflow Labeling Session UI +2. The pipeline detects new labels and pulls traces with both SME feedback and baseline LLM judge scores +3. Judge alignment runs with MemAlign, producing a new judge version +4. Prompt optimization runs with GEPA, using the aligned judge +5. Conditional promotion pushes the new prompt to production if it exceeds performance thresholds +6. The agent improves automatically as the prompt registry serves the optimized version + +Manual review can be injected at any step, giving developers complete control over the level of automation. + +### Key Gotchas + +- **Label schema name matching**: The label schema `name` MUST match the judge `name` from `evaluate()`, or `align()` cannot pair the scores +- **Score decrease after alignment**: The aligned judge may give lower scores than the unaligned judge. This is expected -- the judge is now more accurate, not the agent worse +- **MemAlign embedding costs**: Set `embedding_model` explicitly (e.g., `"databricks:/databricks-gte-large-en"`) and filter traces to labeled subset only +- **GEPA expectations**: The optimization dataset must have both `inputs` AND `expectations` per record +- **Episodic memory**: After `get_scorer()`, inspect `.instructions` not `._episodic_memory` (lazy loaded) + +See `GOTCHAS.md` for the complete list. + +### Reference Files + +- `patterns-judge-alignment.md` -- Judge alignment workflow: design judge, evaluate, label, MemAlign, register, re-evaluate +- `patterns-prompt-optimization.md` -- GEPA optimization: build dataset, run optimize_prompts, register/promote +- `GOTCHAS.md` -- MemAlign embedding costs, episodic memory lazy loading, name matching, score interpretation, GEPA expectations + +### Success Indicators + +- Aligned judge instructions include domain-specific guidelines derived from SME ratings +- `result.final_eval_score > result.initial_eval_score` +- Production prompt alias updated only on genuine improvements +- Repeat sessions progressively encode more expert knowledge + +--- + +## Quick Reference + +### Which Journey Am I On? + +| Symptom | Journey | +|---------|---------| +| "It was working before" | Journey 3 (Regression) | +| "It's too slow" | Journey 7 (Performance) | +| "It's not accurate enough" | Journey 8 (Prompt Optimization) | +| "I need traces in Unity Catalog" | Journey 9 (Trace Ingestion) | +| "I want SMEs to improve my judge and prompt" | Journey 10 (Domain Expert Loop) | + +### Common Tools Across Journeys + +| Tool | Purpose | +|------|---------| +| `run_quick_eval.py` | Fast evaluation (8 test cases) | +| `run_full_eval.py` | Full evaluation (23 test cases) | +| `analyze_signatures.py` | Signature/prompt analysis | +| `OptimizationHistory` | Track iterations | +| `/eval:analyze-traces` | Deep trace analysis | +| `/optimize:context` | Full optimization loop | + +### Metric Targets + +| Metric | Target | Critical Threshold | +|--------|--------|-------------------| +| classifier_accuracy | 95%+ | <80% | +| tool_selection_accuracy | 90%+ | <70% | +| follows_instructions | 80%+ | <50% | +| executor_latency | <30s | >60s | diff --git a/.claude/skills/model-serving/1-classical-ml.md b/.claude/skills/databricks-model-serving/1-classical-ml.md similarity index 99% rename from .claude/skills/model-serving/1-classical-ml.md rename to .claude/skills/databricks-model-serving/1-classical-ml.md index 0d7d5ace..4b973e0a 100644 --- a/.claude/skills/model-serving/1-classical-ml.md +++ b/.claude/skills/databricks-model-serving/1-classical-ml.md @@ -143,7 +143,8 @@ endpoint = w.serving_endpoints.create_and_wait( ### Via MCP Tool ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="diabetes-predictor", dataframe_records=[ {"age": 45, "bmi": 25.3, "bp": 120, "s1": 200} diff --git a/.claude/skills/model-serving/2-custom-pyfunc.md b/.claude/skills/databricks-model-serving/2-custom-pyfunc.md similarity index 98% rename from .claude/skills/model-serving/2-custom-pyfunc.md rename to .claude/skills/databricks-model-serving/2-custom-pyfunc.md index afd6e185..b7dbad3f 100644 --- a/.claude/skills/model-serving/2-custom-pyfunc.md +++ b/.claude/skills/databricks-model-serving/2-custom-pyfunc.md @@ -189,7 +189,8 @@ endpoint = client.create_endpoint( ## Query Custom Model ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="custom-model-endpoint", dataframe_records=[ {"age": 25, "income": 50000, "category": "A"} @@ -200,7 +201,8 @@ query_serving_endpoint( Or with inputs format: ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="custom-model-endpoint", inputs={"age": 25, "income": 50000, "category": "A"} ) diff --git a/.claude/skills/model-serving/3-genai-agents.md b/.claude/skills/databricks-model-serving/3-genai-agents.md similarity index 94% rename from .claude/skills/model-serving/3-genai-agents.md rename to .claude/skills/databricks-model-serving/3-genai-agents.md index f408760b..4061dbab 100644 --- a/.claude/skills/model-serving/3-genai-agents.md +++ b/.claude/skills/databricks-model-serving/3-genai-agents.md @@ -155,13 +155,16 @@ mlflow.models.set_model(AGENT) ## Using Databricks-Hosted Models +Use exact endpoint names from the reference table in [SKILL.md](SKILL.md#foundation-model-api-endpoints). + ```python from databricks_langchain import ChatDatabricks -# Foundation Model APIs (pay-per-token) +# Foundation Model APIs (pay-per-token) - use exact endpoint names llm = ChatDatabricks(endpoint="databricks-meta-llama-3-3-70b-instruct") -llm = ChatDatabricks(endpoint="databricks-claude-3-7-sonnet") -llm = ChatDatabricks(endpoint="databricks-dbrx-instruct") +llm = ChatDatabricks(endpoint="databricks-claude-sonnet-4-6") +llm = ChatDatabricks(endpoint="databricks-gpt-5-1") +llm = ChatDatabricks(endpoint="databricks-gemini-3-flash") # Custom fine-tuned model endpoint llm = ChatDatabricks(endpoint="my-finetuned-model-endpoint") @@ -221,7 +224,7 @@ for event in AGENT.predict_stream(request): Run via MCP: ``` -run_python_file_on_databricks(file_path="./my_agent/test_agent.py") +execute_code(file_path="./my_agent/test_agent.py") ``` ## Logging the Agent @@ -272,7 +275,8 @@ agents.deploy( ## Query Deployed Agent ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="my-agent-endpoint", messages=[{"role": "user", "content": "What is Databricks?"}], max_tokens=500 diff --git a/.claude/skills/model-serving/4-tools-integration.md b/.claude/skills/databricks-model-serving/4-tools-integration.md similarity index 98% rename from .claude/skills/model-serving/4-tools-integration.md rename to .claude/skills/databricks-model-serving/4-tools-integration.md index a9081056..50491ee0 100644 --- a/.claude/skills/model-serving/4-tools-integration.md +++ b/.claude/skills/databricks-model-serving/4-tools-integration.md @@ -39,7 +39,6 @@ uc_toolkit = UCFunctionToolkit( | Function | Purpose | |----------|---------| | `system.ai.python_exec` | Execute Python code | -| `system.ai.similarity_search` | Vector similarity search | ### Creating a UC Function diff --git a/.claude/skills/model-serving/5-development-testing.md b/.claude/skills/databricks-model-serving/5-development-testing.md similarity index 85% rename from .claude/skills/model-serving/5-development-testing.md rename to .claude/skills/databricks-model-serving/5-development-testing.md index cbc4f76d..2a3806cf 100644 --- a/.claude/skills/model-serving/5-development-testing.md +++ b/.claude/skills/databricks-model-serving/5-development-testing.md @@ -13,17 +13,17 @@ MCP-based workflow for developing and testing agents on Databricks. ▼ ┌─────────────────────────────────────────────────────────────┐ │ Step 2: Upload to workspace │ -│ → upload_folder MCP tool │ +│ → manage_workspace_files MCP tool │ └─────────────────────────────────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Step 3: Install packages │ -│ → execute_databricks_command MCP tool │ +│ → execute_code MCP tool │ └─────────────────────────────────────────────────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────┐ │ Step 4: Test agent (iterate) │ -│ → run_python_file_on_databricks MCP tool │ +│ → execute_code MCP tool (with file_path) │ │ → If error: fix locally, re-upload, re-run │ └─────────────────────────────────────────────────────────────┘ ``` @@ -85,12 +85,13 @@ print("Response:", result.model_dump(exclude_none=True)) ## Step 2: Upload to Workspace -Use the `upload_folder` MCP tool: +Use the `manage_workspace_files` MCP tool: ``` -upload_folder( - local_folder="./my_agent", - workspace_folder="/Workspace/Users/you@company.com/my_agent" +manage_workspace_files( + action="upload", + local_path="./my_agent", + workspace_path="/Workspace/Users/you@company.com/my_agent" ) ``` @@ -98,10 +99,10 @@ This uploads all files in parallel. ## Step 3: Install Packages -Use `execute_databricks_command` to install dependencies: +Use `execute_code` to install dependencies: ``` -execute_databricks_command( +execute_code( code="%pip install -U mlflow==3.6.0 databricks-langchain langgraph==0.3.4 databricks-agents pydantic" ) ``` @@ -111,7 +112,7 @@ execute_databricks_command( ### Follow-up Commands (Reuse Context) ``` -execute_databricks_command( +execute_code( code="dbutils.library.restartPython()", cluster_id="", context_id="" @@ -120,10 +121,10 @@ execute_databricks_command( ## Step 4: Test the Agent -Use `run_python_file_on_databricks`: +Use `execute_code` with `file_path`: ``` -run_python_file_on_databricks( +execute_code( file_path="./my_agent/test_agent.py", cluster_id="", context_id="" @@ -134,8 +135,8 @@ run_python_file_on_databricks( 1. Read the error from the output 2. Fix the local file (`agent.py` or `test_agent.py`) -3. Re-upload: `upload_folder(...)` -4. Re-run: `run_python_file_on_databricks(...)` +3. Re-upload: `manage_workspace_files(action="upload", ...)` +4. Re-run: `execute_code(file_path=...)` ### Iteration Tips @@ -148,7 +149,7 @@ run_python_file_on_databricks( ### Check if packages are installed ``` -execute_databricks_command( +execute_code( code="import mlflow; print(mlflow.__version__)", cluster_id="", context_id="" @@ -158,7 +159,7 @@ execute_databricks_command( ### List available endpoints ``` -execute_databricks_command( +execute_code( code=""" from databricks.sdk import WorkspaceClient w = WorkspaceClient() @@ -173,7 +174,7 @@ for ep in list(w.serving_endpoints.list())[:10]: ### Test LLM endpoint directly ``` -execute_databricks_command( +execute_code( code=""" from databricks_langchain import ChatDatabricks llm = ChatDatabricks(endpoint="databricks-meta-llama-3-3-70b-instruct") @@ -189,11 +190,11 @@ print(response.content) | Step | MCP Tool | Purpose | |------|----------|---------| -| Upload files | `upload_folder` | Sync local files to workspace | -| Install packages | `execute_databricks_command` | Set up dependencies | -| Restart Python | `execute_databricks_command` | Apply package changes | -| Test agent | `run_python_file_on_databricks` | Run test script | -| Debug | `execute_databricks_command` | Quick checks | +| Upload files | `manage_workspace_files` (action="upload") | Sync local files to workspace | +| Install packages | `execute_code` | Set up dependencies | +| Restart Python | `execute_code` | Apply package changes | +| Test agent | `execute_code` (with `file_path`) | Run test script | +| Debug | `execute_code` | Quick checks | ## Next Steps diff --git a/.claude/skills/model-serving/6-logging-registration.md b/.claude/skills/databricks-model-serving/6-logging-registration.md similarity index 98% rename from .claude/skills/model-serving/6-logging-registration.md rename to .claude/skills/databricks-model-serving/6-logging-registration.md index f2344aff..cd687358 100644 --- a/.claude/skills/model-serving/6-logging-registration.md +++ b/.claude/skills/databricks-model-serving/6-logging-registration.md @@ -63,7 +63,7 @@ print(f"Registered: {uc_model_info.name} version {uc_model_info.version}") Run via MCP: ``` -run_python_file_on_databricks(file_path="./my_agent/log_model.py") +execute_code(file_path="./my_agent/log_model.py") ``` ## Resources for Auto Authentication diff --git a/.claude/skills/model-serving/7-deployment.md b/.claude/skills/databricks-model-serving/7-deployment.md similarity index 62% rename from .claude/skills/model-serving/7-deployment.md rename to .claude/skills/databricks-model-serving/7-deployment.md index 63b8c8b7..666cb168 100644 --- a/.claude/skills/model-serving/7-deployment.md +++ b/.claude/skills/databricks-model-serving/7-deployment.md @@ -41,10 +41,11 @@ print(f"Endpoint: {deployment.endpoint_name}") ### Step 2: Create Deployment Job (One-Time) -Use the `create_job` MCP tool: +Use the `manage_jobs` MCP tool with action="create": ``` -create_job( +manage_jobs( + action="create", name="deploy-agent-job", tasks=[ { @@ -66,10 +67,11 @@ Save the returned `job_id`. ### Step 3: Run Deployment (Async) -Use `run_job_now` - returns immediately: +Use `manage_job_runs` with action="run_now" - returns immediately: ``` -run_job_now( +manage_job_runs( + action="run_now", job_id="", job_parameters={"model_name": "main.agents.my_agent", "version": "1"} ) @@ -82,13 +84,13 @@ Save the returned `run_id`. Check job run status: ``` -get_run(run_id="") +manage_job_runs(action="get", run_id="") ``` Or check endpoint directly: ``` -get_serving_endpoint_status(name="") +manage_serving_endpoint(action="get", name="") ``` ## Classical ML Deployment @@ -142,13 +144,61 @@ endpoint = w.serving_endpoints.create_and_wait( ) ``` -## Endpoint Naming +## Endpoint Naming and Visibility -For agents deployed with `databricks.agents.deploy()`: +### Auto-generated Names -- Endpoint name is derived from model name -- `main.agents.my_agent` → `agents_my_agent` or similar -- Check with `list_serving_endpoints()` after deployment +When you call `agents.deploy()`, the endpoint name is auto-derived from the UC model path by replacing dots with underscores and prefixing with `agents_`: + +| UC Model Path | Auto-generated Endpoint Name | +|---------------|------------------------------| +| `main.agents.my_agent` | `agents_main-agents-my_agent` | +| `catalog.schema.model` | `agents_catalog-schema-model` | +| `users.jane.demo_bot` | `agents_users-jane-demo_bot` | + +The exact format can vary. To avoid surprises, **always specify the endpoint name explicitly**: + +```python +deployment = agents.deploy( + "main.agents.my_agent", + "1", + endpoint_name="my-agent-endpoint", # Control the name + tags={"source": "mcp", "environment": "dev"} +) +``` + +### Finding Endpoints in the UI + +Endpoints created via `agents.deploy()` appear under **Serving** in the Databricks UI. If you don't see your endpoint: + +1. **Check the filter** - The Serving page defaults to "Owned by me". If the deployment ran as a service principal (e.g., via a job), switch to "All" to see it. +2. **Verify via API** - Use `manage_serving_endpoint(action="list")` or `manage_serving_endpoint(action="get", name="...")` to confirm the endpoint exists and check its state. +3. **Check the name** - The auto-generated name may not be what you expect. Print `deployment.endpoint_name` in the deploy script or check the job run output. + +### Deployment Script with Explicit Naming + +```python +# deploy_agent.py - recommended pattern +import sys +from databricks import agents + +model_name = sys.argv[1] if len(sys.argv) > 1 else "main.agents.my_agent" +version = sys.argv[2] if len(sys.argv) > 2 else "1" +endpoint_name = sys.argv[3] if len(sys.argv) > 3 else None + +deploy_kwargs = { + "tags": {"source": "mcp", "environment": "dev"} +} +if endpoint_name: + deploy_kwargs["endpoint_name"] = endpoint_name + +print(f"Deploying {model_name} version {version}...") +deployment = agents.deploy(model_name, version, **deploy_kwargs) + +print(f"Deployment complete!") +print(f"Endpoint name: {deployment.endpoint_name}") +print(f"Query URL: {deployment.query_endpoint}") +``` ## Deployment Job Template @@ -213,16 +263,16 @@ client.update_endpoint( | Step | MCP Tool | Waits? | |------|----------|--------| -| Upload deploy script | `upload_folder` | Yes | -| Create job (one-time) | `create_job` | Yes | -| Run deployment | `run_job_now` | **No** - returns immediately | -| Check job status | `get_run` | Yes | -| Check endpoint status | `get_serving_endpoint_status` | Yes | +| Upload deploy script | `manage_workspace_files` (action="upload") | Yes | +| Create job (one-time) | `manage_jobs` (action="create") | Yes | +| Run deployment | `manage_job_runs` (action="run_now") | **No** - returns immediately | +| Check job status | `manage_job_runs` (action="get") | Yes | +| Check endpoint status | `manage_serving_endpoint` (action="get") | Yes | ## After Deployment Once endpoint is READY: -1. **Test with MCP**: `query_serving_endpoint(name="...", messages=[...])` +1. **Test with MCP**: `manage_serving_endpoint(action="query", name="...", messages=[...])` 2. **Share with team**: Endpoint URL in Databricks UI 3. **Integrate in apps**: Use REST API or SDK diff --git a/.claude/skills/model-serving/8-querying-endpoints.md b/.claude/skills/databricks-model-serving/8-querying-endpoints.md similarity index 96% rename from .claude/skills/model-serving/8-querying-endpoints.md rename to .claude/skills/databricks-model-serving/8-querying-endpoints.md index 9c655a14..4dfa2f91 100644 --- a/.claude/skills/model-serving/8-querying-endpoints.md +++ b/.claude/skills/databricks-model-serving/8-querying-endpoints.md @@ -11,7 +11,7 @@ Send requests to deployed Model Serving endpoints. Before querying, verify the endpoint is ready: ``` -get_serving_endpoint_status(name="my-agent-endpoint") +manage_serving_endpoint(action="get", name="my-agent-endpoint") ``` Response: @@ -28,7 +28,8 @@ Response: ### Query Chat/Agent Endpoint ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="my-agent-endpoint", messages=[ {"role": "user", "content": "What is Databricks?"} @@ -61,7 +62,8 @@ Response: ### Query ML Model Endpoint ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="sklearn-classifier", dataframe_records=[ {"age": 25, "income": 50000, "credit_score": 720}, @@ -80,7 +82,7 @@ Response: ### List All Endpoints ``` -list_serving_endpoints(limit=20) +manage_serving_endpoint(action="list", limit=20) ``` ## Python SDK diff --git a/.claude/skills/model-serving/9-package-requirements.md b/.claude/skills/databricks-model-serving/9-package-requirements.md similarity index 97% rename from .claude/skills/model-serving/9-package-requirements.md rename to .claude/skills/databricks-model-serving/9-package-requirements.md index f78a1129..f9ceb7a9 100644 --- a/.claude/skills/model-serving/9-package-requirements.md +++ b/.claude/skills/databricks-model-serving/9-package-requirements.md @@ -139,10 +139,10 @@ export DATABRICKS_CONFIG_PROFILE="your-profile" ## Installing Packages via MCP -Use `execute_databricks_command`: +Use `execute_code`: ``` -execute_databricks_command( +execute_code( code="%pip install -U mlflow==3.6.0 databricks-langchain langgraph==0.3.4 databricks-agents pydantic" ) ``` @@ -150,7 +150,7 @@ execute_databricks_command( Then restart Python: ``` -execute_databricks_command( +execute_code( code="dbutils.library.restartPython()", cluster_id="", context_id="" @@ -174,7 +174,7 @@ for pkg in packages: Via MCP: ``` -execute_databricks_command( +execute_code( code=""" import pkg_resources for pkg in ['mlflow', 'langchain', 'langgraph', 'pydantic', 'databricks-langchain']: diff --git a/.claude/skills/model-serving/SKILL.md b/.claude/skills/databricks-model-serving/SKILL.md similarity index 50% rename from .claude/skills/model-serving/SKILL.md rename to .claude/skills/databricks-model-serving/SKILL.md index e8287fd9..74160298 100644 --- a/.claude/skills/model-serving/SKILL.md +++ b/.claude/skills/databricks-model-serving/SKILL.md @@ -1,5 +1,5 @@ --- -name: model-serving +name: databricks-model-serving description: "Deploy and query Databricks Model Serving endpoints. Use when (1) deploying MLflow models or AI agents to endpoints, (2) creating ChatAgent/ResponsesAgent agents, (3) integrating UC Functions or Vector Search tools, (4) querying deployed endpoints, (5) checking endpoint status. Covers classical ML models, custom pyfunc, and GenAI agents." --- @@ -21,6 +21,59 @@ Deploy MLflow models and AI agents to scalable REST API endpoints. - Unity Catalog enabled workspace - Model Serving enabled +## Foundation Model API Endpoints + +ALWAYS use exact endpoint names from this table. NEVER guess or abbreviate. + +### Chat / Instruct Models + +| Endpoint Name | Provider | Notes | +|--------------|----------|-------| +| `databricks-gpt-5-2` | OpenAI | Latest GPT, 400K context | +| `databricks-gpt-5-1` | OpenAI | Instant + Thinking modes | +| `databricks-gpt-5-1-codex-max` | OpenAI | Code-specialized (high perf) | +| `databricks-gpt-5-1-codex-mini` | OpenAI | Code-specialized (cost-opt) | +| `databricks-gpt-5` | OpenAI | 400K context, reasoning | +| `databricks-gpt-5-mini` | OpenAI | Cost-optimized reasoning | +| `databricks-gpt-5-nano` | OpenAI | High-throughput, lightweight | +| `databricks-gpt-oss-120b` | OpenAI | Open-weight, 128K context | +| `databricks-gpt-oss-20b` | OpenAI | Lightweight open-weight | +| `databricks-claude-opus-4-6` | Anthropic | Most capable, 1M context | +| `databricks-claude-sonnet-4-6` | Anthropic | Hybrid reasoning | +| `databricks-claude-sonnet-4-5` | Anthropic | Hybrid reasoning | +| `databricks-claude-opus-4-5` | Anthropic | Deep analysis, 200K context | +| `databricks-claude-sonnet-4` | Anthropic | Hybrid reasoning | +| `databricks-claude-opus-4-1` | Anthropic | 200K context, 32K output | +| `databricks-claude-haiku-4-5` | Anthropic | Fastest, cost-effective | +| `databricks-claude-3-7-sonnet` | Anthropic | Retiring April 2026 | +| `databricks-meta-llama-3-3-70b-instruct` | Meta | 128K context, multilingual | +| `databricks-meta-llama-3-1-405b-instruct` | Meta | Retiring May 2026 (PT) | +| `databricks-meta-llama-3-1-8b-instruct` | Meta | Lightweight, 128K context | +| `databricks-llama-4-maverick` | Meta | MoE architecture | +| `databricks-gemini-3-1-pro` | Google | 1M context, hybrid reasoning | +| `databricks-gemini-3-pro` | Google | 1M context, hybrid reasoning | +| `databricks-gemini-3-flash` | Google | Fast, cost-efficient | +| `databricks-gemini-2-5-pro` | Google | 1M context, Deep Think | +| `databricks-gemini-2-5-flash` | Google | 1M context, hybrid reasoning | +| `databricks-gemma-3-12b` | Google | 128K context, multilingual | +| `databricks-qwen3-next-80b-a3b-instruct` | Alibaba | Efficient MoE | + +### Embedding Models + +| Endpoint Name | Dimensions | Max Tokens | Notes | +|--------------|-----------|------------|-------| +| `databricks-gte-large-en` | 1024 | 8192 | English, not normalized | +| `databricks-bge-large-en` | 1024 | 512 | English, normalized | +| `databricks-qwen3-embedding-0-6b` | up to 1024 | ~32K | 100+ languages, instruction-aware | + +### Common Defaults + +- **Agent LLM**: `databricks-meta-llama-3-3-70b-instruct` (good balance of quality/cost) +- **Embedding**: `databricks-gte-large-en` +- **Code tasks**: `databricks-gpt-5-1-codex-mini` or `databricks-gpt-5-1-codex-max` + +> These are pay-per-token endpoints available in every workspace. For production, consider provisioned throughput mode. See [supported models](https://docs.databricks.com/aws/en/machine-learning/foundation-model-apis/supported-models). + ## Reference Files | Topic | File | When to Read | @@ -48,7 +101,7 @@ dbutils.library.restartPython() Or via MCP: ``` -execute_databricks_command(code="%pip install -U mlflow==3.6.0 databricks-langchain langgraph==0.3.4 databricks-agents pydantic") +execute_code(code="%pip install -U mlflow==3.6.0 databricks-langchain langgraph==0.3.4 databricks-agents pydantic") ``` ### Step 2: Create Agent File @@ -58,16 +111,17 @@ Create `agent.py` locally with `ResponsesAgent` pattern (see [3-genai-agents.md] ### Step 3: Upload to Workspace ``` -upload_folder( - local_folder="./my_agent", - workspace_folder="/Workspace/Users/you@company.com/my_agent" +manage_workspace_files( + action="upload", + local_path="./my_agent", + workspace_path="/Workspace/Users/you@company.com/my_agent" ) ``` ### Step 4: Test Agent ``` -run_python_file_on_databricks( +execute_code( file_path="./my_agent/test_agent.py", cluster_id="" ) @@ -76,7 +130,7 @@ run_python_file_on_databricks( ### Step 5: Log Model ``` -run_python_file_on_databricks( +execute_code( file_path="./my_agent/log_model.py", cluster_id="" ) @@ -89,7 +143,8 @@ See [7-deployment.md](7-deployment.md) for job-based deployment that doesn't tim ### Step 7: Query Endpoint ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="my-agent-endpoint", messages=[{"role": "user", "content": "Hello!"}] ) @@ -127,25 +182,48 @@ Then deploy via UI or SDK. See [1-classical-ml.md](1-classical-ml.md). | Tool | Purpose | |------|---------| -| `upload_folder` | Upload agent files to workspace | -| `run_python_file_on_databricks` | Test agent, log model | -| `execute_databricks_command` | Install packages, quick tests | +| `manage_workspace_files` (action="upload") | Upload agent files to workspace | +| `execute_code` | Install packages, test agent, log model | ### Deployment | Tool | Purpose | |------|---------| -| `create_job` | Create deployment job (one-time) | -| `run_job_now` | Kick off deployment (async) | -| `get_run` | Check deployment job status | +| `manage_jobs` (action="create") | Create deployment job (one-time) | +| `manage_job_runs` (action="run_now") | Kick off deployment (async) | +| `manage_job_runs` (action="get") | Check deployment job status | -### Querying +### manage_serving_endpoint - Querying -| Tool | Purpose | -|------|---------| -| `get_serving_endpoint_status` | Check if endpoint is READY | -| `query_serving_endpoint` | Send requests to endpoint | -| `list_serving_endpoints` | List all endpoints | +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `get` | Check endpoint status (READY/NOT_READY/NOT_FOUND) | name | +| `list` | List all endpoints | (none, optional limit) | +| `query` | Send requests to endpoint | name + one of: messages, inputs, dataframe_records | + +**Example usage:** +```python +# Check endpoint status +manage_serving_endpoint(action="get", name="my-agent-endpoint") + +# List all endpoints +manage_serving_endpoint(action="list") + +# Query a chat/agent endpoint +manage_serving_endpoint( + action="query", + name="my-agent-endpoint", + messages=[{"role": "user", "content": "Hello!"}], + max_tokens=500 +) + +# Query a traditional ML endpoint +manage_serving_endpoint( + action="query", + name="sklearn-classifier", + dataframe_records=[{"age": 25, "income": 50000, "credit_score": 720}] +) +``` --- @@ -154,7 +232,7 @@ Then deploy via UI or SDK. See [1-classical-ml.md](1-classical-ml.md). ### Check Endpoint Status After Deployment ``` -get_serving_endpoint_status(name="my-agent-endpoint") +manage_serving_endpoint(action="get", name="my-agent-endpoint") ``` Returns: @@ -169,7 +247,8 @@ Returns: ### Query a Chat/Agent Endpoint ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="my-agent-endpoint", messages=[ {"role": "user", "content": "What is Databricks?"} @@ -181,7 +260,8 @@ query_serving_endpoint( ### Query a Traditional ML Endpoint ``` -query_serving_endpoint( +manage_serving_endpoint( + action="query", name="sklearn-classifier", dataframe_records=[ {"age": 25, "income": 50000, "credit_score": 720} @@ -196,7 +276,7 @@ query_serving_endpoint( | Issue | Solution | |-------|----------| | **Invalid output format** | Use `self.create_text_output_item(text, id)` - NOT raw dicts! | -| **Endpoint NOT_READY** | Deployment takes ~15 min. Use `get_serving_endpoint_status` to poll. | +| **Endpoint NOT_READY** | Deployment takes ~15 min. Use `manage_serving_endpoint(action="get")` to poll. | | **Package not found** | Specify exact versions in `pip_requirements` when logging model | | **Tool timeout** | Use job-based deployment, not synchronous calls | | **Auth error on endpoint** | Ensure `resources` specified in `log_model` for auto passthrough | @@ -223,6 +303,14 @@ Available helper methods: --- +## Related Skills + +- **[databricks-agent-bricks](../databricks-agent-bricks/SKILL.md)** - Pre-built agent tiles that deploy to model-serving endpoints +- **[databricks-vector-search](../databricks-vector-search/SKILL.md)** - Create vector indexes used as retriever tools in agents +- **[databricks-genie](../databricks-genie/SKILL.md)** - Genie Spaces can serve as agents in multi-agent setups +- **[databricks-mlflow-evaluation](../databricks-mlflow-evaluation/SKILL.md)** - Evaluate model and agent quality before deployment +- **[databricks-jobs](../databricks-jobs/SKILL.md)** - Job-based async deployment used for agent endpoints + ## Resources - [Model Serving Documentation](https://docs.databricks.com/machine-learning/model-serving/) diff --git a/.claude/skills/databricks-python-sdk/SKILL.md b/.claude/skills/databricks-python-sdk/SKILL.md index c5937eec..eaf7cd66 100644 --- a/.claude/skills/databricks-python-sdk/SKILL.md +++ b/.claude/skills/databricks-python-sdk/SKILL.md @@ -613,3 +613,13 @@ If I'm unsure about a method, I should: | Pipelines | https://databricks-sdk-py.readthedocs.io/en/latest/workspace/pipelines/pipelines.html | | Secrets | https://databricks-sdk-py.readthedocs.io/en/latest/workspace/workspace/secrets.html | | DBUtils | https://databricks-sdk-py.readthedocs.io/en/latest/dbutils.html | + +## Related Skills + +- **[databricks-config](../databricks-config/SKILL.md)** - profile and authentication setup +- **[databricks-bundles](../databricks-bundles/SKILL.md)** - deploying resources via DABs +- **[databricks-jobs](../databricks-jobs/SKILL.md)** - job orchestration patterns +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - catalog governance +- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** - serving endpoint management +- **[databricks-vector-search](../databricks-vector-search/SKILL.md)** - vector index operations +- **[databricks-lakebase-provisioned](../databricks-lakebase-provisioned/SKILL.md)** - managed PostgreSQL via SDK diff --git a/.claude/skills/databricks-python-sdk/examples/5-serving-and-vector-search.py b/.claude/skills/databricks-python-sdk/examples/5-serving-and-vector-search.py index 2a47c2b0..597aedee 100644 --- a/.claude/skills/databricks-python-sdk/examples/5-serving-and-vector-search.py +++ b/.claude/skills/databricks-python-sdk/examples/5-serving-and-vector-search.py @@ -168,10 +168,15 @@ # Query with embedding vector directly +# query_vector must be a list[float] whose length matches your index's +# embedding dimension (e.g. 768 for bge-small, 1024 for bge-large, 1536 for +# text-embedding-3-small / ada-002). The [0.0] * N below is a stand-in; +# replace with the actual vector returned by your embedding model. +query_vector = [0.0] * 768 results = w.vector_search_indexes.query_index( index_name="main.default.my_index", columns=["id", "text"], - query_vector=[0.1, 0.2, 0.3, ...], # Your embedding vector + query_vector=query_vector, num_results=10 ) diff --git a/.claude/skills/databricks-spark-declarative-pipelines/SKILL.md b/.claude/skills/databricks-spark-declarative-pipelines/SKILL.md new file mode 100644 index 00000000..a1bdd7c3 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/SKILL.md @@ -0,0 +1,389 @@ +--- +name: databricks-spark-declarative-pipelines +description: "Creates, configures, and updates Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) using serverless compute. Handles data ingestion with streaming tables, materialized views, CDC, SCD Type 2, and Auto Loader ingestion patterns. Use when building data pipelines, working with Delta Live Tables, ingesting streaming data, implementing change data capture, or when the user mentions SDP, LDP, DLT, Lakeflow pipelines, streaming tables, or bronze/silver/gold medallion architectures." +--- + +# Lakeflow Spark Declarative Pipelines (SDP) + +--- + +## Critical Rules (always follow) + +### Syntax: CREATE OR REFRESH (not CREATE OR REPLACE) +- **MUST** use `CREATE OR REFRESH` for SDP objects: + - `CREATE OR REFRESH STREAMING TABLE` - for streaming tables + - `CREATE OR REFRESH MATERIALIZED VIEW` - for materialized views +- **NEVER** use `CREATE OR REPLACE` - that is standard SQL syntax, not SDP syntax + +### Simplicity First +- **MUST** create the minimal number of tables to solve the task +- Simplicity first: prefer single pipeline even for multi-schema setups - use fully qualified names (`catalog.schema.table`) +- When asked to "create a silver table" or "create a gold table", create **ONE table** - not a multi-layer pipeline +- Don't add intermediate tables, staging tables, or helper views unless explicitly requested +- A silver transformation = 1 streaming table reading from bronze +- A gold aggregation = 1 materialized view reading from silver +- Create bronze→silver→gold chains when the user asks for a "pipeline" or "medallion architecture" or full/detailed ingestion. Otherwise keep it simple - don't over engineer. + +### Language Selection +- **MUST** know the language (Python or SQL). For simple task / pipeline / table creation, pick SQL. For complex pipeline with parametrized information, or if the user mentions python-related items pick python. If you have a doubt, ask the user. Stick with that language unless told otherwise. + +| User Says | Action | +|-----------|--------| +| "Python pipeline", "Python SDP", "use Python", "udf", "pandas", "ml inference", "pyspark" | **User wants Python** | +| "SQL pipeline", "SQL files", "use SQL" | **User wants SQL** | +| "Create a simple pipeline", "create a table", "an aggregation" | **Pick SQL as it's simple** | + +### Other Rules +- **MUST** create serverless pipelines by default. Only use classic clusters if user explicitly requires R language, Spark RDD APIs, or JAR libraries. +- **MUST** choose the right workflow based on context (see below). +- When the user provides table schema and asks for code, respond directly with the code. Don't ask clarifying questions if the request is clear. + +## Tools +- List files in volume: `databricks fs ls dbfs:/Volumes/{catalog}/{schema}/{volume}/{path} --profile {PROFILE}` +- Query data: `databricks experimental aitools tools query --profile {PROFILE} --warehouse abc123 "SELECT 1 FROM catalog.schema.table"` +- Discover schema: `databricks experimental aitools tools discover-schema --profile {PROFILE} catalog.schema.table1 catalog.schema.table2` +- Pipelines CLI: `databricks pipelines init|deploy|run|logs|stop` or use `databricks pipelines --help` for more options + +## Choose Your Workflow + +**First, determine which workflow to use:** + +### Option A: Standalone New Pipeline Project (use `databricks pipelines init`) + +Use this when the user wants to **create a new, standalone SDP project** that will have its own DAB: +- User asks: "Create a new pipeline", "Build me an SDP", "Set up a new data pipeline" +- No existing `databricks.yml` in the workspace +- The pipeline IS the project (not part of a larger demo/app) + + +Use `databricks pipeline` CLI commands: +```bash +databricks pipelines init --output-dir . --config-file init-config.json +``` + +**Example init-config.json:** +```json +{ + "project_name": "customer_pipeline", + "initial_catalog": "prod_catalog", + "use_personal_schema": "no", + "initial_language": "sql" +} +``` + +→ See [1-project-initialization.md](references/1-project-initialization.md) +→ + + +### Option B: Pipeline within Existing Bundle (edit the bundle) + +Use this when the pipeline is **part of an existing DAB project**: +- There's already a `databricks.yml` file in the project +- User is adding a pipeline to an existing app/demo + +→ See [1-project-initialization.md](references/1-project-initialization.md) for adding pipelines to existing bundles + +### Option C: Rapid Iteration with MCP Tools (no bundle management) + +Use this when you need to **quickly create, test, and iterate** on a pipeline without managing bundle files: +- User wants to "just run a pipeline and see if it works" +- Part of a larger demo where bundle is managed separately, or the DAB bundle will be created at the end as you want to quickly test the project first +- Prototyping or experimenting with pipeline logic +- User explicitly asks to use MCP tools + +→ See [2-mcp-approach.md](references/2-mcp-approach.md) for MCP-based workflow + +--- + +## Required Checklist + +Before writing pipeline code, make sure you have: +``` +- [ ] Language selected: Python or SQL +- [ ] Read the syntax basics: **SQL**: Always Read [sql/1-syntax-basics.md](references/sql/1-syntax-basics.md), **Python**: Always Read [python/1-syntax-basics.md](references/python/1-syntax-basics.md) +- [ ] Workflow chosen: Standalone DAB / Existing DAB / MCP iteration +- [ ] Compute type: serverless (default) or classic +- [ ] Schema strategy: single schema with prefixes vs. multi-schema +- [ ] Consider [Multi-Schema Patterns](#multi-schema-patterns) and [Modern Defaults](#modern-defaults) +``` + +**Then read additional guides based on what the pipeline needs, when you need it:** +| If the pipeline needs... | Read | +|--------------------------|------| +| File ingestion (Auto Loader, JSON, CSV, Parquet) | `references/sql/2-ingestion.md` or `references/python/2-ingestion.md` | +| Kafka, Event Hub, or Kinesis streaming | `references/sql/2-ingestion.md` or `references/python/2-ingestion.md` | +| Deduplication, windowed aggregations, joins | `references/sql/3-streaming-patterns.md` or `references/python/3-streaming-patterns.md` | +| CDC, SCD Type 1/2, or history tracking | `references/sql/4-cdc-patterns.md` or `references/python/4-cdc-patterns.md` | +| Performance tuning, Liquid Clustering | `references/sql/5-performance.md` or `references/python/5-performance.md` | + +--- + +## Quick Reference + +| Concept | Details | +|---------|---------| +| **Names** | SDP = Spark Declarative Pipelines = LDP = Lakeflow Declarative Pipelines (all interchangeable) | +| **SQL Syntax** | `CREATE OR REFRESH STREAMING TABLE`, `CREATE OR REFRESH MATERIALIZED VIEW` | +| **Python Import** | `from pyspark import pipelines as dp` | +| **Primary Decorators** | `@dp.table()`, `@dp.materialized_view()`, `@dp.temporary_view()` | + +### Legacy APIs (Do NOT Use) + +| Legacy | Modern Replacement | +|--------|-------------------| +| `import dlt` | `from pyspark import pipelines as dp` | +| `dlt.apply_changes()` | `dp.create_auto_cdc_flow()` | +| `dlt.read()` / `dlt.read_stream()` | `spark.read` / `spark.readStream` | +| `CREATE LIVE XXX` | `CREATE OR REFRESH STREAMING TABLE\|MATERIALIZED VIEW` | +| `PARTITION BY` + `ZORDER` | `CLUSTER BY` (Liquid Clustering) | +| `input_file_name()` | `_metadata.file_path` | +| `target` parameter | `schema` parameter | + +### Streaming Table vs Materialized View + +| Use Case | Type | Pattern | +|----------|------|---------| +| Windowed aggregations (tumbling, sliding, session) | Streaming Table | `FROM stream(source)` + `GROUP BY window()` | +| Full-table aggregations (totals, daily counts) | Materialized View | `FROM source` (no stream wrapper) | +| CDC / SCD Type 2 | Streaming Table | `AUTO CDC INTO` or `dp.create_auto_cdc_flow()` | + +Use streaming tables for windowed aggregations to enable incremental processing. Use materialized views for simple aggregations that recompute fully on each refresh. + +--- + +## Task-Based Routing + +After choosing your workflow (see [Choose Your Workflow](#choose-your-workflow)), determine the specific task: + +**Choose documentation by language:** + +### SQL Documentation +| Task | Guide | +|------|-------| +| **SQL syntax basics** | [sql/1-syntax-basics.md](references/sql/1-syntax-basics.md) | +| **Data ingestion (Auto Loader, Kafka)** | [sql/2-ingestion.md](references/sql/2-ingestion.md) | +| **Streaming patterns (deduplication, windows)** | [sql/3-streaming-patterns.md](references/sql/3-streaming-patterns.md) | +| **CDC patterns (AUTO CDC, SCD, queries)** | [sql/4-cdc-patterns.md](references/sql/4-cdc-patterns.md) | +| **Performance tuning** | [sql/5-performance.md](references/sql/5-performance.md) | + +### Python Documentation +| Task | Guide | +|------|-------| +| **Python syntax basics** | [python/1-syntax-basics.md](references/python/1-syntax-basics.md) | +| **Data ingestion (Auto Loader, Kafka)** | [python/2-ingestion.md](references/python/2-ingestion.md) | +| **Streaming patterns (deduplication, windows)** | [python/3-streaming-patterns.md](references/python/3-streaming-patterns.md) | +| **CDC patterns (AUTO CDC, SCD, queries)** | [python/4-cdc-patterns.md](references/python/4-cdc-patterns.md) | +| **Performance tuning** | [python/5-performance.md](references/python/5-performance.md) | + +### General Documentation +| Task | Guide | +|------|-------| +| **Setting up standalone pipeline project** | [1-project-initialization.md](references/1-project-initialization.md) | +| **Rapid iteration with MCP tools** | [2-mcp-approach.md](references/2-mcp-approach.md) | +| **Advanced configuration** | [3-advanced-configuration.md](references/3-advanced-configuration.md) | +| **Migrating from DLT** | [4-dlt-migration.md](references/4-dlt-migration.md) | + +--- + +## Official Documentation + +- **[Lakeflow Spark Declarative Pipelines Overview](https://docs.databricks.com/aws/en/ldp/)** - Main documentation hub +- **[SQL Language Reference](https://docs.databricks.com/aws/en/ldp/developer/sql-dev)** - SQL syntax for streaming tables and materialized views +- **[Python Language Reference](https://docs.databricks.com/aws/en/ldp/developer/python-ref)** - `pyspark.pipelines` API +- **[Loading Data](https://docs.databricks.com/aws/en/ldp/load)** - Auto Loader, Kafka, Kinesis ingestion +- **[Change Data Capture (CDC)](https://docs.databricks.com/aws/en/ldp/cdc)** - AUTO CDC, SCD Type 1/2 + + +### Medallion Architecture + +| Layer | SDP Pattern | Common Practices | +|-------|-------------|------------------| +| **Bronze** | `STREAM read_files()` → streaming table | Often adds `_metadata.file_path`, `_ingested_at`. Minimal transforms, append-only. | +| **Silver** | `stream(bronze)` → streaming table | Clean/validate, type casting, quality filters. Prefer `DECIMAL(p,s)` for money. Dedup can happen here or gold. | +| **Gold** | `AUTO CDC INTO` or materialized view | Aggregated, denormalized. SCD/dedup often via `AUTO CDC`. Star schema typically uses `dim_*`/`fact_*`. | + +#### Gold Layer: Preserve Key Dimensions + +When aggregating data in gold tables, **keep the main business dimensions** to enable flexible analysis. Over-aggregating loses information that analysts may need later. + +**Guidance based on context:** +- **If a dashboard is mentioned**: Include all dimensions that appear as filters. Dashboard filters only work if the underlying data has those columns. +- **If analysis by dimension is mentioned** (e.g., "analyze by store", "breakdown by department"): Include those dimensions in the aggregation. +- **If no specific instructions**: Default to keeping key business dimensions (location, department, product line, customer segment, time period) rather than aggregating them away. This preserves flexibility for future analysis. + +**Rule of thumb**: If users might want to slice the data by a dimension, include it in the gold table. It's easier to aggregate further in queries than to recover lost dimensions. + +**For medallion architecture** (bronze/silver/gold), two approaches work: +- **Flat with naming** (template default): `bronze_*.sql`, `silver_*.sql`, `gold_*.sql` +- **Subdirectories**: `bronze/orders.sql`, `silver/cleaned.sql`, `gold/summary.sql` + +Both work with the `transformations/**` glob pattern. Choose based on preference/existing. + +See **[1-project-initialization.md](references/1-project-initialization.md)** for complete details on bundle initialization, migration, and troubleshooting. + +--- +## General SDP development guidance + +**SQL Example:** +```sql +CREATE OR REFRESH STREAMING TABLE bronze_orders +CLUSTER BY (order_date) +AS SELECT *, current_timestamp() AS _ingested_at +FROM STREAM read_files('/Volumes/catalog/schema/raw/orders/', format => 'json'); +``` + +**Python Example:** +```python +from pyspark import pipelines as dp + +@dp.table(name="bronze_events", cluster_by=["event_date"]) +def bronze_events(): + return spark.readStream.format("cloudFiles").option("cloudFiles.format", "json").load("/Volumes/...") +``` + +For detailed syntax, see [sql/1-syntax-basics.md](references/sql/1-syntax-basics.md) or [python/1-syntax-basics.md](references/python/1-syntax-basics.md). + +## Best Practices (2026) + +### Project Structure +- **Standalone pipeline projects**: Use `databricks pipelines init` for Asset Bundle with multi-environment support +- **Pipeline in existing bundle**: Add to `resources/*.pipeline.yml` +- **Rapid iteration/prototyping**: Use MCP tools, formalize in bundle later +- See **[1-project-initialization.md](references/1-project-initialization.md)** for project setup details + +### Minimal pipeline config pointers +- Define parameters in your pipeline’s configuration and access them in code with spark.conf.get("key"). +- In Databricks Asset Bundles, set these under resources.pipelines..configuration; validate with databricks bundle validate. + +### Modern Defaults +- **Always use raw `.sql`/`.py` files for the transformations files** - NO notebooks in your pipeline. Pipeline code must be plain files. +- **Databricks notebook source for explorations** - Use `# Databricks notebook source` format with `# COMMAND ----------` separators for ad-hoc queries. See [examples/exploration_notebook.py](scripts/exploration_notebook.py). +- **Serverless compute** - Do not use classic clusters unless explicitly required (R, RDD APIs, JAR libraries) +- **Unity Catalog** (required for serverless) +- **CLUSTER BY** (Liquid Clustering), not PARTITION BY with ZORDER - see [sql/5-performance.md](references/sql/5-performance.md) or [python/5-performance.md](references/python/5-performance.md) +- **read_files()** for SQL cloud storage ingestion - always consume a folder, not a single file - see [sql/2-ingestion.md](references/sql/2-ingestion.md) + +### Multi-Schema Patterns + +**Preferred: One pipeline writing to multiple schemas** using fully qualified table names (`catalog.schema.table`). This keeps dependencies clear and is simpler to manage than multiple pipelines. + +- **Python**: `@dp.table(name="catalog.bronze_schema.orders")` +- **SQL**: `CREATE OR REFRESH STREAMING TABLE catalog.silver_schema.orders_clean AS ...` + +For detailed examples, see **[3-advanced-configuration.md](references/3-advanced-configuration.md#multi-schema-patterns)**. + +**Fallback**: If all tables must be in the same schema, use name prefixes (`bronze_*`, `silver_*`, `gold_*`). + +--- + +## Post-Run Validation (Required) + +After running a pipeline (via DAB or MCP), you **MUST** validate both the execution status AND the actual data. + +### Step 1: Check Pipeline Execution Status + +**From MCP (`manage_pipeline(action="run")` or `manage_pipeline(action="create_or_update")`):** +- Check `result["success"]` and `result["state"]` +- If failed, check `result["message"]` and `result["errors"]` for details + +**From DAB (`databricks bundle run`):** +- Check the command output for success/failure +- Use `manage_pipeline(action="get", pipeline_id=...)` to get detailed status and recent events + +### Step 2: Validate Output Data + +Even if the pipeline reports SUCCESS, you **MUST** verify the data is correct: + +``` +# MCP Tool: get_table_stats_and_schema - validates schema, row counts, and stats +get_table_stats_and_schema( + catalog="my_catalog", + schema="my_schema", + table_names=["bronze_*", "silver_*", "gold_*"] # Use glob patterns +) +``` + +**Check for:** +- Empty tables (row_count = 0) - indicates ingestion or filtering issues +- Unexpected row counts - joins may have exploded or filtered too much +- Missing columns - schema mismatch or transformation errors +- NULL values in key columns - data quality issues + +### Step 3: Debug Data Issues + +If validation reveals problems, trace upstream to find the root cause: + +1. **Start from the problematic table** - identify what's wrong (empty, wrong counts, bad data) +2. **Check its source table** - use `get_table_stats_and_schema` on the upstream table +3. **Trace back to bronze** - continue until you find where the issue originates +4. **Common causes:** + - Bronze empty → source files missing or path incorrect + - Silver empty → filter too aggressive or join condition wrong + - Gold wrong counts → aggregation logic error or duplicate keys + - Data mismatch → type casting issues or NULL handling + +5. **Fix the SQL/Python code**, re-upload, and re-run the pipeline + +**Do NOT use `execute_sql` with COUNT queries for validation** - `get_table_stats_and_schema` is faster and returns more information in a single call. + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **Empty output tables** | Use `get_table_stats_and_schema` to check upstream sources. Verify source files exist and paths are correct. | +| **Pipeline stuck INITIALIZING** | Normal for serverless, wait a few minutes | +| **"Column not found"** | Check `schemaHints` match actual data | +| **Streaming reads fail** | For file ingestion in a streaming table, you must use the `STREAM` keyword with `read_files`: `FROM STREAM read_files(...)`. For table streams use `FROM stream(table)`. See [read_files — Usage in streaming tables](https://docs.databricks.com/aws/en/sql/language-manual/functions/read_files#usage-in-streaming-tables). | +| **Timeout during run** | Increase `timeout`, or use `wait_for_completion=False` and check status with `manage_pipeline(action="get")` | +| **MV doesn't refresh** | Enable row tracking on source tables | +| **SCD2: query column not found** | Lakeflow uses `__START_AT` and `__END_AT` (double underscore), not `START_AT`/`END_AT`. Use `WHERE __END_AT IS NULL` for current rows. See [sql/4-cdc-patterns.md](references/sql/4-cdc-patterns.md). | +| **AUTO CDC parse error at APPLY/SEQUENCE** | Put `APPLY AS DELETE WHEN` **before** `SEQUENCE BY`. Only list columns in `COLUMNS * EXCEPT (...)` that exist in the source (omit `_rescued_data` unless bronze uses rescue data). Omit `TRACK HISTORY ON *` if it causes "end of input" errors; default is equivalent. See [sql/4-cdc-patterns.md](references/sql/4-cdc-patterns.md). | +| **"Cannot create streaming table from batch query"** | In a streaming table query, use `FROM STREAM read_files(...)` so `read_files` leverages Auto Loader; `FROM read_files(...)` alone is batch. See [sql/2-ingestion.md](references/sql/2-ingestion.md) and [read_files — Usage in streaming tables](https://docs.databricks.com/aws/en/sql/language-manual/functions/read_files#usage-in-streaming-tables). | + +**For detailed errors**, the `result["message"]` from `manage_pipeline(action="create_or_update")` includes suggested next steps. Use `manage_pipeline(action="get", pipeline_id=...)` which includes recent events and error details. + +--- + +## Advanced Pipeline Configuration + +For advanced configuration options (development mode, continuous pipelines, custom clusters, notifications, Python dependencies, etc.), see **[3-advanced-configuration.md](references/3-advanced-configuration.md)**. + +--- + +## Platform Constraints + +### Serverless Pipeline Requirements (Default) +| Requirement | Details | +|-------------|---------| +| **Unity Catalog** | Required - serverless pipelines always use UC | +| **Workspace Region** | Must be in serverless-enabled region | +| **Serverless Terms** | Must accept serverless terms of use | +| **CDC Features** | Requires serverless (or Pro/Advanced with classic clusters) | + +### Serverless Limitations (When Classic Clusters Required) +| Limitation | Workaround | +|------------|-----------| +| **R language** | Not supported - use classic clusters if required | +| **Spark RDD APIs** | Not supported - use classic clusters if required | +| **JAR libraries** | Not supported - use classic clusters if required | +| **Maven coordinates** | Not supported - use classic clusters if required | +| **DBFS root access** | Limited - must use Unity Catalog external locations | +| **Global temp views** | Not supported | + +### General Constraints +| Constraint | Details | +|------------|---------| +| **Schema Evolution** | Streaming tables require full refresh for incompatible changes | +| **SQL Limitations** | PIVOT clause unsupported | +| **Sinks** | Python only, streaming only, append flows only | + +**Default to serverless** unless user explicitly requires R, RDD APIs, or JAR libraries. + +## Related Skills + +- **[databricks-jobs](../databricks-jobs/SKILL.md)** - for orchestrating and scheduling pipeline runs +- **[databricks-bundles](../databricks-bundles/SKILL.md)** - for multi-environment deployment of pipeline projects +- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** - for generating test data to feed into pipelines +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - for catalog/schema/volume management and governance diff --git a/.claude/skills/spark-declarative-pipelines/8-project-initialization.md b/.claude/skills/databricks-spark-declarative-pipelines/references/1-project-initialization.md similarity index 55% rename from .claude/skills/spark-declarative-pipelines/8-project-initialization.md rename to .claude/skills/databricks-spark-declarative-pipelines/references/1-project-initialization.md index 44850f67..fbab69b3 100644 --- a/.claude/skills/spark-declarative-pipelines/8-project-initialization.md +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/1-project-initialization.md @@ -1,24 +1,21 @@ -# Project Initialization with databricks pipelines init +# Project Initialization -## Overview +Two approaches for creating SDP pipelines with Declarative Automation Bundles (DABs): +- **Option A**: Standalone new project using `databricks pipelines init` +- **Option B**: Adding a pipeline to an existing bundle -The `databricks pipelines init` command scaffolds a complete Databricks Asset Bundle project for Lakeflow Spark Declarative Pipelines, providing a production-ready structure with multi-environment support, pipeline configuration, and sample transformation files. +--- -**Benefits of Asset Bundles:** -- Multi-environment deployments (dev/staging/prod) -- Infrastructure as code with `databricks.yml` -- Built-in CI/CD integration -- Version control for pipeline configuration -- Automated deployment workflows +## Option A: Standalone New Pipeline Project ---- +Use `databricks pipelines init` to scaffold a complete DAB project with multi-environment support, pipeline configuration, and sample transformation files. -## Command Reference +### Command Reference ### Interactive Mode ```bash -databricks pipelines init --output-dir ./my_pipeline +databricks pipelines init --output-dir . ``` **Interactive Prompts:** @@ -43,7 +40,7 @@ databricks pipelines init --output-dir ./my_pipeline ```bash databricks pipelines init \ - --output-dir ./customer_pipeline \ + --output-dir . \ --config-file init-config.json ``` @@ -70,7 +67,7 @@ databricks pipelines init \ ### SQL Project ``` -customer_pipeline/ +project_root/ ├── databricks.yml # Bundle configuration ├── resources/ │ ├── customer_pipeline_etl.pipeline.yml # Pipeline resource definition @@ -89,7 +86,7 @@ customer_pipeline/ ### Python Project ``` -customer_pipeline/ +project_root/ ├── databricks.yml # Bundle configuration ├── pyproject.toml # Python dependencies ├── resources/ @@ -241,173 +238,84 @@ databricks pipelines start-update --pipeline-id --- -## Language Detection (for Claude) +## Medallion Architecture -When a user requests a new Lakeflow pipeline, Claude should detect the appropriate language from keywords in the prompt. +For bronze/silver/gold organization, two file structure approaches work with Declarative Automation Bundles (DABs): -### SQL Indicators (Default Choice) +### Option 1: Flat Structure with Prefixes (Recommended) -**Keywords:** -- "SQL", "sql files", ".sql" -- "simple", "basic", "straightforward" -- "aggregations", "joins", "transformations" -- "materialized view", "CREATE OR REFRESH" -- "SELECT", "GROUP BY", "WHERE" +``` +transformations/ +├── bronze_orders.sql +├── bronze_events.sql +├── silver_orders.sql +├── silver_events.sql +├── gold_daily_metrics.sql +└── gold_summary.sql +``` -**Context:** -- User mentions only data transformations without complex logic -- Request focuses on filtering, joining, aggregating data -- No mention of custom functions or external integrations +### Option 2: Subdirectories by Layer -**Default Behavior**: Prefer SQL when ambiguous (covers 90% of use cases) +``` +transformations/ +├── bronze/ +│ └── orders.sql +├── silver/ +│ └── orders.sql +└── gold/ + └── daily_metrics.sql +``` -### Python Indicators +Both work with `transformations/**` glob pattern. Choose based on team preference. -**Keywords:** -- "Python", "python files", ".py", "@dp.table" -- "UDF", "user-defined function", "custom function" -- "complex logic", "complex transformations" -- "ML", "machine learning", "inference", "model" -- "API", "external API", "REST", "HTTP" -- "pandas", "numpy", "pyspark" -- "decorator", "pyspark.pipelines" +For syntax examples, see: +- **[sql/1-syntax-basics.md](sql/1-syntax-basics.md)** - SQL table definitions +- **[python/1-syntax-basics.md](python/1-syntax-basics.md)** - Python decorators +- **[sql/2-ingestion.md](sql/2-ingestion.md)** - Bronze layer ingestion patterns -**Context:** -- User needs custom data processing beyond SQL capabilities -- Request mentions integrating with external services -- Task requires ML model inference or scoring -- Dynamic schema or path generation needed +--- -### Ambiguous Cases (Ask User) +## Option B: Adding a Pipeline to an Existing Bundle -**Indicators:** -- Both SQL and Python keywords mentioned -- "mixed pipeline", "some Python, some SQL" -- "complex pipeline" without specifics -- Unclear requirements or vague description +If you already have a `databricks.yml` for a larger project (e.g., an app with jobs, dashboards, etc.) and want to add a pipeline: -**Response:** -``` -I can create this pipeline using either SQL or Python: +### Step 1: Create Pipeline Resource File -- **SQL**: Best for transformations, aggregations, joins (simpler, faster to develop) -- **Python**: Best for custom logic, UDFs, ML inference, external APIs +Create `resources/my_pipeline.pipeline.yml`: -Which would you prefer? +```yaml +resources: + pipelines: + my_pipeline: + name: my_pipeline + catalog: ${var.catalog} + schema: ${var.schema} + serverless: true + libraries: + - file: + path: ../src/pipelines/my_pipeline/ ``` ---- - -## Medallion Architecture - -For bronze/silver/gold organization, Asset Bundles support two approaches. Both work with the `transformations/**` glob pattern in pipeline configuration. +### Step 2: Add Pipeline Source Files -### Option 1: Flat Structure with Naming (Template Default) +Create your pipeline transformation files: ``` -transformations/ -├── bronze_raw_orders.sql # Raw data ingestion -├── bronze_raw_events.sql -├── bronze_raw_customers.sql -├── silver_cleaned_orders.sql # Cleaned and validated -├── silver_joined_data.sql -├── silver_customer_profiles.sql -├── gold_daily_metrics.sql # Business aggregations -├── gold_customer_summary.sql -└── gold_revenue_analysis.sql -``` - -**Advantages:** -- Matches the official `databricks pipelines init` template structure -- All files visible at one level -- Simple file listing and discovery -- Clear naming provides logical organization +src/pipelines/my_pipeline/ +├── bronze_ingest.sql +├── silver_clean.sql +└── gold_summary.sql +``` -### Option 2: Subdirectories by Layer +### Step 3: Deploy -``` -transformations/ -├── bronze/ -│ ├── raw_orders.sql -│ ├── raw_events.sql -│ └── raw_customers.sql -├── silver/ -│ ├── cleaned_orders.sql -│ ├── joined_data.sql -│ └── customer_profiles.sql -└── gold/ - ├── daily_metrics.sql - ├── customer_summary.sql - └── revenue_analysis.sql -``` - -**Advantages:** -- Physical separation of layers -- Familiar structure for teams using manual workflow -- Easier to navigate large projects with many files -- Works with `transformations/**` glob pattern - -**Both approaches are technically valid** - the `**` in the glob pattern matches files recursively. Choose based on team preference and project size. - -### Example Bronze Layer (SQL) - -```sql --- File: bronze_raw_orders.sql -CREATE OR REFRESH STREAMING TABLE bronze_raw_orders -CLUSTER BY (order_date) -COMMENT "Raw order data ingested from cloud storage" -AS -SELECT - *, - current_timestamp() AS _ingested_at, - _metadata.file_path AS _source_file -FROM read_files( - '/Volumes/main/raw_data/orders/', - format => 'json', - schemaHints => 'order_id STRING, customer_id STRING, amount DECIMAL(10,2), order_date DATE' -); -``` - -### Example Silver Layer (SQL) - -```sql --- File: silver_cleaned_orders.sql -CREATE OR REFRESH MATERIALIZED VIEW silver_cleaned_orders -CLUSTER BY (order_date) -COMMENT "Cleaned and validated orders with customer enrichment" -AS -SELECT - o.order_id, - o.customer_id, - o.amount, - o.order_date, - c.customer_name, - c.customer_segment -FROM LIVE.bronze_raw_orders o -INNER JOIN LIVE.bronze_raw_customers c - ON o.customer_id = c.customer_id -WHERE o.amount > 0 -- Remove invalid orders - AND o.order_date >= '2020-01-01'; -``` - -### Example Gold Layer (SQL) - -```sql --- File: gold_daily_metrics.sql -CREATE OR REFRESH MATERIALIZED VIEW gold_daily_metrics -CLUSTER BY (metric_date) -COMMENT "Daily business metrics for reporting" -AS -SELECT - order_date AS metric_date, - COUNT(DISTINCT customer_id) AS unique_customers, - COUNT(*) AS total_orders, - SUM(amount) AS total_revenue, - AVG(amount) AS avg_order_value -FROM LIVE.silver_cleaned_orders -GROUP BY order_date; +```bash +databricks bundle deploy +databricks bundle run my_pipeline ``` +That's it - the pipeline is now part of your existing bundle and shares the same targets/variables. + --- ## Migration from Manual Structure @@ -429,73 +337,6 @@ my_pipeline/ └── summary.sql ``` -**Migration Steps:** - -1. **Initialize new bundle project** - ```bash - databricks pipelines init --output-dir my_pipeline_bundle - cd my_pipeline_bundle/src/my_pipeline_bundle_etl/transformations/ - ``` - -2. **Copy files using either approach** - - **Option A: Flat structure with naming** - ```bash - # Remove sample files - rm sample_*.sql - - # Copy and rename with medallion prefix - cp ../../../../my_pipeline/bronze/orders.sql bronze_orders.sql - cp ../../../../my_pipeline/bronze/events.sql bronze_events.sql - cp ../../../../my_pipeline/silver/cleaned.sql silver_cleaned.sql - cp ../../../../my_pipeline/silver/joined.sql silver_joined.sql - cp ../../../../my_pipeline/gold/summary.sql gold_summary.sql - ``` - - **Option B: Keep subdirectories** - ```bash - # Remove sample files - rm sample_*.sql - - # Copy entire directory structure - cp -r ../../../../my_pipeline/bronze . - cp -r ../../../../my_pipeline/silver . - cp -r ../../../../my_pipeline/gold . - ``` - -3. **Update file references (if needed)** - - If files reference each other by path, update to use table names - - Example: Change `FROM ../bronze/orders` to `FROM LIVE.bronze_orders` - - Table names are derived from filenames or view/table definitions, not folder structure - -4. **Deploy bundle** - ```bash - cd ../../.. # Back to bundle root - databricks bundle deploy - databricks bundle run my_pipeline_bundle_etl - ``` - -**Benefits:** -- Multi-environment support (dev/staging/prod) -- Version control for configuration -- CI/CD integration -- Professional project structure - -### Option 2: Keep Manual Structure (Legacy) - -Continue using the manual workflow if: -- Quick prototyping without multi-environment needs -- Existing workflow is working well -- Team prefers manual control - -**Legacy Workflow:** -1. Write files in bronze/silver/gold folders -2. Upload with `upload_folder` MCP tool -3. Create pipeline with `create_or_update_pipeline` MCP tool -4. Update files and re-upload - -See [SKILL.md](SKILL.md) "Alternative: Manual Workflow" section for details. - --- ## Python Project: Dependency Management @@ -664,7 +505,7 @@ For advanced pipeline configuration options beyond the bundle initialization: - **Custom notifications**: Email or webhook alerts - **Non-serverless clusters**: When serverless limitations apply -See [7-advanced-configuration.md](7-advanced-configuration.md) for detailed examples. +See [3-advanced-configuration.md](3-advanced-configuration.md) for detailed examples. --- @@ -723,45 +564,22 @@ resources: ## Best Practices -### Project Organization - -1. **Use descriptive file names**: `bronze_orders_raw.sql` not just `orders.sql` -2. **Choose structure approach**: - - **Flat with prefixes**: `bronze_*`, `silver_*`, `gold_*` (template default) - - **Subdirectories**: `bronze/`, `silver/`, `gold/` folders (also valid) - - Both work with `transformations/**` glob pattern -3. **One table per file**: Each file defines a single table or view -4. **Be consistent**: Pick one approach and use it throughout the project - -### Configuration Management - -1. **Use variables**: Parameterize catalog and schema names -2. **Separate environments**: Define dev/staging/prod targets -3. **Version control**: Track `databricks.yml` and pipeline configs in git -4. **Sensitive data**: Use secrets, not hardcoded values - -### Development Workflow - -1. **Start with dev**: Always test in development environment first -2. **Validate locally**: Run `databricks bundle validate` before deploy -3. **Incremental changes**: Deploy and test small changes frequently -4. **Use explorations**: Ad-hoc notebooks for data exploration - -### Deployment Strategy +1. **One table per file** - Each `.sql` or `.py` file defines a single table/view +2. **Use variables** - Parameterize catalog and schema names for environment portability +3. **Sensitive data** - Use secrets (`{{secrets/scope/key}}`), not hardcoded values +4. **Test in dev first** - Run `databricks bundle validate` before deploy +5. **Version control** - Track `databricks.yml` and pipeline configs in git -1. **CI/CD integration**: Automate deployments with GitHub Actions, GitLab CI -2. **Approval gates**: Require approval for production deployments -3. **Rollback plan**: Keep previous bundle versions for quick rollback -4. **Monitor pipelines**: Set up notifications for failures +For technical best practices (Liquid Clustering, serverless, etc.), see **[SKILL.md](SKILL.md#best-practices-2026)**. --- ## References -- **[SKILL.md](SKILL.md)** - Main development workflow and MCP tools -- **[Databricks Asset Bundles Documentation](https://docs.databricks.com/dev-tools/bundles/)** - Official bundle reference +- **[SKILL.md](../SKILL.md)** - Main development workflow and MCP tools +- **[Declarative Automation Bundles (DABs) Documentation](https://docs.databricks.com/dev-tools/bundles/)** - Official bundle reference - **[Pipeline Configuration Reference](https://docs.databricks.com/aws/en/ldp/configure-pipeline)** - Pipeline settings - **[Databricks CLI Reference](https://docs.databricks.com/dev-tools/cli/)** - CLI commands and options -- **[1-ingestion-patterns.md](1-ingestion-patterns.md)** - Data ingestion patterns -- **[2-streaming-patterns.md](2-streaming-patterns.md)** - Streaming transformations -- **[7-advanced-configuration.md](7-advanced-configuration.md)** - Advanced pipeline settings +- **[sql/2-ingestion.md](sql/2-ingestion.md)** or **[python/2-ingestion.md](python/2-ingestion.md)** - Data ingestion patterns +- **[sql/3-streaming-patterns.md](sql/3-streaming-patterns.md)** or **[python/3-streaming-patterns.md](python/3-streaming-patterns.md)** - Streaming transformations +- **[3-advanced-configuration.md](3-advanced-configuration.md)** - Advanced pipeline settings diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/2-mcp-approach.md b/.claude/skills/databricks-spark-declarative-pipelines/references/2-mcp-approach.md new file mode 100644 index 00000000..87e0ed70 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/2-mcp-approach.md @@ -0,0 +1,163 @@ +Use MCP tools to create, run, and iterate on **SDP pipelines**. The **primary tool is `manage_pipeline`** which handles the entire lifecycle. + +**IMPORTANT: Default to serverless pipelines.** Only use classic clusters if user explicitly requires R language, Spark RDD APIs, or JAR libraries. + +### Step 1: Write Pipeline Files Locally + +Create `.sql` or `.py` files in a local folder. For syntax examples, see: +- [sql/1-syntax-basics.md](sql/1-syntax-basics.md) for SQL syntax +- [python/1-syntax-basics.md](python/1-syntax-basics.md) for Python syntax + +### Step 2: Upload to Databricks Workspace + +``` +# MCP Tool: manage_workspace_files +manage_workspace_files( + action="upload", + local_path="/path/to/my_pipeline", + workspace_path="/Workspace/Users/user@example.com/my_pipeline" +) +``` + +### Step 3: Create/Update and Run Pipeline + +Use **`manage_pipeline`** with `action="create_or_update"` to manage the resource: + +``` +# MCP Tool: manage_pipeline +manage_pipeline( + action="create_or_update", + name="my_orders_pipeline", + root_path="/Workspace/Users/user@example.com/my_pipeline", + catalog="my_catalog", + schema="my_schema", + workspace_file_paths=[ + "/Workspace/Users/user@example.com/my_pipeline/bronze/ingest_orders.sql", + "/Workspace/Users/user@example.com/my_pipeline/silver/clean_orders.sql", + "/Workspace/Users/user@example.com/my_pipeline/gold/daily_summary.sql" + ], + start_run=True, # Automatically run after create/update + wait_for_completion=True, # Wait for run to finish + full_refresh=True # Reprocess all data +) +``` + +**Result contains actionable information:** +```json +{ + "success": true, + "pipeline_id": "abc-123", + "pipeline_name": "my_orders_pipeline", + "created": true, + "state": "COMPLETED", + "catalog": "my_catalog", + "schema": "my_schema", + "duration_seconds": 45.2, + "message": "Pipeline created and completed successfully in 45.2s. Tables written to my_catalog.my_schema", + "error_message": null, + "errors": [] +} +``` + +### Alternative: Run Pipeline Separately + +If you want to run an existing pipeline or control the run separately: + +``` +# MCP Tool: manage_pipeline_run +manage_pipeline_run( + action="start", + pipeline_id="", + full_refresh=True, + wait=True, # Wait for completion + timeout=1800 # 30 minute timeout +) +``` + +### Step 4: Validate Results + +**On Success** - Use `get_table_stats_and_schema` to verify tables (NOT manual SQL COUNT queries): +``` +# MCP Tool: get_table_stats_and_schema +get_table_stats_and_schema( + catalog="my_catalog", + schema="my_schema", + table_names=["bronze_orders", "silver_orders", "gold_daily_summary"] +) +# Returns schema, row counts, and column stats for all tables in one call +``` + +**On Failure** - Check `run_result["message"]` for suggested next steps, then get detailed errors: +``` +# MCP Tool: manage_pipeline +manage_pipeline(action="get", pipeline_id="") +# Returns pipeline details enriched with recent events and error messages + +# Or get events/logs directly: +# MCP Tool: manage_pipeline_run +manage_pipeline_run( + action="get_events", + pipeline_id="", + event_log_level="ERROR", # ERROR, WARN, or INFO + max_results=10 +) +``` + +### Step 5: Iterate Until Working + +1. Review errors from run result or `manage_pipeline(action="get")` +2. Fix issues in local files +3. Re-upload with `manage_workspace_files(action="upload")` +4. Run `manage_pipeline(action="create_or_update", start_run=True)` again (it will update, not recreate) +5. Repeat until `result["success"] == True` + +--- + +## Quick Reference: MCP Tools + +### manage_pipeline - Pipeline Lifecycle + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create` | Create new pipeline | name, root_path, catalog, schema, workspace_file_paths | +| `create_or_update` | **Main entry point.** Idempotent create/update, optionally run | name, root_path, catalog, schema, workspace_file_paths | +| `get` | Get pipeline details by ID | pipeline_id | +| `update` | Update pipeline config | pipeline_id + fields to change | +| `delete` | Delete a pipeline | pipeline_id | +| `find_by_name` | Find pipeline by name | name | + +**create_or_update options:** +- `start_run=True`: Automatically run after create/update +- `wait_for_completion=True`: Block until run finishes +- `full_refresh=True`: Reprocess all data (default) +- `timeout=1800`: Max wait time in seconds + +### manage_pipeline_run - Run Management + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `start` | Start pipeline update | pipeline_id | +| `get` | Get run status | pipeline_id, update_id | +| `stop` | Stop running pipeline | pipeline_id | +| `get_events` | Get events/logs for debugging | pipeline_id | + +**start options:** +- `wait=True`: Block until complete (default) +- `full_refresh=True`: Reprocess all data +- `validate_only=True`: Dry run without writing data +- `refresh_selection=["table1", "table2"]`: Refresh specific tables only + +**get_events options:** +- `event_log_level`: "ERROR", "WARN" (default), "INFO" +- `max_results`: Number of events (default 5) +- `update_id`: Filter to specific run + +### Supporting Tools + +| Tool | Description | +|------|-------------| +| `manage_workspace_files(action="upload")` | Upload files/folders to workspace | +| `get_table_stats_and_schema` | **Use this to validate tables** - returns schema, row counts, and stats in one call | +| `execute_sql` | Run ad-hoc SQL to inspect actual data content (not for row counts) | + +--- diff --git a/.claude/skills/spark-declarative-pipelines/7-advanced-configuration.md b/.claude/skills/databricks-spark-declarative-pipelines/references/3-advanced-configuration.md similarity index 76% rename from .claude/skills/spark-declarative-pipelines/7-advanced-configuration.md rename to .claude/skills/databricks-spark-declarative-pipelines/references/3-advanced-configuration.md index a6c8ecf3..b637f469 100644 --- a/.claude/skills/spark-declarative-pipelines/7-advanced-configuration.md +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/3-advanced-configuration.md @@ -142,7 +142,7 @@ Install pip dependencies for serverless pipelines: | Field | Type | Description | |-------|------|-------------| -| `kind` | str | `"BUNDLE"` (Databricks Asset Bundles) or `"DEFAULT"` | +| `kind` | str | `"BUNDLE"` (DABs) or `"DEFAULT"` | | `metadata_file_path` | str | Path to deployment metadata file | ### Edition Comparison @@ -159,7 +159,7 @@ Install pip dependencies for serverless pipelines: ### Development Mode Pipeline -Use `create_or_update_pipeline` tool with: +Use `manage_pipeline(action="create_or_update")` tool with: - `name`: "my_dev_pipeline" - `root_path`: "/Workspace/Users/user@example.com/my_pipeline" - `catalog`: "dev_catalog" @@ -176,7 +176,7 @@ Use `create_or_update_pipeline` tool with: ### Non-Serverless with Dedicated Cluster -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "serverless": false, @@ -193,7 +193,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Continuous Streaming Pipeline -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "continuous": true, @@ -205,7 +205,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Using Instance Pool -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "serverless": false, @@ -220,7 +220,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Custom Event Log Location -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "event_log": { @@ -233,7 +233,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Pipeline with Email Notifications -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "notifications": [{ @@ -245,7 +245,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Production Pipeline with Autoscaling -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "serverless": false, @@ -274,7 +274,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Run as Service Principal -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "run_as": { @@ -285,7 +285,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Continuous Pipeline with Restart Window -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "continuous": true, @@ -299,7 +299,7 @@ Use `create_or_update_pipeline` tool with `extra_settings`: ### Serverless with Python Dependencies -Use `create_or_update_pipeline` tool with `extra_settings`: +Use `manage_pipeline(action="create_or_update")` tool with `extra_settings`: ```json { "serverless": true, @@ -348,3 +348,77 @@ You can copy pipeline settings from the Databricks UI (Pipeline Settings > JSON) ``` **Note**: Explicit tool parameters (`name`, `root_path`, `catalog`, `schema`, `workspace_file_paths`) always take precedence over values in `extra_settings`. + +--- + +## Multi-Schema Patterns + +**Recommended: One pipeline writing to multiple schemas** using fully qualified table names. This is simpler than creating multiple pipelines and keeps all dependencies in one place. + +For simple cases where all tables go to the same schema, use name prefixes (`bronze_*`, `silver_*`, `gold_*`). + +### Option 1: Same Catalog, Separate Schemas + +Set pipeline defaults to bronze, use parameters for silver/gold: + +```python +from pyspark import pipelines as dp +from pyspark.sql.functions import col + +# Pull variables from pipeline configuration +silver_schema = spark.conf.get("silver_schema") # e.g., "silver" +gold_schema = spark.conf.get("gold_schema") # e.g., "gold" +landing_schema = spark.conf.get("landing_schema") # e.g., "landing" + +# Bronze → uses default catalog/schema (set to bronze in pipeline settings) +@dp.table(name="orders_bronze") +def orders_bronze(): + return spark.readStream.table(f"{landing_schema}.orders_raw") + +# Silver → same catalog, schema from parameter +@dp.table(name=f"{silver_schema}.orders_clean") +def orders_clean(): + return spark.read.table("orders_bronze").filter(col("order_id").isNotNull()) + +# Gold → same catalog, schema from parameter +@dp.materialized_view(name=f"{gold_schema}.orders_by_date") +def orders_by_date(): + return (spark.read.table(f"{silver_schema}.orders_clean") + .groupBy("order_date").count()) +``` + +### Option 2: Custom Catalog/Schema Per Layer + +For cross-catalog scenarios: + +```python +from pyspark import pipelines as dp +from pyspark.sql.functions import col + +# Pull variables from pipeline configuration +silver_catalog = spark.conf.get("silver_catalog") +silver_schema = spark.conf.get("silver_schema") +gold_catalog = spark.conf.get("gold_catalog") +gold_schema = spark.conf.get("gold_schema") + +# Bronze → uses pipeline defaults +@dp.table(name="orders_bronze") +def orders_bronze(): + return spark.readStream.format("cloudFiles").load("/Volumes/...") + +# Silver → custom catalog + schema +@dp.table(name=f"{silver_catalog}.{silver_schema}.orders_clean") +def orders_clean(): + return spark.read.table("orders_bronze").filter(col("order_id").isNotNull()) + +# Gold → custom catalog + schema +@dp.materialized_view(name=f"{gold_catalog}.{gold_schema}.orders_by_date") +def orders_by_date(): + return (spark.read.table(f"{silver_catalog}.{silver_schema}.orders_clean") + .groupBy("order_date").count()) +``` + +**Key points:** +- Multipart names in `@dp.table(name=...)` let you publish to explicit catalog.schema targets +- Unqualified names use pipeline defaults +- Use fully-qualified names when crossing catalogs diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/4-dlt-migration.md b/.claude/skills/databricks-spark-declarative-pipelines/references/4-dlt-migration.md new file mode 100644 index 00000000..dbde0d9c --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/4-dlt-migration.md @@ -0,0 +1,447 @@ +# Migration Guide: DLT to SDP + +Guide for migrating from Delta Live Tables (DLT) to Spark Declarative Pipelines (SDP). + +**Two migration paths:** +1. **DLT Python → SDP Python** (dlt → dp): Same language, new API +2. **DLT Python → SDP SQL**: Change language for simpler pipelines + +--- + +## Migration Path 1: DLT Python → SDP Python (dlt → dp) + +Use this when staying with Python but moving to the modern `pyspark.pipelines` API. + +### Quick Reference + +| Aspect | Legacy (`dlt`) | Modern (`dp`) | +|--------|---------------|----------------| +| **Import** | `import dlt` | `from pyspark import pipelines as dp` | +| **Table decorator** | `@dlt.table()` | `@dp.table()` | +| **Read table** | `dlt.read("table")` | `spark.read.table("table")` | +| **Read stream** | `dlt.read_stream("table")` | `spark.readStream.table("table")` | +| **CDC/SCD** | `dlt.apply_changes()` | `dp.create_auto_cdc_flow()` | +| **Clustering** | `partition_cols=["date"]` | `cluster_by=["date", "col2"]` | + +### Step-by-Step Migration + +#### Step 1: Update Imports + +```python +# Before +import dlt + +# After +from pyspark import pipelines as dp +``` + +#### Step 2: Update Decorators + +```python +# Before +@dlt.table(name="my_table") + +# After +@dp.table(name="my_table") +``` + +#### Step 3: Update Table Reads + +```python +# Before +@dlt.table(name="silver_events") +def silver_events(): + return dlt.read("bronze_events").filter(...) + +# After +@dp.table(name="silver_events") +def silver_events(): + return spark.read.table("bronze_events").filter(...) +``` + +```python +# Before (streaming) +@dlt.table(name="silver_events") +def silver_events(): + return dlt.read_stream("bronze_events").filter(...) + +# After (streaming) +@dp.table(name="silver_events") +def silver_events(): + return spark.readStream.table("bronze_events").filter(...) +``` + +#### Step 4: Update Expectations + +```python +# Before +@dlt.table(name="silver") +@dlt.expect_or_drop("valid_id", "id IS NOT NULL") + +# After (identical syntax, just change dlt → dp) +@dp.table(name="silver") +@dp.expect_or_drop("valid_id", "id IS NOT NULL") +``` + +#### Step 5: Update CDC/SCD Operations + +```python +# Before +dlt.create_streaming_table("customers_history") +dlt.apply_changes( + target="customers_history", + source="customers_cdc", + keys=["customer_id"], + sequence_by="event_timestamp", + stored_as_scd_type="2" +) + +# After +from pyspark.sql.functions import col + +dp.create_streaming_table("customers_history") +dp.create_auto_cdc_flow( + target="customers_history", + source="customers_cdc", + keys=["customer_id"], + sequence_by=col("event_timestamp"), # Note: use col() + stored_as_scd_type=2 # Note: integer, not string +) +``` + +**Key differences:** +- `apply_changes()` → `create_auto_cdc_flow()` +- `sequence_by` takes a Column object (`col("...")`) not a string +- `stored_as_scd_type` is integer `2` for Type 2, string `"1"` for Type 1 + +#### Step 6: Update Clustering (Partitioning → Liquid Clustering) + +```python +# Before (legacy partitioning) +@dlt.table( + name="bronze_events", + partition_cols=["event_date"], + table_properties={"pipelines.autoOptimize.zOrderCols": "event_type"} +) + +# After (Liquid Clustering) +@dp.table( + name="bronze_events", + cluster_by=["event_date", "event_type"] +) +``` + +### Complete Before/After Example + +**Before (DLT):** +```python +import dlt +from pyspark.sql import functions as F + +@dlt.table(name="bronze_orders", partition_cols=["order_date"]) +def bronze_orders(): + return spark.readStream.format("cloudFiles").load("/data/orders") + +@dlt.table(name="silver_orders") +@dlt.expect_or_drop("valid_amount", "amount > 0") +def silver_orders(): + return dlt.read_stream("bronze_orders").filter(F.col("status") == "completed") + +dlt.create_streaming_table("dim_customers") +dlt.apply_changes( + target="dim_customers", + source="customers_cdc", + keys=["customer_id"], + sequence_by="updated_at", + stored_as_scd_type="2" +) +``` + +**After (SDP):** +```python +from pyspark import pipelines as dp +from pyspark.sql import functions as F + +@dp.table(name="bronze_orders", cluster_by=["order_date"]) +def bronze_orders(): + return spark.readStream.format("cloudFiles").load("/data/orders") + +@dp.table(name="silver_orders") +@dp.expect_or_drop("valid_amount", "amount > 0") +def silver_orders(): + return spark.readStream.table("bronze_orders").filter(F.col("status") == "completed") + +dp.create_streaming_table("dim_customers") +dp.create_auto_cdc_flow( + target="dim_customers", + source="customers_cdc", + keys=["customer_id"], + sequence_by=F.col("updated_at"), + stored_as_scd_type=2 +) +``` + +--- + +## Migration Path 2: DLT Python → SDP SQL + +Use this when simplifying pipelines by converting to SQL. + +### Decision Matrix + +| Feature/Pattern | DLT Python | SDP SQL | Recommendation | +|-----------------|------------|---------|----------------| +| Simple transformations | ✓ | ✓ | **Migrate to SQL** | +| Aggregations | ✓ | ✓ | **Migrate to SQL** | +| Filtering, WHERE clauses | ✓ | ✓ | **Migrate to SQL** | +| CASE expressions | ✓ | ✓ | **Migrate to SQL** | +| SCD Type 1/2 | ✓ | ✓ | **Migrate to SQL** (AUTO CDC) | +| Simple joins | ✓ | ✓ | **Migrate to SQL** | +| Auto Loader | ✓ | ✓ | **Migrate to SQL** (read_files) | +| Streaming sources (Kafka) | ✓ | ✓ | **Migrate to SQL** (read_kafka) | +| Complex Python UDFs | ✓ | ❌ | **Stay in Python** | +| External API calls | ✓ | ❌ | **Stay in Python** | +| Custom libraries | ✓ | ❌ | **Stay in Python** | +| ML model inference | ✓ | ❌ | **Stay in Python** | + +**Rule**: If 80%+ is SQL-expressible, migrate to SDP SQL. If heavy Python logic, stay with Python (use modern `dp` API). + +### Side-by-Side Conversions + +#### Basic Streaming Table + +**DLT Python:** +```python +@dlt.table(name="bronze_sales", comment="Raw sales") +def bronze_sales(): + return ( + spark.readStream.format("cloudFiles") + .option("cloudFiles.format", "json") + .load("/Volumes/my_catalog/my_schema/raw/sales") + .withColumn("_ingested_at", F.current_timestamp()) + ) +``` + +**SDP SQL:** +```sql +CREATE OR REFRESH STREAMING TABLE bronze_sales +COMMENT 'Raw sales' +AS +SELECT *, current_timestamp() AS _ingested_at +FROM STREAM read_files('/Volumes/my_catalog/my_schema/raw/sales', format => 'json'); +``` + +#### Filtering and Transformations + +**DLT Python:** +```python +@dlt.table(name="silver_sales") +@dlt.expect_or_drop("valid_amount", "amount > 0") +@dlt.expect_or_drop("valid_sale_id", "sale_id IS NOT NULL") +def silver_sales(): + return ( + dlt.read_stream("bronze_sales") + .withColumn("sale_date", F.to_date("sale_date")) + .withColumn("amount", F.col("amount").cast("decimal(10,2)")) + .select("sale_id", "customer_id", "amount", "sale_date") + ) +``` + +**SDP SQL:** +```sql +CREATE OR REFRESH STREAMING TABLE silver_sales AS +SELECT + sale_id, customer_id, + CAST(amount AS DECIMAL(10,2)) AS amount, + CAST(sale_date AS DATE) AS sale_date +FROM STREAM bronze_sales +WHERE amount > 0 AND sale_id IS NOT NULL; +``` + +#### SCD Type 2 + +**DLT Python:** +```python +dlt.create_streaming_table("customers_history") + +dlt.apply_changes( + target="customers_history", + source="customers_cdc_clean", + keys=["customer_id"], + sequence_by="event_timestamp", + stored_as_scd_type="2", + track_history_column_list=["*"] +) +``` + +**SDP SQL:** +```sql +CREATE OR REFRESH STREAMING TABLE customers_history; + +CREATE FLOW customers_scd2_flow AS +AUTO CDC INTO customers_history +FROM stream(customers_cdc_clean) +KEYS (customer_id) +APPLY AS DELETE WHEN operation = "DELETE" +SEQUENCE BY event_timestamp +COLUMNS * EXCEPT (operation, _ingested_at, _source_file) +STORED AS SCD TYPE 2; +``` + +**Note:** In SQL, put `APPLY AS DELETE WHEN` before `SEQUENCE BY`. Only list columns in `COLUMNS * EXCEPT (...)` that exist in the source. + +#### Joins + +**DLT Python:** +```python +@dlt.table(name="silver_sales_enriched") +def silver_sales_enriched(): + sales = dlt.read_stream("silver_sales") + products = dlt.read("dim_products") + return sales.join(products, "product_id", "left") +``` + +**SDP SQL:** +```sql +CREATE OR REFRESH STREAMING TABLE silver_sales_enriched AS +SELECT s.*, p.product_name, p.category +FROM STREAM silver_sales s +LEFT JOIN dim_products p ON s.product_id = p.product_id; +``` + +### Handling Expectations + +**DLT Python:** +```python +@dlt.expect_or_drop("valid_amount", "amount > 0") +@dlt.expect_or_fail("critical_id", "id IS NOT NULL") +``` + +**SDP SQL - Basic** (equivalent to expect_or_drop): +```sql +WHERE amount > 0 AND id IS NOT NULL +``` + +**SDP SQL - Quarantine Pattern** (for auditing dropped records): +```sql +-- Flag invalid records +CREATE OR REFRESH STREAMING TABLE bronze_data_flagged AS +SELECT *, + CASE WHEN amount <= 0 OR id IS NULL THEN TRUE ELSE FALSE END AS is_invalid +FROM STREAM bronze_data; + +-- Clean for downstream +CREATE OR REFRESH STREAMING TABLE silver_data_clean AS +SELECT * FROM STREAM bronze_data_flagged WHERE NOT is_invalid; + +-- Quarantine for investigation +CREATE OR REFRESH STREAMING TABLE silver_data_quarantine AS +SELECT * FROM STREAM bronze_data_flagged WHERE is_invalid; +``` + +### Handling UDFs + +#### Simple UDFs → SQL CASE + +**DLT Python:** +```python +@F.udf(returnType=StringType()) +def categorize_amount(amount): + if amount > 1000: return "High" + elif amount > 100: return "Medium" + else: return "Low" + +@dlt.table(name="sales_categorized") +def sales_categorized(): + return dlt.read("sales").withColumn("category", categorize_amount(F.col("amount"))) +``` + +**SDP SQL:** +```sql +CREATE OR REFRESH MATERIALIZED VIEW sales_categorized AS +SELECT *, + CASE + WHEN amount > 1000 THEN 'High' + WHEN amount > 100 THEN 'Medium' + ELSE 'Low' + END AS category +FROM sales; +``` + +#### Complex UDFs → Stay in Python + +Keep in Python if: +- Complex conditional logic +- External API calls +- Custom algorithms +- ML inference + +Use modern `dp` API instead of `dlt`. + +--- + +## Migration Process + +### Step 1: Inventory + +Document: +- Number of tables/views +- Python UDFs (simple vs complex) +- External dependencies +- Expectations and quality rules + +### Step 2: Choose Path + +- **80%+ SQL-expressible** → Migrate to SDP SQL +- **Heavy Python logic** → Migrate to SDP Python (`dp` API) +- **Mixed** → Hybrid (SQL for most, Python for complex) + +### Step 3: Migrate by Layer + +1. **Bronze** (ingestion): `cloudFiles` → `read_files()` or keep `cloudFiles` with `dp` +2. **Silver** (cleansing): `dlt.expect*` → WHERE clause or `dp.expect*` +3. **Gold** (aggregations): Usually straightforward +4. **SCD/CDC**: `apply_changes` → AUTO CDC or `create_auto_cdc_flow` + +### Step 4: Test + +- Run both pipelines in parallel +- Compare outputs for correctness +- Validate performance +- Check quality metrics + +--- + +## When NOT to Migrate + +**Stay with current approach if:** +1. Pipeline works well and team is comfortable +2. Heavy Python UDF usage (>30% of logic) +3. External API calls required +4. Custom ML model inference +5. Complex stateful operations not expressible in SQL +6. Limited time/resources for migration + +**Key**: DLT and SDP are both fully supported. Migrate for simplicity or new features, not necessity. + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| `sequence_by` type error | Use `col("column")` not string in `dp.create_auto_cdc_flow()` | +| UDF doesn't translate | Keep in Python or refactor with SQL built-ins | +| Expectations differ | Use quarantine pattern to audit dropped records | +| Performance degradation | Use `CLUSTER BY` for Liquid Clustering | +| Schema evolution different | Use `mode => 'PERMISSIVE'` in `read_files()` | +| AUTO CDC parse error | Put `APPLY AS DELETE WHEN` before `SEQUENCE BY` | + +--- + +## Related Documentation + +- **[python/1-syntax-basics.md](python/1-syntax-basics.md)** - Modern `dp` API reference +- **[python/4-cdc-patterns.md](python/4-cdc-patterns.md)** - Python CDC patterns +- **[sql/4-cdc-patterns.md](sql/4-cdc-patterns.md)** - SQL CDC patterns +- **[SKILL.md](../SKILL.md)** - Main skill entry point diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/python/1-syntax-basics.md b/.claude/skills/databricks-spark-declarative-pipelines/references/python/1-syntax-basics.md new file mode 100644 index 00000000..9d00cdec --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/python/1-syntax-basics.md @@ -0,0 +1,321 @@ +# Python Syntax Basics + +Core Python syntax for Spark Declarative Pipelines (SDP) using the modern `pyspark.pipelines` API. + +**Import**: `from pyspark import pipelines as dp` + +--- + +## Decorators + +### `@dp.table()` + +Creates a streaming table or batch table. + +```python +from pyspark import pipelines as dp +from pyspark.sql import functions as F + +@dp.table( + name="bronze_events", # Table name (can be fully qualified: catalog.schema.table) + comment="Raw event data", # Optional description + cluster_by=["event_type", "date"], # Liquid Clustering columns (recommended) + table_properties={ # Delta table properties + "delta.autoOptimize.optimizeWrite": "true", + "delta.autoOptimize.autoCompact": "true" + }, + schema="col1 STRING, col2 INT", # Optional explicit schema + path="/path/to/external/location" # Optional external location +) +def bronze_events(): + return ( + spark.readStream.format("cloudFiles") + .option("cloudFiles.format", "json") + .load("/Volumes/catalog/schema/raw/events/") + .withColumn("_ingested_at", F.current_timestamp()) + .withColumn("_source_file", F.col("_metadata.file_path")) + ) +``` + +**Parameters:** +| Parameter | Type | Description | +|-----------|------|-------------| +| `name` | str | Table name. Can be unqualified (`my_table`), schema-qualified (`schema.table`), or fully qualified (`catalog.schema.table`). | +| `comment` | str | Table description | +| `cluster_by` | list | Columns for Liquid Clustering. Use `["AUTO"]` for automatic selection. | +| `table_properties` | dict | Delta table properties | +| `schema` | str/StructType | Explicit schema (optional, usually inferred) | +| `path` | str | External storage location (optional) | + +**Streaming vs Batch:** +- Return `spark.readStream...` for streaming table +- Return `spark.read...` for batch table + +### `@dp.materialized_view()` + +Creates a materialized view (batch, incrementally refreshed). + +```python +@dp.materialized_view( + name="gold_daily_summary", + comment="Daily aggregated metrics", + cluster_by=["report_date"] +) +def gold_daily_summary(): + return ( + spark.read.table("silver_orders") + .groupBy("report_date") + .agg(F.sum("amount").alias("total_amount")) + ) +``` + +**Parameters:** Same as `@dp.table()`. + +### `@dp.temporary_view()` + +Creates a pipeline-scoped temporary view (not persisted, exists only during pipeline execution). + +```python +@dp.temporary_view() +def orders_with_calculations(): + """Intermediate view for complex logic before AUTO CDC.""" + return ( + spark.readStream.table("bronze_orders") + .withColumn("total", F.col("quantity") * F.col("price")) + .filter(F.col("total") > 0) + ) +``` + +**Constraints:** +- Cannot specify `catalog` or `schema` (pipeline-scoped only) +- Cannot use `cluster_by` (not persisted) +- Useful for intermediate transformations before AUTO CDC + +--- + +## Expectation Decorators (Data Quality) + +```python +@dp.table(name="silver_validated") +@dp.expect("valid_id", "id IS NOT NULL") # Warn only, keep all rows +@dp.expect_or_drop("valid_amount", "amount > 0") # Drop invalid rows +@dp.expect_or_fail("critical_field", "timestamp IS NOT NULL") # Fail pipeline if violated +def silver_validated(): + return spark.read.table("bronze_events") +``` + +| Decorator | Behavior | +|-----------|----------| +| `@dp.expect(name, condition)` | Log warning, keep all rows | +| `@dp.expect_or_drop(name, condition)` | Drop rows that violate | +| `@dp.expect_or_fail(name, condition)` | Fail pipeline if any row violates | + +--- + +## Functions + +### `dp.create_streaming_table()` + +Creates an empty streaming table (typically used before `create_auto_cdc_flow`). + +```python +dp.create_streaming_table( + name="customers_history", + comment="SCD Type 2 customer dimension" +) +``` + +### `dp.create_auto_cdc_flow()` + +Creates a Change Data Capture flow for SCD Type 1 or Type 2. + +```python +from pyspark.sql.functions import col + +dp.create_streaming_table("dim_customers") + +dp.create_auto_cdc_flow( + target="dim_customers", + source="customers_cdc_clean", + keys=["customer_id"], + sequence_by=col("event_timestamp"), # Note: use col(), not string + stored_as_scd_type=2, # Integer for Type 2 + apply_as_deletes=col("operation") == "DELETE", # Optional + except_column_list=["operation", "_ingested_at"], # Columns to exclude + track_history_column_list=["price", "status"] # Type 2: only track these +) +``` + +**Parameters:** +| Parameter | Type | Description | +|-----------|------|-------------| +| `target` | str | Target table name | +| `source` | str | Source table/view name | +| `keys` | list | Primary key columns | +| `sequence_by` | Column | Column for ordering changes (**use `col()`**) | +| `stored_as_scd_type` | int/str | `2` for Type 2 (history), `"1"` for Type 1 (overwrite) | +| `apply_as_deletes` | Column | Condition identifying delete operations | +| `apply_as_truncates` | Column | Condition identifying truncate operations | +| `except_column_list` | list | Columns to exclude from target | +| `track_history_column_list` | list | Type 2 only: columns that trigger new versions | + +**Important:** `stored_as_scd_type` is integer `2` for Type 2, string `"1"` for Type 1. + +### `dp.create_auto_cdc_from_snapshot_flow()` + +Creates CDC from periodic snapshots (compares consecutive snapshots to detect changes). + +```python +dp.create_streaming_table("dim_products") + +dp.create_auto_cdc_from_snapshot_flow( + target="dim_products", + source="products_snapshot", + keys=["product_id"], + stored_as_scd_type=2 +) +``` + +### `dp.append_flow()` + +Appends data from a source to a target table. + +```python +dp.create_streaming_table("events_archive") + +dp.append_flow( + target="events_archive", + source="old_events_source" +) +``` + +### `dp.create_sink()` + +Creates a custom sink for streaming data. + +```python +def write_to_kafka(batch_df, batch_id): + batch_df.write.format("kafka").option("topic", "output").save() + +dp.create_sink( + name="kafka_sink", + sink_fn=write_to_kafka +) +``` + +--- + +## Reading Data + +**Use standard Spark APIs** - SDP automatically tracks dependencies: + +```python +# Batch read (for materialized views or batch tables) +df = spark.read.table("catalog.schema.source_table") + +# Streaming read (for streaming tables) +df = spark.readStream.table("catalog.schema.source_table") + +# Unqualified name (uses pipeline's default catalog/schema) +df = spark.read.table("source_table") + +# Read from file with Auto Loader (schema location managed automatically in SDP) +df = spark.readStream.format("cloudFiles") \ + .option("cloudFiles.format", "json") \ + .load("/Volumes/catalog/schema/raw/data/") +``` + +**Do NOT use:** +- `dp.read()` or `dp.read_stream()` - not part of modern API +- `dlt.read()` or `dlt.read_stream()` - legacy API +- `dlt.apply_changes()` - legacy API; use `dp.create_auto_cdc_flow()` instead +- `import dlt` - legacy module; use `from pyspark import pipelines as dp` + +--- + +## Table Name Resolution + +| Level | Example | When to Use | +|-------|---------|-------------| +| Unqualified | `spark.read.table("my_table")` | Tables in same pipeline (recommended) | +| Schema-qualified | `spark.read.table("other_schema.my_table")` | Different schema, same catalog | +| Fully-qualified | `spark.read.table("other_catalog.schema.table")` | External catalogs | + +**Best practice:** Use unqualified names for pipeline-internal tables. + +### Multi-Schema Pattern (One Pipeline) + +Write to multiple schemas from a single pipeline using fully qualified names: + +```python +from pyspark import pipelines as dp + +# Bronze → writes to bronze schema +@dp.table(name="my_catalog.bronze.raw_orders") +def bronze_orders(): + return spark.readStream.format("cloudFiles") \ + .option("cloudFiles.format", "json") \ + .load("/Volumes/my_catalog/raw/orders/") + +# Silver → writes to silver schema, reads from bronze +@dp.table(name="my_catalog.silver.clean_orders") +def silver_orders(): + return spark.readStream.table("my_catalog.bronze.raw_orders") \ + .filter("order_id IS NOT NULL") +``` + +--- + +## Pipeline Parameters + +Access configuration values set in pipeline settings: + +```python +# Get parameter value +catalog = spark.conf.get("target_catalog") +schema = spark.conf.get("target_schema") + +# With default +env = spark.conf.get("environment", "dev") + +@dp.table(name=f"{catalog}.{schema}.my_table") +def my_table(): + return spark.readStream.format("cloudFiles") \ + .option("cloudFiles.format", "json") \ + .load("/Volumes/...") +``` + +--- + +## Prohibited Operations + +**Do NOT include these in dataset definitions:** + +```python +# These cause unexpected behavior +@dp.table(name="bad_example") +def bad_example(): + df = spark.read.table("source") + df.collect() # No collect() + df.count() # No count() + df.toPandas() # No toPandas() + df.save(...) # No save() + df.saveAsTable(...) # No saveAsTable() + return df +``` + +Dataset functions should only contain code to define the transformation, not execute actions. + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| `sequence_by` type error | Use `col("column")` not string in `create_auto_cdc_flow()` | +| SCD type syntax error | Type 2 uses integer `2`, Type 1 uses string `"1"` | +| Table not found | Check catalog/schema qualification or pipeline default settings | +| Parameter not resolved | Use `spark.conf.get("param_name")` | +| Actions in definition | Remove `collect()`, `count()`, `save()` from table functions | +| Using legacy `dlt` API | Replace `import dlt` with `from pyspark import pipelines as dp` | +| Using `input_file_name()` | Use `F.col("_metadata.file_path")` | diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/python/2-ingestion.md b/.claude/skills/databricks-spark-declarative-pipelines/references/python/2-ingestion.md new file mode 100644 index 00000000..06ddad2f --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/python/2-ingestion.md @@ -0,0 +1,150 @@ +# Python Data Ingestion + +Data ingestion patterns using the modern `pyspark.pipelines` API. + +**Official Documentation:** +- [Auto Loader options](https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options) +- [Structured Streaming + Kafka](https://docs.databricks.com/aws/en/structured-streaming/kafka) + +--- + +## Auto Loader (Cloud Files) + +Auto Loader incrementally processes new files. In SDP pipelines, schema location and checkpoints are managed automatically. + +### Basic Pattern + +```python +from pyspark import pipelines as dp +from pyspark.sql import functions as F + +@dp.table(name="bronze_orders", cluster_by=["order_date"]) +def bronze_orders(): + return ( + spark.readStream + .format("cloudFiles") + .option("cloudFiles.format", "json") + .option("cloudFiles.inferColumnTypes", "true") + .load("/Volumes/my_catalog/my_schema/raw/orders/") + .withColumn("_ingested_at", F.current_timestamp()) + .withColumn("_source_file", F.col("_metadata.file_path")) + ) +``` + +**Key options:** +- `cloudFiles.format`: `json`, `csv`, `parquet`, `avro`, `text`, `binaryFile` +- `cloudFiles.inferColumnTypes`: Infer types (default strings) +- `cloudFiles.schemaHints`: Hint specific column types + +### Rescue Data (Quarantine Pattern) + +```python +@dp.table(name="bronze_events", cluster_by=["ingestion_date"]) +def bronze_events(): + return ( + spark.readStream + .format("cloudFiles") + .option("cloudFiles.format", "json") + .option("rescuedDataColumn", "_rescued_data") + .load("/Volumes/catalog/schema/raw/events/") + .withColumn("_ingested_at", F.current_timestamp()) + .withColumn("_has_errors", F.col("_rescued_data").isNotNull()) + ) + +@dp.table(name="bronze_quarantine") +def bronze_quarantine(): + return spark.readStream.table("bronze_events").filter("_has_errors = true") + +@dp.table(name="silver_clean") +def silver_clean(): + return spark.readStream.table("bronze_events").filter("_has_errors = false") +``` + +--- + +## Streaming Sources + +### Kafka + +```python +@dp.table(name="bronze_kafka_events") +def bronze_kafka_events(): + kafka_brokers = spark.conf.get("kafka_brokers") + return ( + spark.readStream + .format("kafka") + .option("kafka.bootstrap.servers", kafka_brokers) + .option("subscribe", "events-topic") + .option("startingOffsets", "latest") + .load() + .selectExpr( + "CAST(key AS STRING) AS event_key", + "CAST(value AS STRING) AS event_value", + "topic", "partition", "offset", + "timestamp AS kafka_timestamp" + ) + .withColumn("_ingested_at", F.current_timestamp()) + ) +``` + +### Parse JSON from Kafka + +```python +from pyspark.sql.types import StructType, StructField, StringType, TimestampType + +event_schema = StructType([ + StructField("event_id", StringType()), + StructField("event_type", StringType()), + StructField("timestamp", TimestampType()) +]) + +@dp.table(name="silver_events") +def silver_events(): + return ( + spark.readStream.table("bronze_kafka_events") + .withColumn("data", F.from_json("event_value", event_schema)) + .select("data.*", "kafka_timestamp", "_ingested_at") + ) +``` + +--- + +## Authentication + +### Databricks Secrets + +```python +username = dbutils.secrets.get(scope="kafka", key="username") +password = dbutils.secrets.get(scope="kafka", key="password") +``` + +### Pipeline Parameters + +```python +kafka_brokers = spark.conf.get("kafka_brokers") +input_path = spark.conf.get("input_path") +``` + +--- + +## Best Practices + +1. **Add ingestion metadata:** +```python +.withColumn("_ingested_at", F.current_timestamp()) +.withColumn("_source_file", F.col("_metadata.file_path")) +``` + +2. **Handle rescue data** - route malformed records to quarantine + +3. **Use pipeline parameters** for paths and connection strings + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| Files not picked up | Verify path and format match actual files | +| Schema evolution breaking | Use `rescuedDataColumn` and monitor `_rescued_data` | +| Kafka lag increasing | Check downstream bottlenecks | diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/python/3-streaming-patterns.md b/.claude/skills/databricks-spark-declarative-pipelines/references/python/3-streaming-patterns.md new file mode 100644 index 00000000..44fd6191 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/python/3-streaming-patterns.md @@ -0,0 +1,382 @@ +# Python Streaming Patterns + +Streaming-specific patterns including deduplication, windowed aggregations, late-arriving data handling, and stateful operations. + +**Import**: `from pyspark import pipelines as dp` + +--- + +## Deduplication Patterns + +### By Key + +```python +from pyspark import pipelines as dp +from pyspark.sql import functions as F +from pyspark.sql.window import Window + +@dp.table(name="silver_events_dedup", cluster_by=["event_date"]) +def silver_events_dedup(): + """Deduplicate by event_id, keeping first occurrence.""" + window_spec = Window.partitionBy("event_id").orderBy("event_timestamp") + return ( + spark.readStream.table("bronze_events") + .withColumn("rn", F.row_number().over(window_spec)) + .filter(F.col("rn") == 1) + .drop("rn") + ) +``` + +### With Time Window + +Deduplicate within time window to handle late arrivals: + +```python +@dp.table(name="silver_events_dedup") +def silver_events_dedup(): + return ( + spark.readStream.table("bronze_events") + .groupBy( + "event_id", "user_id", "event_type", "event_timestamp", + F.window("event_timestamp", "1 hour") + ) + .agg(F.min("_ingested_at").alias("first_seen_at")) + ) +``` + +### Composite Key + +```python +@dp.table(name="silver_transactions_dedup") +def silver_transactions_dedup(): + return ( + spark.readStream.table("bronze_transactions") + .groupBy("transaction_id", "customer_id", "amount", "transaction_timestamp") + .agg(F.min("_ingested_at").alias("_ingested_at")) + ) +``` + +--- + +## Windowed Aggregations + +### Tumbling Windows + +Non-overlapping fixed-size windows: + +```python +@dp.table(name="silver_sensor_5min", cluster_by=["sensor_id"]) +def silver_sensor_5min(): + """5-minute tumbling window aggregations.""" + return ( + spark.readStream.table("bronze_sensor_events") + .groupBy( + F.col("sensor_id"), + F.window("event_timestamp", "5 minutes") + ) + .agg( + F.avg("temperature").alias("avg_temperature"), + F.min("temperature").alias("min_temperature"), + F.max("temperature").alias("max_temperature"), + F.count("*").alias("event_count") + ) + ) +``` + +### Multiple Window Sizes + +```python +# 1-minute for real-time monitoring +@dp.table(name="gold_sensor_1min") +def gold_sensor_1min(): + return ( + spark.readStream.table("silver_sensor_data") + .groupBy( + "sensor_id", + F.window("event_timestamp", "1 minute") + ) + .agg( + F.avg("value").alias("avg_value"), + F.count("*").alias("event_count") + ) + .select( + "sensor_id", + F.col("window.start").alias("window_start"), + F.col("window.end").alias("window_end"), + "avg_value", + "event_count" + ) + ) + +# 1-hour for trend analysis +@dp.table(name="gold_sensor_1hour") +def gold_sensor_1hour(): + return ( + spark.readStream.table("silver_sensor_data") + .groupBy( + "sensor_id", + F.window("event_timestamp", "1 hour") + ) + .agg( + F.avg("value").alias("avg_value"), + F.stddev("value").alias("stddev_value") + ) + ) +``` + +### Session Windows + +Group events into sessions based on inactivity gaps: + +```python +@dp.table(name="silver_user_sessions") +def silver_user_sessions(): + """Group user events into sessions with 30-minute inactivity timeout.""" + return ( + spark.readStream.table("bronze_user_events") + .groupBy( + F.col("user_id"), + F.session_window("event_timestamp", "30 minutes") + ) + .agg( + F.min("event_timestamp").alias("session_start"), + F.max("event_timestamp").alias("session_end"), + F.count("*").alias("event_count"), + F.collect_list("event_type").alias("event_sequence") + ) + ) +``` + +--- + +## Late-Arriving Data + +### Event-Time vs Processing-Time + +Always use event timestamp for business logic: + +```python +@dp.table(name="gold_daily_orders") +def gold_daily_orders(): + return ( + spark.readStream.table("silver_orders") + .groupBy(F.to_date("order_timestamp").alias("order_date")) # Event time + .agg( + F.count("*").alias("order_count"), + F.sum("amount").alias("total_amount") + ) + ) +``` + +**Keep processing time for debugging:** +```python +.select( + "order_id", "order_timestamp", # Event time (business logic) + "customer_id", "amount", + "_ingested_at" # Processing time (debugging only) +) +``` + +--- + +## Joins + +### Stream-to-Static Joins + +Enrich streaming data with dimension tables: + +```python +@dp.table(name="silver_sales_enriched", cluster_by=["product_id"]) +def silver_sales_enriched(): + """Enrich streaming sales with static product dimension.""" + sales = spark.readStream.table("bronze_sales") + products = spark.read.table("dim_products") + return ( + sales.join(products, "product_id", "left") + .select( + "sale_id", "product_id", "quantity", "sale_timestamp", + "product_name", "category", "price" + ) + .withColumn("total_amount", F.col("quantity") * F.col("price")) + ) +``` + +### Stream-to-Stream Joins + +```python +@dp.table(name="silver_orders_with_payments") +def silver_orders_with_payments(): + """Join orders with payments within 1-hour window.""" + orders = spark.readStream.table("bronze_orders") + payments = spark.readStream.table("bronze_payments") + + return ( + orders.join( + payments, + (orders.order_id == payments.order_id) & + (payments.payment_timestamp >= orders.order_timestamp) & + (payments.payment_timestamp <= orders.order_timestamp + F.expr("INTERVAL 1 HOUR")), + "inner" + ) + .select( + orders.order_id, + orders.customer_id, + orders.order_timestamp, + orders.amount.alias("order_amount"), + payments.payment_id, + payments.payment_timestamp, + payments.amount.alias("payment_amount") + ) + ) +``` + +**Important:** Use time bounds in join condition to limit state retention. + +--- + +## Incremental Aggregations + +### Running Totals + +```python +@dp.table(name="silver_customer_running_totals") +def silver_customer_running_totals(): + return ( + spark.readStream.table("bronze_transactions") + .groupBy("customer_id") + .agg( + F.sum("amount").alias("total_spent"), + F.count("*").alias("transaction_count"), + F.max("transaction_timestamp").alias("last_transaction_at") + ) + ) +``` + +--- + +## Anomaly Detection + +### Real-Time Outlier Detection + +```python +@dp.table(name="silver_sensor_with_anomalies") +def silver_sensor_with_anomalies(): + window_spec = Window.partitionBy("sensor_id").orderBy("event_timestamp").rowsBetween(-100, 0) + + return ( + spark.readStream.table("bronze_sensor_events") + .withColumn("rolling_avg", F.avg("temperature").over(window_spec)) + .withColumn("rolling_stddev", F.stddev("temperature").over(window_spec)) + .withColumn("anomaly_flag", + F.when(F.col("temperature") > F.col("rolling_avg") + (3 * F.col("rolling_stddev")), "HIGH_OUTLIER") + .when(F.col("temperature") < F.col("rolling_avg") - (3 * F.col("rolling_stddev")), "LOW_OUTLIER") + .otherwise("NORMAL") + ) + ) + +@dp.table(name="silver_sensor_anomalies") +def silver_sensor_anomalies(): + return ( + spark.readStream.table("silver_sensor_with_anomalies") + .filter(F.col("anomaly_flag").isin("HIGH_OUTLIER", "LOW_OUTLIER")) + ) +``` + +### Threshold-Based Filtering + +```python +@dp.table(name="silver_high_value_transactions") +def silver_high_value_transactions(): + return ( + spark.readStream.table("bronze_transactions") + .filter(F.col("amount") > 10000) + ) +``` + +--- + +## Monitoring Lag + +```python +@dp.table(name="monitoring_lag") +def monitoring_lag(): + return ( + spark.readStream.table("bronze_kafka_events") + .groupBy(F.window("kafka_timestamp", "1 minute")) + .agg( + F.lit("kafka_events").alias("source"), + F.max("kafka_timestamp").alias("max_event_timestamp"), + F.current_timestamp().alias("processing_timestamp") + ) + .withColumn("lag_seconds", + F.unix_timestamp("processing_timestamp") - F.unix_timestamp("max_event_timestamp") + ) + ) +``` + +--- + +## Best Practices + +### 1. Use Event Timestamps + +```python +# Correct: Event timestamp for logic +.groupBy(F.date_trunc("hour", "event_timestamp")) + +# Avoid: Processing timestamp +# .groupBy(F.date_trunc("hour", "_ingested_at")) +``` + +### 2. Window Size Selection + +- **1-5 minutes**: Real-time monitoring +- **15-60 minutes**: Operational dashboards +- **1-24 hours**: Analytical reports + +### 3. State Management + +Higher cardinality = more state: + +```python +# High state: 1M users x 10K products x 100M sessions +.groupBy("user_id", "product_id", "session_id") + +# Lower state: 1M users x 100 categories x days +.groupBy("user_id", "product_category", F.to_date("event_time")) +``` + +Use time windows to bound state retention. + +### 4. Deduplicate Early + +Apply at bronze → silver transition: + +```python +# Bronze: Accept duplicates +@dp.table(name="bronze_events") +def bronze_events(): + return spark.readStream.format("cloudFiles")... + +# Silver: Deduplicate immediately +@dp.table(name="silver_events") +def silver_events(): + return spark.readStream.table("bronze_events").dropDuplicates(["event_id"]) + +# Gold: Work with clean data +@dp.table(name="gold_metrics") +def gold_metrics(): + return spark.readStream.table("silver_events")... +``` + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| High memory with windows | Use larger windows, reduce group-by cardinality | +| Duplicate events in output | Add explicit deduplication by unique key | +| Missing late-arriving events | Increase window size or use longer retention | +| Stream-to-stream join empty | Verify join conditions and time bounds | +| State growth over time | Add time windows, reduce cardinality, materialize intermediates | diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/python/4-cdc-patterns.md b/.claude/skills/databricks-spark-declarative-pipelines/references/python/4-cdc-patterns.md new file mode 100644 index 00000000..9e053700 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/python/4-cdc-patterns.md @@ -0,0 +1,449 @@ +# Python CDC Patterns (AUTO CDC & SCD) + +Change Data Capture patterns using AUTO CDC for SCD Type 1 and Type 2, plus querying SCD history tables. + +**Import**: `from pyspark import pipelines as dp` + +--- + +## Overview + +AUTO CDC automatically handles Change Data Capture to track changes using Slow Changing Dimensions (SCD). It provides automatic deduplication, change tracking, and handles late-arriving data correctly. + +**Where to apply AUTO CDC:** +- **Silver layer**: When business users need deduplicated or historical data +- **Gold layer**: When implementing dimensional modeling (star schema) + +--- + +## SCD Type 1 vs Type 2 + +### SCD Type 1 (In-place updates) +- **Overwrites** old values with new values +- **No history preserved** - only current state +- **Use for**: Error corrections, attributes where history doesn't matter +- **Syntax**: `stored_as_scd_type="1"` (string) + +### SCD Type 2 (History tracking) +- **Creates new row** for each change +- **Preserves full history** with `__START_AT` and `__END_AT` timestamps +- **Use for**: Tracking changes over time (addresses, prices, roles) +- **Syntax**: `stored_as_scd_type=2` (integer) + +**Important:** Type 2 uses integer `2`, Type 1 uses string `"1"`. + +--- + +## Creating AUTO CDC Flows + +### SCD Type 2 + +```python +from pyspark import pipelines as dp +from pyspark.sql.functions import col + +target_schema = spark.conf.get("target_schema") +source_schema = spark.conf.get("source_schema") + +# Step 1: Create target table +dp.create_streaming_table(f"{target_schema}.dim_customers") + +# Step 2: Create AUTO CDC flow +dp.create_auto_cdc_flow( + target=f"{target_schema}.dim_customers", + source=f"{source_schema}.customers_cdc_clean", + keys=["customer_id"], + sequence_by=col("event_timestamp"), # Note: use col(), not string + stored_as_scd_type=2, # Integer for Type 2 + apply_as_deletes=col("operation") == "DELETE", + except_column_list=["operation", "_ingested_at", "_source_file"] +) +``` + +### SCD Type 1 + +```python +dp.create_streaming_table(f"{target_schema}.orders_current") + +dp.create_auto_cdc_flow( + target=f"{target_schema}.orders_current", + source=f"{source_schema}.orders_clean", + keys=["order_id"], + sequence_by=col("updated_timestamp"), + stored_as_scd_type="1" # String for Type 1 +) +``` + +### Selective History Tracking + +Track history only when specific columns change: + +```python +dp.create_auto_cdc_flow( + target="gold.dim_products", + source="silver.products_clean", + keys=["product_id"], + sequence_by=col("modified_at"), + stored_as_scd_type=2, + track_history_column_list=["price", "cost"] # Only track these columns +) +``` + +When `price` or `cost` changes, a new version is created. Other column changes update the current record without new versions. + +--- + +## Complete Pattern: Clean + AUTO CDC + +### Step 1: Clean and Validate Source Data + +```python +from pyspark import pipelines as dp +from pyspark.sql import functions as F + +schema = spark.conf.get("schema") + +@dp.table( + name=f"{schema}.users_clean", + comment="Cleaned and validated user data", + cluster_by=["user_id"] +) +def users_clean(): + """ + Clean data with proper typing and quality checks. + """ + return ( + spark.readStream.table("bronze_users") + .filter(F.col("user_id").isNotNull()) + .filter(F.col("email").isNotNull()) + .filter(F.col("email").rlike(r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")) + .withColumn("created_timestamp", F.to_timestamp("created_timestamp")) + .withColumn("updated_timestamp", F.to_timestamp("updated_timestamp")) + .drop("_rescued_data") + .select( + "user_id", "email", "name", "subscription_tier", "country", + "created_timestamp", "updated_timestamp", + "_ingested_at", "_source_file" + ) + ) +``` + +### Step 2: Apply AUTO CDC + +```python +from pyspark.sql.functions import col + +target_schema = spark.conf.get("target_schema") +source_schema = spark.conf.get("source_schema") + +dp.create_streaming_table(f"{target_schema}.dim_users") + +dp.create_auto_cdc_flow( + target=f"{target_schema}.dim_users", + source=f"{source_schema}.users_clean", + keys=["user_id"], + sequence_by=col("updated_timestamp"), + stored_as_scd_type=2, + except_column_list=["_ingested_at", "_source_file"] +) +``` + +--- + +## Using Temporary Views with AUTO CDC + +`@dp.temporary_view()` creates in-pipeline temporary views useful for intermediate transformations before AUTO CDC. + +**Key Constraints:** +- Cannot specify `catalog` or `schema` (pipeline-scoped only) +- Cannot use `cluster_by` (not persisted) +- Only exists during pipeline execution + +```python +from pyspark import pipelines as dp +from pyspark.sql import functions as F + +# Step 1: Temporary view for complex business logic +@dp.temporary_view() +def orders_with_calculated_fields(): + """ + Temporary view for complex calculations. + No catalog/schema needed - exists only in pipeline. + """ + return ( + spark.readStream.table("bronze.orders") + .withColumn("order_total", F.col("quantity") * F.col("unit_price")) + .withColumn("discount_amount", F.col("order_total") * F.col("discount_rate")) + .withColumn("final_amount", F.col("order_total") - F.col("discount_amount")) + .withColumn("order_category", + F.when(F.col("final_amount") > 1000, "large") + .when(F.col("final_amount") > 100, "medium") + .otherwise("small") + ) + .filter(F.col("order_id").isNotNull()) + .filter(F.col("final_amount") > 0) + ) + +# Step 2: Apply AUTO CDC using the temporary view as source +target_schema = spark.conf.get("target_schema") + +dp.create_streaming_table(f"{target_schema}.orders_current") +dp.create_auto_cdc_flow( + target=f"{target_schema}.orders_current", + source="orders_with_calculated_fields", # Reference temporary view by name + keys=["order_id"], + sequence_by=col("order_date"), + stored_as_scd_type="1" +) +``` + +--- + +## Querying SCD Type 2 Tables + +SCD Type 2 tables include temporal columns: +- `__START_AT` - When this version became effective +- `__END_AT` - When this version expired (NULL for current) + +### Current State + +```python +@dp.materialized_view(name="dim_customers_current") +def dim_customers_current(): + """All current records.""" + return ( + spark.read.table("dim_customers") + .filter(F.col("__END_AT").isNull()) + .select( + "customer_id", "customer_name", "email", "phone", "address", + F.col("__START_AT").alias("valid_from") + ) + ) +``` + +### Point-in-Time Queries + +Get state as of a specific date: + +```python +@dp.materialized_view(name="products_as_of_date") +def products_as_of_date(): + """Products as of January 1, 2024.""" + as_of_date = "2024-01-01" + return ( + spark.read.table("products_history") + .filter(F.col("__START_AT") <= as_of_date) + .filter( + (F.col("__END_AT") > as_of_date) | + F.col("__END_AT").isNull() + ) + ) +``` + +### Change Analysis + +Track all changes for an entity: + +```python +def get_customer_history(customer_id: str): + """Get complete history for a customer.""" + return ( + spark.read.table("dim_customers") + .filter(F.col("customer_id") == customer_id) + .withColumn("days_active", + F.coalesce( + F.datediff("__END_AT", "__START_AT"), + F.datediff(F.current_timestamp(), "__START_AT") + ) + ) + .orderBy(F.col("__START_AT").desc()) + ) +``` + +--- + +## Joining Facts with Historical Dimensions + +### At Transaction Time + +```python +@dp.materialized_view(name="sales_with_historical_prices") +def sales_with_historical_prices(): + """Join sales with product prices at time of sale.""" + sales = spark.read.table("sales_fact") + products = spark.read.table("products_history") + + return ( + sales.join( + products, + (sales.product_id == products.product_id) & + (sales.sale_date >= products.__START_AT) & + ((sales.sale_date < products.__END_AT) | products.__END_AT.isNull()), + "inner" + ) + .select( + sales.sale_id, + sales.product_id, + sales.sale_date, + sales.quantity, + products.product_name, + products.price.alias("unit_price_at_sale_time"), + (sales.quantity * products.price).alias("calculated_amount"), + products.category + ) + ) +``` + +### With Current Dimension + +```python +@dp.materialized_view(name="sales_with_current_prices") +def sales_with_current_prices(): + """Join sales with current product information.""" + sales = spark.read.table("sales_fact") + products_current = spark.read.table("products_history").filter(F.col("__END_AT").isNull()) + + return ( + sales.join(products_current, "product_id", "inner") + .select( + "sale_id", "product_id", "sale_date", "quantity", + sales.amount.alias("amount_at_sale"), + products_current.product_name.alias("current_product_name"), + products_current.price.alias("current_price") + ) + ) +``` + +--- + +## Common Patterns + +### Pattern 1: Gold Dimensional Model + +```python +# Silver: Cleaned streaming tables +@dp.table(name="silver.customers_clean") +def customers_clean(): + return spark.readStream.table("bronze.customers").filter(...) + +# Gold: SCD Type 2 dimension +dp.create_streaming_table("gold.dim_customers") +dp.create_auto_cdc_flow( + target="gold.dim_customers", + source="silver.customers_clean", + keys=["customer_id"], + sequence_by=col("updated_at"), + stored_as_scd_type=2 +) + +# Gold: Fact table (no AUTO CDC) +@dp.table(name="gold.fact_orders") +def fact_orders(): + return spark.read.table("silver.orders_clean") +``` + +### Pattern 2: Silver Deduplication for Joins + +```python +# Silver: AUTO CDC for deduplication +dp.create_streaming_table("silver.products_dedupe") +dp.create_auto_cdc_flow( + target="silver.products_dedupe", + source="bronze.products", + keys=["product_id"], + sequence_by=col("modified_at"), + stored_as_scd_type="1" # Type 1: just dedupe, no history +) + +# Silver: Join with deduplicated data +@dp.table(name="silver.orders_enriched") +def orders_enriched(): + orders = spark.readStream.table("bronze.orders") + products = spark.read.table("silver.products_dedupe") + return orders.join(products, "product_id") +``` + +### Pattern 3: Mixed SCD Types + +```python +# SCD Type 2: Need history +dp.create_auto_cdc_flow( + target="gold.dim_customers", + source="silver.customers", + keys=["customer_id"], + sequence_by=col("updated_at"), + stored_as_scd_type=2 # Track address changes over time +) + +# SCD Type 1: Corrections only +dp.create_auto_cdc_flow( + target="gold.dim_products", + source="silver.products", + keys=["product_id"], + sequence_by=col("modified_at"), + stored_as_scd_type="1" # Current product info only +) +``` + +--- + +## Best Practices + +### 1. Clean Data Before AUTO CDC + +Apply type casting, validation, and filtering first: + +```python +@dp.table(name="users_clean") +def users_clean(): + return ( + spark.readStream.table("bronze_users") + .filter(F.col("user_id").isNotNull()) + .filter(F.col("email").isNotNull()) + .withColumn("updated_at", F.to_timestamp("updated_at")) + ) + +# Then apply AUTO CDC +dp.create_auto_cdc_flow( + target="dim_users", + source="users_clean", + keys=["user_id"], + sequence_by=col("updated_at"), + stored_as_scd_type=2 +) +``` + +### 2. Use col() for sequence_by + +```python +# Correct +sequence_by=col("event_timestamp") + +# Wrong - causes error +# sequence_by="event_timestamp" +``` + +### 3. Choose the Right SCD Type + +- **Type 2** (`stored_as_scd_type=2`): Need to query historical states +- **Type 1** (`stored_as_scd_type="1"`): Only need current state or deduplication + +### 4. Use meaningful sequence_by column + +Should reflect true chronological order of changes: +- `updated_timestamp` +- `modified_at` +- `event_timestamp` + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| `sequence_by` type error | Use `col("column")` not string | +| SCD type syntax error | Type 2 uses integer `2`, Type 1 uses string `"1"` | +| Duplicates still appearing | Check `keys` include all business key columns | +| Missing `__START_AT`/`__END_AT` | These only appear in SCD Type 2, not Type 1 | +| Late data not handled | Ensure `sequence_by` reflects true event time | +| Performance issues | Use `track_history_column_list` to limit version triggers | diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/python/5-performance.md b/.claude/skills/databricks-spark-declarative-pipelines/references/python/5-performance.md new file mode 100644 index 00000000..0cdcc942 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/python/5-performance.md @@ -0,0 +1,423 @@ +# Python Performance Tuning + +Performance optimization strategies including Liquid Clustering, materialized view refresh, state management, and compute configuration. + +**Import**: `from pyspark import pipelines as dp` + +--- + +## Liquid Clustering (Recommended) + +Liquid Clustering is the recommended approach for data layout optimization. It replaces manual partitioning and Z-ORDER. + +### Benefits + +- **Adaptive**: Adjusts to data distribution changes +- **Multi-dimensional**: Clusters on multiple columns simultaneously +- **Automatic file sizing**: Maintains optimal file sizes +- **Self-optimizing**: Reduces manual OPTIMIZE commands + +### Basic Syntax + +```python +from pyspark import pipelines as dp + +@dp.table(cluster_by=["event_type", "event_date"]) +def bronze_events(): + return spark.readStream.format("cloudFiles").load("/data") +``` + +### Automatic Key Selection + +```python +@dp.table(cluster_by=["AUTO"]) +def bronze_events(): + return spark.readStream.format("cloudFiles").load("/data") +``` + +**When to use AUTO**: Learning phase, unknown access patterns, prototyping +**When to define manually**: Well-known query patterns, production workloads + +--- + +## Cluster Key Selection by Layer + +### Bronze Layer + +Cluster by event type + date: + +```python +@dp.table( + name="bronze_events", + cluster_by=["event_type", "ingestion_date"], + table_properties={"delta.autoOptimize.optimizeWrite": "true"} +) +def bronze_events(): + return ( + spark.readStream.format("cloudFiles") + .option("cloudFiles.format", "json") + .load("/Volumes/my_catalog/my_schema/raw/events/") + .withColumn("_ingested_at", F.current_timestamp()) + .withColumn("ingestion_date", F.current_date()) + ) +``` + +**Why**: Bronze filtered by event type for processing and by date for incremental loads. + +### Silver Layer + +Cluster by primary key + business dimension: + +```python +@dp.table( + name="silver_orders", + cluster_by=["customer_id", "order_date"] +) +def silver_orders(): + return ( + spark.readStream.table("bronze_orders") + .withColumn("order_date", F.to_date("order_timestamp")) + .select("order_id", "customer_id", "product_id", "amount", "order_date") + ) +``` + +**Why**: Entity lookups (by ID) and time-range queries (by date). + +### Gold Layer + +Cluster by aggregation dimensions: + +```python +@dp.materialized_view( + name="gold_sales_summary", + cluster_by=["product_category", "year_month"] +) +def gold_sales_summary(): + return ( + spark.read.table("silver_orders") + .withColumn("year_month", F.date_format("order_date", "yyyy-MM")) + .groupBy("product_category", "year_month") + .agg( + F.sum("amount").alias("total_sales"), + F.count("*").alias("transaction_count"), + F.avg("amount").alias("avg_order_value") + ) + ) +``` + +**Why**: Dashboard filters (category, region, time period). + +### Selection Guidelines + +| Layer | Good Keys | Rationale | +|-------|-----------|-----------| +| **Bronze** | event_type, ingestion_date | Filter by type; date for incremental | +| **Silver** | primary_key, business_date | Entity lookups + time ranges | +| **Gold** | aggregation_dimensions | Dashboard filters | + +**Best practices:** +- First key: Most selective filter (e.g., customer_id) +- Second key: Next common filter (e.g., date) +- Order matters: Most selective first +- Limit to 4 keys: Diminishing returns beyond 4 +- **Use `["AUTO"]` if unsure** + +--- + +## Table Properties + +### Auto-Optimize + +```python +@dp.table( + name="bronze_events", + table_properties={ + "delta.autoOptimize.optimizeWrite": "true", + "delta.autoOptimize.autoCompact": "true" + } +) +def bronze_events(): + return spark.readStream.format("cloudFiles").load(...) +``` + +### Change Data Feed + +```python +@dp.table( + name="silver_customers", + table_properties={"delta.enableChangeDataFeed": "true"} +) +def silver_customers(): + return spark.readStream.table("bronze_customers") +``` + +**Use when**: Downstream systems need efficient change tracking. + +### Retention Periods + +```python +@dp.table( + name="bronze_high_volume", + table_properties={ + "delta.logRetentionDuration": "7 days", + "delta.deletedFileRetentionDuration": "7 days" + } +) +def bronze_high_volume(): + return spark.readStream.format("cloudFiles").load(...) +``` + +**Use for**: High-volume tables to reduce storage costs. + +--- + +## State Management for Streaming + +### Understand State Growth + +Higher cardinality = more state: + +```python +# High state: 1M users x 10K products x 100M sessions - Massive state! +.groupBy("user_id", "product_id", "session_id") +``` + +### Reduce State Size + +**Strategy 1: Reduce cardinality** + +```python +@dp.table(name="user_category_stats") +def user_category_stats(): + return ( + spark.readStream.table("bronze_events") + .groupBy( + "user_id", + "product_category", # 100 categories (not 10K products) + F.to_date("event_time").alias("event_date") + ) + .agg(F.count("*").alias("events")) + ) +``` + +**Strategy 2: Use time windows** + +```python +@dp.table(name="user_hourly_stats") +def user_hourly_stats(): + return ( + spark.readStream.table("bronze_events") + .groupBy( + "user_id", + F.window("event_time", "1 hour") + ) + .agg(F.count("*").alias("events")) + ) +``` + +**Strategy 3: Materialize intermediates** + +```python +# Streaming aggregation (maintains state) +@dp.table(name="user_daily_stats") +def user_daily_stats(): + return ( + spark.readStream.table("bronze_events") + .groupBy("user_id", F.to_date("event_time").alias("event_date")) + .agg(F.count("*").alias("event_count")) + ) + +# Batch aggregation (no streaming state) +@dp.materialized_view(name="user_monthly_stats") +def user_monthly_stats(): + return ( + spark.read.table("user_daily_stats") + .groupBy("user_id", F.date_trunc("month", "event_date").alias("month")) + .agg(F.sum("event_count").alias("total_events")) + ) +``` + +--- + +## Join Optimization + +### Stream-to-Static (Efficient) + +```python +@dp.table(name="sales_enriched") +def sales_enriched(): + """Small static dimension, large streaming fact.""" + sales = spark.readStream.table("bronze_sales") + products = spark.read.table("dim_products") # Small, broadcast + + return ( + sales.join(products, "product_id", "left") + .select("sale_id", "product_id", "amount", "product_name", "category") + ) +``` + +**Best practice**: Keep static dimensions small (<10K rows) for broadcast. + +### Stream-to-Stream (Stateful) + +```python +@dp.table(name="orders_with_payments") +def orders_with_payments(): + """Time bounds limit state retention.""" + orders = spark.readStream.table("bronze_orders") + payments = spark.readStream.table("bronze_payments") + + return orders.join( + payments, + (orders.order_id == payments.order_id) & + (payments.payment_time >= orders.order_time) & + (payments.payment_time <= orders.order_time + F.expr("INTERVAL 1 HOUR")), + "inner" + ) +``` + +--- + +## Query Optimization + +### Filter Early + +```python +# Filter at source +@dp.table(name="silver_recent") +def silver_recent(): + return ( + spark.readStream.table("bronze_events") + .filter(F.col("event_date") >= F.current_date() - 7) + ) + +# Avoid filtering late in separate table +# @dp.table(name="silver_all") +# def silver_all(): return spark.readStream.table("bronze_events") +# @dp.materialized_view(name="gold_recent") +# def gold_recent(): return spark.read.table("silver_all").filter(...) +``` + +### Select Specific Columns + +```python +# Only needed columns +.select("customer_id", "order_date", "amount") + +# Avoid SELECT * +# .select("*") +``` + +--- + +## Pre-Aggregation + +```python +@dp.materialized_view(name="orders_monthly") +def orders_monthly(): + """Pre-aggregate for fast queries.""" + return ( + spark.read.table("large_orders_table") + .groupBy( + "customer_id", + F.year("order_date").alias("year"), + F.month("order_date").alias("month") + ) + .agg(F.sum("amount").alias("total")) + ) + +# Query the MV directly - much faster than querying large_orders_table +``` + +--- + +## Compute Configuration + +### Serverless vs Classic + +| Aspect | Serverless | Classic | +|--------|-----------|---------| +| Startup | Fast (seconds) | Slower (minutes) | +| Scaling | Automatic, instant | Manual/autoscaling | +| Cost | Pay-per-use | Pay for cluster time | +| Best for | Variable workloads, dev/test | Steady workloads | + +### Serverless (Recommended) + +Enable at pipeline level: + +```yaml +execution_mode: continuous # or triggered +serverless: true +``` + +**Advantages**: No cluster management, instant scaling, lower cost for bursty workloads. + +--- + +## Complete Example + +```python +from pyspark import pipelines as dp +from pyspark.sql import functions as F + +# Bronze: Optimized ingestion +@dp.table( + name="bronze_orders", + cluster_by=["order_date"], + table_properties={ + "delta.autoOptimize.optimizeWrite": "true", + "delta.autoOptimize.autoCompact": "true" + } +) +def bronze_orders(): + return ( + spark.readStream.format("cloudFiles") + .option("cloudFiles.format", "json") + .load("/Volumes/my_catalog/my_schema/raw/orders/") + .withColumn("_ingested_at", F.current_timestamp()) + .withColumn("order_date", F.to_date("order_timestamp")) + ) + +# Silver: Efficient clustering for joins +@dp.table( + name="silver_orders", + cluster_by=["customer_id", "order_date"] +) +@dp.expect_or_drop("valid_amount", "amount > 0") +def silver_orders(): + return ( + spark.readStream.table("bronze_orders") + .filter(F.col("order_date") >= F.current_date() - 90) # Filter early + .withColumn("amount", F.col("amount").cast("decimal(10,2)")) # DECIMAL for monetary + .select("order_id", "customer_id", "amount", "order_date") # Select specific + ) + +# Gold: Pre-aggregated for dashboards +@dp.materialized_view( + name="gold_daily_revenue", + cluster_by=["order_date"] +) +def gold_daily_revenue(): + return ( + spark.read.table("silver_orders") + .groupBy("order_date") + .agg( + F.sum("amount").alias("total_revenue"), + F.count("order_id").alias("order_count"), + F.countDistinct("customer_id").alias("unique_customers") + ) + ) +``` + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| Pipeline running slowly | Check clustering, state size, join patterns | +| High memory usage | Unbounded state - add time windows, reduce cardinality | +| Many small files | Enable auto-optimize table properties | +| Expensive queries on large tables | Add clustering, create filtered MVs | +| MV refresh slow | Enable row tracking on source | diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/sql/1-syntax-basics.md b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/1-syntax-basics.md new file mode 100644 index 00000000..54e45df4 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/1-syntax-basics.md @@ -0,0 +1,243 @@ +# SQL Syntax Basics + +Core SQL syntax for Spark Declarative Pipelines (SDP). + +--- + +## Table Types + +### Streaming Table + +Processes data incrementally. Use for continuous ingestion and transformations. + +```sql +CREATE OR REFRESH STREAMING TABLE bronze_events +COMMENT 'Raw event data' +CLUSTER BY (event_type, event_date) +TBLPROPERTIES ( + 'delta.autoOptimize.optimizeWrite' = 'true', + 'delta.autoOptimize.autoCompact' = 'true' +) +AS +SELECT + *, + current_timestamp() AS _ingested_at, + _metadata.file_path AS _source_file +FROM STREAM read_files('/Volumes/my_catalog/my_schema/raw/events/', format => 'json'); +``` + +**Key points:** +- Use `STREAM` keyword with source for incremental processing +- `CLUSTER BY` enables Liquid Clustering (recommended over PARTITION BY) +- Returns streaming DataFrame + +### Materialized View + +Batch table with automatic incremental refresh. + +```sql +CREATE OR REFRESH MATERIALIZED VIEW gold_daily_summary +COMMENT 'Daily aggregated metrics' +CLUSTER BY (report_date) +AS +SELECT + report_date, + SUM(amount) AS total_amount, + COUNT(*) AS transaction_count +FROM silver_orders +GROUP BY report_date; +``` + +**Key points:** +- No `STREAM` keyword - reads batch +- Automatically refreshes incrementally when source changes +- Use for aggregations and reporting tables + +### View (Persisted) + +A regular view published to Unity Catalog. Unlike materialized views, it doesn't store data - the query runs each time the view is accessed. + +```sql +CREATE VIEW taxi_raw AS +SELECT * FROM read_files("/Volumes/catalog/schema/raw/taxi/"); + +CREATE VIEW active_customers AS +SELECT customer_id, name, email +FROM dim_customers +WHERE status = 'active'; +``` + +**Key points:** +- Persisted in Unity Catalog (visible outside pipeline) +- No data storage - query executes on access +- Cannot use streaming queries or constraints +- Requires Unity Catalog pipeline with default publishing mode + +**Documentation:** [CREATE VIEW reference](https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-view) + +### Temporary View + +Pipeline-scoped view, not persisted. Useful for intermediate transformations. + +```sql +CREATE TEMPORARY VIEW orders_with_calculations AS +SELECT + *, + quantity * price AS total, + quantity * price * discount_rate AS discount_amount +FROM STREAM bronze_orders +WHERE quantity > 0; +``` + +**Key points:** +- Exists only during pipeline execution +- No storage cost +- Not visible outside pipeline +- Useful before AUTO CDC flows + +### Choosing Between View Types + +| Type | Persisted | Stores Data | Streaming | Use Case | +|------|-----------|-------------|-----------|----------| +| **Materialized View** | Yes | Yes | No | Aggregations, reporting tables | +| **View** | Yes | No | No | Simple transformations, external access | +| **Temporary View** | No | No | Yes | Intermediate steps, before AUTO CDC | + +--- + +## Data Quality (Expectations) +**Documentation:** [Expectations]https://docs.databricks.com/aws/en/ldp/expectations) + +### Constraint Syntax + +```sql +CREATE OR REFRESH STREAMING TABLE silver_orders ( + CONSTRAINT valid_amount EXPECT (amount > 0) ON VIOLATION DROP ROW, + CONSTRAINT valid_customer EXPECT (customer_id IS NOT NULL) ON VIOLATION DROP ROW, + CONSTRAINT critical_field EXPECT (order_id IS NOT NULL) ON VIOLATION FAIL UPDATE +) +AS +SELECT * FROM STREAM bronze_orders; +``` + +| Violation Action | Behavior | +|-----------------|----------| +| `ON VIOLATION DROP ROW` | Drop rows that violate | +| `ON VIOLATION FAIL UPDATE` | Fail pipeline if any row violates | +| (no action) | Log warning, keep all rows | + +### WHERE Clause Alternative + +For simple filtering without tracking: + +```sql +CREATE OR REFRESH STREAMING TABLE silver_orders AS +SELECT * FROM STREAM bronze_orders +WHERE amount > 0 AND customer_id IS NOT NULL; +``` + +--- + +## Liquid Clustering + +Use `CLUSTER BY` instead of legacy `PARTITION BY`. See **[5-performance.md](5-performance.md#liquid-clustering-recommended)** for detailed guidance on key selection by layer. + +```sql +CREATE OR REFRESH STREAMING TABLE bronze_events +CLUSTER BY (event_type, event_date) +AS SELECT ...; +``` + +--- + +## Table Properties + +```sql +CREATE OR REFRESH STREAMING TABLE bronze_events +TBLPROPERTIES ( + 'delta.autoOptimize.optimizeWrite' = 'true', -- Optimize file sizes on write + 'delta.autoOptimize.autoCompact' = 'true', -- Automatic compaction + 'delta.enableChangeDataFeed' = 'true', -- Enable CDF for downstream + 'delta.logRetentionDuration' = '7 days', -- Log retention + 'delta.deletedFileRetentionDuration' = '7 days' -- Deleted file retention +) +AS SELECT ...; +``` + +--- + +## Refresh Scheduling (Materialized Views) + +```sql +-- Near-real-time +CREATE OR REFRESH MATERIALIZED VIEW gold_live_metrics +REFRESH EVERY 5 MINUTES +AS SELECT ...; + +-- Daily +CREATE OR REFRESH MATERIALIZED VIEW gold_daily_summary +REFRESH EVERY 1 DAY +AS SELECT ...; +``` + +--- + +## Table Name Resolution + +| Level | Example | When to Use | +|-------|---------|-------------| +| Unqualified | `FROM bronze_orders` | Tables in same pipeline (recommended) | +| Schema-qualified | `FROM other_schema.orders` | Different schema, same catalog | +| Fully-qualified | `FROM other_catalog.schema.orders` | External catalogs | + +**Best practice:** Use unqualified names for pipeline-internal tables. + +### Multi-Schema Pattern (One Pipeline) + +Write to multiple schemas from a single pipeline using fully qualified names: + +```sql +-- bronze_orders.sql → writes to bronze schema +CREATE OR REFRESH STREAMING TABLE my_catalog.bronze.raw_orders +AS SELECT *, current_timestamp() AS _ingested_at +FROM STREAM read_files('/Volumes/my_catalog/raw/orders/', format => 'json'); + +-- silver_orders.sql → writes to silver schema, reads from bronze +CREATE OR REFRESH STREAMING TABLE my_catalog.silver.clean_orders +AS SELECT * FROM STREAM my_catalog.bronze.raw_orders +WHERE order_id IS NOT NULL; +``` + +--- + +## Pipeline Parameters + +Reference configuration values in SQL: + +```sql +-- In SQL, use ${variable_name} syntax +CREATE OR REFRESH STREAMING TABLE bronze_orders AS +SELECT * FROM STREAM read_files( + '${input_path}/orders/', + format => 'json' +); +``` + +Define in pipeline configuration (YAML): +```yaml +configuration: + input_path: /Volumes/my_catalog/my_schema/raw +``` + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| Missing `STREAM` keyword | Use `FROM STREAM table_name` for streaming tables | +| Constraint syntax error | Use `CONSTRAINT name EXPECT (condition)` | +| Cluster key not working | Verify column exists, limit to 4 keys | +| Parameter not resolved | Check `${var}` syntax and pipeline configuration | +| Using legacy `LIVE` keyword | Use `CREATE OR REFRESH STREAMING TABLE` \| `MATERIALIZED VIEW`, not `CREATE LIVE TABLE` \| `STREAMING LIVE TABLE` | +| Using `input_file_name()` | Use `_metadata.file_path` | diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/sql/2-ingestion.md b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/2-ingestion.md new file mode 100644 index 00000000..61f98f69 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/2-ingestion.md @@ -0,0 +1,161 @@ +# SQL Data Ingestion + +Data ingestion patterns for cloud storage and streaming sources. + +**Official Documentation:** +- [read_files function reference](https://docs.databricks.com/aws/en/sql/language-manual/functions/read_files) +- [Auto Loader options](https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options) + +--- + +## Auto Loader (Cloud Files) + +Auto Loader incrementally processes new files. Use `STREAM read_files()` in streaming table queries. + +### Basic Pattern + +```sql +CREATE OR REFRESH STREAMING TABLE bronze_orders AS +SELECT + *, + current_timestamp() AS _ingested_at, + _metadata.file_path AS _source_file +FROM STREAM read_files( + '/Volumes/my_catalog/my_schema/raw/orders/', + format => 'json', + schemaHints => 'order_id STRING, amount DECIMAL(10,2)' +); +``` + +**Key points:** +- Use `FROM STREAM read_files(...)` for streaming tables (not `FROM read_files(...)` which is batch) +- `format` supports: `json`, `csv`, `parquet`, `avro`, `text`, `binaryFile` +- `schemaHints` recommended for production to prevent schema drift +- `_metadata` provides file path, modification time, size + +### Schema Handling + +```sql +-- Explicit hints (recommended for production) +FROM STREAM read_files( + '/Volumes/catalog/schema/raw/', + format => 'json', + schemaHints => 'id STRING, amount DECIMAL(10,2), date DATE' +) + +-- Schema evolution with rescue data +FROM STREAM read_files( + '/Volumes/catalog/schema/raw/', + format => 'json', + schemaHints => 'id STRING', + mode => 'PERMISSIVE' +) +``` + +### Rescue Data (Quarantine Pattern) + +Handle malformed records: + +```sql +-- Flag records with parsing errors +CREATE OR REFRESH STREAMING TABLE bronze_events AS +SELECT + *, + current_timestamp() AS _ingested_at, + CASE WHEN _rescued_data IS NOT NULL THEN TRUE ELSE FALSE END AS _has_errors +FROM STREAM read_files('/Volumes/catalog/schema/raw/events/', format => 'json'); + +-- Quarantine bad records +CREATE OR REFRESH STREAMING TABLE bronze_quarantine AS +SELECT * FROM STREAM bronze_events WHERE _rescued_data IS NOT NULL; + +-- Clean records for downstream +CREATE OR REFRESH STREAMING TABLE silver_clean AS +SELECT * FROM STREAM bronze_events WHERE _rescued_data IS NULL; +``` + +--- + +## Streaming Sources + +### Kafka + +```sql +CREATE OR REFRESH STREAMING TABLE bronze_kafka_events AS +SELECT + CAST(key AS STRING) AS event_key, + CAST(value AS STRING) AS event_value, + topic, partition, offset, + timestamp AS kafka_timestamp, + current_timestamp() AS _ingested_at +FROM read_kafka( + bootstrapServers => '${kafka_brokers}', + subscribe => 'events-topic', + startingOffsets => 'latest' +); +``` + +**Documentation:** [read_kafka function](https://docs.databricks.com/aws/en/sql/language-manual/functions/read_kafka) + +### Parse JSON from Kafka + +```sql +CREATE OR REFRESH STREAMING TABLE silver_events AS +SELECT + from_json(event_value, 'event_id STRING, event_type STRING, timestamp TIMESTAMP') AS data, + kafka_timestamp, _ingested_at +FROM STREAM bronze_kafka_events; +``` + +--- + +## Authentication + +### Databricks Secrets + +```sql +-- Kafka +`kafka.sasl.jaas.config` => '...username="{{secrets/kafka/username}}" password="{{secrets/kafka/password}}";' + +-- Event Hub +`eventhubs.connectionString` => '{{secrets/eventhub/connection-string}}' +``` + +### Pipeline Variables + +```sql +-- Reference in SQL +FROM STREAM read_files('${input_path}/orders/', format => 'json') +``` + +Define in pipeline configuration: +```yaml +configuration: + input_path: /Volumes/my_catalog/my_schema/raw +``` + +--- + +## Best Practices + +1. **Always add ingestion metadata:** +```sql +SELECT *, current_timestamp() AS _ingested_at, _metadata.file_path AS _source_file +``` + +2. **Use schemaHints for production** - prevents unexpected schema changes + +3. **Handle rescue data** - route malformed records to quarantine table + +4. **Use STREAM keyword** - `FROM STREAM read_files(...)` for streaming tables + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| Files not picked up | Verify path and format match actual files | +| "Cannot create streaming table from batch query" | Use `FROM STREAM read_files(...)` not `FROM read_files(...)` | +| Schema evolution breaking | Use `mode => 'PERMISSIVE'` and monitor `_rescued_data` | +| Kafka lag increasing | Check downstream bottlenecks | diff --git a/.claude/skills/spark-declarative-pipelines/2-streaming-patterns.md b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md similarity index 72% rename from .claude/skills/spark-declarative-pipelines/2-streaming-patterns.md rename to .claude/skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md index 1f87076a..fc427025 100644 --- a/.claude/skills/spark-declarative-pipelines/2-streaming-patterns.md +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md @@ -1,4 +1,4 @@ -# Streaming Patterns for SDP +# SQL Streaming Patterns Streaming-specific patterns including deduplication, windowed aggregations, late-arriving data handling, and stateful operations. @@ -10,12 +10,12 @@ Streaming-specific patterns including deduplication, windowed aggregations, late ```sql -- Bronze: Ingest all (may contain duplicates) -CREATE OR REPLACE STREAMING TABLE bronze_events AS +CREATE OR REFRESH STREAMING TABLE bronze_events AS SELECT *, current_timestamp() AS _ingested_at -FROM read_stream(...); +FROM STREAM read_files(...); -- Silver: Deduplicate by event_id -CREATE OR REPLACE STREAMING TABLE silver_events_dedup AS +CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS SELECT event_id, user_id, event_type, event_timestamp, _ingested_at FROM ( @@ -32,21 +32,21 @@ WHERE rn = 1; Deduplicate within time window to handle late arrivals: ```sql -CREATE OR REPLACE STREAMING TABLE silver_events_dedup AS +CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS SELECT event_id, user_id, event_type, event_timestamp, MIN(_ingested_at) AS first_seen_at FROM STREAM bronze_events GROUP BY event_id, user_id, event_type, event_timestamp, - window(event_timestamp, '1 hour') -- Deduplicate within 1-hour windows + window(event_timestamp, '1 hour') HAVING COUNT(*) >= 1; ``` ### Composite Key ```sql -CREATE OR REPLACE STREAMING TABLE silver_transactions_dedup AS +CREATE OR REFRESH STREAMING TABLE silver_transactions_dedup AS SELECT transaction_id, customer_id, amount, transaction_timestamp, MIN(_ingested_at) AS _ingested_at @@ -60,9 +60,11 @@ GROUP BY transaction_id, customer_id, amount, transaction_timestamp; ### Tumbling Windows +Non-overlapping fixed-size windows: + ```sql --- 5-minute non-overlapping windows -CREATE OR REPLACE STREAMING TABLE silver_sensor_5min AS +-- 5-minute windows +CREATE OR REFRESH STREAMING TABLE silver_sensor_5min AS SELECT sensor_id, window(event_timestamp, '5 minutes') AS time_window, @@ -78,7 +80,7 @@ GROUP BY sensor_id, window(event_timestamp, '5 minutes'); ```sql -- 1-minute for real-time monitoring -CREATE OR REPLACE STREAMING TABLE gold_sensor_1min AS +CREATE OR REFRESH STREAMING TABLE gold_sensor_1min AS SELECT sensor_id, window(event_timestamp, '1 minute').start AS window_start, @@ -89,7 +91,7 @@ FROM STREAM silver_sensor_data GROUP BY sensor_id, window(event_timestamp, '1 minute'); -- 1-hour for trend analysis -CREATE OR REPLACE STREAMING TABLE gold_sensor_1hour AS +CREATE OR REFRESH STREAMING TABLE gold_sensor_1hour AS SELECT sensor_id, window(event_timestamp, '1 hour').start AS window_start, @@ -99,25 +101,35 @@ FROM STREAM silver_sensor_data GROUP BY sensor_id, window(event_timestamp, '1 hour'); ``` +### Session Windows + +Group events into sessions based on inactivity gaps: + +```sql +-- 30-minute inactivity timeout +CREATE OR REFRESH STREAMING TABLE silver_user_sessions AS +SELECT + user_id, + session_window(event_timestamp, '30 minutes') AS session, + MIN(event_timestamp) AS session_start, + MAX(event_timestamp) AS session_end, + COUNT(*) AS event_count, + COLLECT_LIST(event_type) AS event_sequence +FROM STREAM bronze_user_events +GROUP BY user_id, session_window(event_timestamp, '30 minutes'); +``` + --- ## Late-Arriving Data ### Event-Time vs Processing-Time -Always use event timestamp for business logic, not ingestion timestamp: +Always use event timestamp for business logic: ```sql --- ✅ Use event timestamp -CREATE OR REPLACE STREAMING TABLE silver_orders AS -SELECT - order_id, order_timestamp, -- Event time from source - customer_id, amount, - _ingested_at -- Processing time (debugging only) -FROM STREAM bronze_orders; - --- Group by event time -CREATE OR REPLACE STREAMING TABLE gold_daily_orders AS +-- Use event timestamp for aggregations +CREATE OR REFRESH STREAMING TABLE gold_daily_orders AS SELECT CAST(order_timestamp AS DATE) AS order_date, -- Event time COUNT(*) AS order_count, @@ -126,33 +138,23 @@ FROM STREAM silver_orders GROUP BY CAST(order_timestamp AS DATE); ``` -### Handling Out-of-Order with SCD2 - -Use SEQUENCE BY with event timestamp: - +**Keep processing time for debugging:** ```sql -CREATE OR REFRESH STREAMING TABLE silver_customers_history; - -CREATE FLOW customers_scd2_flow AS -AUTO CDC INTO silver_customers_history -FROM stream(bronze_customer_cdc) -KEYS (customer_id) -SEQUENCE BY event_timestamp -- Handles out-of-order -APPLY AS DELETE WHEN operation = "DELETE" -COLUMNS * EXCEPT (operation, _rescued_data) -STORED AS SCD TYPE 2 -TRACK HISTORY ON *; +SELECT + order_id, order_timestamp, -- Event time (business logic) + customer_id, amount, + _ingested_at -- Processing time (debugging only) +FROM STREAM bronze_orders; ``` --- -## Stateful Operations +## Joins ### Stream-to-Stream Joins ```sql --- Join two streaming sources -CREATE OR REPLACE STREAMING TABLE silver_orders_with_payments AS +CREATE OR REFRESH STREAMING TABLE silver_orders_with_payments AS SELECT o.order_id, o.customer_id, o.order_timestamp, o.amount AS order_amount, p.payment_id, p.payment_timestamp, p.payment_method, p.amount AS payment_amount @@ -162,17 +164,19 @@ INNER JOIN STREAM bronze_payments p AND p.payment_timestamp BETWEEN o.order_timestamp AND o.order_timestamp + INTERVAL 1 HOUR; ``` +**Important:** Use time bounds in join condition to limit state retention. + ### Stream-to-Static Joins Enrich streaming data with dimension tables: ```sql --- Static dimension (changes infrequently) +-- Static dimension CREATE OR REPLACE TABLE dim_products AS SELECT * FROM catalog.schema.products; -- Stream-to-static join -CREATE OR REPLACE STREAMING TABLE silver_sales_enriched AS +CREATE OR REFRESH STREAMING TABLE silver_sales_enriched AS SELECT s.sale_id, s.product_id, s.quantity, s.sale_timestamp, p.product_name, p.category, p.price, @@ -181,11 +185,14 @@ FROM STREAM bronze_sales s LEFT JOIN dim_products p ON s.product_id = p.product_id; ``` -### Incremental Aggregations +--- + +## Incremental Aggregations + +### Running Totals ```sql --- Running totals by customer (stateful) -CREATE OR REPLACE STREAMING TABLE silver_customer_running_totals AS +CREATE OR REFRESH STREAMING TABLE silver_customer_running_totals AS SELECT customer_id, SUM(amount) AS total_spent, @@ -197,32 +204,12 @@ GROUP BY customer_id; --- -## Session Windows - -Group events into sessions based on inactivity gaps: - -```sql --- 30-minute inactivity timeout -CREATE OR REPLACE STREAMING TABLE silver_user_sessions AS -SELECT - user_id, - session_window(event_timestamp, '30 minutes') AS session, - MIN(event_timestamp) AS session_start, - MAX(event_timestamp) AS session_end, - COUNT(*) AS event_count, - COLLECT_LIST(event_type) AS event_sequence -FROM STREAM bronze_user_events -GROUP BY user_id, session_window(event_timestamp, '30 minutes'); -``` - ---- - ## Anomaly Detection ### Real-Time Outlier Detection ```sql -CREATE OR REPLACE STREAMING TABLE silver_sensor_with_anomalies AS +CREATE OR REFRESH STREAMING TABLE silver_sensor_with_anomalies AS SELECT sensor_id, event_timestamp, temperature, AVG(temperature) OVER ( @@ -241,7 +228,7 @@ SELECT FROM STREAM bronze_sensor_events; -- Route anomalies for alerting -CREATE OR REPLACE STREAMING TABLE silver_sensor_anomalies AS +CREATE OR REFRESH STREAMING TABLE silver_sensor_anomalies AS SELECT * FROM STREAM silver_sensor_with_anomalies WHERE anomaly_flag IN ('HIGH_OUTLIER', 'LOW_OUTLIER'); @@ -250,7 +237,7 @@ WHERE anomaly_flag IN ('HIGH_OUTLIER', 'LOW_OUTLIER'); ### Threshold-Based Filtering ```sql -CREATE OR REPLACE STREAMING TABLE silver_high_value_transactions AS +CREATE OR REFRESH STREAMING TABLE silver_high_value_transactions AS SELECT transaction_id, customer_id, amount, transaction_timestamp FROM STREAM bronze_transactions WHERE amount > 10000; @@ -258,38 +245,51 @@ WHERE amount > 10000; --- +## Monitoring Lag + +```sql +CREATE OR REFRESH STREAMING TABLE monitoring_lag AS +SELECT + 'kafka_events' AS source, + MAX(kafka_timestamp) AS max_event_timestamp, + current_timestamp() AS processing_timestamp, + (unix_timestamp(current_timestamp()) - unix_timestamp(MAX(kafka_timestamp))) AS lag_seconds +FROM STREAM bronze_kafka_events +GROUP BY window(kafka_timestamp, '1 minute'); +``` + +--- + ## Execution Modes Configure at pipeline level (not in SQL): -**Continuous** (real-time, sub-second latency): ```yaml +# Continuous (real-time, sub-second latency) execution_mode: continuous serverless: true -``` -**Triggered** (scheduled, cost-optimized): -```yaml +# Triggered (scheduled, cost-optimized) execution_mode: triggered schedule: "0 * * * *" # Hourly ``` -**When to use**: +**When to use:** - **Continuous**: Real-time dashboards, alerting, sub-minute SLAs - **Triggered**: Daily/hourly reports, batch processing --- -## Key Patterns +## Best Practices ### 1. Use Event Timestamps ```sql --- ✅ Event timestamp for logic +-- Correct: Event timestamp for logic GROUP BY date_trunc('hour', event_timestamp) --- ❌ Processing timestamp -GROUP BY date_trunc('hour', _ingested_at) +-- Avoid: Processing timestamp +-- GROUP BY date_trunc('hour', _ingested_at) ``` ### 2. Window Size Selection @@ -303,10 +303,10 @@ GROUP BY date_trunc('hour', _ingested_at) Higher cardinality = more state: ```sql --- High state: 1M users × 10K products × 100M sessions +-- High state: 1M users x 10K products x 100M sessions GROUP BY user_id, product_id, session_id --- Lower state: 1M users × 100 categories × days +-- Lower state: 1M users x 100 categories x days GROUP BY user_id, product_category, DATE(event_time) ``` @@ -318,32 +318,19 @@ Apply at bronze → silver transition: ```sql -- Bronze: Accept duplicates -CREATE OR REPLACE STREAMING TABLE bronze_events AS -SELECT * FROM read_stream(...); +CREATE OR REFRESH STREAMING TABLE bronze_events AS +SELECT * FROM STREAM read_files(...); -- Silver: Deduplicate immediately -CREATE OR REPLACE STREAMING TABLE silver_events AS +CREATE OR REFRESH STREAMING TABLE silver_events AS SELECT DISTINCT event_id, event_type, event_timestamp, user_id FROM STREAM bronze_events; -- Gold: Work with clean data -CREATE OR REPLACE STREAMING TABLE gold_metrics AS +CREATE OR REFRESH STREAMING TABLE gold_metrics AS SELECT ... FROM STREAM silver_events; ``` -### 5. Monitor Lag - -```sql -CREATE OR REPLACE STREAMING TABLE monitoring_lag AS -SELECT - 'kafka_events' AS source, - MAX(kafka_timestamp) AS max_event_timestamp, - current_timestamp() AS processing_timestamp, - (unix_timestamp(current_timestamp()) - unix_timestamp(MAX(kafka_timestamp))) AS lag_seconds -FROM STREAM bronze_kafka_events -GROUP BY window(kafka_timestamp, '1 minute'); -``` - --- ## Common Issues diff --git a/.claude/skills/databricks-spark-declarative-pipelines/references/sql/4-cdc-patterns.md b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/4-cdc-patterns.md new file mode 100644 index 00000000..d9977c23 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/4-cdc-patterns.md @@ -0,0 +1,323 @@ +# SQL CDC Patterns (AUTO CDC & SCD) + +Change Data Capture patterns using AUTO CDC for SCD Type 1 and Type 2, plus querying SCD history tables. + +--- + +## Overview + +AUTO CDC automatically handles Change Data Capture to track changes using Slow Changing Dimensions (SCD). It provides automatic deduplication, change tracking, and handles late-arriving data correctly. + +**Where to apply AUTO CDC:** +- **Silver layer**: When business users need deduplicated or historical data +- **Gold layer**: When implementing dimensional modeling (star schema) + +--- + +## SCD Type 1 vs Type 2 + +### SCD Type 1 (In-place updates) +- **Overwrites** old values with new values +- **No history preserved** - only current state +- **Use for**: Error corrections, attributes where history doesn't matter +- **Syntax**: `STORED AS SCD TYPE 1` + +### SCD Type 2 (History tracking) +- **Creates new row** for each change +- **Preserves full history** with `__START_AT` and `__END_AT` timestamps +- **Use for**: Tracking changes over time (addresses, prices, roles) +- **Syntax**: `STORED AS SCD TYPE 2` + +--- + +## Creating AUTO CDC Flows + +### SCD Type 2 + +```sql +-- Step 1: Create target table +CREATE OR REFRESH STREAMING TABLE dim_customers; + +-- Step 2: Create AUTO CDC flow +CREATE FLOW customers_scd2_flow AS +AUTO CDC INTO dim_customers +FROM stream(customers_cdc_clean) +KEYS (customer_id) +APPLY AS DELETE WHEN operation = "DELETE" +SEQUENCE BY event_timestamp +COLUMNS * EXCEPT (operation, _ingested_at, _source_file) +STORED AS SCD TYPE 2; +``` + +**Important:** Put `APPLY AS DELETE WHEN` before `SEQUENCE BY`. Only list columns in `COLUMNS * EXCEPT (...)` that exist in the source. + +### SCD Type 1 + +```sql +-- Step 1: Create target table +CREATE OR REFRESH STREAMING TABLE orders_current; + +-- Step 2: Create AUTO CDC flow +CREATE FLOW orders_scd1_flow AS +AUTO CDC INTO orders_current +FROM stream(orders_clean) +KEYS (order_id) +SEQUENCE BY updated_timestamp +COLUMNS * EXCEPT (_ingested_at) +STORED AS SCD TYPE 1; +``` + +### Selective History Tracking + +Track history only when specific columns change: + +```sql +CREATE FLOW products_scd2_flow AS +AUTO CDC INTO products_history +FROM stream(products_clean) +KEYS (product_id) +SEQUENCE BY modified_at +COLUMNS * EXCEPT (operation) +STORED AS SCD TYPE 2 +TRACK HISTORY ON price, cost; +``` + +When `price` or `cost` changes, a new version is created. Other column changes update the current record without new versions. + +--- + +## Complete Pattern: Clean + AUTO CDC + +### Step 1: Clean and Validate Source Data + +```sql +CREATE OR REFRESH STREAMING TABLE customers_cdc_clean AS +SELECT + customer_id, + customer_name, + email, + phone, + address, + CAST(updated_at AS TIMESTAMP) AS event_timestamp, + operation +FROM STREAM bronze_customers_cdc +WHERE customer_id IS NOT NULL + AND email IS NOT NULL; +``` + +### Step 2: Apply AUTO CDC + +```sql +CREATE OR REFRESH STREAMING TABLE dim_customers; + +CREATE FLOW customers_scd2_flow AS +AUTO CDC INTO dim_customers +FROM stream(customers_cdc_clean) +KEYS (customer_id) +APPLY AS DELETE WHEN operation = "DELETE" +SEQUENCE BY event_timestamp +COLUMNS * EXCEPT (operation) +STORED AS SCD TYPE 2; +``` + +--- + +## Querying SCD Type 2 Tables + +SCD Type 2 tables include temporal columns: +- `__START_AT` - When this version became effective +- `__END_AT` - When this version expired (NULL for current) + +### Current State + +```sql +-- All current records +CREATE OR REFRESH MATERIALIZED VIEW dim_customers_current AS +SELECT + customer_id, customer_name, email, phone, address, + __START_AT AS valid_from +FROM dim_customers +WHERE __END_AT IS NULL; + +-- Specific customer +SELECT * +FROM dim_customers +WHERE customer_id = '12345' + AND __END_AT IS NULL; +``` + +### Point-in-Time Queries + +Get state as of a specific date: + +```sql +-- Products as of January 1, 2024 +CREATE OR REFRESH MATERIALIZED VIEW products_as_of_2024_01_01 AS +SELECT + product_id, product_name, price, category, + __START_AT, __END_AT +FROM products_history +WHERE __START_AT <= '2024-01-01' + AND (__END_AT > '2024-01-01' OR __END_AT IS NULL); +``` + +### Change Analysis + +Track all changes for an entity: + +```sql +SELECT + customer_id, customer_name, email, phone, + __START_AT, __END_AT, + COALESCE( + DATEDIFF(DAY, __START_AT, __END_AT), + DATEDIFF(DAY, __START_AT, CURRENT_TIMESTAMP()) + ) AS days_active +FROM dim_customers +WHERE customer_id = '12345' +ORDER BY __START_AT DESC; +``` + +Changes within a time period: + +```sql +-- Customers who changed during Q1 2024 +SELECT + customer_id, customer_name, + __START_AT AS change_timestamp, + 'UPDATE' AS change_type +FROM dim_customers +WHERE __START_AT BETWEEN '2024-01-01' AND '2024-03-31' + AND __START_AT != ( + SELECT MIN(__START_AT) + FROM dim_customers ch2 + WHERE ch2.customer_id = dim_customers.customer_id + ) +ORDER BY __START_AT; +``` + +--- + +## Joining Facts with Historical Dimensions + +### At Transaction Time + +```sql +-- Join sales with product prices at time of sale +CREATE OR REFRESH MATERIALIZED VIEW sales_with_historical_prices AS +SELECT + s.sale_id, s.product_id, s.sale_date, s.quantity, + p.product_name, p.price AS unit_price_at_sale_time, + s.quantity * p.price AS calculated_amount, + p.category +FROM sales_fact s +INNER JOIN products_history p + ON s.product_id = p.product_id + AND s.sale_date >= p.__START_AT + AND (s.sale_date < p.__END_AT OR p.__END_AT IS NULL); +``` + +### With Current Dimension + +```sql +CREATE OR REFRESH MATERIALIZED VIEW sales_with_current_prices AS +SELECT + s.sale_id, s.product_id, s.sale_date, s.quantity, + s.amount AS amount_at_sale, + p.product_name AS current_product_name, + p.price AS current_price +FROM sales_fact s +INNER JOIN products_history p + ON s.product_id = p.product_id + AND p.__END_AT IS NULL; +``` + +--- + +## Optimization Patterns + +### Pre-Filter Materialized Views + +```sql +-- Current state view (most common pattern) +CREATE OR REFRESH MATERIALIZED VIEW dim_products_current AS +SELECT * FROM products_history WHERE __END_AT IS NULL; + +-- Recent changes only +CREATE OR REFRESH MATERIALIZED VIEW dim_recent_changes AS +SELECT * FROM products_history +WHERE __START_AT >= CURRENT_DATE() - INTERVAL 90 DAYS; + +-- Change frequency stats +CREATE OR REFRESH MATERIALIZED VIEW product_change_stats AS +SELECT + product_id, + COUNT(*) AS version_count, + MIN(__START_AT) AS first_seen, + MAX(__START_AT) AS last_updated +FROM products_history +GROUP BY product_id; +``` + +--- + +## Best Practices + +### 1. Filter by __END_AT for Current + +```sql +-- Efficient +WHERE __END_AT IS NULL + +-- Less efficient +WHERE __START_AT = (SELECT MAX(__START_AT) FROM table WHERE ...) +``` + +### 2. Use Inclusive Lower, Exclusive Upper + +```sql +WHERE __START_AT <= '2024-01-01' + AND (__END_AT > '2024-01-01' OR __END_AT IS NULL) +``` + +### 3. Clean Data Before AUTO CDC + +Apply type casting, validation, and filtering first: + +```sql +-- Clean source +CREATE OR REFRESH STREAMING TABLE users_clean AS +SELECT + user_id, + TRIM(email) AS email, + CAST(updated_at AS TIMESTAMP) AS updated_timestamp +FROM STREAM bronze_users +WHERE user_id IS NOT NULL AND email IS NOT NULL; + +-- Then apply AUTO CDC +CREATE FLOW users_scd2_flow AS +AUTO CDC INTO dim_users +FROM stream(users_clean) +KEYS (user_id) +SEQUENCE BY updated_timestamp +STORED AS SCD TYPE 2; +``` + +### 4. Choose the Right SCD Type + +- **Type 2**: Need to query historical states +- **Type 1**: Only need current state or deduplication + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| Multiple rows for same key | Missing `__END_AT IS NULL` filter for current state | +| Point-in-time no results | Use `__START_AT <= date AND (__END_AT > date OR __END_AT IS NULL)` | +| Slow temporal join | Create materialized view for specific time period | +| Unexpected duplicates | Multiple changes same day - use SEQUENCE BY with high precision | +| Parse error on AUTO CDC | Put `APPLY AS DELETE WHEN` before `SEQUENCE BY` | +| Columns not in target | Only list existing columns in `COLUMNS * EXCEPT (...)` | +| Type syntax error | Use `SCD TYPE 1` or `SCD TYPE 2` (not quoted) | diff --git a/.claude/skills/spark-declarative-pipelines/4-performance-tuning.md b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/5-performance.md similarity index 72% rename from .claude/skills/spark-declarative-pipelines/4-performance-tuning.md rename to .claude/skills/databricks-spark-declarative-pipelines/references/sql/5-performance.md index bd1c1dc1..aa9ffafb 100644 --- a/.claude/skills/spark-declarative-pipelines/4-performance-tuning.md +++ b/.claude/skills/databricks-spark-declarative-pipelines/references/sql/5-performance.md @@ -1,14 +1,14 @@ -# Performance Tuning for SDP +# SQL Performance Tuning -Performance optimization strategies including **Liquid Clustering** (modern approach), materialized view refresh, state management, and compute configuration. +Performance optimization strategies including Liquid Clustering, materialized view refresh, state management, and compute configuration. --- ## Liquid Clustering (Recommended) -**Liquid Clustering** is the recommended approach for data layout optimization. It replaces manual `PARTITION BY` and `Z-ORDER`. +Liquid Clustering is the recommended approach for data layout optimization. It replaces manual `PARTITION BY` and `Z-ORDER`. -### What is Liquid Clustering? +### Benefits - **Adaptive**: Adjusts to data distribution changes - **Multi-dimensional**: Clusters on multiple columns simultaneously @@ -17,32 +17,22 @@ Performance optimization strategies including **Liquid Clustering** (modern appr ### Basic Syntax -**SQL**: ```sql -CREATE OR REPLACE STREAMING TABLE bronze_events +CREATE OR REFRESH STREAMING TABLE bronze_events CLUSTER BY (event_type, event_date) AS SELECT *, current_timestamp() AS _ingested_at, CAST(current_date() AS DATE) AS event_date -FROM read_files('/mnt/raw/events/', format => 'json'); +FROM STREAM read_files('/Volumes/my_catalog/my_schema/raw/events/', format => 'json'); ``` -**Python**: -```python -from pyspark import pipelines as dp - -@dp.table(cluster_by=["event_type", "event_date"]) -def bronze_events(): - return spark.readStream.format("cloudFiles").load("/data") -``` - -### Automatic Cluster Key Selection +### Automatic Key Selection ```sql -- Let Databricks choose based on query patterns -CREATE OR REPLACE STREAMING TABLE bronze_events +CREATE OR REFRESH STREAMING TABLE bronze_events CLUSTER BY (AUTO) AS SELECT ...; ``` @@ -59,7 +49,7 @@ AS SELECT ...; Cluster by event type + date: ```sql -CREATE OR REPLACE STREAMING TABLE bronze_events +CREATE OR REFRESH STREAMING TABLE bronze_events CLUSTER BY (event_type, ingestion_date) TBLPROPERTIES ('delta.autoOptimize.optimizeWrite' = 'true') AS @@ -67,7 +57,7 @@ SELECT *, current_timestamp() AS _ingested_at, CAST(current_date() AS DATE) AS ingestion_date -FROM read_files('/mnt/raw/events/', format => 'json'); +FROM STREAM read_files('/Volumes/my_catalog/my_schema/raw/events/', format => 'json'); ``` **Why**: Bronze filtered by event type for processing and by date for incremental loads. @@ -77,11 +67,12 @@ FROM read_files('/mnt/raw/events/', format => 'json'); Cluster by primary key + business dimension: ```sql -CREATE OR REPLACE STREAMING TABLE silver_orders +CREATE OR REFRESH STREAMING TABLE silver_orders CLUSTER BY (customer_id, order_date) AS SELECT - order_id, customer_id, product_id, amount, + order_id, customer_id, product_id, + CAST(amount AS DECIMAL(10,2)) AS amount, -- DECIMAL for monetary values CAST(order_timestamp AS DATE) AS order_date, order_timestamp FROM STREAM bronze_orders; @@ -94,7 +85,7 @@ FROM STREAM bronze_orders; Cluster by aggregation dimensions: ```sql -CREATE OR REPLACE MATERIALIZED VIEW gold_sales_summary +CREATE OR REFRESH MATERIALIZED VIEW gold_sales_summary CLUSTER BY (product_category, year_month) AS SELECT @@ -117,7 +108,7 @@ GROUP BY product_category, DATE_FORMAT(order_date, 'yyyy-MM'); | **Silver** | primary_key, business_date | Entity lookups + time ranges | | **Gold** | aggregation_dimensions | Dashboard filters | -**Best practices**: +**Best practices:** - First key: Most selective filter (e.g., customer_id) - Second key: Next common filter (e.g., date) - Order matters: Most selective first @@ -131,7 +122,7 @@ GROUP BY product_category, DATE_FORMAT(order_date, 'yyyy-MM'); ### Before (Legacy) ```sql -CREATE OR REPLACE STREAMING TABLE events +CREATE OR REFRESH STREAMING TABLE events PARTITIONED BY (date DATE) TBLPROPERTIES ('pipelines.autoOptimize.zOrderCols' = 'user_id,event_type') AS SELECT ...; @@ -139,10 +130,10 @@ AS SELECT ...; **Issues**: Fixed keys, small file problem, skewed distribution, manual OPTIMIZE required. -### After (Modern with Liquid Clustering) +### After (Modern) ```sql -CREATE OR REPLACE STREAMING TABLE events +CREATE OR REFRESH STREAMING TABLE events CLUSTER BY (date, user_id, event_type) AS SELECT ...; ``` @@ -157,8 +148,6 @@ AS SELECT ...; 3. **Compatibility**: Older Delta Lake versions (< DBR 13.3) 4. **Existing large tables**: Migration cost outweighs benefits -**Otherwise, prefer Liquid Clustering.** - --- ## Table Properties @@ -166,20 +155,18 @@ AS SELECT ...; ### Auto-Optimize ```sql -CREATE OR REPLACE STREAMING TABLE bronze_events +CREATE OR REFRESH STREAMING TABLE bronze_events TBLPROPERTIES ( 'delta.autoOptimize.optimizeWrite' = 'true', 'delta.autoOptimize.autoCompact' = 'true' ) -AS SELECT * FROM read_files(...); +AS SELECT * FROM STREAM read_files(...); ``` -**Benefits**: Reduces small files, improves reads, automatic compaction. - ### Change Data Feed ```sql -CREATE OR REPLACE STREAMING TABLE silver_customers +CREATE OR REFRESH STREAMING TABLE silver_customers TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true') AS SELECT * FROM STREAM bronze_customers; ``` @@ -189,12 +176,12 @@ AS SELECT * FROM STREAM bronze_customers; ### Retention Periods ```sql -CREATE OR REPLACE STREAMING TABLE bronze_high_volume +CREATE OR REFRESH STREAMING TABLE bronze_high_volume TBLPROPERTIES ( 'delta.logRetentionDuration' = '7 days', 'delta.deletedFileRetentionDuration' = '7 days' ) -AS SELECT * FROM read_files(...); +AS SELECT * FROM STREAM read_files(...); ``` **Use for**: High-volume tables to reduce storage costs. @@ -206,8 +193,8 @@ AS SELECT * FROM read_files(...); ### Refresh Frequency ```sql --- Near-real-time (frequent) -CREATE OR REPLACE MATERIALIZED VIEW gold_live_metrics +-- Near-real-time +CREATE OR REFRESH MATERIALIZED VIEW gold_live_metrics REFRESH EVERY 5 MINUTES AS SELECT @@ -217,8 +204,8 @@ SELECT FROM silver_metrics GROUP BY metric_name; --- Daily reports (scheduled) -CREATE OR REPLACE MATERIALIZED VIEW gold_daily_summary +-- Daily reports +CREATE OR REFRESH MATERIALIZED VIEW gold_daily_summary REFRESH EVERY 1 DAY AS SELECT report_date, SUM(amount) AS total_amount @@ -226,13 +213,12 @@ FROM silver_sales GROUP BY report_date; ``` -### Incremental Refresh (Automatic) +### Incremental Refresh Materialized views auto-use incremental refresh when possible: ```sql --- Refreshes incrementally if source has row tracking -CREATE OR REPLACE MATERIALIZED VIEW gold_aggregates AS +CREATE OR REFRESH MATERIALIZED VIEW gold_aggregates AS SELECT product_id, SUM(quantity) AS total_quantity, @@ -246,8 +232,8 @@ GROUP BY product_id; ### Pre-Aggregation ```sql --- Instead of querying large table repeatedly -CREATE OR REPLACE MATERIALIZED VIEW orders_monthly AS +-- Create pre-aggregated MV for fast queries +CREATE OR REFRESH MATERIALIZED VIEW orders_monthly AS SELECT customer_id, YEAR(order_date) AS year, @@ -282,7 +268,6 @@ GROUP BY user_id, product_id, session_id; -- Massive state! **Strategy 1: Reduce cardinality** ```sql --- Aggregate at higher level SELECT user_id, product_category, -- 100 categories (not 10K products) @@ -295,7 +280,6 @@ GROUP BY user_id, product_category, DATE(event_time); **Strategy 2: Use time windows** ```sql --- Bounded state with windows SELECT user_id, window(event_time, '1 hour') AS time_window, @@ -308,7 +292,7 @@ GROUP BY user_id, window(event_time, '1 hour'); ```sql -- Streaming aggregation (maintains state) -CREATE OR REPLACE STREAMING TABLE user_daily_stats AS +CREATE OR REFRESH STREAMING TABLE user_daily_stats AS SELECT user_id, DATE(event_time) AS event_date, @@ -317,7 +301,7 @@ FROM STREAM bronze_events GROUP BY user_id, DATE(event_time); -- Batch aggregation (no streaming state) -CREATE OR REPLACE MATERIALIZED VIEW user_monthly_stats AS +CREATE OR REFRESH MATERIALIZED VIEW user_monthly_stats AS SELECT user_id, DATE_TRUNC('month', event_date) AS month, @@ -334,10 +318,10 @@ GROUP BY user_id, DATE_TRUNC('month', event_date); ```sql -- Small static dimension, large streaming fact -CREATE OR REPLACE STREAMING TABLE sales_enriched AS +CREATE OR REFRESH STREAMING TABLE sales_enriched AS SELECT s.sale_id, s.product_id, s.amount, - p.product_name, p.category -- From small static table + p.product_name, p.category FROM STREAM bronze_sales s LEFT JOIN dim_products p ON s.product_id = p.product_id; ``` @@ -348,7 +332,7 @@ LEFT JOIN dim_products p ON s.product_id = p.product_id; ```sql -- Time bounds limit state retention -CREATE OR REPLACE STREAMING TABLE orders_with_payments AS +CREATE OR REFRESH STREAMING TABLE orders_with_payments AS SELECT o.order_id, o.amount AS order_amount, p.payment_id, p.amount AS payment_amount @@ -358,32 +342,6 @@ INNER JOIN STREAM bronze_payments p AND p.payment_time BETWEEN o.order_time AND o.order_time + INTERVAL 1 HOUR; ``` -**Optimization**: Use time bounds in join condition. - ---- - -## Compute Configuration - -### Serverless vs Classic - -| Aspect | Serverless | Classic | -|--------|-----------|---------| -| Startup | Fast (seconds) | Slower (minutes) | -| Scaling | Automatic, instant | Manual/autoscaling | -| Cost | Pay-per-use | Pay for cluster time | -| Best for | Variable workloads, dev/test | Steady workloads | - -### Serverless (Recommended) - -Enable at pipeline level: - -```yaml -execution_mode: continuous # or triggered -serverless: true -``` - -**Advantages**: No cluster management, instant scaling, lower cost for bursty workloads. - --- ## Query Optimization @@ -391,47 +349,53 @@ serverless: true ### Filter Early ```sql --- ✅ Filter at source -CREATE OR REPLACE STREAMING TABLE silver_recent AS +-- Filter at source +CREATE OR REFRESH STREAMING TABLE silver_recent AS SELECT * FROM STREAM bronze_events WHERE event_date >= CURRENT_DATE() - INTERVAL 7 DAYS; --- ❌ Filter late -CREATE OR REPLACE STREAMING TABLE silver_all AS -SELECT * FROM STREAM bronze_events; - -CREATE OR REPLACE MATERIALIZED VIEW gold_recent AS -SELECT * FROM silver_all -WHERE event_date >= CURRENT_DATE() - INTERVAL 7 DAYS; +-- Avoid filtering late +-- CREATE OR REFRESH STREAMING TABLE silver_all AS SELECT * FROM STREAM bronze_events; +-- CREATE OR REFRESH MATERIALIZED VIEW gold_recent AS SELECT * FROM silver_all WHERE ...; ``` ### Select Specific Columns ```sql --- ❌ Reads all columns -SELECT * FROM large_table; - --- ✅ Only needed columns +-- Only needed columns SELECT customer_id, order_date, amount FROM large_table; + +-- Avoid SELECT * +-- SELECT * FROM large_table; ``` -### Use GROUP BY Over DISTINCT +--- -```sql --- ❌ Expensive on high-cardinality -SELECT DISTINCT transaction_id FROM huge_table; +## Compute Configuration + +### Serverless vs Classic + +| Aspect | Serverless | Classic | +|--------|-----------|---------| +| Startup | Fast (seconds) | Slower (minutes) | +| Scaling | Automatic, instant | Manual/autoscaling | +| Cost | Pay-per-use | Pay for cluster time | +| Best for | Variable workloads, dev/test | Steady workloads | + +### Serverless (Recommended) --- ✅ Better -SELECT transaction_id, COUNT(*) FROM huge_table GROUP BY transaction_id; +Enable at pipeline level: + +```yaml +execution_mode: continuous # or triggered +serverless: true ``` --- ## Monitoring -Track key metrics: - ```sql -- Data freshness SELECT @@ -455,7 +419,7 @@ GROUP BY table_name; | Issue | Solution | |-------|----------| -| Pipeline running slowly | Check partitioning, state size, join patterns | +| Pipeline running slowly | Check clustering, state size, join patterns | | High memory usage | Unbounded state - add time windows, reduce cardinality | | Many small files | Enable auto-optimize, run OPTIMIZE command | | Expensive queries on large tables | Add clustering, create filtered MVs | diff --git a/.claude/skills/databricks-spark-declarative-pipelines/scripts/exploration_notebook.py b/.claude/skills/databricks-spark-declarative-pipelines/scripts/exploration_notebook.py new file mode 100644 index 00000000..f3f67857 --- /dev/null +++ b/.claude/skills/databricks-spark-declarative-pipelines/scripts/exploration_notebook.py @@ -0,0 +1,81 @@ +# Databricks notebook source +# MAGIC %md +# MAGIC # Data Exploration Notebook +# MAGIC +# MAGIC Explore raw data in Volumes before building pipeline transformations. +# MAGIC +# MAGIC **Note:** Pipeline transformations should use raw `.sql` or `.py` files, NOT notebooks. + +# COMMAND ---------- + +# MAGIC %md +# MAGIC ## 1. Explore Raw Files in Volume +# MAGIC +# MAGIC Query raw parquet/json files directly to understand the data structure. + +# COMMAND ---------- + +# MAGIC %sql +# MAGIC -- Preview raw orders data +# MAGIC SELECT * FROM parquet.`/Volumes/my_catalog/my_schema/raw/orders/` LIMIT 100 + +# COMMAND ---------- + +# MAGIC %sql +# MAGIC -- Check schema and sample values +# MAGIC DESCRIBE SELECT * FROM parquet.`/Volumes/my_catalog/my_schema/raw/orders/` + +# COMMAND ---------- + +# MAGIC %sql +# MAGIC -- Data quality: nulls, distinct values, date range +# MAGIC SELECT +# MAGIC COUNT(*) AS total_rows, +# MAGIC COUNT(order_id) AS non_null_order_id, +# MAGIC COUNT(DISTINCT customer_id) AS unique_customers, +# MAGIC MIN(order_date) AS min_date, +# MAGIC MAX(order_date) AS max_date +# MAGIC FROM parquet.`/Volumes/my_catalog/my_schema/raw/orders/` + +# COMMAND ---------- + +# MAGIC %md +# MAGIC ## 2. Explore Another Raw Source + +# COMMAND ---------- + +# MAGIC %sql +# MAGIC -- Preview raw customers data +# MAGIC SELECT * FROM parquet.`/Volumes/my_catalog/my_schema/raw/customers/` LIMIT 100 + +# COMMAND ---------- + +# MAGIC %md +# MAGIC ## 3. Join Raw Data for Exploration +# MAGIC +# MAGIC Test joins before building the pipeline. + +# COMMAND ---------- + +# MAGIC %sql +# MAGIC -- Join orders with customers to validate keys +# MAGIC SELECT +# MAGIC o.order_id, +# MAGIC o.order_date, +# MAGIC o.amount, +# MAGIC c.customer_name, +# MAGIC c.email +# MAGIC FROM parquet.`/Volumes/my_catalog/my_schema/raw/orders/` o +# MAGIC LEFT JOIN parquet.`/Volumes/my_catalog/my_schema/raw/customers/` c +# MAGIC ON o.customer_id = c.customer_id +# MAGIC LIMIT 100 + +# COMMAND ---------- + +# MAGIC %sql +# MAGIC -- Check for orphan orders (no matching customer) +# MAGIC SELECT COUNT(*) AS orphan_orders +# MAGIC FROM parquet.`/Volumes/my_catalog/my_schema/raw/orders/` o +# MAGIC LEFT JOIN parquet.`/Volumes/my_catalog/my_schema/raw/customers/` c +# MAGIC ON o.customer_id = c.customer_id +# MAGIC WHERE c.customer_id IS NULL diff --git a/.claude/skills/databricks-spark-structured-streaming/SKILL.md b/.claude/skills/databricks-spark-structured-streaming/SKILL.md new file mode 100644 index 00000000..ddb52a06 --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/SKILL.md @@ -0,0 +1,65 @@ +--- +name: databricks-spark-structured-streaming +description: "Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance." +--- + +# Spark Structured Streaming + +Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices. + +## Quick Start + +```python +from pyspark.sql.functions import col, from_json + +# Basic Kafka to Delta streaming +df = (spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", "broker:9092") + .option("subscribe", "topic") + .load() + .select(from_json(col("value").cast("string"), schema).alias("data")) + .select("data.*") +) + +df.writeStream \ + .format("delta") \ + .outputMode("append") \ + .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \ + .trigger(processingTime="30 seconds") \ + .start("/delta/target_table") +``` + +## Core Patterns + +| Pattern | Description | Reference | +|---------|-------------|-----------| +| **Kafka Streaming** | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See [kafka-streaming.md](kafka-streaming.md) | +| **Stream Joins** | Stream-stream joins, stream-static joins | See [stream-stream-joins.md](stream-stream-joins.md), [stream-static-joins.md](stream-static-joins.md) | +| **Multi-Sink Writes** | Write to multiple tables, parallel merges | See [multi-sink-writes.md](multi-sink-writes.md) | +| **Merge Operations** | MERGE performance, parallel merges, optimizations | See [merge-operations.md](merge-operations.md) | + +## Configuration + +| Topic | Description | Reference | +|-------|-------------|-----------| +| **Checkpoints** | Checkpoint management and best practices | See [checkpoint-best-practices.md](checkpoint-best-practices.md) | +| **Stateful Operations** | Watermarks, state stores, RocksDB configuration | See [stateful-operations.md](stateful-operations.md) | +| **Trigger & Cost** | Trigger selection, cost optimization, RTM | See [trigger-and-cost-optimization.md](trigger-and-cost-optimization.md) | + +## Best Practices + +| Topic | Description | Reference | +|-------|-------------|-----------| +| **Production Checklist** | Comprehensive best practices | See [streaming-best-practices.md](streaming-best-practices.md) | + +## Production Checklist + +- [ ] Checkpoint location is persistent (UC volumes, not DBFS) +- [ ] Unique checkpoint per stream +- [ ] Fixed-size cluster (no autoscaling for streaming) +- [ ] Monitoring configured (input rate, lag, batch duration) +- [ ] Exactly-once verified (txnVersion/txnAppId) +- [ ] Watermark configured for stateful operations +- [ ] Left joins for stream-static (not inner) diff --git a/.claude/skills/databricks-spark-structured-streaming/checkpoint-best-practices.md b/.claude/skills/databricks-spark-structured-streaming/checkpoint-best-practices.md new file mode 100644 index 00000000..349cb9bf --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/checkpoint-best-practices.md @@ -0,0 +1,316 @@ +--- +name: checkpoint-best-practices +description: Configure and manage checkpoint locations for reliable Spark Structured Streaming. Use when setting up new streaming jobs, troubleshooting checkpoint issues, migrating checkpoints, or ensuring exactly-once semantics with proper checkpoint storage and organization. +--- + +# Checkpoint Best Practices + +Configure checkpoint locations for reliable streaming with exactly-once semantics. Checkpoints track progress and enable fault tolerance. + +## Quick Start + +```python +def get_checkpoint_location(table_name): + """Checkpoint tied to target table""" + return f"/Volumes/catalog/checkpoints/{table_name}" + +# Example: +# Table: prod.analytics.orders +# Checkpoint: /Volumes/prod/checkpoints/orders + +query = (df + .writeStream + .format("delta") + .option("checkpointLocation", get_checkpoint_location("orders")) + .start("/delta/orders") +) +``` + +## Checkpoint Storage + +### Use Persistent Storage + +```python +# DO: Use Unity Catalog volumes (S3/ADLS-backed) +checkpoint_path = "/Volumes/catalog/checkpoints/stream_name" + +# DON'T: Use DBFS (ephemeral, workspace-local) +checkpoint_path = "/dbfs/checkpoints/stream_name" # Avoid +``` + +### Target-Tied Organization + +```python +def get_checkpoint_location(table_name): + """Checkpoint should be tied to TARGET, not source""" + return f"/Volumes/catalog/checkpoints/{table_name}" + +# Why target-tied? +# - Checkpoint already contains source information +# - Systematic organization +# - Easy backup and restore +# - Clear ownership +``` + +### Unique Checkpoint Per Stream + +```python +# CORRECT: Each stream has its own checkpoint +stream1.writeStream \ + .option("checkpointLocation", "/checkpoints/stream1") \ + .start() + +stream2.writeStream \ + .option("checkpointLocation", "/checkpoints/stream2") \ + .start() + +# WRONG: Never share checkpoints between streams +# This causes data loss and corruption +``` + +## Checkpoint Structure + +### Folder Contents + +``` +checkpoint_location/ +├── metadata/ # Query ID +├── offsets/ # What to process (intent) +├── commits/ # What completed (confirmation) +├── sources/ # Source metadata +└── state/ # Stateful operations (if any) +``` + +### Stateless vs Stateful + +```python +# Stateless (read from Kafka, write to Delta) +# Checkpoint: metadata, offsets, commits, sources +# No state folder + +df = (spark.readStream + .format("kafka") + .option("subscribe", "topic") + .load()) + +# Stateful (with watermark and deduplication) +# Checkpoint: + state folder +df_stateful = (df + .withWatermark("timestamp", "10 minutes") + .dropDuplicates(["partition", "offset"]) +) +``` + +## Reading Checkpoint Contents + +### Read Offset Files + +```python +import json + +# Read offset file +offset_file = "/checkpoints/stream/offsets/223" +content = dbutils.fs.head(offset_file) +offset_data = json.loads(content) + +# Pretty print +print(json.dumps(offset_data, indent=2)) + +# Key fields: +# - batchWatermarkMs: Watermark timestamp +# - batchTimestampMs: When batch started +# - source[0].startOffset: Beginning of batch (inclusive) +# - source[0].endOffset: End of batch (exclusive) +# - source[0].latestOffset: Current position in source +``` + +### Read State Store + +```python +# Query state store directly +state_df = (spark + .read + .format("statestore") + .load("/checkpoints/stream/state") +) + +state_df.show() +# Shows: key, value, partitionId, expiration timestamp + +# Read state metadata +state_metadata = (spark + .read + .format("state-metadata") + .load("/checkpoints/stream") +) +state_metadata.show() +# Shows: operatorName, numPartitions, minBatchId, maxBatchId +``` + +## Recovery Scenarios + +### Lost Checkpoint + +```python +# Steps to recover: +# 1. Delete checkpoint folder +dbutils.fs.rm("/checkpoints/stream", recurse=True) + +# 2. Restart stream with startingOffsets=earliest +df.writeStream \ + .format("delta") \ + .option("checkpointLocation", "/checkpoints/stream") \ + .option("startingOffsets", "earliest") \ + .start() + +# 3. Stream reprocesses from beginning +# 4. Delta sink handles deduplication (if idempotent writes configured) +``` + +### Corrupted Checkpoint + +```python +# Same as lost checkpoint: +# 1. Delete checkpoint folder +# 2. Restart with startingOffsets=earliest +# 3. Or restore from backup if available + +# Backup checkpoint before major changes +dbutils.fs.cp( + "/checkpoints/stream", + "/checkpoints/stream_backup_20240101", + recurse=True +) +``` + +### Crash During Batch + +```python +# Scenario: Crash during batch processing +# - Latest offset = 223 (written at start) +# - Commit 223 missing (crash before finish) +# - On restart: Spark reprocesses offset 223 +# - Delta deduplication prevents duplicates (if txnVersion configured) +``` + +## Monitoring + +### Checkpoint Size + +```python +# Track checkpoint folder size +checkpoint_size = dbutils.fs.ls("/checkpoints/stream") +total_size = sum([f.size for f in checkpoint_size if f.isFile()]) +print(f"Checkpoint size: {total_size / (1024*1024):.2f} MB") + +# Alert on checkpoint access failures +try: + dbutils.fs.ls("/checkpoints/stream") +except Exception as e: + print(f"Checkpoint access failed: {e}") + # Send alert +``` + +### State Store Growth + +```python +# Monitor state store size (stateful jobs) +state_df = spark.read.format("statestore").load("/checkpoints/stream/state") + +# Check partition balance +state_df.groupBy("partitionId").count().orderBy(desc("count")).show() + +# Look for skew - one partition with 10x others = problem +# State size = f(watermark duration, key cardinality) +``` + +### Offset vs Commit Sync + +```python +# Check if offsets have matching commits +import json + +# Read latest offset +latest_offset_file = sorted(dbutils.fs.ls("/checkpoints/stream/offsets"))[-1].path +offset_data = json.loads(dbutils.fs.head(latest_offset_file)) +batch_id = latest_offset_file.split("/")[-1] + +# Check if commit exists +commit_file = f"/checkpoints/stream/commits/{batch_id}" +if dbutils.fs.exists(commit_file): + print(f"Batch {batch_id}: Committed") +else: + print(f"Batch {batch_id}: Not committed (will reprocess)") +``` + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **State growing too large** | Long watermark duration or high cardinality keys | Reduce watermark duration; reduce key cardinality | +| **Checkpoint corruption** | File system issues or manual deletion | Delete checkpoint and restart; restore from backup | +| **Slow state operations** | Partition imbalance | Check partition balance; ensure keys are evenly distributed | +| **Can't find commit file** | Normal if job crashed | Spark will reprocess on restart | +| **Offsets out of sync** | Offsets without matching commits | Indicates unprocessed batch; will reprocess | + +## Production Best Practices + +### Checkpoint Location Pattern + +```python +def get_checkpoint_path(table_name, environment="prod"): + """ + Checkpoint should be: + 1. Tied to TARGET table (not source) + 2. In persistent storage (UC Volume, S3, ADLS) + 3. Organized systematically + """ + return f"/Volumes/{environment}/checkpoints/{table_name}" + +# Usage +checkpoint = get_checkpoint_path("orders", "prod") +``` + +### Backup Strategy + +```python +# Backup checkpoint before major changes +def backup_checkpoint(checkpoint_path, backup_suffix): + backup_path = f"{checkpoint_path}_backup_{backup_suffix}" + dbutils.fs.cp(checkpoint_path, backup_path, recurse=True) + return backup_path + +# Before code changes or migrations +backup_checkpoint("/checkpoints/stream", "20240101") +``` + +### Migration + +```python +# Migrate checkpoint to new location +def migrate_checkpoint(old_path, new_path): + # Copy checkpoint folder + dbutils.fs.cp(old_path, new_path, recurse=True) + + # Update code to use new path + # Old checkpoint remains for rollback + + # Restart stream with new checkpoint location +``` + +## Production Checklist + +- [ ] Checkpoint location is persistent (S3/ADLS, not DBFS) +- [ ] Unique checkpoint per stream +- [ ] Target-tied checkpoint organization +- [ ] Backup strategy defined +- [ ] Monitoring configured (checkpoint size, access failures) +- [ ] State store growth monitored (if stateful) +- [ ] Recovery procedure documented +- [ ] Migration procedure documented + +## Related Skills + +- `kafka-to-delta` - Kafka ingestion with checkpoint management +- `stream-stream-joins` - Stateful operations and state stores +- `state-store-management` - Deep dive on state store optimization diff --git a/.claude/skills/databricks-spark-structured-streaming/kafka-streaming.md b/.claude/skills/databricks-spark-structured-streaming/kafka-streaming.md new file mode 100644 index 00000000..9731434e --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/kafka-streaming.md @@ -0,0 +1,417 @@ +--- +name: kafka-streaming +description: Comprehensive Kafka streaming patterns including Kafka-to-Delta ingestion, Kafka-to-Kafka pipelines, and Real-Time Mode for sub-second latency. Use when building Kafka ingestion pipelines, implementing event enrichment, format transformation, or low-latency streaming workloads. +--- + +# Kafka Streaming Patterns + +Comprehensive guide to Kafka streaming with Spark Structured Streaming: ingestion to Delta, Kafka-to-Kafka pipelines, and Real-Time Mode for sub-second latency. + +## Quick Start + +### Kafka to Delta + +```python +from pyspark.sql.functions import col, from_json + +# Read from Kafka +df = (spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", "broker1:9092,broker2:9092") + .option("subscribe", "topic_name") + .option("startingOffsets", "earliest") + .option("minPartitions", "6") # Match Kafka partitions + .load() +) + +# Parse JSON value +df_parsed = df.select( + col("key").cast("string"), + from_json(col("value").cast("string"), event_schema).alias("data"), + col("topic"), col("partition"), col("offset"), + col("timestamp").alias("kafka_timestamp") +).select("key", "data.*", "topic", "partition", "offset", "kafka_timestamp") + +# Write to Delta +df_parsed.writeStream \ + .format("delta") \ + .outputMode("append") \ + .option("checkpointLocation", "/Volumes/catalog/checkpoints/kafka_stream") \ + .trigger(processingTime="30 seconds") \ + .start("/delta/bronze_events") +``` + +### Kafka to Kafka + +```python +from pyspark.sql.functions import col, from_json, to_json, struct, current_timestamp + +# Read from source Kafka +source_df = (spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", "broker1:9092") + .option("subscribe", "input-events") + .option("startingOffsets", "latest") + .load() +) + +# Parse and transform +parsed_df = source_df.select( + col("key").cast("string"), + from_json(col("value").cast("string"), event_schema).alias("data"), + col("topic").alias("source_topic") +).select("key", "data.*", "source_topic") + +# Transform events +enriched_df = parsed_df.withColumn( + "processed_at", current_timestamp() +).withColumn( + "value", to_json(struct("event_id", "user_id", "event_type", "processed_at")) +) + +# Write to output Kafka topic +enriched_df.select("key", "value").writeStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", "broker1:9092") \ + .option("topic", "output-events") \ + .option("checkpointLocation", "/checkpoints/kafka-to-kafka") \ + .trigger(processingTime="30 seconds") \ + .start() +``` + +## Common Patterns + +### Pattern 1: Bronze Layer Ingestion (Kafka to Delta) + +Minimal transformation, preserve original columns: + +```python +# Best practice: Minimal transformation, preserve original columns +# Why: Kafka retention is expensive (default 7 days) +# Delta provides permanent storage with full history + +df_bronze = (spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", servers) + .option("subscribe", topic) + .option("startingOffsets", "earliest") + .option("maxOffsetsPerTrigger", 10000) # Control batch size + .load() + .select( + col("key").cast("string"), + col("value").cast("string"), + col("topic"), col("partition"), col("offset"), + col("timestamp").alias("kafka_timestamp"), + current_timestamp().alias("ingestion_timestamp") + ) +) + +df_bronze.writeStream \ + .format("delta") \ + .outputMode("append") \ + .option("checkpointLocation", "/Volumes/catalog/checkpoints/bronze_events") \ + .trigger(processingTime="30 seconds") \ + .start("/delta/bronze_events") +``` + +### Pattern 2: Scheduled Streaming (Cost-Optimized) + +Run periodically instead of continuously: + +```python +# Run every 4 hours, not continuously +# Same code, just change trigger in job scheduler + +df_bronze.writeStream \ + .format("delta") \ + .outputMode("append") \ + .option("checkpointLocation", "/Volumes/catalog/checkpoints/bronze_events") \ + .trigger(availableNow=True) \ # Process all available, then stop + .start("/delta/bronze_events") + +# In Databricks Jobs: +# - Schedule: Every 4 hours +# - Cluster: Fixed size (no autoscaling for streaming) +# - Same streaming code, batch-style execution +``` + +### Pattern 3: Real-Time Mode (Sub-Second Latency) + +Use RTM for sub-second (as low as 5ms) latency requirements. Requires DBR 16.4 LTS+: + +```python +# Real-time trigger (DBR 16.4 LTS+) +# Requirements: dedicated cluster, no autoscaling, no Photon, outputMode("update") +# Spark config on cluster: spark.databricks.streaming.realTimeMode.enabled = true +query = (enriched_df + .select(col("key"), col("value")) + .writeStream + .format("kafka") + .option("kafka.bootstrap.servers", brokers) + .option("topic", "output-events") + .outputMode("update") # RTM only supports update mode + .trigger(realTime="5 minutes") # PySpark requires specifying the checkpoint interval + .option("checkpointLocation", checkpoint_path) + .start() +) + +# When to use RTM: +# - Sub-second latency required (achieves as low as 5ms E2E) +# - Photon must be DISABLED (not supported with RTM) +# - Autoscaling must be DISABLED +# - Dedicated (single-user) cluster only +# - forEachBatch is NOT supported in RTM +``` + +### Pattern 4: Event Enrichment (Kafka to Kafka with Delta) + +Enrich events with dimension data: + +```python +# Read reference data (Delta table - auto-refreshed each microbatch) +user_dim = spark.table("users.dimension") + +# Stream-static join for enrichment +enriched = (parsed_df + .join(user_dim, "user_id", "left") + .withColumn("enriched_value", to_json(struct( + col("event_id"), + col("user_id"), + col("user_name"), # From dimension table + col("user_segment"), # From dimension table + col("event_type"), + col("timestamp") + ))) +) + +# Write enriched events to Kafka +enriched.select(col("key"), col("enriched_value").alias("value")).writeStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", brokers) \ + .option("topic", "enriched-events") \ + .trigger(realTime=True) \ + .option("checkpointLocation", "/checkpoints/enrichment") \ + .start() +``` + +### Pattern 5: Multi-Topic Routing + +Route events to different Kafka topics: + +```python +def route_events(batch_df, batch_id): + """Route events to different Kafka topics""" + + # High priority → urgent topic + high_priority = batch_df.filter(col("priority") == "high") + if high_priority.count() > 0: + high_priority.select("key", "value").write \ + .format("kafka") \ + .option("kafka.bootstrap.servers", brokers) \ + .option("topic", "urgent-events") \ + .save() + + # Errors → DLQ topic + errors = batch_df.filter(col("event_type") == "error") + if errors.count() > 0: + errors.select("key", "value").write \ + .format("kafka") \ + .option("kafka.bootstrap.servers", brokers) \ + .option("topic", "error-events-dlq") \ + .save() + + # All events → standard topic + batch_df.select("key", "value").write \ + .format("kafka") \ + .option("kafka.bootstrap.servers", brokers) \ + .option("topic", "standard-events") \ + .save() + +parsed_df.writeStream \ + .foreachBatch(route_events) \ + .trigger(realTime=True) \ + .option("checkpointLocation", "/checkpoints/routing") \ + .start() +``` + +### Pattern 6: Schema Validation with DLQ + +Validate schema and route invalid records: + +```python +from pyspark.sql.functions import from_json, col, lit, to_json, struct, current_timestamp + +def validate_and_route(batch_df, batch_id): + """Validate schema, route bad records to DLQ""" + + # Try to parse with strict schema + parsed = batch_df.withColumn( + "parsed", + from_json(col("value").cast("string"), validated_schema) + ) + + # Valid records + valid = parsed.filter(col("parsed").isNotNull()).select("key", "value") + + # Invalid records → DLQ + invalid = parsed.filter(col("parsed").isNull()).select( + col("key"), + to_json(struct( + col("value"), + lit("SCHEMA_VALIDATION_FAILED").alias("dlq_reason"), + current_timestamp().alias("dlq_timestamp") + )).alias("value") + ) + + # Write valid to main topic + if valid.count() > 0: + valid.write.format("kafka") \ + .option("kafka.bootstrap.servers", brokers) \ + .option("topic", "valid-events") \ + .save() + + # Write invalid to DLQ + if invalid.count() > 0: + invalid.write.format("kafka") \ + .option("kafka.bootstrap.servers", brokers) \ + .option("topic", "dlq-events") \ + .save() + +source_df.writeStream \ + .foreachBatch(validate_and_route) \ + .trigger(realTime=True) \ + .option("checkpointLocation", "/checkpoints/validation") \ + .start() +``` + +## Configuration + +### Consumer Options (Reading from Kafka) + +```python +(spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", "host1:9092,host2:9092") + .option("subscribe", "source-topic") + .option("startingOffsets", "latest") # latest, earliest, or specific JSON + .option("maxOffsetsPerTrigger", "10000") # Control batch size + .option("minPartitions", "6") # Match Kafka partitions + .option("kafka.auto.offset.reset", "latest") + .option("kafka.enable.auto.commit", "false") # Spark manages offsets + .load() +) +``` + +### Producer Options (Writing to Kafka) + +```python +(df + .select("key", "value") + .writeStream + .format("kafka") + .option("kafka.bootstrap.servers", "host1:9092,host2:9092") + .option("topic", "target-topic") + .option("kafka.acks", "all") # Durability: all, 1, 0 + .option("kafka.retries", "3") + .option("kafka.batch.size", "16384") + .option("kafka.linger.ms", "5") + .option("kafka.compression.type", "lz4") # lz4, snappy, gzip + .option("checkpointLocation", checkpoint_path) + .start() +) +``` + +### Security (SASL/SSL) + +```python +# Using Databricks secrets +kafka_username = dbutils.secrets.get("kafka-scope", "username") +kafka_password = dbutils.secrets.get("kafka-scope", "password") + +# SASL/PLAIN Authentication +df.writeStream \ + .format("kafka") \ + .option("kafka.bootstrap.servers", brokers) \ + .option("topic", target_topic) \ + .option("kafka.security.protocol", "SASL_SSL") \ + .option("kafka.sasl.mechanism", "PLAIN") \ + .option("kafka.sasl.jaas.config", + f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{kafka_username}" password="{kafka_password}";') \ + .option("checkpointLocation", checkpoint_path) \ + .start() +``` + +## Performance Tuning + +| Parameter | Recommendation | Why | +|-----------|---------------|-----| +| minPartitions | Match Kafka partitions | Optimal parallelism | +| maxOffsetsPerTrigger | 10,000-100,000 | Balance latency vs throughput | +| trigger interval | Business SLA / 3 | Recovery time buffer | +| RTM | Only if < 800ms required | Microbatch more cost-effective | + +## Monitoring + +### Key Metrics + +```python +# Programmatic monitoring +for stream in spark.streams.active: + progress = stream.lastProgress + if progress: + print(f"Input rate: {progress.get('inputRowsPerSecond', 0)} rows/sec") + print(f"Processing rate: {progress.get('processedRowsPerSecond', 0)} rows/sec") + + # Kafka-specific metrics + sources = progress.get("sources", []) + for source in sources: + end_offset = source.get("endOffset", {}) + latest_offset = source.get("latestOffset", {}) + + # Calculate lag per partition + for topic, partitions in end_offset.items(): + for partition, end in partitions.items(): + latest = latest_offset.get(topic, {}).get(partition, end) + lag = int(latest) - int(end) + print(f"Topic {topic}, Partition {partition}: Lag = {lag}") +``` + +### Spark UI Checks + +- **Input Rate vs Processing Rate**: Processing must be > Input +- **Max Offsets Behind Latest**: Should be consistent or dropping +- **Batch Duration**: Should be < trigger interval + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **No data being read** | `startingOffsets` default is "latest" | Use "earliest" for existing data | +| **High latency** | Microbatch overhead | Use RTM (trigger(realTime=True)) | +| **Consumer lag** | Processing < Input rate | Scale cluster; reduce maxOffsetsPerTrigger | +| **Duplicate messages** | Exactly-once not configured | Enable idempotent producer (acks=all) | +| **Falling behind** | Processing < Input rate | Increase cluster size | +| **Can't use autoscaling** | Streaming requirement | Use fixed-size clusters | + +## Production Checklist + +- [ ] Checkpoint location is persistent (UC volumes, not DBFS) +- [ ] Unique checkpoint per pipeline +- [ ] Fixed-size cluster (no autoscaling for streaming/RTM) +- [ ] RTM enabled only if latency < 800ms required +- [ ] Consumer lag monitored and alerts configured +- [ ] Producer acks=all for durability +- [ ] Schema validation with DLQ configured +- [ ] Security (SASL/SSL) configured for production +- [ ] Exactly-once semantics verified + +## Related Skills + +- `stream-static-joins` - Enrichment patterns with Delta tables +- `stream-stream-joins` - Event correlation across Kafka topics +- `checkpoint-best-practices` - Checkpoint configuration +- `trigger-tuning` - Trigger configuration and RTM setup diff --git a/.claude/skills/databricks-spark-structured-streaming/merge-operations.md b/.claude/skills/databricks-spark-structured-streaming/merge-operations.md new file mode 100644 index 00000000..374239ad --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/merge-operations.md @@ -0,0 +1,358 @@ +--- +name: merge-operations +description: Comprehensive guide to Delta MERGE operations in streaming including performance optimization, parallel merges, and Liquid Clustering configuration. Use when implementing upserts, optimizing merge performance, performing parallel merges to multiple tables, or eliminating optimize pauses. +--- + +# Merge Operations in Streaming + +Comprehensive guide to Delta MERGE operations: performance optimization, parallel merges to multiple tables, and modern Delta features (Liquid Clustering + Deletion Vectors + Row-Level Concurrency). + +## Quick Start + +### Basic MERGE with Optimization + +```python +from delta.tables import DeltaTable + +# Enable modern Delta features +spark.sql(""" + ALTER TABLE target_table SET TBLPROPERTIES ( + 'delta.enableDeletionVectors' = true, + 'delta.enableRowLevelConcurrency' = true, + 'delta.liquid.clustering' = true + ) +""") + +# MERGE in ForEachBatch +def upsert_batch(batch_df, batch_id): + batch_df.createOrReplaceTempView("updates") + spark.sql(""" + MERGE INTO target_table t + USING updates s ON t.id = s.id + WHEN MATCHED THEN UPDATE SET * + WHEN NOT MATCHED THEN INSERT * + """) + # No optimize needed - Liquid Clustering handles it automatically + +stream.writeStream \ + .foreachBatch(upsert_batch) \ + .option("checkpointLocation", "/checkpoints/merge") \ + .start() +``` + +### Parallel MERGE to Multiple Tables + +```python +from delta.tables import DeltaTable +from concurrent.futures import ThreadPoolExecutor, as_completed + +def parallel_merge_multiple_tables(batch_df, batch_id): + """Merge into multiple tables in parallel""" + + batch_df.cache() + + def merge_table(table_name, merge_key): + target = DeltaTable.forName(spark, table_name) + source = batch_df.alias("source") + + (target.alias("target") + .merge(source, f"target.{merge_key} = source.{merge_key}") + .whenMatchedUpdateAll() + .whenNotMatchedInsertAll() + .execute() + ) + return f"Merged {table_name}" + + tables = [ + ("silver.customers", "customer_id"), + ("silver.orders", "order_id"), + ("silver.products", "product_id") + ] + + # Parallel merges + with ThreadPoolExecutor(max_workers=3) as executor: + futures = { + executor.submit(merge_table, table_name, merge_key): table_name + for table_name, merge_key in tables + } + + for future in as_completed(futures): + future.result() # Raise on error + + batch_df.unpersist() + +stream.writeStream \ + .foreachBatch(parallel_merge_multiple_tables) \ + .option("checkpointLocation", "/checkpoints/parallel_merge") \ + .start() +``` + +## Core Concepts + +### Liquid Clustering + DV + RLC + +Enable modern Delta features for optimal merge performance: + +```sql +-- Enable for target table +ALTER TABLE target_table SET TBLPROPERTIES ( + 'delta.enableDeletionVectors' = true, + 'delta.enableRowLevelConcurrency' = true, + 'delta.liquid.clustering' = true +); +``` + +**Benefits:** +- **Deletion Vectors**: Soft deletes without file rewrite +- **Row-Level Concurrency**: Concurrent updates to different rows +- **Liquid Clustering**: Automatic optimization without pauses +- **Result**: Eliminates optimize pauses, lower P99 latency, simpler code + +## Common Patterns + +### Pattern 1: Basic MERGE with Optimization + +```python +def optimized_merge(batch_df, batch_id): + """MERGE with optimized table""" + batch_df.createOrReplaceTempView("updates") + + spark.sql(""" + MERGE INTO target_table t + USING updates s ON t.id = s.id + WHEN MATCHED THEN UPDATE SET * + WHEN NOT MATCHED THEN INSERT * + """) + # No optimize needed - Liquid Clustering handles it + +stream.writeStream \ + .foreachBatch(optimized_merge) \ + .option("checkpointLocation", "/checkpoints/merge") \ + .start() +``` + +### Pattern 2: Parallel MERGE to Multiple Tables + +```python +from concurrent.futures import ThreadPoolExecutor, as_completed + +def parallel_merge(batch_df, batch_id): + """Merge into multiple tables in parallel""" + + batch_df.cache() + + def merge_one_table(table_name, merge_key): + target = DeltaTable.forName(spark, table_name) + source = batch_df.alias("source") + + (target.alias("target") + .merge(source, f"target.{merge_key} = source.{merge_key}") + .whenMatchedUpdateAll() + .whenNotMatchedInsertAll() + .execute() + ) + return table_name + + tables = [ + ("silver.customers", "customer_id"), + ("silver.orders", "order_id"), + ("silver.products", "product_id") + ] + + # Optimal thread count: min(number_of_tables, cluster_cores / 2) + max_workers = min(len(tables), max(2, total_cores // 2)) + + with ThreadPoolExecutor(max_workers=max_workers) as executor: + futures = { + executor.submit(merge_one_table, table_name, merge_key): table_name + for table_name, merge_key in tables + } + + errors = [] + for future in as_completed(futures): + table_name = futures[future] + try: + future.result() + except Exception as e: + errors.append((table_name, str(e))) + + batch_df.unpersist() + + if errors: + raise Exception(f"Merge failures: {errors}") +``` + +### Pattern 3: MERGE with Partition Pruning + +```python +def partition_pruned_merge(batch_df, batch_id): + """MERGE with partition column in condition""" + batch_df.createOrReplaceTempView("updates") + + # Include partition column in merge condition + spark.sql(""" + MERGE INTO target_table t + USING updates s + ON t.id = s.id AND t.date = s.date -- partition column + WHEN MATCHED THEN UPDATE SET * + WHEN NOT MATCHED THEN INSERT * + """) + # Skips irrelevant partitions for faster execution +``` + +### Pattern 4: CDC Multi-Target with Parallel MERGE + +```python +def cdc_parallel_merge(batch_df, batch_id): + """Apply CDC changes to multiple tables in parallel""" + + batch_df.cache() + + # Split by operation type + deletes = batch_df.filter(col("_op") == "DELETE") + upserts = batch_df.filter(col("_op").isin(["INSERT", "UPDATE"])) + + def merge_cdc_table(table_name, merge_key): + target = DeltaTable.forName(spark, table_name) + + # Upserts + if upserts.count() > 0: + (target.alias("target") + .merge(upserts.alias("source"), f"target.{merge_key} = source.{merge_key}") + .whenMatchedUpdateAll() + .whenNotMatchedInsertAll() + .execute() + ) + + # Deletes + if deletes.count() > 0: + (target.alias("target") + .merge(deletes.alias("source"), f"target.{merge_key} = source.{merge_key}") + .whenMatchedDelete() + .execute() + ) + + tables = [ + ("silver.customers", "customer_id"), + ("silver.orders", "order_id") + ] + + with ThreadPoolExecutor(max_workers=2) as executor: + futures = { + executor.submit(merge_cdc_table, table_name, merge_key): table_name + for table_name, merge_key in tables + } + + for future in as_completed(futures): + future.result() + + batch_df.unpersist() +``` + +## Performance Optimization + +### Enable Liquid Clustering + DV + RLC + +```sql +-- Create table with Liquid Clustering +CREATE TABLE target_table ( + id STRING, + name STRING, + updated_at TIMESTAMP +) USING DELTA +CLUSTER BY (id) +TBLPROPERTIES ( + 'delta.enableDeletionVectors' = true, + 'delta.enableRowLevelConcurrency' = true +); + +-- Or alter existing table +ALTER TABLE target_table SET TBLPROPERTIES ( + 'delta.enableDeletionVectors' = true, + 'delta.enableRowLevelConcurrency' = true, + 'delta.liquid.clustering' = true +); +ALTER TABLE target_table CLUSTER BY (id); +``` + +### Z-Ordering on Merge Key + +```sql +-- Z-Order on merge key for faster lookups +OPTIMIZE target_table ZORDER BY (id); + +-- Run periodically or via Predictive Optimization +-- 5-10x faster for targeted lookups +``` + +### File Size Tuning + +```sql +-- Target file size for optimal merge +ALTER TABLE target_table SET TBLPROPERTIES ( + 'delta.targetFileSize' = '128mb' +); +``` + +### Optimal Thread Count + +```python +# Formula: min(number_of_tables, cluster_cores / 2) +# Example: 4 tables, 8 cores → 4 workers +# Example: 2 tables, 4 cores → 2 workers + +max_workers = min(len(tables), max(2, total_cores // 2)) +``` + +## Monitoring + +### Track Merge Performance + +```python +import time + +def monitored_merge(batch_df, batch_id): + start_time = time.time() + + batch_df.createOrReplaceTempView("updates") + spark.sql(""" + MERGE INTO target_table t + USING updates s ON t.id = s.id + WHEN MATCHED THEN UPDATE SET * + WHEN NOT MATCHED THEN INSERT * + """) + + duration = time.time() - start_time + print(f"Merge duration: {duration:.2f}s") + + # Alert if duration exceeds threshold + if duration > 30: + print(f"WARNING: Merge duration {duration:.2f}s exceeds threshold") +``` + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **High P99 latency** | OPTIMIZE pauses | Enable Liquid Clustering (no pauses) | +| **Merge conflicts** | Concurrent updates to same rows | Enable Row-Level Concurrency | +| **Slow merges** | Large files, no optimization | Enable Liquid Clustering; Z-Order on merge key | +| **Too many threads** | Resource contention | Reduce max_workers; match to cluster capacity | +| **Partial failures** | One merge fails | Collect all errors; fail batch if any error | + +## Production Checklist + +- [ ] Liquid Clustering + DV + RLC enabled on all target tables +- [ ] Z-Ordering configured on merge keys +- [ ] Optimal thread count configured (start with 2) +- [ ] Error handling implemented (collect all errors) +- [ ] Performance monitoring per table +- [ ] Cache used to avoid recomputation +- [ ] Unpersist after writes +- [ ] File size tuned (128MB target) + +## Related Skills + +- `multi-sink-writes` - Multi-sink write patterns +- `partitioning-strategy` - Partition optimization for merges +- `checkpoint-best-practices` - Checkpoint configuration diff --git a/.claude/skills/databricks-spark-structured-streaming/multi-sink-writes.md b/.claude/skills/databricks-spark-structured-streaming/multi-sink-writes.md new file mode 100644 index 00000000..6611ab0d --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/multi-sink-writes.md @@ -0,0 +1,427 @@ +--- +name: multi-sink-writes +description: Write a single Spark stream to multiple Delta tables or Kafka topics using ForEachBatch. Use when fanning out streaming data to multiple sinks, implementing medallion architecture (bronze/silver/gold), conditional routing, CDC patterns, or creating materialized views from a single stream. +--- + +# Multi-Sink Writes + +Write a single streaming source to multiple Delta tables or Kafka topics efficiently using ForEachBatch. Read once, write many - avoiding reprocessing the source multiple times. + +## Quick Start + +```python +from pyspark.sql.functions import col, current_timestamp + +def write_multiple_tables(batch_df, batch_id): + """Write batch to multiple sinks""" + # Bronze - raw data + batch_df.write \ + .format("delta") \ + .mode("append") \ + .option("txnVersion", batch_id) \ + .option("txnAppId", "multi_sink_job") \ + .save("/delta/bronze_events") + + # Silver - cleansed + cleansed = batch_df.dropDuplicates(["event_id"]) + cleansed.write \ + .format("delta") \ + .mode("append") \ + .option("txnVersion", batch_id) \ + .option("txnAppId", "multi_sink_job_silver") \ + .save("/delta/silver_events") + + # Gold - aggregated + aggregated = batch_df.groupBy("category").count() + aggregated.write \ + .format("delta") \ + .mode("append") \ + .option("txnVersion", batch_id) \ + .option("txnAppId", "multi_sink_job_gold") \ + .save("/delta/category_counts") + +stream.writeStream \ + .foreachBatch(write_multiple_tables) \ + .option("checkpointLocation", "/checkpoints/multi_sink") \ + .start() +``` + +## Core Concepts + +### One Source, One Checkpoint + +Use a single checkpoint for the entire multi-sink stream: + +```python +# CORRECT: One checkpoint for all sinks +stream.writeStream \ + .foreachBatch(multi_sink_function) \ + .option("checkpointLocation", "/checkpoints/single_source_multi_sink") \ + .start() + +# WRONG: Don't create separate streams +# Each stream would reprocess the source independently +``` + +### Transactional Guarantees + +Each ForEachBatch call represents one epoch. All writes within the batch: +- See the same input data +- Share the same batch_id +- Are idempotent if using txnVersion + +## Common Patterns + +### Pattern 1: Bronze-Silver-Gold Medallion Architecture + +Single stream feeding all three medallion layers: + +```python +from pyspark.sql.functions import window, count, sum, current_timestamp + +def medallion_architecture(batch_df, batch_id): + """Single stream feeding all three medallion layers""" + + # Bronze: Raw ingestion + (batch_df.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "medallion_bronze") + .saveAsTable("bronze.events") + ) + + # Silver: Cleansed and validated + silver_df = (batch_df + .dropDuplicates(["event_id"]) + .filter(col("status").isin(["active", "pending"])) + .withColumn("processed_at", current_timestamp()) + ) + + (silver_df.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "medallion_silver") + .saveAsTable("silver.events") + ) + + # Gold: Business aggregates + gold_df = (silver_df + .groupBy(window(col("timestamp"), "5 minutes"), "category") + .agg( + count("*").alias("event_count"), + sum("amount").alias("total_amount") + ) + ) + + (gold_df.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "medallion_gold") + .saveAsTable("gold.category_metrics") + ) + +stream.writeStream \ + .foreachBatch(medallion_architecture) \ + .trigger(processingTime="30 seconds") \ + .option("checkpointLocation", "/checkpoints/medallion") \ + .start() +``` + +### Pattern 2: Conditional Routing + +Route events to different tables based on criteria: + +```python +def route_by_type(batch_df, batch_id): + """Route events to different tables based on type""" + + # Split by event type + orders = batch_df.filter(col("event_type") == "order") + refunds = batch_df.filter(col("event_type") == "refund") + reviews = batch_df.filter(col("event_type") == "review") + + # Write to respective tables + if orders.count() > 0: + (orders.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "router_orders") + .saveAsTable("orders") + ) + + if refunds.count() > 0: + (refunds.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "router_refunds") + .saveAsTable("refunds") + ) + + if reviews.count() > 0: + (reviews.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "router_reviews") + .saveAsTable("reviews") + ) +``` + +### Pattern 3: Parallel Fan-Out + +Write to multiple sinks in parallel for independent tables: + +```python +from concurrent.futures import ThreadPoolExecutor, as_completed + +def parallel_write(batch_df, batch_id): + """Write to multiple sinks in parallel""" + + # Cache to avoid recomputation + batch_df.cache() + + def write_table(table_name, filter_expr=None): + """Write filtered data to table""" + df = batch_df.filter(filter_expr) if filter_expr else batch_df + (df.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", f"parallel_{table_name}") + .saveAsTable(table_name) + ) + return f"Wrote {table_name}" + + # Define tables and filters + tables = [ + ("bronze.all_events", None), + ("silver.errors", col("level") == "ERROR"), + ("silver.warnings", col("level") == "WARN"), + ("gold.metrics", col("type") == "metric") + ] + + # Parallel writes + with ThreadPoolExecutor(max_workers=4) as executor: + futures = { + executor.submit(write_table, table_name, filter_expr): table_name + for table_name, filter_expr in tables + } + + errors = [] + for future in as_completed(futures): + table_name = futures[future] + try: + future.result() + except Exception as e: + errors.append((table_name, str(e))) + + batch_df.unpersist() + + if errors: + raise Exception(f"Write failures: {errors}") +``` + +### Pattern 4: Materialized Views + +Create multiple derived views from the same stream: + +```python +from pyspark.sql.functions import window, count, sum + +def create_materialized_views(batch_df, batch_id): + """Create multiple derived views from the same stream""" + + # Base: All events + (batch_df.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "views_raw") + .save("/delta/views/raw") + ) + + # View 1: Hourly aggregations + hourly = (batch_df + .withWatermark("event_time", "1 hour") + .groupBy(window(col("event_time"), "1 hour"), col("category")) + .agg( + count("*").alias("event_count"), + sum("value").alias("total_value") + ) + ) + + (hourly.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "views_hourly") + .save("/delta/views/hourly") + ) + + # View 2: User sessions (15 min window) + sessions = (batch_df + .withWatermark("event_time", "15 minutes") + .groupBy(window(col("event_time"), "15 minutes"), col("user_id")) + .agg(count("*").alias("actions")) + ) + + (sessions.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "views_sessions") + .save("/delta/views/sessions") + ) +``` + +### Pattern 5: Error Handling with Dead Letter Queue + +Route invalid records to DLQ: + +```python +from pyspark.sql.functions import when, lit + +def write_with_dlq(batch_df, batch_id): + """Write valid records to target, invalid to dead letter queue""" + + # Validation + valid = batch_df.filter( + col("required_field").isNotNull() & + col("timestamp").isNotNull() + ) + invalid = batch_df.filter( + col("required_field").isNull() | + col("timestamp").isNull() + ) + + # Write valid data + if valid.count() > 0: + (valid.write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "multi_sink_valid") + .saveAsTable("silver.valid_events") + ) + + # Write invalid to DLQ with metadata + if invalid.count() > 0: + dlq_df = (invalid + .withColumn("_error_reason", + when(col("required_field").isNull(), "missing_required_field") + .otherwise("missing_timestamp")) + .withColumn("_batch_id", lit(batch_id)) + .withColumn("_processed_at", current_timestamp()) + ) + + (dlq_df.write + .format("delta") + .mode("append") + .saveAsTable("errors.dead_letter_queue") + ) +``` + +## Performance Optimization + +### Minimize Recomputation + +Cache the batch DataFrame to avoid recomputation: + +```python +def optimized_multi_sink(batch_df, batch_id): + """Cache to avoid recomputation""" + + # Cache the batch + batch_df.cache() + + # Multiple writes from cached data + batch_df.write... # Sink 1 + batch_df.filter(...).write... # Sink 2 + batch_df.filter(...).write... # Sink 3 + + # Unpersist when done + batch_df.unpersist() +``` + +### Parallel Writes + +Use ThreadPoolExecutor for independent writes: + +```python +from concurrent.futures import ThreadPoolExecutor + +def parallel_write(batch_df, batch_id): + """Write to independent tables in parallel""" + + batch_df.cache() + + def write_table(table_name, df): + df.write.format("delta").mode("append").saveAsTable(table_name) + + # Parallel writes + with ThreadPoolExecutor(max_workers=4) as executor: + executor.submit(write_table, "table1", batch_df) + executor.submit(write_table, "table2", batch_df.filter(...)) + executor.submit(write_table, "table3", batch_df.filter(...)) + + batch_df.unpersist() +``` + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **Slow writes** | Sequential processing | Use parallel ThreadPoolExecutor | +| **Recomputation** | Multiple actions on same DataFrame | Cache the batch DataFrame | +| **Partial failures** | One sink fails | Use idempotent writes; Spark retries entire batch | +| **Schema conflicts** | Tables have different schemas | Transform before each write | +| **Resource contention** | Too many concurrent writes | Limit parallelism; batch writes | + +## Production Best Practices + +### Idempotent Writes + +Always use txnVersion with batch_id: + +```python +.write + .format("delta") + .option("txnVersion", batch_id) + .option("txnAppId", "unique_app_id_per_table") + .mode("append") +``` + +### Keep Batch Processing Fast + +```python +# GOOD: Simple filters and writes +def efficient_write(df, batch_id): + df.filter(...).write.save("/delta/table1") + df.filter(...).write.save("/delta/table2") + +# BAD: Expensive aggregations (move to stream definition) +def inefficient_write(df, batch_id): + df.groupBy(...).agg(...).write.save("/delta/table3") # Move to stream! +``` + +## Production Checklist + +- [ ] One checkpoint per multi-sink stream +- [ ] Idempotent writes configured (txnVersion/txnAppId) +- [ ] Cache used to avoid recomputation +- [ ] Parallel writes for independent tables +- [ ] Error handling and DLQ configured +- [ ] Schema evolution handled +- [ ] Performance monitoring per sink + +## Related Skills + +- `merge-operations` - Parallel MERGE operations +- `kafka-streaming` - Kafka ingestion patterns +- `stream-static-joins` - Enrichment before multi-sink writes +- `checkpoint-best-practices` - Checkpoint configuration diff --git a/.claude/skills/databricks-spark-structured-streaming/stateful-operations.md b/.claude/skills/databricks-spark-structured-streaming/stateful-operations.md new file mode 100644 index 00000000..625f53eb --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/stateful-operations.md @@ -0,0 +1,397 @@ +--- +name: stateful-operations +description: Configure watermarks and manage state stores for Spark Structured Streaming stateful operations. Use when setting up stateful operations, tuning watermark duration, handling late-arriving data, configuring RocksDB for large state, monitoring state store size, or optimizing state performance. +--- + +# Stateful Operations: Watermarks and State Stores + +Configure watermarks to handle late-arriving data and manage state stores for stateful streaming operations. Watermarks control state cleanup, while state stores handle the storage and retrieval of stateful data. + +## Quick Start + +```python +# Enable RocksDB for large state stores +spark.conf.set( + "spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider" +) + +# Stateful operation with watermark +df = (spark.readStream + .format("kafka") + .option("subscribe", "events") + .load() + .select(from_json(col("value").cast("string"), schema).alias("data")) + .select("data.*") + .withWatermark("event_time", "10 minutes") # Late data threshold + state cleanup + .dropDuplicates(["event_id"]) # Stateful operation +) + +# Watermark = latest_event_time - 10 minutes +# State automatically expires after watermark duration +``` + +## Watermark Configuration + +### How Watermarks Work + +```python +# Watermark = latest_event_time - delay_threshold +.withWatermark("event_time", "10 minutes") + +# Events with timestamp < watermark are considered "too late" +# State for late events is automatically cleaned up +# Late events may be dropped (outer joins) or processed (inner joins) +``` + +### Watermark Duration Selection + +| Watermark Setting | Effect | Use Case | +|-------------------|--------|----------| +| `"10 minutes"` | Moderate latency | General streaming | +| `"1 hour"` | High completeness | Financial transactions | +| `"5 minutes"` | Low latency | Real-time analytics | +| `"24 hours"` | Batch-like | Backfill scenarios | + +**Rule of thumb**: Start with 2-3× your p95 latency. Monitor late data rate and adjust. + +### Watermark and State Size + +```python +# Watermark directly affects state store size +# State kept for watermark duration + processing time + +# Example calculation: +# - 10 minute watermark +# - 1M events/min +# - State size = ~10M keys × key_size + +# Reduce watermark to reduce state size +.withWatermark("event_time", "5 minutes") # Smaller state + +# State automatically expires after watermark duration +# No manual cleanup needed +``` + +## State Store Configuration + +### Enable RocksDB + +Use RocksDB for state stores exceeding memory capacity: + +```python +# Enable RocksDB state store provider +spark.conf.set( + "spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider" +) + +# Benefits: +# - State stored on disk, reducing memory pressure +# - Recommended for: High cardinality keys, long watermark durations +# - Better performance for large state stores +``` + +### State Store Configuration + +```python +# State store batch retention +spark.conf.set("spark.sql.streaming.stateStore.minBatchesToRetain", "2") + +# State maintenance interval +spark.conf.set("spark.sql.streaming.stateStore.maintenanceInterval", "5m") + +# State store location (default: checkpoint/state) +# Automatically managed by Spark +``` + +## Common Patterns + +### Pattern 1: Basic Stateful Operation with Watermark + +```python +# Watermark for deduplication +df = (spark.readStream + .format("kafka") + .option("subscribe", "events") + .load() + .select(from_json(col("value").cast("string"), schema).alias("data")) + .select("data.*") + .withWatermark("event_time", "10 minutes") + .dropDuplicates(["event_id"]) +) + +# State expires after watermark duration +# Prevents infinite state growth +``` + +### Pattern 2: Join-Specific Watermark Tuning + +Different watermarks for streams with different latencies: + +```python +# Fast source: 5 minute watermark +impressions = (spark.readStream + .format("kafka") + .option("subscribe", "impressions") + .load() + .select(from_json(col("value").cast("string"), impression_schema).alias("data")) + .select("data.*") + .withWatermark("impression_time", "5 minutes") +) + +# Slower source: 15 minute watermark +clicks = (spark.readStream + .format("kafka") + .option("subscribe", "clicks") + .load() + .select(from_json(col("value").cast("string"), click_schema).alias("data")) + .select("data.*") + .withWatermark("click_time", "15 minutes") +) + +# Effective watermark = max(5, 15) = 15 minutes +joined = impressions.join( + clicks, + expr(""" + impressions.ad_id = clicks.ad_id AND + clicks.click_time BETWEEN impressions.impression_time AND + impressions.impression_time + interval 1 hour + """), + "inner" +) +``` + +### Pattern 3: Windowed Aggregations with Watermark + +```python +from pyspark.sql.functions import window, count, sum, max, current_timestamp + +windowed = (df + .withWatermark("event_time", "10 minutes") + .groupBy( + window(col("event_time"), "5 minutes"), + col("user_id") + ) + .agg( + count("*").alias("event_count"), + sum("value").alias("total_value"), + max("event_time").alias("latest_event") + ) + .withColumn("processing_time", current_timestamp()) +) + +# Use update mode for corrected results when late data arrives +windowed.writeStream \ + .outputMode("update") \ + .format("delta") \ + .option("checkpointLocation", "/checkpoints/windowed") \ + .start("/delta/windowed_metrics") +``` + +### Pattern 4: Monitor State Partition Balance + +Check for state store skew: + +```python +def check_state_balance(checkpoint_path): + """Check state store partition balance""" + state_df = spark.read.format("statestore").load(f"{checkpoint_path}/state") + + partition_counts = state_df.groupBy("partitionId").count().orderBy(desc("count")) + partition_counts.show() + + # Calculate skew + counts = [row['count'] for row in partition_counts.collect()] + if counts: + max_count = max(counts) + min_count = min(counts) + skew_ratio = max_count / min_count if min_count > 0 else float('inf') + + print(f"State skew ratio: {skew_ratio:.2f}") + if skew_ratio > 10: + print("WARNING: High state skew detected") + return False + return True +``` + +### Pattern 5: Monitor State Growth + +```python +def monitor_state_growth(checkpoint_path): + """Track state store growth""" + state_df = spark.read.format("statestore").load(f"{checkpoint_path}/state") + + # Current state size + total_rows = state_df.count() + + print(f"State rows: {total_rows}") + + # Check expiration + from pyspark.sql.functions import current_timestamp, col + expired = state_df.filter(col("expirationMs") < current_timestamp().cast("long") * 1000) + expired_count = expired.count() + + print(f"Expired state rows: {expired_count}") + print(f"Active state rows: {total_rows - expired_count}") +``` + +## State Size Control + +### Use Watermarks + +Watermarks automatically clean up expired state: + +```python +# State expires after watermark duration +.withWatermark("event_time", "10 minutes") + +# State size = f(watermark duration, key cardinality) +# 10 min watermark × 1M events/min = manageable +# 72 hour watermark × 1M events/min = very large +``` + +### Reduce Key Cardinality + +```python +# Bad: High cardinality keys +.dropDuplicates(["user_id"]) # Millions of distinct values + +# Good: Lower cardinality or expiring keys +.dropDuplicates(["session_id"]) # Sessions expire naturally +.dropDuplicates(["event_id", "date"]) # Partition by date reduces cardinality +``` + +## Monitoring + +### Programmatic State Monitoring + +```python +# Monitor state size programmatically +for stream in spark.streams.active: + progress = stream.lastProgress + + if progress and "stateOperators" in progress: + for op in progress["stateOperators"]: + print(f"Operator: {op.get('operatorName', 'unknown')}") + print(f"State rows: {op.get('numRowsTotal', 0)}") + print(f"State memory: {op.get('memoryUsedBytes', 0)}") + print(f"State on disk: {op.get('diskBytesUsed', 0)}") +``` + +### Track Late Data Rates + +```python +# Monitor late data impact +late_data_stats = spark.sql(""" + SELECT + date_trunc('hour', event_time) as hour, + COUNT(*) as total_events, + SUM(CASE + WHEN unix_timestamp(processing_time) - unix_timestamp(event_time) > 600 + THEN 1 ELSE 0 + END) as late_events, + AVG(unix_timestamp(processing_time) - unix_timestamp(event_time)) as avg_delay_seconds, + MAX(unix_timestamp(processing_time) - unix_timestamp(event_time)) as max_delay_seconds + FROM events + WHERE processing_time >= current_timestamp() - interval 24 hours + GROUP BY 1 + ORDER BY 1 DESC +""") +``` + +## Late Data Classification + +| Delay | Category | Handling | +|-------|----------|----------| +| < Watermark | On-time | Normal processing | +| Watermark < delay < 2×Watermark | Late | Join with inner match; may still process | +| > 2×Watermark | Very late | DLQ for manual handling | + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **State store explosion** | Watermark too long | Reduce watermark; archive old state | +| **Late data dropped** | Watermark too short | Increase watermark; analyze latency patterns | +| **State too large** | High cardinality keys or long watermark | Reduce key cardinality; decrease watermark duration | +| **State partition skew** | Uneven key distribution | Ensure keys are evenly distributed; consider salting | +| **OOM errors** | State exceeds memory | Enable RocksDB; increase memory; reduce watermark | +| **State not expiring** | Watermark not configured | Add watermark to stateful operations | + +## State Store Recovery + +```python +# Scenario 1: State store corruption +# Solution: Delete state folder, restart stream +# State will rebuild from watermark + +dbutils.fs.rm("/checkpoints/stream/state", recurse=True) + +# Restart stream - state rebuilds automatically +# Note: May reprocess some data within watermark window + +# Scenario 2: State store too large +# Solution: Reduce watermark duration +.withWatermark("event_time", "5 minutes") # Reduced from 10 minutes + +# Scenario 3: State partition imbalance +# Solution: Ensure keys are evenly distributed +# Consider salting keys if needed +``` + +## Production Best Practices + +### Always Use Watermarks for Stateful Operations + +```python +# REQUIRED: Watermark for stateful operations +df.withWatermark("event_time", "10 minutes").dropDuplicates(["id"]) + +# REQUIRED: Watermark for aggregations +df.withWatermark("event_time", "10 minutes").groupBy(...).agg(...) + +# REQUIRED: Watermark for stream-stream joins +stream1.withWatermark("ts", "10 min").join(stream2.withWatermark("ts", "10 min")) +``` + +### Watermark Selection + +```python +# Rule of thumb: 2-3× p95 latency +# Example: p95 latency = 5 minutes → watermark = 10-15 minutes + +# Start conservative, adjust based on monitoring +.withWatermark("event_time", "10 minutes") # Start here +# Monitor late data rate +# Increase if too many late events +# Decrease if state too large +``` + +### Use RocksDB for Large State + +```python +# Enable RocksDB if state > memory capacity +# Typical threshold: > 100M keys or > 10GB state + +spark.conf.set( + "spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider" +) +``` + +## Production Checklist + +- [ ] Watermark configured for all stateful operations +- [ ] Watermark duration matches latency requirements (2-3× p95) +- [ ] RocksDB enabled for large state stores +- [ ] State size monitored and alerts configured +- [ ] State partition balance checked regularly +- [ ] State growth tracked over time +- [ ] Late data monitoring configured +- [ ] Recovery procedure documented + +## Related Skills + +- `stream-stream-joins` - Late data in joins +- `checkpoint-best-practices` - Checkpoint and state recovery diff --git a/.claude/skills/databricks-spark-structured-streaming/stream-static-joins.md b/.claude/skills/databricks-spark-structured-streaming/stream-static-joins.md new file mode 100644 index 00000000..614d87c8 --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/stream-static-joins.md @@ -0,0 +1,519 @@ +--- +name: stream-static-joins +description: Enrich streaming data with Delta dimension tables in real-time. Use when joining fast-moving streaming events with slowly-changing reference data (device dimensions, user profiles, product catalogs), implementing real-time data enrichment, or adding context to streaming events without state management overhead. +--- + +# Stream-Static Joins + +Enrich streaming data with slowly-changing reference data stored in Delta tables. Stream-static joins are stateless and automatically refresh dimension data each microbatch. + +## Quick Start + +```python +from pyspark.sql.functions import col, from_json + +# Streaming source (IoT events from Kafka) +iot_stream = (spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", "broker:9092") + .option("subscribe", "iot-events") + .load() + .select(from_json(col("value").cast("string"), event_schema).alias("data")) + .select("data.*") +) + +# Static Delta dimension table (refreshes each microbatch) +device_dim = spark.table("device_dimensions") + +# Enrich streaming data with left join (recommended) +enriched = iot_stream.join( + device_dim, + "device_id", + "left" # Preserves all streaming events +).select( + iot_stream["*"], + device_dim["device_type"], + device_dim["location"], + device_dim["manufacturer"], + device_dim["updated_at"].alias("dim_updated_at") +) + +# Write enriched data +query = (enriched + .writeStream + .format("delta") + .outputMode("append") + .option("checkpointLocation", "/Volumes/catalog/checkpoints/enriched_events") + .trigger(processingTime="30 seconds") + .start("/delta/enriched_iot_events") +) +``` + +## Core Concepts + +### Why Delta Tables Matter + +Delta tables enable automatic version checking each microbatch: + +```python +# Delta table: Version checked every microbatch +device_dim = spark.table("device_dimensions") # Reads latest version automatically + +# Non-Delta format: Read once at startup (truly static) +device_dim = spark.read.parquet("/path/to/devices") # No refresh +``` + +**Key Insight**: Delta's versioning ensures each microbatch gets the latest dimension data without manual refresh. + +### Join Types and Production Use + +| Join Type | Behavior | Production Use | +|-----------|----------|----------------| +| **Left** | Preserves all stream events | ✅ Recommended - prevents data loss | +| **Inner** | Drops unmatched events | ⚠️ Risk of data loss - avoid in production | +| **Right** | Preserves all dimension rows | Rarely used | +| **Full** | Preserves both sides | Rarely used | + +**Production Rule**: Always use left join to prevent dropping valid streaming events. + +## Common Patterns + +### Pattern 1: Basic Device Enrichment + +Enrich IoT events with device metadata: + +```python +# Streaming IoT events +iot_stream = (spark + .readStream + .format("kafka") + .option("subscribe", "iot-events") + .load() + .select(from_json(col("value").cast("string"), event_schema).alias("data")) + .select("data.*") +) + +# Device dimension table +device_dim = spark.table("device_dimensions") + +# Left join to preserve all events +enriched = iot_stream.join( + device_dim, + "device_id", + "left" +).select( + iot_stream["*"], + device_dim["device_type"], + device_dim["location"], + device_dim["status"] +) + +enriched.writeStream \ + .format("delta") \ + .option("checkpointLocation", "/checkpoints/enriched") \ + .start("/delta/enriched_events") +``` + +### Pattern 2: Multi-Table Enrichment + +Chain multiple dimension joins: + +```python +# Multiple dimension tables +devices = spark.table("device_dimensions") +locations = spark.table("location_dimensions") +categories = spark.table("category_dimensions") + +# Chain joins (each is stateless) +enriched = (iot_stream + .join(devices, "device_id", "left") + .join(locations, "location_id", "left") + .join(categories, "category_id", "left") + .select( + iot_stream["*"], + devices["device_type"], + devices["manufacturer"], + locations["region"], + locations["country"], + categories["category_name"] + ) +) + +# Each join refreshes independently each microbatch +``` + +### Pattern 3: Broadcast Hash Join Optimization + +Optimize joins by ensuring broadcast: + +```python +from pyspark.sql.functions import broadcast + +# Option 1: Select only needed columns +small_dim = device_dim.select("device_id", "device_type", "location") + +# Option 2: Filter to active records +active_dim = device_dim.filter(col("status") == "active") + +# Option 3: Force broadcast hint +enriched = iot_stream.join( + broadcast(active_dim), + "device_id", + "left" +) + +# Verify in Spark UI: Look for "BroadcastHashJoin" in query plan +``` + +### Pattern 4: Audit Dimension Freshness + +Track how fresh dimension data is: + +```python +from pyspark.sql.functions import unix_timestamp, current_timestamp + +enriched = (iot_stream + .join(device_dim, "device_id", "left") + .withColumn( + "dim_lag_seconds", + unix_timestamp(current_timestamp()) - + unix_timestamp(col("dim_updated_at")) + ) + .withColumn( + "dim_fresh", + col("dim_lag_seconds") < 3600 # Less than 1 hour old + ) +) + +# Monitor: Alert if dim_lag_seconds > threshold +# Use for data quality checks +``` + +### Pattern 5: Time-Travel Dimension Lookup + +Join with dimension as-of event time: + +```python +from delta import DeltaTable + +def enrich_with_time_travel(batch_df, batch_id): + """Enrich with dimension version at event time""" + from pyspark.sql.functions import max as spark_max + + # Get latest dimension version + latest_version = DeltaTable.forName(spark, "device_dimensions") \ + .history() \ + .select(spark_max("version").alias("max_version")) \ + .first()[0] + + # Read dimension at specific version + dim_at_version = (spark + .read + .format("delta") + .option("versionAsOf", latest_version) + .table("device_dimensions") + ) + + # Join with batch + enriched = batch_df.join(dim_at_version, "device_id", "left") + + # Write + (enriched + .write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "enrichment_job") + .saveAsTable("enriched_events") + ) + +iot_stream.writeStream \ + .foreachBatch(enrich_with_time_travel) \ + .option("checkpointLocation", "/checkpoints/enriched") \ + .start() +``` + +### Pattern 6: Backfill Missing Dimensions + +Daily job to fix null dimensions from left join: + +```python +# Daily batch job to backfill missing dimensions +spark.sql(""" + MERGE INTO enriched_events target + USING device_dimensions source + ON target.device_id = source.device_id + AND target.device_type IS NULL + WHEN MATCHED THEN + UPDATE SET + device_type = source.device_type, + location = source.location, + manufacturer = source.manufacturer, + dim_updated_at = source.updated_at +""") + +# Run after dimension table updates +# Fixes events that arrived before dimension was available +``` + +### Pattern 7: Dimension Change Detection + +Stream that reacts to dimension changes: + +```python +def update_reference_cache(batch_df, batch_id): + """Update in-memory cache when dimension changes""" + # Dimension table changed + # Update application cache or notify downstream systems + pass + +# Stream dimension table changes +dim_changes = (spark + .readStream + .format("delta") + .table("device_dimensions") + .writeStream + .foreachBatch(update_reference_cache) + .option("checkpointLocation", "/checkpoints/dim_changes") + .start() +) +``` + +## Performance Optimization + +### Checklist + +- [ ] Dimension table < 100MB for broadcast (or increase threshold) +- [ ] Select only needed columns before join +- [ ] Filter dimension to active records only +- [ ] Verify "BroadcastHashJoin" in query plan +- [ ] Partition size 100-200MB in memory +- [ ] Use same region for compute and storage + +### Configuration + +```python +# Increase broadcast threshold if dimension is larger +spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "1g") + +# Control partition size +spark.conf.set("spark.sql.shuffle.partitions", "200") + +# Optimize dimension table reads +spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true") +spark.conf.set("spark.databricks.delta.autoCompact.enabled", "true") +``` + +### Reduce Dimension Size + +```python +# Before join: Select only needed columns +small_dim = device_dim.select( + "device_id", + "device_type", + "location", + "status" +) + +# Filter to active records +active_dim = small_dim.filter(col("status") == "active") + +# Join with smaller dimension +enriched = iot_stream.join(active_dim, "device_id", "left") +``` + +## Monitoring + +### Key Metrics + +```python +# Null rate (left join quality) +spark.sql(""" + SELECT + date_trunc('hour', timestamp) as hour, + count(*) as total_events, + count(device_type) as matched_events, + count(*) - count(device_type) as unmatched_events, + (count(*) - count(device_type)) * 100.0 / count(*) as null_rate_pct + FROM enriched_events + GROUP BY 1 + ORDER BY 1 DESC +""") + +# Dimension freshness +spark.sql(""" + SELECT + date_trunc('hour', timestamp) as hour, + avg(dim_lag_seconds) as avg_lag_seconds, + max(dim_lag_seconds) as max_lag_seconds, + count(*) as events_with_dim + FROM enriched_events + WHERE dim_updated_at IS NOT NULL + GROUP BY 1 + ORDER BY 1 DESC +""") +``` + +### Programmatic Monitoring + +```python +# Monitor stream health +for stream in spark.streams.active: + status = stream.status + progress = stream.lastProgress + + if progress: + print(f"Stream: {stream.name}") + print(f"Input rate: {progress.get('inputRowsPerSecond', 0)} rows/sec") + print(f"Processing rate: {progress.get('processedRowsPerSecond', 0)} rows/sec") + print(f"Batch duration: {progress.get('durationMs', {}).get('triggerExecution', 0)} ms") +``` + +### Spark UI Checks + +- **Streaming Tab**: Input rate vs processing rate (processing must exceed input) +- **SQL Tab**: Look for "BroadcastHashJoin" (not "SortMergeJoin") +- **Jobs Tab**: Check for shuffle operations (should be minimal) +- **Stages Tab**: Verify partition sizes (100-200MB target) + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **Data loss** | Inner join dropping unmatched events | Switch to left join | +| **Slow joins** | Shuffle join instead of broadcast | Reduce dimension size; force broadcast | +| **Stale data** | Non-Delta format | Convert dimension table to Delta | +| **Memory issues** | Large dimension table | Filter before join; increase broadcast threshold | +| **Skewed joins** | Hot keys in dimension | Salt the join key or partition dimension table | +| **High null rate** | Dimension updates lagging | Monitor dimension freshness; backfill job | + +## Production Best Practices + +### Always Use Left Join + +```python +# WRONG: Inner join loses data +enriched = iot_stream.join(device_dim, "device_id", "inner") + +# CORRECT: Left join preserves all events +enriched = iot_stream.join(device_dim, "device_id", "left") + +# Why? New devices may send data before dimension table is updated +# Left join preserves events; backfill dimensions later +``` + +### Handle Null Dimensions + +```python +# Add null handling in transformations +enriched = (iot_stream + .join(device_dim, "device_id", "left") + .withColumn( + "device_type", + coalesce(col("device_type"), lit("UNKNOWN")) + ) + .withColumn( + "location", + coalesce(col("location"), lit("UNKNOWN")) + ) +) + +# Or flag for manual review +enriched = enriched.withColumn( + "needs_review", + col("device_type").isNull() +) +``` + +### Idempotent Writes + +```python +def idempotent_write(batch_df, batch_id): + """Write with transaction version for idempotency""" + (batch_df + .write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "enrichment_job") + .saveAsTable("enriched_events") + ) + +enriched.writeStream \ + .foreachBatch(idempotent_write) \ + .option("checkpointLocation", "/checkpoints/enriched") \ + .start() +``` + +## Production Checklist + +- [ ] Left join used (not inner join) +- [ ] Dimension table is Delta format +- [ ] Broadcast hash join verified in query plan +- [ ] Dimension size optimized (< 100MB or threshold increased) +- [ ] Null rate monitored and alerts configured +- [ ] Dimension freshness tracked +- [ ] Backfill job scheduled for missing dimensions +- [ ] Checkpoint location is unique per query +- [ ] Idempotent writes configured (txnVersion/txnAppId) +- [ ] Performance metrics tracked (input rate, batch duration) + +## Expert Tips + +### Delta Version Checking + +Delta tables automatically refresh each microbatch by checking the latest version: + +```python +# Each microbatch: +# 1. Spark checks Delta table version +# 2. Reads latest version if changed +# 3. Uses cached version if unchanged +# 4. No manual refresh needed + +# This is why Delta tables work better than Parquet for dimensions +# Parquet: Read once at startup (truly static) +# Delta: Version checked each microbatch (semi-static) +``` + +### Broadcast Join Verification + +Always verify broadcast joins in production: + +```python +# Check query plan +enriched.explain(extended=True) + +# Look for: +# - BroadcastHashJoin ✅ (fast, no shuffle) +# - SortMergeJoin ⚠️ (slower, requires shuffle) + +# If seeing SortMergeJoin: +# 1. Reduce dimension size (select columns, filter rows) +# 2. Increase broadcast threshold +# 3. Force broadcast hint +``` + +### Dimension Table Optimization + +Optimize dimension tables for streaming joins: + +```python +# 1. Use Z-order or liquid clustering on join key +spark.sql(""" + OPTIMIZE device_dimensions + ZORDER BY (device_id) +""") + +# 2. Keep dimension tables small (< 100MB ideal) +# 3. Use Delta for automatic version checking +# 4. Partition by frequently filtered columns +``` + +## Related Skills + +- `stream-stream-joins` - Join two streaming sources with state management +- `kafka-to-delta` - Kafka ingestion patterns +- `write-multiple-tables` - Fan-out patterns for multiple sinks +- `checkpoint-best-practices` - Checkpoint configuration and management diff --git a/.claude/skills/databricks-spark-structured-streaming/stream-stream-joins.md b/.claude/skills/databricks-spark-structured-streaming/stream-stream-joins.md new file mode 100644 index 00000000..e5b10aad --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/stream-stream-joins.md @@ -0,0 +1,588 @@ +--- +name: stream-stream-joins +description: Join two streaming sources in real-time with event-time semantics, watermarks, and state management. Use when correlating events from different streams (orders with payments, clicks with conversions, sensor readings), handling late-arriving data, or implementing windowed aggregations across multiple streams. +--- + +# Stream-Stream Joins + +Join two streaming sources in real-time to correlate events that arrive at different times and speeds. Stream-stream joins require watermarks to manage state and handle late-arriving data. + +## Quick Start + +```python +from pyspark.sql.functions import expr, from_json, col +from pyspark.sql.types import StructType + +# Read two streaming sources +orders = (spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", "broker:9092") + .option("subscribe", "orders") + .load() + .select(from_json(col("value").cast("string"), order_schema).alias("data")) + .select("data.*") + .withWatermark("order_time", "10 minutes") +) + +payments = (spark + .readStream + .format("kafka") + .option("kafka.bootstrap.servers", "broker:9092") + .option("subscribe", "payments") + .load() + .select(from_json(col("value").cast("string"), payment_schema).alias("data")) + .select("data.*") + .withWatermark("payment_time", "10 minutes") +) + +# Join with time bounds +matched = (orders + .join( + payments, + expr(""" + orders.order_id = payments.order_id AND + payments.payment_time >= orders.order_time - interval 5 minutes AND + payments.payment_time <= orders.order_time + interval 10 minutes + """), + "inner" + ) +) + +# Write results +query = (matched + .writeStream + .format("delta") + .outputMode("append") + .option("checkpointLocation", "/Volumes/catalog/checkpoints/orders_payments") + .trigger(processingTime="30 seconds") + .start("/delta/order_payments") +) +``` + +## Core Concepts + +### Why Stream-Stream Joins Need Watermarks + +Stream-stream joins are stateful: both sides must buffer events until matches are found or state expires. Watermarks define when state can be safely cleaned up. + +```python +# Watermark = latest_event_time - delay_threshold +.withWatermark("event_time", "10 minutes") + +# Events with timestamp < watermark are considered "too late" +# State for late events is automatically cleaned up +``` + +### Join Types and Behavior + +| Join Type | Matches | Late Events | Use Case | +|-----------|---------|-------------|----------| +| **Inner** | Both sides | May still match if other side hasn't expired | Correlation analysis | +| **Left Outer** | All left + matched right | Dropped from left side after watermark | Enrichment with optional data | +| **Right Outer** | All right + matched left | Dropped from right side after watermark | Rarely used | +| **Full Outer** | All events from both | Dropped after watermark | Complete picture | + +## Common Patterns + +### Pattern 1: Order-Payment Matching + +Match orders with payments within a time window: + +```python +orders = (spark + .readStream + .format("kafka") + .option("subscribe", "orders") + .load() + .select(from_json(col("value").cast("string"), order_schema).alias("data")) + .select("data.*") + .withWatermark("order_time", "10 minutes") +) + +payments = (spark + .readStream + .format("kafka") + .option("subscribe", "payments") + .load() + .select(from_json(col("value").cast("string"), payment_schema).alias("data")) + .select("data.*") + .withWatermark("payment_time", "10 minutes") +) + +# Match payments within 10 minutes of order +matched = (orders + .join( + payments, + expr(""" + orders.order_id = payments.order_id AND + payments.payment_time >= orders.order_time - interval 5 minutes AND + payments.payment_time <= orders.order_time + interval 10 minutes + """), + "leftOuter" # Include orders without payments + ) + .withColumn("matched", col("payment_id").isNotNull()) +) + +matched.writeStream \ + .format("delta") \ + .option("checkpointLocation", "/checkpoints/orders_payments") \ + .start("/delta/order_payments") +``` + +### Pattern 2: Click-Conversion Attribution + +Attribute conversions to clicks within a time window: + +```python +impressions = (spark + .readStream + .format("kafka") + .option("subscribe", "impressions") + .load() + .select(from_json(col("value").cast("string"), impression_schema).alias("data")) + .select("data.*") + .withWatermark("impression_time", "1 hour") +) + +conversions = (spark + .readStream + .format("kafka") + .option("subscribe", "conversions") + .load() + .select(from_json(col("value").cast("string"), conversion_schema).alias("data")) + .select("data.*") + .withWatermark("conversion_time", "1 hour") +) + +# Attribute conversion to last impression within 24 hours +attributed = (impressions + .join( + conversions, + expr(""" + impressions.user_id = conversions.user_id AND + impressions.ad_id = conversions.ad_id AND + conversions.conversion_time >= impressions.impression_time AND + conversions.conversion_time <= impressions.impression_time + interval 24 hours + """), + "inner" + ) + .withColumn("attribution_window_hours", + (col("conversion_time").cast("long") - col("impression_time").cast("long")) / 3600) +) + +attributed.writeStream \ + .format("delta") \ + .option("checkpointLocation", "/checkpoints/attribution") \ + .start("/delta/attributed_conversions") +``` + +### Pattern 3: Sessionization Across Streams + +Group events from multiple streams into sessions: + +```python +from pyspark.sql.functions import session_window + +pageviews = (spark + .readStream + .format("kafka") + .option("subscribe", "pageviews") + .load() + .select(from_json(col("value").cast("string"), pageview_schema).alias("data")) + .select("data.*") + .withWatermark("event_time", "30 minutes") +) + +clicks = (spark + .readStream + .format("kafka") + .option("subscribe", "clicks") + .load() + .select(from_json(col("value").cast("string"), click_schema).alias("data")) + .select("data.*") + .withWatermark("event_time", "30 minutes") +) + +# Create session windows for each stream +pageview_sessions = (pageviews + .groupBy( + col("user_id"), + session_window(col("event_time"), "10 minutes") + ) + .agg( + count("*").alias("pageview_count"), + min("event_time").alias("session_start"), + max("event_time").alias("session_end") + ) +) + +click_sessions = (clicks + .groupBy( + col("user_id"), + session_window(col("event_time"), "10 minutes") + ) + .agg( + count("*").alias("click_count"), + min("event_time").alias("session_start"), + max("event_time").alias("session_end") + ) +) + +# Join sessions +joined_sessions = (pageview_sessions + .join( + click_sessions, + ["user_id", "session_window"], + "outer" + ) + .withColumn("total_events", + coalesce(col("pageview_count"), lit(0)) + + coalesce(col("click_count"), lit(0))) +) + +joined_sessions.writeStream \ + .format("delta") \ + .option("checkpointLocation", "/checkpoints/sessions") \ + .start("/delta/user_sessions") +``` + +### Pattern 4: Late Data Handling with Dead Letter Queue + +Route late-arriving events to a separate table: + +```python +def write_with_late_data_handling(batch_df, batch_id): + """Separate on-time and late data""" + from pyspark.sql.functions import current_timestamp, unix_timestamp + + # Calculate delay + processed = batch_df.withColumn( + "processing_delay_seconds", + unix_timestamp(current_timestamp()) - unix_timestamp(col("event_time")) + ) + + # On-time data (within watermark) + on_time = processed.filter(col("processing_delay_seconds") < 600) # 10 minutes + + # Late data + late = processed.filter(col("processing_delay_seconds") >= 600) + + # Write on-time data + (on_time + .drop("processing_delay_seconds") + .write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "stream_join_job") + .saveAsTable("matched_events") + ) + + # Write late data to DLQ + if late.count() > 0: + (late + .withColumn("dlq_reason", lit("LATE_ARRIVAL")) + .withColumn("dlq_timestamp", current_timestamp()) + .write + .format("delta") + .mode("append") + .saveAsTable("late_data_dlq") + ) + +matched.writeStream \ + .foreachBatch(write_with_late_data_handling) \ + .option("checkpointLocation", "/checkpoints/orders_payments") \ + .start() +``` + +## State Management + +### Configure RocksDB for Large State + +For state stores exceeding memory capacity, use RocksDB: + +```python +# Enable RocksDB state store provider +spark.conf.set( + "spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider" +) + +# State is stored on disk, reducing memory pressure +# Recommended for: High cardinality keys, long watermark durations +``` + +### Monitor State Size + +```python +# Read state store directly +state_df = (spark + .read + .format("statestore") + .load("/checkpoints/orders_payments/state") +) + +# Check partition balance +state_df.groupBy("partitionId").count().orderBy(desc("count")).show() + +# Check state size +state_metadata = (spark + .read + .format("state-metadata") + .load("/checkpoints/orders_payments") +) +state_metadata.show() + +# Programmatic monitoring +for stream in spark.streams.active: + progress = stream.lastProgress + if progress and "stateOperators" in progress: + for op in progress["stateOperators"]: + print(f"State rows: {op.get('numRowsTotal', 0)}") + print(f"State memory: {op.get('memoryUsedBytes', 0)}") +``` + +### Control State Growth + +```python +# 1. Use watermarks (automatic cleanup) +.withWatermark("event_time", "10 minutes") # State expires after watermark + +# 2. Reduce key cardinality +# Bad: user_id (millions of distinct values) +# Good: session_id (expires naturally) + +# 3. Set reasonable time bounds +# Bad: unbounded time range +expr("s2.ts >= s1.ts") # State grows forever! + +# Good: bounded time range +expr("s2.ts BETWEEN s1.ts AND s1.ts + interval 1 hour") +``` + +## Watermark Configuration + +### Choosing Watermark Duration + +Balance between latency and completeness: + +```python +# Rule of thumb: 2-3x the expected delay +# If 99th percentile delay is 5 minutes → use 10-15 minute watermark + +# High tolerance (more matches, larger state) +.withWatermark("event_time", "2 hours") + +# Low tolerance (faster results, smaller state) +.withWatermark("event_time", "10 minutes") +``` + +### Multiple Watermarks + +When joining streams with different latencies: + +```python +# Stream 1: Fast, low latency +stream1 = stream1.withWatermark("ts", "5 minutes") + +# Stream 2: Slow, high latency +stream2 = stream2.withWatermark("ts", "15 minutes") + +# Effective watermark = max(5, 15) = 15 minutes +joined = stream1.join(stream2, join_condition, "inner") +``` + +## Production Best Practices + +### Idempotent Writes + +Ensure exactly-once semantics: + +```python +def idempotent_write(batch_df, batch_id): + """Write with transaction version for idempotency""" + (batch_df + .write + .format("delta") + .mode("append") + .option("txnVersion", batch_id) + .option("txnAppId", "stream_join_job") + .saveAsTable("matched_events") + ) + +matched.writeStream \ + .foreachBatch(idempotent_write) \ + .option("checkpointLocation", "/checkpoints/orders_payments") \ + .start() +``` + +### Multi-Stream Joins (3+ Streams) + +Chain joins carefully - each adds state overhead: + +```python +# Step 1: Join streams A and B +ab = (stream_a + .withWatermark("ts", "10 minutes") + .join( + stream_b.withWatermark("ts", "10 minutes"), + expr("a.key = b.key AND b.ts BETWEEN a.ts - interval 5 min AND a.ts + interval 5 min"), + "inner" + ) +) + +# Step 2: Join result with stream C +abc = ab.join( + stream_c.withWatermark("ts", "10 minutes"), + expr("ab.key = c.key AND c.ts BETWEEN ab.ts - interval 5 min AND ab.ts + interval 5 min"), + "inner" +) + +# Note: Result watermark comes from left side (ab) +``` + +### Performance Tuning + +```python +# State store batch retention +spark.conf.set("spark.sql.streaming.stateStore.minBatchesToRetain", "2") + +# State maintenance interval +spark.conf.set("spark.sql.streaming.stateStore.maintenanceInterval", "5m") + +# Shuffle partitions (match worker cores) +spark.conf.set("spark.sql.shuffle.partitions", "200") +``` + +## Monitoring + +### Key Metrics + +```python +# Programmatic monitoring +for stream in spark.streams.active: + status = stream.status + progress = stream.lastProgress + + if progress: + print(f"Stream: {stream.name}") + print(f"Input rate: {progress.get('inputRowsPerSecond', 0)} rows/sec") + print(f"Processing rate: {progress.get('processedRowsPerSecond', 0)} rows/sec") + + # State metrics + if "stateOperators" in progress: + for op in progress["stateOperators"]: + print(f"State rows: {op.get('numRowsTotal', 0)}") + print(f"State memory: {op.get('memoryUsedBytes', 0)}") + + # Watermark + if "eventTime" in progress: + print(f"Watermark: {progress['eventTime'].get('watermark', 'N/A')}") +``` + +### Spark UI Checks + +- **Streaming Tab**: Input rate vs processing rate (processing must exceed input) +- **State Operators**: State size and memory usage +- **Watermark**: Current watermark timestamp +- **Batch Duration**: Should be < trigger interval + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **State too large** | High cardinality keys or long watermark | Reduce key space; decrease watermark duration | +| **Late events dropped** | Watermark too aggressive | Increase watermark delay | +| **No matches** | Time condition wrong | Check time bounds and units (minutes vs hours) | +| **OOM errors** | State explosion | Use RocksDB; increase memory; reduce watermark | +| **Missing watermarks** | State grows forever | Always define watermarks on both sides | +| **Unbounded state** | Open-ended time range | Use bounded time range in join condition | + +## Production Checklist + +- [ ] Watermark configured on both streaming sources +- [ ] Join condition includes explicit time bounds +- [ ] State store provider set (RocksDB for large state) +- [ ] State size monitored and alerts configured +- [ ] Late data handling strategy defined (DLQ or tolerance) +- [ ] Output mode is "append" (required for streaming joins) +- [ ] Checkpoint location is unique per query +- [ ] Idempotent writes configured (txnVersion/txnAppId) +- [ ] Time zones normalized across streams +- [ ] Performance metrics tracked (input rate, state size, watermark lag) + +## Expert Tips + +### Event Time vs Processing Time + +Always use event time for stream-stream joins: + +```python +# ✅ CORRECT: Event time (deterministic) +.withWatermark("event_time", "10 minutes") + +# ❌ WRONG: Processing time (non-deterministic) +# Processing time varies based on system load +# Results are not reproducible +``` + +### Watermark Semantics Deep Dive + +Understanding watermark behavior: + +```python +# Watermark = max_event_time - delay_threshold +# Example: max_event_time = 10:15, delay = 10 min +# Watermark = 10:05 + +# Events with timestamp < 10:05 are "too late" +# - Inner join: May still match if other side hasn't expired +# - Outer join: Dropped from outer side after watermark passes + +# Effective watermark = max(left_watermark, right_watermark) +``` + +### State Store Backend Selection + +Choose the right state store backend: + +```python +# Default: In-memory (fast but limited) +# Use for: Small state (< 10GB), low cardinality keys + +# RocksDB: Disk-backed (slower but scalable) +spark.conf.set( + "spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider" +) +# Use for: Large state (> 10GB), high cardinality keys + +# Monitor state size to decide when to switch +``` + +### Join Condition Best Practices + +Always include explicit time bounds: + +```python +# ❌ BAD: Unbounded (state grows forever) +expr("s1.key = s2.key AND s2.ts >= s1.ts") + +# ✅ GOOD: Bounded (state bounded by watermark) +expr(""" + s1.key = s2.key AND + s2.ts >= s1.ts - interval 5 minutes AND + s2.ts <= s1.ts + interval 10 minutes +""") + +# Why? Bounded ranges allow state cleanup +# Unbounded ranges cause state to grow indefinitely +``` + +## Related Skills + +- `stream-static-joins` - Enrich streams with Delta dimension tables +- `kafka-to-delta` - Kafka ingestion patterns +- `watermark-configuration` - Deep dive on watermark semantics +- `state-store-management` - State store optimization and monitoring diff --git a/.claude/skills/databricks-spark-structured-streaming/streaming-best-practices.md b/.claude/skills/databricks-spark-structured-streaming/streaming-best-practices.md new file mode 100644 index 00000000..9f3927a9 --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/streaming-best-practices.md @@ -0,0 +1,265 @@ +--- +name: "streaming-best-practices" +description: "Production-proven best practices for Spark Streaming: trigger intervals, partitioning, checkpoint management, and cluster configuration for reliable pipelines." +tags: ["spark-streaming", "best-practices", "production", "performance", "expert"] +--- + +# Streaming Best Practices Expert Pack + +## Overview + +A comprehensive checklist distilled from production experience. These practices should hold true in almost all scenarios. + +**Source**: Canadian Data Guy — "Spark Streaming Best Practices" + +## Beginner Checklist + +### 1. Always Set a Trigger Interval + +```python +# ✅ Good: Controls API costs and listing operations +stream.writeStream \ + .trigger(processingTime='5 seconds') \ + .start() + +# ❌ Bad: No trigger means continuous microbatches +# Can cause excessive S3/ADLS listing costs +``` + +**Why**: Fast processing (<1 sec) repeats listing operations, causing unintended costs. + +### 2. Use Auto Loader Notification Mode + +```python +# Switch from file listing to event-based +spark.readStream \ + .format("cloudFiles") \ + .option("cloudFiles.useNotifications", "true") \ + .load("/path/to/data") +``` + +[Auto Loader File Notification Mode](https://docs.databricks.com/ingestion/auto-loader/file-notification-mode.html) + +### 3. Disable S3 Versioning + +```python +# ❌ Don't enable versioning on S3 buckets with Delta +# ✅ Delta has time travel — no need for S3 versioning +# Versioning adds significant latency at scale +``` + +### 4. Co-Locate Compute and Storage + +```python +# ✅ Keep compute and storage in the same region +# Cross-region = latency + egress costs +``` + +### 5. Use ADLS Gen2 on Azure + +```python +# ✅ ADLS Gen2 is optimized for big data analytics +# ❌ Regular blob storage = slower performance +``` + +### 6. Partition Strategy + +```python +# ✅ Partition on low-cardinality columns: date, region, country +# ❌ Avoid high-cardinality: user_id, transaction_id + +# Rule of thumb: < 100,000 partitions +# Example: 10 years × 365 days × 20 countries = 73,000 partitions ✅ +``` + +### 7. Name Your Streaming Query + +```python +# ✅ Easily identifiable in Spark UI +stream.writeStream \ + .option("queryName", "IngestFromKafka") \ + .start() + +# Shows up as "IngestFromKafka" in Streaming tab +``` + +### 8. One Checkpoint Per Stream + +```python +# ✅ Each stream has its own checkpoint +# ❌ Never share checkpoints between streams + +# Example: Two sources → one target +# Source 1 → checkpoint_1 → target +# Source 2 → checkpoint_2 → target +``` + +### 9. Don't Multiplex Streams + +```python +# ❌ Don't run multiple streams on same driver +# Can cause stability issues + +# ✅ Use separate jobs or benchmark thoroughly +``` + +### 10. Optimal Partition Size + +```python +# Target: 100-200MB per partition in memory + +# Tune with: +.option("maxFilesPerTrigger", "100") +.option("maxBytesPerTrigger", "100MB") + +# Monitor in Spark UI → Stages → Partition size +``` + +### 11. Prefer Broadcast Hash Join + +```python +# ✅ BroadcastHashJoin is faster than SortMergeJoin +# Spark auto-broadcasts tables < 100MB + +# Increase threshold if needed: +spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "1g") +``` + +## Advanced Checklist + +### 12. Checkpoint Naming Convention + +```python +# Structure: {table_location}/_checkpoints/_{target_table_name}_starting_{identifier} + +# Examples: +# 1. By timestamp: /delta/events/_checkpoints/_events_starting_2024_01_15 +# 2. By version: /delta/events/_checkpoints/_events_startingVersion_12345 + +# Why: Multiple checkpoints over table lifetime (upgrades, logic changes) +``` + +### 13. Minimize Shuffle Spill + +```python +# ✅ Goal: Shuffle spill (disk) = 0 +# ✅ Only shuffle read should exist + +# Check: Spark UI → SQL → Exchange operators +# If spill > 0: Increase memory or reduce partition size +``` + +### 14. Use RocksDB for Stateful Operations + +```python +# For large state stores, use RocksDB backend +spark.conf.set( + "spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider" +) +``` + +### 15. Event Hubs via Kafka Connector + +```python +# ✅ Use Kafka protocol for Azure Event Hubs +# More flexible partition handling + +# Note: With EventHubs Kafka connector +# Number of cores can differ from partitions +# (vs native EventHubs: cores == partitions) +``` + +### 16. Watermark for State Cleanup + +```python +# ✅ Always use watermark with stateful ops +# Prevents infinite state growth + +stream.withWatermark("timestamp", "10 minutes") \ + .groupBy("user_id") \ + .agg(sum("amount")) + +# Exception: If infinite state needed, store in Delta + ZORDER +``` + +### 17. Deduplication at Scale + +```python +# At trillion-record scale: +# ✅ Delta merge over dropDuplicates + +# dropDuplicates: State store grows very large +# Delta merge: Use table for lookup + +# Example: +spark.sql(""" + MERGE INTO target t + USING source s ON t.event_id = s.event_id + WHEN NOT MATCHED THEN INSERT * +""") +``` + +### 18. Azure Instance Family Selection + +| Workload | Instance Family | +|----------|----------------| +| Map-heavy (parsing, JSON) | F-series | +| Multiple streams from same source | Fsv2-series | +| Joins/aggregations/optimize | DS_v2-series | +| Delta caching | L-series (SSD) | + +### 19. Shuffle Partitions + +```python +# Set equal to total worker cores +spark.conf.set("spark.sql.shuffle.partitions", "200") + +# ❌ Don't set too high +# If changing: Clear checkpoint (stores the old value) +``` + +## Quick Reference + +### Trigger Selection + +| Latency Requirement | Trigger | +|---------------------|---------| +| < 1 second | Real-Time Mode (RTM) | +| 1-10 seconds | processingTime('5 seconds') | +| 1-60 minutes | processingTime based on SLA/3 | +| Batch-like | availableNow=True | + +### Cluster Sizing + +```python +# Fixed-size cluster recommended for streaming +# ❌ Don't use auto-scaling for streaming workloads + +# Why: Pre-allocated resources = predictable latency +``` + +## Monitoring Checklist + +- [ ] Input rate vs processing rate (processing > input) +- [ ] Max offsets behind latest (should decrease over time) +- [ ] Batch duration vs trigger interval (headroom exists) +- [ ] State store size (if using stateful ops) +- [ ] Shuffle spill = 0 +- [ ] Null rate in left joins (data quality) + +## Common Mistakes + +| Mistake | Impact | Fix | +|---------|--------|-----| +| Shared checkpoint | Data loss/corruption | Separate checkpoints | +| No watermark | State explosion | Add watermark | +| S3 versioning | Latency | Disable versioning | +| Autoscaling clusters | Unpredictable latency | Fixed-size clusters | +| High-cardinality partitions | Small files | Partition by date | + +## Related Skills + +- `spark-streaming-master-class-kafka-to-delta` — End-to-end patterns +- `mastering-checkpoints-in-spark-streaming` — Checkpoint deep dive +- `scaling-spark-streaming-jobs` — Performance tuning diff --git a/.claude/skills/databricks-spark-structured-streaming/trigger-and-cost-optimization.md b/.claude/skills/databricks-spark-structured-streaming/trigger-and-cost-optimization.md new file mode 100644 index 00000000..92ba4cb3 --- /dev/null +++ b/.claude/skills/databricks-spark-structured-streaming/trigger-and-cost-optimization.md @@ -0,0 +1,517 @@ +--- +name: trigger-and-cost-optimization +description: Select and tune triggers for Spark Structured Streaming to balance latency and cost. Use when choosing between processingTime, availableNow, and Real-Time Mode (RTM), calculating optimal trigger intervals, optimizing costs through cluster right-sizing, scheduled streaming, multi-stream clusters, or managing latency vs cost trade-offs. +--- + +# Trigger and Cost Optimization + +Select and tune triggers to balance latency requirements with cost. Optimize streaming job costs through trigger tuning, cluster right-sizing, multi-stream clusters, storage optimization, and scheduled execution patterns. + +## Quick Start + +```python +# Cost-optimized: Scheduled streaming instead of continuous +df.writeStream \ + .format("delta") \ + .option("checkpointLocation", "/checkpoints/stream") \ + .trigger(availableNow=True) \ # Process all, then stop + .start("/delta/target") + +# Schedule via Databricks Jobs: Every 15 minutes +# Cost: ~$20/day for 100 tables on 8-core cluster +``` + +## Trigger Types + +### ProcessingTime Trigger + +Process at fixed intervals: + +```python +# Process every 30 seconds +.trigger(processingTime="30 seconds") + +# Process every 5 minutes +.trigger(processingTime="5 minutes") + +# Latency: Trigger interval + processing time +# Cost: Continuous cluster running +``` + +### AvailableNow Trigger + +Process all available data, then stop: + +```python +# Process all available data, then stop +.trigger(availableNow=True) + +# Schedule via Databricks Jobs: +# - Every 15 minutes: Near real-time +# - Every 4 hours: Batch-style + +# Latency: Schedule interval + processing time +# Cost: Cluster runs only during processing +``` + +### Real-Time Mode (RTM) + +Sub-second latency with Photon: + +```python +# Real-Time Mode (Databricks 13.3+) +.trigger(realTime=True) + +# Requirements: +# - Photon enabled +# - Fixed-size cluster (no autoscaling) +# - Latency: < 800ms + +# Cost: Continuous cluster with Photon +``` + +## Trigger Selection Guide + +| Latency Requirement | Trigger | Cost | Use Case | +|---------------------|---------|------|----------| +| < 800ms | RTM | $$$ | Real-time analytics, alerts | +| 1-30 seconds | processingTime | $$ | Near real-time dashboards | +| 15-60 minutes | availableNow (scheduled) | $ | Batch-style SLA | +| > 1 hour | availableNow (scheduled) | $ | ETL pipelines | + +## Trigger Interval Calculation + +### Rule of Thumb: SLA / 3 + +```python +# Calculate trigger interval from SLA +business_sla_minutes = 60 # 1 hour SLA +trigger_interval_minutes = business_sla_minutes / 3 # 20 minutes + +.trigger(processingTime=f"{trigger_interval_minutes} minutes") + +# Why /3? +# - Processing time buffer +# - Recovery time buffer +# - Safety margin +``` + +### Example Calculations + +```python +# Example 1: 1 hour SLA +sla = 60 # minutes +trigger = sla / 3 # 20 minutes +.trigger(processingTime="20 minutes") + +# Example 2: 15 minute SLA +sla = 15 # minutes +trigger = sla / 3 # 5 minutes +.trigger(processingTime="5 minutes") + +# Example 3: Real-time requirement +.trigger(realTime=True) # < 800ms +``` + +## Cost Optimization Strategies + +### Strategy 1: Trigger Interval Tuning + +Balance latency and cost: + +```python +# Shorter interval = higher cost +.trigger(processingTime="5 seconds") # Expensive - continuous processing + +# Longer interval = lower cost +.trigger(processingTime="5 minutes") # Cheaper - less frequent processing + +# Use availableNow for batch-style (cheapest) +.trigger(availableNow=True) # Process backlog, then stop + +# Rule of thumb: SLA / 3 +# Example: 1 hour SLA → 20 minute trigger +``` + +### Strategy 2: Scheduled vs Continuous + +Choose execution pattern based on SLA: + +| Pattern | Cost | Latency | Use Case | +|---------|------|---------|----------| +| Continuous | $$$ | < 1 minute | Real-time requirements | +| 15-min schedule | $$ | 15-30 minutes | Near real-time | +| 4-hour schedule | $ | 4-5 hours | Batch-style SLA | + +```python +# Continuous (expensive) +.trigger(processingTime="30 seconds") + +# Scheduled (cost-effective) +.trigger(availableNow=True) # Schedule via Jobs: Every 15 minutes + +# Batch-style (cheapest) +.trigger(availableNow=True) # Schedule via Jobs: Every 4 hours +``` + +### Strategy 3: Cluster Right-Sizing + +Right-size clusters based on workload: + +```python +# Don't oversize: +# - Monitor CPU utilization (target 60-80%) +# - Check for idle time +# - Use fixed-size clusters (no autoscaling for streaming) + +# Scale test approach: +# 1. Start small +# 2. Monitor lag (max offsets behind latest) +# 3. Scale up if falling behind +# 4. Right-size based on steady state +``` + +### Strategy 4: Multi-Stream Clusters + +Run multiple streams on one cluster: + +```python +# Run multiple streams on one cluster +# Tested: 100 streams on 8-core single-node cluster +# Cost: ~$20/day for 100 tables + +# Example: Multiple streams on same cluster +stream1.writeStream.option("checkpointLocation", "/checkpoints/stream1").start() +stream2.writeStream.option("checkpointLocation", "/checkpoints/stream2").start() +stream3.writeStream.option("checkpointLocation", "/checkpoints/stream3").start() +# ... up to 100+ streams + +# Monitor: CPU/memory per stream +# Scale cluster if aggregate utilization > 80% +``` + +### Strategy 5: Storage Optimization + +Reduce storage costs: + +```sql +-- VACUUM old files +VACUUM table RETAIN 24 HOURS; + +-- Enable auto-optimize to reduce small files +ALTER TABLE table SET TBLPROPERTIES ( + 'delta.autoOptimize.optimizeWrite' = true, + 'delta.autoOptimize.autoCompact' = true +); + +-- Archive old data to cheaper storage +-- Use data retention policies +``` + +## Cost Formula + +``` +Daily Cost = + (Cluster DBU/hour × Hours running) + + (Storage GB × Storage rate) + + (Network egress if applicable) + +Optimization levers: +- Reduce hours running (scheduled triggers) +- Reduce cluster size (right-sizing) +- Reduce storage (VACUUM, compression) +- Reduce network egress (co-locate compute and storage) +``` + +## Common Patterns + +### Pattern 1: Cost-Optimized Scheduled Streaming + +Convert continuous to scheduled: + +```python +# Before: Continuous (expensive) +df.writeStream \ + .trigger(processingTime="30 seconds") \ + .start() + +# After: Scheduled (cost-effective) +df.writeStream \ + .trigger(availableNow=True) \ # Process all, then stop + .start() + +# Schedule via Databricks Jobs: +# - Every 15 minutes: Near real-time +# - Every 4 hours: Batch-style +# Same code, different schedule +``` + +### Pattern 2: Multi-Stream Cluster + +Optimize cluster utilization: + +```python +# Run multiple streams on one cluster +def start_all_streams(): + streams = [] + + # Start multiple streams + for i in range(100): + stream = (spark + .readStream + .table(f"source_{i}") + .writeStream + .format("delta") + .option("checkpointLocation", f"/checkpoints/stream_{i}") + .trigger(availableNow=True) + .start(f"/delta/target_{i}") + ) + streams.append(stream) + + return streams + +# Monitor aggregate CPU/memory +# Scale cluster if needed +``` + +### Pattern 3: RTM for Sub-Second Latency + +Use RTM for real-time requirements: + +```python +# Real-Time Mode for sub-second latency +df.writeStream \ + .format("kafka") + .option("topic", "output") + .trigger(realTime=True) \ + .start() + +# Required configurations: +spark.conf.set("spark.databricks.photon.enabled", "true") +spark.conf.set("spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider") + +# Latency: < 800ms +# Cost: Continuous cluster with Photon +``` + +## Real-Time Mode (RTM) Configuration + +### Enable RTM + +```python +# Enable Real-Time Mode +.trigger(realTime=True) + +# Required configurations: +spark.conf.set("spark.databricks.photon.enabled", "true") +spark.conf.set("spark.sql.streaming.stateStore.providerClass", + "com.databricks.sql.streaming.state.RocksDBStateProvider") + +# Cluster requirements: +# - Fixed-size cluster (no autoscaling) +# - Photon enabled +# - Driver: Minimum 4 cores +``` + +### RTM Use Cases + +```python +# Good for RTM: +# - Sub-second latency requirements +# - Simple transformations +# - Stateless operations +# - Kafka-to-Kafka pipelines + +# Not recommended for RTM: +# - Stateful operations (aggregations, joins) +# - Complex transformations +# - Large batch sizes +``` + +## Performance Considerations + +### Batch Duration vs Trigger Interval + +```python +# Batch duration should be < trigger interval +# Example: +trigger_interval = 30 # seconds +batch_duration = 10 # seconds + +# Healthy: batch_duration < trigger_interval +# Unhealthy: batch_duration >= trigger_interval + +# Monitor in Spark UI: +# - Batch duration +# - Trigger interval +# - Alert if batch duration >= trigger interval +``` + +### Trigger Interval Tuning + +```python +# Start conservative, optimize based on monitoring +# Step 1: Start with SLA / 3 +trigger_interval = business_sla / 3 + +# Step 2: Monitor batch duration +# If batch duration < trigger_interval / 2: Can increase trigger +# If batch duration >= trigger_interval: Decrease trigger + +# Step 3: Optimize for cost vs latency +# Increase trigger interval to reduce cost +# Decrease trigger interval to reduce latency +``` + +## Cost Monitoring + +### Track Per-Stream Costs + +```python +# Tag jobs with stream name +job_tags = { + "stream_name": "orders_stream", + "environment": "prod", + "cost_center": "analytics" +} + +# Use DBU consumption metrics +# Monitor by workspace/cluster +# Track cost per stream over time +``` + +### Monitor Cluster Utilization + +```python +# Check CPU utilization +# Target: 60-80% utilization +# Below 60%: Consider downsizing +# Above 80%: Consider upsizing + +# Check memory utilization +# Monitor for OOM errors +# Adjust cluster size accordingly +``` + +## Latency vs Cost Trade-offs + +### Continuous Processing + +```python +# High cost, low latency +.trigger(processingTime="30 seconds") + +# Cost: Continuous cluster running +# Latency: 30 seconds + processing time +# Use when: Real-time requirements +``` + +### Scheduled Processing + +```python +# Lower cost, higher latency +.trigger(availableNow=True) # Schedule: Every 15 minutes + +# Cost: Cluster runs only during processing +# Latency: Schedule interval + processing time +# Use when: Batch-style SLA acceptable +``` + +### Real-Time Mode + +```python +# Highest cost, lowest latency +.trigger(realTime=True) + +# Cost: Continuous cluster with Photon +# Latency: < 800ms +# Use when: Sub-second latency required +``` + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| **High latency** | Trigger interval too long | Decrease trigger interval or use RTM | +| **High cost** | Continuous processing | Use scheduled (availableNow) | +| **Batch duration > trigger** | Processing too slow | Optimize processing or increase trigger | +| **RTM not working** | Photon not enabled | Enable Photon and configure cluster | + +## Quick Wins + +1. **Change from continuous to 15-minute schedule** - Significant cost reduction +2. **Run multiple streams per cluster** - Better cluster utilization +3. **Enable auto-optimize** - Reduce storage costs +4. **Use Spot instances** - For non-critical streams (with caution) +5. **Archive old data** - Move to cheaper storage tiers + +## Trade-offs + +| Cost Reduction | Impact | Mitigation | +|----------------|--------|------------| +| Longer trigger | Higher latency | Acceptable if SLA allows | +| Smaller cluster | May fall behind | Monitor lag; scale if needed | +| Aggressive VACUUM | Less time travel | Balance retention vs cost | +| Spot instances | Possible interruptions | Use for non-critical streams | +| Scheduled vs continuous | Higher latency | Match to business SLA | + +## Production Best Practices + +### Match Trigger to SLA + +```python +# Calculate trigger from business SLA +def calculate_trigger_interval(sla_minutes): + """Calculate optimal trigger interval""" + return max(30, sla_minutes / 3) # Minimum 30 seconds + +trigger_interval = calculate_trigger_interval(business_sla_minutes) +.trigger(processingTime=f"{trigger_interval} seconds") +``` + +### Cluster Configuration + +```python +# Fixed-size cluster (no autoscaling for streaming) +cluster_config = { + "num_workers": 4, + "node_type_id": "i3.xlarge", + "autotermination_minutes": 60, # Terminate if idle + "enable_elastic_disk": True # Reduce storage costs +} +``` + +### Storage Management + +```sql +-- Enable auto-optimize +ALTER TABLE table SET TBLPROPERTIES ( + 'delta.autoOptimize.optimizeWrite' = true, + 'delta.autoOptimize.autoCompact' = true +); + +-- Periodic VACUUM +VACUUM table RETAIN 7 DAYS; -- Balance retention vs cost + +-- Archive old partitions +-- Move to cheaper storage tier +``` + +## Production Checklist + +- [ ] Trigger type selected based on latency requirements +- [ ] Trigger interval calculated from SLA (SLA / 3) +- [ ] Batch duration monitored (< trigger interval) +- [ ] Cluster right-sized (60-80% utilization) +- [ ] Multiple streams per cluster (if applicable) +- [ ] Scheduled execution (if SLA allows) +- [ ] RTM configured if sub-second latency required +- [ ] Auto-optimize enabled +- [ ] Storage costs monitored +- [ ] Cost per stream tracked + +## Related Skills + +- `kafka-streaming` - RTM configuration for Kafka pipelines +- `checkpoint-best-practices` - Checkpoint management diff --git a/.claude/skills/databricks-synthetic-data-gen/SKILL.md b/.claude/skills/databricks-synthetic-data-gen/SKILL.md new file mode 100644 index 00000000..c046e488 --- /dev/null +++ b/.claude/skills/databricks-synthetic-data-gen/SKILL.md @@ -0,0 +1,261 @@ +--- +name: databricks-synthetic-data-gen +description: "Generate realistic synthetic data using Spark + Faker (strongly recommended). Supports serverless execution, multiple output formats (Parquet/JSON/CSV/Delta), and scales from thousands to millions of rows. For small datasets (<10K rows), can optionally generate locally and upload to volumes. Use when user mentions 'synthetic data', 'test data', 'generate data', 'demo dataset', 'Faker', or 'sample data'." +--- + +> Catalog and schema are **always user-supplied** — never default to any value. If the user hasn't provided them, ask. For any UC write, **always create the schema if it doesn't exist** before writing data. + +# Databricks Synthetic Data Generation + +Generate realistic, story-driven synthetic data for Databricks using **Spark + Faker + Pandas UDFs** (strongly recommended). + +## Data Must Tell a Business Story + +Synthetic data should demonstrate how Databricks helps solve real business problems. + +**The pattern:** Something goes wrong → business impact ($) → analyze root cause → identify affected customers → fix and prevent. + +**Key principles:** +- **Problem → Impact → Analysis → Solution** — Include an incident, anomaly, or issue that causes measurable business impact. The data lets you find the root cause and act on it. +- **Industry-relevant but simple** — Use domain terms (e.g., "SLA breach", "churn", "stockout") but keep the schema easy to understand. A few tables, clear relationships. +- **Business metrics with $ impact** — Revenue, MRR, cost, conversion rate. Every story needs a dollar sign to show why it matters. +- **Tables explain each other** — Ticket spike? Incident table shows the outage. Revenue drop? Churn table shows who left and why. All data connects. +- **Actionable insights** — Data should answer: What happened? Who's affected? How much did it cost? How do we prevent it? + +**Why no flat distributions:** Uniform data has no story — no spikes, no anomalies, no cohort, no 20/80, no skew, nothing to investigate. It can't show Databricks' value for root cause analysis. + +## References + +| When | Guide | +|------|-------| +| User mentions **ML model training** or complex time patterns | [references/1-data-patterns.md](references/1-data-patterns.md) — ML-ready data, time multipliers, row coherence | +| Errors during generation | [references/2-troubleshooting.md](references/2-troubleshooting.md) — Fixing common issues | + +## Critical Rules + +1. **Data tells a story** — Something goes wrong, impacts $, can be analyzed and fixed. Show Databricks value. +2. **All data serves the story** — Every table and column must be coherent and usable in dashboards or ML models. No orphan data, no random noise — if it doesn't help explain or plot a futur dashboard or predict, don't generate it. +3. **Industry terms, simple schema** — Use domain-specific vocabulary but keep it easy to understand (few tables, clear relationships) +4. **Never uniform distributions** — Skewed categories, log-normal amounts, 80/20 patterns. Flat = no story = useless +5. **Enough data for trends** — ~100K+ rows for main tables so patterns survive aggregation +6. **Ask for catalog/schema** — Never default, always confirm before generating +7. **Present plan for approval** — Show tables, distributions, assumptions before writing code +8. **Master tables first** — Generate parent tables, write to Delta, then create children with valid FKs +9. **Use Spark + Faker + Pandas UDFs** — Scalable, parallel. Polars only if user explicitly wants local + <30K rows +10. **Use Databricks Connect Serverless by default to generate data** — Update databricks-connect on python 3.12 if required (avoid using execute_code unless instructed to not use Databricks Connect) +11. **No `.cache()` or `.persist()`** — Not supported on serverless. Write to Delta, read back for joins +12. **No Python loops or `.collect()`** — Use Spark parallelism. No driver-side iteration, avoid Pandas↔Spark conversions + +## Generation Planning Workflow + +**Before generating any code, you MUST present a plan for user approval.** + +### ⚠️ MUST DO: Confirm Catalog Before Proceeding + +**You MUST explicitly ask the user which catalog to use.** Do not assume or proceed without confirmation. + +Example prompt to user: +> "Which Unity Catalog should I use for this data?" + +When presenting your plan, always show the selected catalog prominently: +``` +📍 Output Location: catalog_name.schema_name + Volume: /Volumes/catalog_name/schema_name/raw_data/ +``` + +This makes it easy for the user to spot and correct if needed. + +### Step 1: Gather Requirements + +Ask the user about: +- **Catalog/Schema** — Which catalog to use? +- **Domain** — E-commerce, support tickets, IoT, financial? (Use industry terms) + +**If user doesn't specify a story:** Propose one. Don't generate bland data — suggest an incident, anomaly, or trend that shows Databricks value (e.g., "I'll include a system outage that causes ticket spike and churn — this lets you demo root cause analysis"). + +### Step 2: Present Plan with Story + +Show a clear specification with **the business story and your assumptions surfaced**: + +``` +📍 Output Location: {user_catalog}.support_demo + Volume: /Volumes/{user_catalog}/support_demo/raw_data/ + +📖 Story: A payment system outage causes support ticket spike. Resolution times + degrade, enterprise customers churn, revenue drops $2.3M. With Databricks we + identify the root cause, affected customers, and prevent future impact. +``` + +| Table | Description | Rows | Key Assumptions | +|-------|-------------|------|-----------------| +| customers | Customer profiles with tier, MRR | 10,000 | Enterprise 10% but 60% of revenue | +| tickets | Support tickets with priority, resolution_time | 80,000 | Spike during outage, SLA breaches | +| incidents | System events (outages, deployments) | 50 | Payment outage mid-month | +| churn_events | Customer cancellations with reason | 500 | Spike after poor support experience | + +**Business metrics:** +- `customers.mrr` — Revenue at risk ($) +- `tickets.resolution_hours` — SLA performance +- `churn_events.lost_mrr` — Churn impact ($) + +**The story this data tells:** +- Incident table shows payment outage on March 15 +- Tickets spike 5x during outage, resolution time degrades from 4h → 18h +- Enterprise customers with SLA breaches churn 3 weeks later +- Total impact: $2.3M lost MRR, traceable to one incident +- **Databricks value:** Root cause analysis, identify at-risk customers, build alerting + +**Ask user**: "Does this story work? Any adjustments?" + +### Step 3: Ask About Data Features + +- [x] Skew (non-uniform distributions) - **Enabled by default** +- [x] Joins (referential integrity) - **Enabled by default** +- [ ] Bad data injection (for data quality testing) +- [ ] Multi-language text +- [ ] Incremental mode (append instead of overwrite) + +### Pre-Generation Checklist + +- [ ] **Catalog confirmed** - User explicitly approved which catalog to use +- [ ] Output location shown prominently in plan (easy to spot/change) +- [ ] Table specification shown and approved +- [ ] Assumptions about distributions confirmed +- [ ] User confirmed compute preference (Databricks Connect on serverless recommended) +- [ ] Data features selected + +**Do NOT proceed to code generation until user approves the plan, including the catalog.** + +### Post-Generation Checklist + +After generating data, use `get_volume_folder_details` to validate the output matches requirements: +- Row counts match the plan +- Schema matches expected columns and types +- Data distributions look reasonable (check column stats) + +## Use Databricks Connect Spark + Faker Pattern + +```python +from databricks.connect import DatabricksSession, DatabricksEnv +from pyspark.sql import functions as F +from pyspark.sql.types import StringType +import pandas as pd + +# Setup serverless with dependencies (MUST list all libs used in UDFs) +env = DatabricksEnv().withDependencies("faker", "holidays") +spark = DatabricksSession.builder.withEnvironment(env).serverless(True).getOrCreate() + +# Pandas UDF pattern - import lib INSIDE the function +@F.pandas_udf(StringType()) +def fake_name(ids: pd.Series) -> pd.Series: + from faker import Faker # Import inside UDF + fake = Faker() + return pd.Series([fake.name() for _ in range(len(ids))]) + +# Generate with spark.range, apply UDFs +customers_df = spark.range(0, 10000, numPartitions=16).select( + F.concat(F.lit("CUST-"), F.lpad(F.col("id").cast("string"), 5, "0")).alias("customer_id"), + fake_name(F.col("id")).alias("name"), +) + +# Write to Volume as Parquet (default for raw data) +# Path is a folder with table name: /Volumes/catalog/schema/raw_data/customers/ +spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}") +spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data") +customers_df.write.mode("overwrite").parquet(f"/Volumes/{CATALOG}/{SCHEMA}/raw_data/customers") +``` + +**Partitions by scale:** `spark.range(N, numPartitions=P)` +- <100K rows: 8 partitions +- 100K-500K: 16 partitions +- 500K-1M: 32 partitions +- 1M+: 64+ partitions + +**Output formats:** +- **Parquet to Volume** (default): `df.write.parquet("/Volumes/.../raw_data/table")` — raw data for pipelines +- **Delta Table**: `df.write.saveAsTable("catalog.schema.table")` — if user wants queryable tables +- **JSON/CSV**: small dimension tables, replicate legacy systems + +## Performance Rules + +Generated scripts must be highly performant. **Never** do these: + +| Anti-Pattern | Why It's Slow | Do This Instead | +|--------------|---------------|-----------------| +| Python loops on driver | Single-threaded, no parallelism | Use `spark.range()` + Spark operations | +| `.collect()` then iterate | Brings all data to driver memory | Keep data in Spark, use DataFrame ops | +| Pandas → Spark → Pandas | Serialization overhead, defeats distribution | Stay in Spark, use `pandas_udf` only for UDFs | +| Read/write temp files | Unnecessary I/O | Chain DataFrame transformations | +| Scalar UDFs | Row-by-row processing | Use `pandas_udf` for batch processing | + +**Good pattern:** `spark.range()` → Spark transforms → `pandas_udf` for Faker → write directly + +## Common Patterns + +### Weighted Categories (never uniform) +```python +F.when(F.rand() < 0.6, "Free").when(F.rand() < 0.9, "Pro").otherwise("Enterprise") +``` + +### Log-Normal Amounts (in a pandas UDF) +Use `np.random.lognormal(mean, sigma)` — always positive, long tail: +- Enterprise: `lognormal(7.5, 0.8)` → ~$1800 median +- Pro: `lognormal(5.5, 0.7)` → ~$245 median +- Free: `lognormal(4.0, 0.6)` → ~$55 median + +### Date Range (Last 6 Months) +```python +END_DATE = datetime.now() +START_DATE = END_DATE - timedelta(days=180) +``` + +### Infrastructure (always create in script) +```python +spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}") +spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data") +``` + +### Referential Integrity (FK pattern) +Write master table to Delta first, then read back for FK joins (no `.cache()` on serverless): +```python +# 1. Write master table +customers_df.write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.customers") + +# 2. Read back for FK lookup +customer_lookup = spark.table(f"{CATALOG}.{SCHEMA}.customers").select("customer_idx", "customer_id") + +# 3. Generate child table with valid FKs via join +orders_df = spark.range(N_ORDERS).select( + (F.abs(F.hash(F.col("id"))) % N_CUSTOMERS).alias("customer_idx") +) +orders_with_fk = orders_df.join(customer_lookup, on="customer_idx") +``` + +## Setup + +Requires Python 3.12 and databricks-connect>=16.4. Use `uv`: + +```bash +uv pip install "databricks-connect>=16.4,<17.4" faker numpy pandas holidays +``` + +## Related Skills + +- **databricks-unity-catalog** — Managing catalogs, schemas, and volumes +- **databricks-bundles** — DABs for production deployment + +## Common Issues + +| Issue | Solution | +|-------|----------| +| `ImportError: cannot import name 'DatabricksEnv'` | Upgrade: `uv pip install "databricks-connect>=16.4"` | +| Python 3.11 instead of 3.12 | Python 3.12 required. Use `uv` to create env with correct version | +| `ModuleNotFoundError: faker` | Add to `withDependencies()`, import inside UDF | +| Faker UDF is slow | Use `pandas_udf` for batch processing | +| Out of memory | Increase `numPartitions` in `spark.range()` | +| Referential integrity errors | Write master table to Delta first, read back for FK joins | +| `PERSIST TABLE is not supported on serverless` | **NEVER use `.cache()` or `.persist()` with serverless** - write to Delta table first, then read back | +| `F.window` vs `Window` confusion | Use `from pyspark.sql.window import Window` for `row_number()`, `rank()`, etc. `F.window` is for streaming only. | +| Broadcast variables not supported | **NEVER use `spark.sparkContext.broadcast()` with serverless** | + +See [references/2-troubleshooting.md](references/2-troubleshooting.md) for full troubleshooting guide. diff --git a/.claude/skills/databricks-synthetic-data-gen/references/1-data-patterns.md b/.claude/skills/databricks-synthetic-data-gen/references/1-data-patterns.md new file mode 100644 index 00000000..eba64916 --- /dev/null +++ b/.claude/skills/databricks-synthetic-data-gen/references/1-data-patterns.md @@ -0,0 +1,146 @@ +# Data Patterns Guide + +Creating realistic synthetic data that tells a story. + +> **Note:** This guide provides principles and simplified examples. Actual implementations should be more sophisticated — use domain-specific distributions, realistic business rules, and correlations that reflect the user's actual use case. Ask clarifying questions to understand the business context before generating. + +## Core Principles + +### 1. Data Must Be Interesting + +Synthetic data should reveal patterns humans can see in dashboards and ML models can learn from: + +- **Visible trends** — Revenue growth, seasonal spikes, degradation over time +- **Actionable segments** — Clear differences between customer tiers, regions, product categories +- **Anomalies to detect** — Fraud patterns, equipment failures, churn signals +- **Correlations to discover** — Higher tier = more spend, faster resolution = better CSAT + +**Anti-pattern:** Uniform random data with no story — useless for demos and ML. + +### 2. Non-Uniform Distributions + +Real data is never uniformly distributed. Use appropriate distributions: + +| Distribution | When to Use | Examples | +|--------------|-------------|----------| +| **Log-normal** | Monetary values, sizes | Order amounts, salaries, file sizes | +| **Pareto (80/20)** | Popularity, wealth | 20% of customers = 80% of revenue | +| **Exponential** | Time between events | Support resolution time, session duration | +| **Weighted categorical** | Skewed categories | Status (70% complete, 5% failed), tiers | + +```python +# Log-normal for amounts (long tail, always positive) +amount = np.random.lognormal(mean=5.5, sigma=0.8) # ~$245 median + +# Pareto for power-law (few large, many small) +value = (np.random.pareto(a=1.5) + 1) * base_value + +# Exponential for time-to-event +hours = np.random.exponential(scale=24) # avg 24h, skewed right +``` + +### 3. Row Coherence + +Attributes within a row must make business sense together. Generate correlated attributes in a single UDF for example: + +| If This... | Then This... | +|------------|--------------| +| Enterprise tier | Higher order amounts, more activity, priority support | +| Critical priority | Faster resolution, more interactions | +| Older equipment | Higher failure rate, more anomalies | +| Large transaction + unusual hour | Higher fraud probability | +| Fast resolution | Higher CSAT score | + +```python +@F.pandas_udf("struct") +def generate_coherent_ticket(tiers: pd.Series) -> pd.DataFrame: + """All attributes correlate logically within each row.""" + results = [] + for tier in tiers: + # Priority depends on tier + priority = "Critical" if tier == "Enterprise" and random() < 0.3 else "Medium" + # Resolution depends on priority + resolution = np.random.exponential(4 if priority == "Critical" else 36) + # CSAT depends on resolution + csat = 5 if resolution < 4 else (3 if resolution < 24 else 2) + results.append({"priority": priority, "resolution_hours": resolution, "csat": csat}) + return pd.DataFrame(results) +``` + +### 4. The 80/20 Rule + +Apply power-law distributions where appropriate: + +- **20% of customers** generate 80% of orders/revenue +- **20% of products** account for 80% of sales +- **20% of support agents** handle 80% of tickets + +Implementation: Use weighted sampling when assigning FKs, not uniform random. + +### 5. Time-Based Patterns + +Most data has temporal patterns: + +- **Weekday vs weekend** — B2B drops on weekends, B2C peaks +- **Business hours** — Support tickets cluster 9am-5pm +- **Seasonality** — Q4 retail spike, summer travel peak +- **Trends** — Growth over time, degradation curves + +```python +def get_volume_multiplier(date): + multiplier = 1.0 + if date.weekday() >= 5: multiplier *= 0.6 # Weekend drop + if date.month in [11, 12]: multiplier *= 1.5 # Holiday spike + return multiplier +``` + +### 6. ML-Ready Data + +If data will train ML models, ensure: + +- **Signal exists** — The patterns you want the model to learn are present +- **Noise is realistic** — Not too clean (overfitting) or too noisy (unlearnable) +- **Class balance** — Fraud at 0.1-1%, not 50/50 (unrealistic) +- **Temporal validity** — Train/test split respects time (no future leakage) + +## Referential Integrity + +Generate master tables first, write to Delta, then join for FKs: + +```python +# 1. Generate and write master table +customers_df.write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.customers") + +# 2. Read back for FK joins (NOT cache - unsupported on serverless) +customer_lookup = spark.table(f"{CATALOG}.{SCHEMA}.customers") + +# 3. Generate child table with valid FKs via join +orders_df = spark.range(N_ORDERS).select( + (F.abs(F.hash(F.col("id"))) % N_CUSTOMERS).alias("customer_idx") +) +orders_with_fk = orders_df.join(customer_lookup, on="customer_idx") +``` + +## Data Volume + +Generate enough rows so patterns survive aggregation: + +| Analysis Type | Minimum Rows | Rationale | +|---------------|--------------|-----------| +| Daily dashboard | 50-100/day | Trends visible after weekly rollup | +| Category comparison | 500+ per category | Statistical significance | +| ML training | 10K-100K+ | Enough signal for model learning | +| Customer-level | 5-20 events/customer | Individual patterns visible | + +**Rule of thumb:** If you'll GROUP BY a column, ensure each group has 100+ rows. + +--- + +## Remember + +These are guiding principles, not templates. Real implementations should: +- Reflect the user's specific business domain and terminology +- Use realistic parameter values (research typical ranges for the industry) +- Include edge cases relevant to the use case (returns, cancellations, failures) +- Have more complex correlations than shown in examples above +- **Never use flat/uniform distributions** — categories, tiers, regions, statuses should always be skewed (e.g., 60/30/10 not 33/33/33) diff --git a/.claude/skills/databricks-synthetic-data-gen/references/2-troubleshooting.md b/.claude/skills/databricks-synthetic-data-gen/references/2-troubleshooting.md new file mode 100644 index 00000000..420b3500 --- /dev/null +++ b/.claude/skills/databricks-synthetic-data-gen/references/2-troubleshooting.md @@ -0,0 +1,324 @@ +# Troubleshooting Guide + +Common issues and solutions for synthetic data generation. + +## Environment Issues + +### ModuleNotFoundError: faker (or other library) + +**Problem:** Dependencies not available in execution environment. + +**Solutions by execution mode:** + +| Mode | Solution | +|------|----------| +| **DB Connect 16.4+** | Use `DatabricksEnv().withDependencies("faker", "pandas", ...)` | +| **Older DB Connect with Serverless** | Create job with `environments` parameter | +| **Databricks Runtime** | Use Databricks CLI to install `faker holidays` | +| **Classic cluster** | Use Databricks CLI to install libraries. `databricks libraries install --json '{"cluster_id": "", "libraries": [{"pypi": {"package": "faker"}}, {"pypi": {"package": "holidays"}}]}'` | + +```python +# For DB Connect 16.4+ +from databricks.connect import DatabricksSession, DatabricksEnv + +env = DatabricksEnv().withDependencies("faker", "pandas", "numpy", "holidays") +spark = DatabricksSession.builder.withEnvironment(env).serverless(True).getOrCreate() +``` + +### DatabricksEnv not found + +**Problem:** Using older databricks-connect version. + +**Solution:** Upgrade to 16.4+ or use job-based approach: + +```bash +# Upgrade (prefer uv, fall back to pip) +uv pip install "databricks-connect>=16.4,<17.4" +# or: pip install "databricks-connect>=16.4,<17.4" + +# Or use job with environments parameter instead +``` + +### serverless_compute_id error + +**Problem:** Missing serverless configuration. + +**Solution:** Add to `~/.databrickscfg`: + +```ini +[DEFAULT] +host = https://your-workspace.cloud.databricks.com/ +serverless_compute_id = auto +auth_type = databricks-cli +``` + +--- + +## Execution Issues + +### CRITICAL: cache() and persist() NOT supported on serverless + +**Problem:** Using `.cache()` or `.persist()` on serverless compute fails with: +``` +AnalysisException: [NOT_SUPPORTED_WITH_SERVERLESS] PERSIST TABLE is not supported on serverless compute. +``` + +**Why this happens:** Serverless compute does not support caching DataFrames in memory. This is a fundamental limitation of the serverless architecture. + +**Solution:** Write master tables to Delta first, then read them back for FK joins: + +```python +# BAD - will fail on serverless +customers_df = spark.range(0, N_CUSTOMERS)... +customers_df.cache() # ❌ FAILS: "PERSIST TABLE is not supported on serverless compute" + +# GOOD - write to Delta, then read back +customers_df = spark.range(0, N_CUSTOMERS)... +customers_df.write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.customers") +customer_lookup = spark.table(f"{CATALOG}.{SCHEMA}.customers") # ✓ Read from Delta +``` + +**Best practice for referential integrity:** +1. Generate master table (e.g., customers) +2. Write to Delta table +3. Read back for FK lookup joins +4. Generate child tables (e.g., orders, tickets) with valid FKs +5. Write child tables to Delta + +--- + +### Serverless job fails to start + +**Possible causes:** +1. Workspace doesn't have serverless enabled +2. Unity Catalog permissions missing +3. Invalid environment configuration + +**Solutions:** +```python +# Verify serverless is available +# Try creating a simple job first to test + +# Check Unity Catalog permissions +spark.sql("SELECT current_catalog(), current_schema()") +``` + +### Classic cluster startup slow (3-8 minutes) + +**Problem:** Clusters take time to start. + +**Solution:** Switch to serverless: + +```python +# Instead of: +# spark = DatabricksSession.builder.clusterId("xxx").getOrCreate() + +# Use: +spark = DatabricksSession.builder.serverless(True).getOrCreate() +``` + +### "Either base environment or version must be provided" + +**Problem:** Missing `client` in job environment spec. + +**Solution:** Add `"client": "4"` to the spec: + +```python +{ + "environments": [{ + "environment_key": "datagen_env", + "spec": { + "client": "4", # Required! + "dependencies": ["faker", "numpy", "pandas"] + } + }] +} +``` + +--- + +## Data Generation Issues + +### AttributeError: 'function' object has no attribute 'partitionBy' + +**Problem:** Using `F.window` instead of `Window` for analytical window functions. + +```python +# WRONG - F.window is for time-based tumbling/sliding windows (streaming) +window_spec = F.window.partitionBy("account_id").orderBy("contact_id") +# Error: AttributeError: 'function' object has no attribute 'partitionBy' + +# CORRECT - Window is for analytical window specifications +from pyspark.sql.window import Window +window_spec = Window.partitionBy("account_id").orderBy("contact_id") +``` + +**When to use Window:** For analytical functions like `row_number()`, `rank()`, `lead()`, `lag()`: + +```python +from pyspark.sql.window import Window + +# Mark first contact per account as primary +window_spec = Window.partitionBy("account_id").orderBy("contact_id") +contacts_df = contacts_df.withColumn( + "is_primary", + F.row_number().over(window_spec) == 1 +) +``` + +--- + +### Faker UDF is slow + +**Problem:** Single-row UDFs don't parallelize well. + +**Solution:** Use `pandas_udf` for batch processing: + +```python +# SLOW - scalar UDF +@F.udf(returnType=StringType()) +def slow_fake_name(): + return Faker().name() + +# FAST - pandas UDF (batch processing) +@F.pandas_udf(StringType()) +def fast_fake_name(ids: pd.Series) -> pd.Series: + fake = Faker() + return pd.Series([fake.name() for _ in range(len(ids))]) +``` + +### Out of memory with large data + +**Problem:** Not enough partitions for data size. + +**Solution:** Increase partitions: + +```python +# For large datasets (1M+ rows) +customers_df = spark.range(0, N_CUSTOMERS, numPartitions=64) # Increase from default +``` + +| Data Size | Recommended Partitions | +|-----------|----------------------| +| < 100K | 8 | +| 100K - 500K | 16 | +| 500K - 1M | 32 | +| 1M+ | 64+ | + +### Context corrupted on classic cluster + +**Problem:** Stale execution context. + +**Solution:** Create fresh context (omit context_id), reinstall libraries: + +```python +# Don't reuse context_id if you see strange errors +# Let it create a new context +``` + +### Referential integrity violations + +**Problem:** Foreign keys reference non-existent parent records. + +**Solution:** Write master table to Delta first, then read back for FK joins: + +```python +# 1. Generate and WRITE master table (do NOT use cache with serverless!) +customers_df = spark.range(0, N_CUSTOMERS)... +customers_df.write.mode("overwrite").saveAsTable(f"{CATALOG}.{SCHEMA}.customers") + +# 2. Read back for FK lookups +customer_lookup = spark.table(f"{CATALOG}.{SCHEMA}.customers").select("customer_id", "tier") + +# 3. Generate child table with valid FKs +orders_df = spark.range(0, N_ORDERS).join( + customer_lookup, + on=, + how="left" +) +``` + +> **WARNING:** Do NOT use `.cache()` or `.persist()` with serverless compute. See the dedicated section above. + +--- + +## Data Quality Issues + +### Uniform distributions (unrealistic) + +**Problem:** All customers have similar order counts, amounts are evenly distributed. + +**Solution:** Use non-linear distributions: + +```python +# BAD - uniform +amounts = np.random.uniform(10, 1000, N) + +# GOOD - log-normal (realistic) +amounts = np.random.lognormal(mean=5, sigma=0.8, N) +``` + +### Missing time-based patterns + +**Problem:** Data doesn't reflect weekday/weekend or seasonal patterns. + +**Solution:** Add multipliers: + +```python +import holidays + +US_HOLIDAYS = holidays.US(years=[2024, 2025]) + +def get_multiplier(date): + mult = 1.0 + if date.weekday() >= 5: # Weekend + mult *= 0.6 + if date in US_HOLIDAYS: + mult *= 0.3 + return mult +``` + +### Incoherent row attributes + +**Problem:** Enterprise customer has low-value orders, critical ticket has slow resolution. + +**Solution:** Correlate attributes: + +```python +# Priority based on tier +if tier == 'Enterprise': + priority = np.random.choice(['Critical', 'High'], p=[0.4, 0.6]) +else: + priority = np.random.choice(['Medium', 'Low'], p=[0.6, 0.4]) + +# Resolution based on priority +resolution_scale = {'Critical': 4, 'High': 12, 'Medium': 36, 'Low': 72} +resolution_hours = np.random.exponential(scale=resolution_scale[priority]) +``` + +--- + +## Validation Steps + +After generation, verify your data: + +```python +# 1. Check row counts +print(f"Customers: {customers_df.count():,}") +print(f"Orders: {orders_df.count():,}") + +# 2. Verify distributions +customers_df.groupBy("tier").count().show() +orders_df.describe("amount").show() + +# 3. Check referential integrity +orphans = orders_df.join( + customers_df, + orders_df.customer_id == customers_df.customer_id, + "left_anti" +) +print(f"Orphan orders: {orphans.count()}") + +# 4. Verify date range +orders_df.select(F.min("order_date"), F.max("order_date")).show() +``` diff --git a/.claude/skills/databricks-synthetic-data-gen/scripts/generate_synthetic_data.py b/.claude/skills/databricks-synthetic-data-gen/scripts/generate_synthetic_data.py new file mode 100644 index 00000000..b9f953fa --- /dev/null +++ b/.claude/skills/databricks-synthetic-data-gen/scripts/generate_synthetic_data.py @@ -0,0 +1,390 @@ +"""Generate synthetic data using Spark + Faker + Pandas UDFs. + +This is the recommended approach for ALL data generation tasks: +- Scales from thousands to millions of rows +- Parallel execution via Spark +- Direct write to Unity Catalog +- Works with serverless and classic compute + +Auto-detects environment and uses: +- DatabricksEnv with managed dependencies if databricks-connect >= 16.4 (local) +- Standard session if running on Databricks Runtime or older databricks-connect +""" +import sys +import os +from pyspark.sql import functions as F +from pyspark.sql.window import Window +from pyspark.sql.types import StringType, DoubleType, StructType, StructField, IntegerType +import numpy as np +import pandas as pd +from datetime import datetime, timedelta + +# ============================================================================= +# CONFIGURATION +# ============================================================================= +# Compute - Serverless strongly recommended +USE_SERVERLESS = True # Set to False and provide CLUSTER_ID for classic compute +CLUSTER_ID = None # Only used if USE_SERVERLESS=False + +# Storage - Update these for your environment +CATALOG = "" # REQUIRED: replace with your catalog +SCHEMA = "" # REQUIRED: replace with your schema +VOLUME_PATH = f"/Volumes/{CATALOG}/{SCHEMA}/raw_data" + +# Data sizes +N_CUSTOMERS = 10_000 +N_ORDERS = 50_000 +PARTITIONS = 16 # Adjust: 8 for <100K, 32 for 1M+ + +# Date range - last 6 months from today +END_DATE = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0) +START_DATE = END_DATE - timedelta(days=180) + +# Write mode - "overwrite" for one-time, "append" for incremental +WRITE_MODE = "overwrite" + +# Bad data injection for testing data quality rules +INJECT_BAD_DATA = False # Set to True to inject bad data +BAD_DATA_CONFIG = { + "null_rate": 0.02, # 2% nulls in required fields + "outlier_rate": 0.01, # 1% impossible values + "orphan_fk_rate": 0.01, # 1% orphan foreign keys +} + +# Reproducibility +SEED = 42 + +# Tier distribution: Free 60%, Pro 30%, Enterprise 10% +TIER_PROBS = [0.6, 0.3, 0.1] + +# Region distribution +REGION_PROBS = [0.4, 0.25, 0.2, 0.15] + +# ============================================================================= +# ENVIRONMENT DETECTION AND SESSION CREATION +# ============================================================================= + +def is_databricks_runtime(): + """Check if running on Databricks Runtime vs locally.""" + return "DATABRICKS_RUNTIME_VERSION" in os.environ + +def get_databricks_connect_version(): + """Get databricks-connect version as (major, minor) tuple or None.""" + try: + import importlib.metadata + version_str = importlib.metadata.version('databricks-connect') + parts = version_str.split('.') + return (int(parts[0]), int(parts[1])) + except Exception: + return None + +# Detect environment +on_runtime = is_databricks_runtime() +db_version = get_databricks_connect_version() + +print("=" * 80) +print("ENVIRONMENT DETECTION") +print("=" * 80) +print(f"Running on Databricks Runtime: {on_runtime}") +if db_version: + print(f"databricks-connect version: {db_version[0]}.{db_version[1]}") +else: + print("databricks-connect: not available") + +# Use DatabricksEnv with managed dependencies if: +# - Running locally (not on Databricks Runtime) +# - databricks-connect >= 16.4 +use_managed_deps = (not on_runtime) and db_version and db_version >= (16, 4) + +if use_managed_deps: + print("Using DatabricksEnv with managed dependencies") + print("=" * 80) + from databricks.connect import DatabricksSession, DatabricksEnv + + env = DatabricksEnv().withDependencies("faker", "pandas", "numpy", "holidays") + + if USE_SERVERLESS: + spark = DatabricksSession.builder.withEnvironment(env).serverless(True).getOrCreate() + print("Connected to serverless compute with managed dependencies!") + else: + if not CLUSTER_ID: + raise ValueError("CLUSTER_ID must be set when USE_SERVERLESS=False") + spark = DatabricksSession.builder.withEnvironment(env).clusterId(CLUSTER_ID).getOrCreate() + print(f"Connected to cluster with managed dependencies!") +else: + print("Using standard session (dependencies must be pre-installed)") + print("=" * 80) + + # Check that UDF dependencies are available + print("\nChecking UDF dependencies...") + missing_deps = [] + + try: + from faker import Faker + print(" faker: OK") + except ImportError: + missing_deps.append("faker") + print(" faker: MISSING") + + try: + import pandas as pd + print(" pandas: OK") + except ImportError: + missing_deps.append("pandas") + print(" pandas: MISSING") + + if missing_deps: + print("\n" + "=" * 80) + print("ERROR: Missing dependencies for UDFs") + print("=" * 80) + print(f"Missing: {', '.join(missing_deps)}") + if on_runtime: + print('\nSolution: Install libraries via Databricks CLI:') + print(' databricks libraries install --json \'{"cluster_id": "", "libraries": [{"pypi": {"package": "faker"}}, {"pypi": {"package": "holidays"}}]}\'') + else: + print("\nSolution: Upgrade to databricks-connect >= 16.4 for managed deps") + print(" Or create a job with environment settings") + print("=" * 80) + sys.exit(1) + + print("\nAll dependencies available") + print("=" * 80) + + from databricks.connect import DatabricksSession + + if USE_SERVERLESS: + spark = DatabricksSession.builder.serverless(True).getOrCreate() + print("Connected to serverless compute") + else: + if not CLUSTER_ID: + raise ValueError("CLUSTER_ID must be set when USE_SERVERLESS=False") + spark = DatabricksSession.builder.clusterId(CLUSTER_ID).getOrCreate() + print(f"Connected to cluster ") + +# Import Faker for UDF definitions +from faker import Faker + +# ============================================================================= +# DEFINE PANDAS UDFs FOR FAKER DATA +# ============================================================================= + +@F.pandas_udf(StringType()) +def fake_name(ids: pd.Series) -> pd.Series: + """Generate realistic person names.""" + fake = Faker() + Faker.seed(SEED) + return pd.Series([fake.name() for _ in range(len(ids))]) + +@F.pandas_udf(StringType()) +def fake_company(ids: pd.Series) -> pd.Series: + """Generate realistic company names.""" + fake = Faker() + Faker.seed(SEED) + return pd.Series([fake.company() for _ in range(len(ids))]) + +@F.pandas_udf(StringType()) +def fake_address(ids: pd.Series) -> pd.Series: + """Generate realistic addresses.""" + fake = Faker() + Faker.seed(SEED) + return pd.Series([fake.address().replace('\n', ', ') for _ in range(len(ids))]) + +@F.pandas_udf(StringType()) +def fake_email(names: pd.Series) -> pd.Series: + """Generate email based on name.""" + emails = [] + for name in names: + if name: + domain = name.lower().replace(" ", ".").replace(",", "")[:20] + emails.append(f"{domain}@example.com") + else: + emails.append("unknown@example.com") + return pd.Series(emails) + +@F.pandas_udf(DoubleType()) +def generate_lognormal_amount(tiers: pd.Series) -> pd.Series: + """Generate amount based on tier using log-normal distribution.""" + np.random.seed(SEED) + amounts = [] + for tier in tiers: + if tier == "Enterprise": + amounts.append(float(np.random.lognormal(mean=7.5, sigma=0.8))) # ~$1800 avg + elif tier == "Pro": + amounts.append(float(np.random.lognormal(mean=5.5, sigma=0.7))) # ~$245 avg + else: + amounts.append(float(np.random.lognormal(mean=4.0, sigma=0.6))) # ~$55 avg + return pd.Series(amounts) + +# ============================================================================= +# CREATE INFRASTRUCTURE +# ============================================================================= +print("\nCreating infrastructure...") +spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}") +spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data") +print(f"Infrastructure ready: {VOLUME_PATH}") + +# ============================================================================= +# GENERATE CUSTOMERS (Master Table) +# ============================================================================= +print(f"\nGenerating {N_CUSTOMERS:,} customers...") + +customers_df = ( + spark.range(0, N_CUSTOMERS, numPartitions=PARTITIONS) + .select( + F.concat(F.lit("CUST-"), F.lpad(F.col("id").cast("string"), 5, "0")).alias("customer_id"), + fake_name(F.col("id")).alias("name"), + fake_company(F.col("id")).alias("company"), + fake_address(F.col("id")).alias("address"), + # Tier distribution: Free 60%, Pro 30%, Enterprise 10% + F.when(F.rand(SEED) < TIER_PROBS[0], "Free") + .when(F.rand(SEED) < TIER_PROBS[0] + TIER_PROBS[1], "Pro") + .otherwise("Enterprise").alias("tier"), + # Region distribution + F.when(F.rand(SEED) < REGION_PROBS[0], "North") + .when(F.rand(SEED) < REGION_PROBS[0] + REGION_PROBS[1], "South") + .when(F.rand(SEED) < REGION_PROBS[0] + REGION_PROBS[1] + REGION_PROBS[2], "East") + .otherwise("West").alias("region"), + # Created date (within last 2 years before start date) + F.date_sub(F.lit(START_DATE.date()), (F.rand(SEED) * 730).cast("int")).alias("created_at"), + ) +) + +# Add tier-based ARR and email +customers_df = ( + customers_df + .withColumn("arr", F.round(generate_lognormal_amount(F.col("tier")), 2)) + .withColumn("email", fake_email(F.col("name"))) +) + +# Save customers +customers_df.write.mode(WRITE_MODE).parquet(f"{VOLUME_PATH}/customers") +print(f" Saved customers to {VOLUME_PATH}/customers") + +# Show tier distribution +print("\n Tier distribution:") +customers_df.groupBy("tier").count().orderBy("tier").show() + +# ============================================================================= +# GENERATE ORDERS (Child Table with Referential Integrity) +# ============================================================================= +print(f"\nGenerating {N_ORDERS:,} orders with referential integrity...") + +# Write customer lookup to temp Delta table (no .cache() on serverless!) +customers_tmp_table = f"{CATALOG}.{SCHEMA}._tmp_customers_lookup" +customers_df.select("customer_id", "tier").write.mode("overwrite").saveAsTable(customers_tmp_table) +customer_lookup = spark.table(customers_tmp_table) + +# Generate orders base +orders_df = ( + spark.range(0, N_ORDERS, numPartitions=PARTITIONS) + .select( + F.concat(F.lit("ORD-"), F.lpad(F.col("id").cast("string"), 6, "0")).alias("order_id"), + # Generate customer_idx for FK join (hash-based distribution) + (F.abs(F.hash(F.col("id"), F.lit(SEED))) % N_CUSTOMERS).alias("customer_idx"), + # Order status + F.when(F.rand(SEED) < 0.65, "delivered") + .when(F.rand(SEED) < 0.80, "shipped") + .when(F.rand(SEED) < 0.90, "processing") + .when(F.rand(SEED) < 0.95, "pending") + .otherwise("cancelled").alias("status"), + # Order date within date range + F.date_add(F.lit(START_DATE.date()), (F.rand(SEED) * 180).cast("int")).alias("order_date"), + ) +) + +# Add customer_idx to lookup for join +customer_lookup_with_idx = customer_lookup.withColumn( + "customer_idx", + (F.row_number().over(Window.orderBy(F.monotonically_increasing_id())) - 1).cast("int") +) + +# Join to get customer_id and tier as foreign key +orders_with_fk = ( + orders_df + .join(customer_lookup_with_idx, on="customer_idx", how="left") + .drop("customer_idx") +) + +# Add tier-based amount +orders_with_fk = orders_with_fk.withColumn( + "amount", + F.round(generate_lognormal_amount(F.col("tier")), 2) +) + +# ============================================================================= +# INJECT BAD DATA (OPTIONAL) +# ============================================================================= +if INJECT_BAD_DATA: + print("\nInjecting bad data for quality testing...") + + # Calculate counts + null_count = int(N_ORDERS * BAD_DATA_CONFIG["null_rate"]) + outlier_count = int(N_ORDERS * BAD_DATA_CONFIG["outlier_rate"]) + orphan_count = int(N_ORDERS * BAD_DATA_CONFIG["orphan_fk_rate"]) + + # Add bad data flags + orders_with_fk = orders_with_fk.withColumn( + "row_num", + F.row_number().over(Window.orderBy(F.monotonically_increasing_id())) + ) + + # Inject nulls in customer_id for first null_count rows + orders_with_fk = orders_with_fk.withColumn( + "customer_id", + F.when(F.col("row_num") <= null_count, None).otherwise(F.col("customer_id")) + ) + + # Inject negative amounts for next outlier_count rows + orders_with_fk = orders_with_fk.withColumn( + "amount", + F.when( + (F.col("row_num") > null_count) & (F.col("row_num") <= null_count + outlier_count), + F.lit(-999.99) + ).otherwise(F.col("amount")) + ) + + # Inject orphan FKs for next orphan_count rows + orders_with_fk = orders_with_fk.withColumn( + "customer_id", + F.when( + (F.col("row_num") > null_count + outlier_count) & + (F.col("row_num") <= null_count + outlier_count + orphan_count), + F.lit("CUST-NONEXISTENT") + ).otherwise(F.col("customer_id")) + ) + + orders_with_fk = orders_with_fk.drop("row_num") + + print(f" Injected {null_count} null customer_ids") + print(f" Injected {outlier_count} negative amounts") + print(f" Injected {orphan_count} orphan foreign keys") + +# Drop tier column (not needed in final output) +orders_final = orders_with_fk.drop("tier") + +# Save orders +orders_final.write.mode(WRITE_MODE).parquet(f"{VOLUME_PATH}/orders") +print(f" Saved orders to {VOLUME_PATH}/orders") + +# Show status distribution +print("\n Status distribution:") +orders_final.groupBy("status").count().orderBy("status").show() + +# ============================================================================= +# CLEANUP AND SUMMARY +# ============================================================================= +spark.sql(f"DROP TABLE IF EXISTS {customers_tmp_table}") + +print("\n" + "=" * 80) +print("GENERATION COMPLETE") +print("=" * 80) +print(f"Catalog: {CATALOG}") +print(f"Schema: {SCHEMA}") +print(f"Volume: {VOLUME_PATH}") +print(f"\nGenerated data:") +print(f" - customers: {N_CUSTOMERS:,} rows") +print(f" - orders: {N_ORDERS:,} rows") +if INJECT_BAD_DATA: + print(f" - Bad data injected: nulls, outliers, orphan FKs") +print(f"\nDate range: {START_DATE.date()} to {END_DATE.date()}") +print("=" * 80) diff --git a/.claude/skills/databricks-unity-catalog/6-volumes.md b/.claude/skills/databricks-unity-catalog/6-volumes.md index 1eae49a2..497b6090 100644 --- a/.claude/skills/databricks-unity-catalog/6-volumes.md +++ b/.claude/skills/databricks-unity-catalog/6-volumes.md @@ -39,69 +39,16 @@ All volume operations use the path format: ## MCP Tools -### List Files in Volume - -```python -# List files and directories -list_volume_files( - volume_path="/Volumes/main/default/my_volume/data/" -) -# Returns: [{"name": "file.csv", "path": "...", "is_directory": false, "file_size": 1024, "last_modified": "..."}] -``` - -### Upload File to Volume - -```python -# Upload a local file -upload_to_volume( - local_path="/tmp/data.csv", - volume_path="/Volumes/main/default/my_volume/data.csv", - overwrite=True -) -# Returns: {"local_path": "...", "volume_path": "...", "success": true} -``` - -### Download File from Volume - -```python -# Download to local path -download_from_volume( - volume_path="/Volumes/main/default/my_volume/data.csv", - local_path="/tmp/downloaded.csv", - overwrite=True -) -# Returns: {"volume_path": "...", "local_path": "...", "success": true} -``` - -### Create Directory - -```python -# Create directory (creates parents like mkdir -p) -create_volume_directory( - volume_path="/Volumes/main/default/my_volume/data/2024/01" -) -# Returns: {"volume_path": "...", "success": true} -``` - -### Delete File - -```python -# Delete a file -delete_volume_file( - volume_path="/Volumes/main/default/my_volume/old_data.csv" -) -# Returns: {"volume_path": "...", "success": true} -``` - -### Get File Info - -```python -# Get file metadata -get_volume_file_info( - volume_path="/Volumes/main/default/my_volume/data.csv" -) -# Returns: {"name": "data.csv", "file_size": 1024, "last_modified": "...", "success": true} -``` +| Tool | Usage | +|------|-------| +| `list_volume_files` | `list_volume_files(volume_path="/Volumes/catalog/schema/volume/path/")` | +| `get_volume_folder_details` | `get_volume_folder_details(volume_path="catalog/schema/volume/path", format="parquet")` - schema, row counts, stats | +| `upload_to_volume` | `upload_to_volume(local_path="/tmp/data/*", volume_path="/Volumes/.../dest")` - supports files, folders, globs | +| `download_from_volume` | `download_from_volume(volume_path="/Volumes/.../file.csv", local_path="/tmp/file.csv")` | +| `create_volume_directory` | `create_volume_directory(volume_path="/Volumes/.../new_folder")` - creates parents like `mkdir -p` | +| `delete_volume_file` | `delete_volume_file(volume_path="/Volumes/.../file.csv")` | +| `delete_volume_directory` | `delete_volume_directory(volume_path="/Volumes/.../folder")` - directory must be empty | +| `get_volume_file_info` | `get_volume_file_info(volume_path="/Volumes/.../file.csv")` - returns size, modified date | --- diff --git a/.claude/skills/databricks-unity-catalog/7-data-profiling.md b/.claude/skills/databricks-unity-catalog/7-data-profiling.md new file mode 100644 index 00000000..23a2b62f --- /dev/null +++ b/.claude/skills/databricks-unity-catalog/7-data-profiling.md @@ -0,0 +1,309 @@ +# Data Profiling (formerly Lakehouse Monitoring) + +Comprehensive reference for Data Profiling: create quality monitors on Unity Catalog tables to track data profiles, detect drift, and monitor ML model performance. + +## Overview + +Data profiling automatically computes statistical profiles and drift metrics for tables over time. When you create a monitor, Databricks generates two output Delta tables (profile metrics + drift metrics) and an optional dashboard. + +| Component | Description | +|-----------|-------------| +| **Monitor** | Configuration attached to a UC table | +| **Profile Metrics Table** | Summary statistics computed per column | +| **Drift Metrics Table** | Statistical drift compared to baseline or previous time window | +| **Dashboard** | Auto-generated visualization of metrics | + +### Requirements + +- Unity Catalog enabled workspace +- Databricks SQL access +- Privileges: `USE CATALOG`, `USE SCHEMA`, `SELECT`, and `MANAGE` on the table +- Only Delta tables supported (managed, external, views, materialized views, streaming tables) + +--- + +## Profile Types + +| Type | Use Case | Key Params | Limitations | +|------|----------|------------|-------------| +| **Snapshot** | General-purpose tables without time column | None required | Max 4TB table size | +| **TimeSeries** | Tables with a timestamp column | `timestamp_column`, `granularities` | Last 30 days only | +| **InferenceLog** | ML model monitoring | `timestamp_column`, `granularities`, `model_id_column`, `problem_type`, `prediction_column` | Last 30 days only | + +### Granularities (for TimeSeries and InferenceLog) + +Supported `AggregationGranularity` values: `AGGREGATION_GRANULARITY_5_MINUTES`, `AGGREGATION_GRANULARITY_30_MINUTES`, `AGGREGATION_GRANULARITY_1_HOUR`, `AGGREGATION_GRANULARITY_1_DAY`, `AGGREGATION_GRANULARITY_1_WEEK` – `AGGREGATION_GRANULARITY_4_WEEKS`, `AGGREGATION_GRANULARITY_1_MONTH`, `AGGREGATION_GRANULARITY_1_YEAR` + +--- + +## MCP Tools + +Use the `manage_uc_monitors` tool for all monitor operations: + +| Action | Description | +|--------|-------------| +| `create` | Create a quality monitor on a table | +| `get` | Get monitor details and status | +| `run_refresh` | Trigger a metric refresh | +| `list_refreshes` | List refresh history | +| `delete` | Delete the monitor (assets are not deleted) | + +### Create a Monitor + +> **Note:** The MCP tool currently only creates **snapshot** monitors. For TimeSeries or InferenceLog monitors, use the Python SDK directly (see below). + +```python +manage_uc_monitors( + action="create", + table_name="catalog.schema.my_table", + output_schema_name="catalog.schema", +) +``` + +### Get Monitor Status + +```python +manage_uc_monitors( + action="get", + table_name="catalog.schema.my_table", +) +``` + +### Trigger a Refresh + +```python +manage_uc_monitors( + action="run_refresh", + table_name="catalog.schema.my_table", +) +``` + +### Delete a Monitor + +```python +manage_uc_monitors( + action="delete", + table_name="catalog.schema.my_table", +) +``` + +--- + +## Python SDK Examples + +**Doc:** https://databricks-sdk-py.readthedocs.io/en/stable/workspace/dataquality/data_quality.html + +The new SDK provides full control over all profile types via `w.data_quality`. + +### Create Snapshot Monitor + +```python +from databricks.sdk import WorkspaceClient +from databricks.sdk.service.dataquality import ( + Monitor, DataProfilingConfig, SnapshotConfig, +) + +w = WorkspaceClient() + +# Look up UUIDs — the new API uses object_id and output_schema_id (both UUIDs) +table_info = w.tables.get("catalog.schema.my_table") +schema_info = w.schemas.get(f"{table_info.catalog_name}.{table_info.schema_name}") + +monitor = w.data_quality.create_monitor( + monitor=Monitor( + object_type="table", + object_id=table_info.table_id, + data_profiling_config=DataProfilingConfig( + assets_dir="/Workspace/Users/user@example.com/monitoring/my_table", + output_schema_id=schema_info.schema_id, + snapshot=SnapshotConfig(), + ), + ), +) +print(f"Monitor status: {monitor.data_profiling_config.status}") +``` + +### Create TimeSeries Monitor + +```python +from databricks.sdk.service.dataquality import ( + Monitor, DataProfilingConfig, TimeSeriesConfig, AggregationGranularity, +) + +table_info = w.tables.get("catalog.schema.events") +schema_info = w.schemas.get(f"{table_info.catalog_name}.{table_info.schema_name}") + +monitor = w.data_quality.create_monitor( + monitor=Monitor( + object_type="table", + object_id=table_info.table_id, + data_profiling_config=DataProfilingConfig( + assets_dir="/Workspace/Users/user@example.com/monitoring/events", + output_schema_id=schema_info.schema_id, + time_series=TimeSeriesConfig( + timestamp_column="event_timestamp", + granularities=[AggregationGranularity.AGGREGATION_GRANULARITY_1_DAY], + ), + ), + ), +) +``` + +### Create InferenceLog Monitor + +```python +from databricks.sdk.service.dataquality import ( + Monitor, DataProfilingConfig, InferenceLogConfig, + AggregationGranularity, InferenceProblemType, +) + +table_info = w.tables.get("catalog.schema.model_predictions") +schema_info = w.schemas.get(f"{table_info.catalog_name}.{table_info.schema_name}") + +monitor = w.data_quality.create_monitor( + monitor=Monitor( + object_type="table", + object_id=table_info.table_id, + data_profiling_config=DataProfilingConfig( + assets_dir="/Workspace/Users/user@example.com/monitoring/predictions", + output_schema_id=schema_info.schema_id, + inference_log=InferenceLogConfig( + timestamp_column="prediction_timestamp", + granularities=[AggregationGranularity.AGGREGATION_GRANULARITY_1_HOUR], + model_id_column="model_version", + problem_type=InferenceProblemType.INFERENCE_PROBLEM_TYPE_CLASSIFICATION, + prediction_column="prediction", + label_column="label", + ), + ), + ), +) +``` + +### Schedule a Monitor + +```python +from databricks.sdk.service.dataquality import ( + Monitor, DataProfilingConfig, SnapshotConfig, CronSchedule, +) + +table_info = w.tables.get("catalog.schema.my_table") +schema_info = w.schemas.get(f"{table_info.catalog_name}.{table_info.schema_name}") + +monitor = w.data_quality.create_monitor( + monitor=Monitor( + object_type="table", + object_id=table_info.table_id, + data_profiling_config=DataProfilingConfig( + assets_dir="/Workspace/Users/user@example.com/monitoring/my_table", + output_schema_id=schema_info.schema_id, + snapshot=SnapshotConfig(), + schedule=CronSchedule( + quartz_cron_expression="0 0 12 * * ?", # Daily at noon + timezone_id="UTC", + ), + ), + ), +) +``` + +### Get, Refresh, and Delete + +```python +# Get monitor details +monitor = w.data_quality.get_monitor( + object_type="table", + object_id=table_info.table_id, +) + +# Trigger refresh +from databricks.sdk.service.dataquality import Refresh + +refresh = w.data_quality.create_refresh( + object_type="table", + object_id=table_info.table_id, + refresh=Refresh( + object_type="table", + object_id=table_info.table_id, + ), +) + +# Delete monitor (does not delete output tables or dashboard) +w.data_quality.delete_monitor( + object_type="table", + object_id=table_info.table_id, +) +``` + +--- + +## Anomaly Detection + +Anomaly detection is enabled at the **schema level**, not per table. Once enabled, Databricks automatically scans all tables in the schema at the same frequency they are updated. + +```python +from databricks.sdk.service.dataquality import Monitor, AnomalyDetectionConfig + +schema_info = w.schemas.get("catalog.schema") + +monitor = w.data_quality.create_monitor( + monitor=Monitor( + object_type="schema", + object_id=schema_info.schema_id, + anomaly_detection_config=AnomalyDetectionConfig(), + ), +) +``` + +> **Note:** Anomaly detection requires `MANAGE SCHEMA` or `MANAGE CATALOG` privileges and serverless compute enabled on the workspace. + +--- + +## Output Tables + +When a monitor is created, two metric tables are generated in the specified output schema: + +| Table | Naming Convention | Contents | +|-------|-------------------|----------| +| **Profile Metrics** | `{table_name}_profile_metrics` | Per-column statistics (nulls, min, max, mean, distinct count, etc.) | +| **Drift Metrics** | `{table_name}_drift_metrics` | Statistical tests comparing current vs. baseline or previous window | + +### Query Output Tables + +```sql +-- View latest profile metrics +SELECT * +FROM catalog.schema.my_table_profile_metrics +ORDER BY window_end DESC +LIMIT 100; + +-- View latest drift metrics +SELECT * +FROM catalog.schema.my_table_drift_metrics +ORDER BY window_end DESC +LIMIT 100; +``` + +--- + +## Common Issues + +| Issue | Cause | Solution | +|-------|-------|----------| +| `FEATURE_NOT_ENABLED` | Data profiling not enabled on workspace | Contact workspace admin to enable the feature | +| `PERMISSION_DENIED` | Missing `MANAGE` privilege on the table | Grant `MANAGE` on the table to your user/group | +| Monitor refresh stuck in `PENDING` | No SQL warehouse available | Ensure a SQL warehouse is running or set `warehouse_id` | +| Profile metrics table empty | Refresh has not completed yet | Check refresh state with `list_refreshes`; wait for `SUCCESS` | +| Snapshot monitor on large table fails | Table exceeds 4TB limit | Switch to TimeSeries profile type instead | +| TimeSeries shows limited data | Only processes last 30 days | Expected behavior; contact account team to adjust | + +--- + +> **Note:** Data profiling was formerly known as Lakehouse Monitoring. The legacy SDK accessor +> `w.lakehouse_monitors` and the MCP tool `manage_uc_monitors` still use the previous API. + +## Resources + +- [Data Quality Monitoring Documentation](https://docs.databricks.com/aws/en/data-quality-monitoring/) +- [Data Quality SDK Reference](https://databricks-sdk-py.readthedocs.io/en/stable/workspace/dataquality/data_quality.html) +- [Legacy Lakehouse Monitors SDK Reference](https://databricks-sdk-py.readthedocs.io/en/stable/workspace/catalog/lakehouse_monitors.html) diff --git a/.claude/skills/databricks-unity-catalog/SKILL.md b/.claude/skills/databricks-unity-catalog/SKILL.md index b8dbbc20..2e3d05fa 100644 --- a/.claude/skills/databricks-unity-catalog/SKILL.md +++ b/.claude/skills/databricks-unity-catalog/SKILL.md @@ -17,6 +17,7 @@ Use this skill when: - Tracking **compute resources** (cluster usage, warehouse metrics) - Reviewing **job execution** (run history, success rates, failures) - Analyzing **query performance** (slow queries, warehouse utilization) +- Profiling **data quality** (data profiling, drift detection, metric tables) ## Reference Files @@ -24,30 +25,19 @@ Use this skill when: |-------|------|-------------| | System Tables | [5-system-tables.md](5-system-tables.md) | Lineage, audit, billing, compute, jobs, query history | | Volumes | [6-volumes.md](6-volumes.md) | Volume file operations, permissions, best practices | +| Data Profiling | [7-data-profiling.md](7-data-profiling.md) | Data profiling, drift detection, profile metrics | ## Quick Start ### Volume File Operations (MCP Tools) -```python -# List files in a volume -list_volume_files(volume_path="/Volumes/catalog/schema/volume/folder/") - -# Upload file to volume -upload_to_volume( - local_path="/tmp/data.csv", - volume_path="/Volumes/catalog/schema/volume/data.csv" -) - -# Download file from volume -download_from_volume( - volume_path="/Volumes/catalog/schema/volume/data.csv", - local_path="/tmp/downloaded.csv" -) - -# Create directory -create_volume_directory(volume_path="/Volumes/catalog/schema/volume/new_folder") -``` +| Tool | Usage | +|------|-------| +| `list_volume_files` | `list_volume_files(volume_path="/Volumes/catalog/schema/volume/path/")` | +| `get_volume_folder_details` | `get_volume_folder_details(volume_path="catalog/schema/volume/path", format="parquet")` - schema, row counts, stats | +| `upload_to_volume` | `upload_to_volume(local_path="/tmp/data/*", volume_path="/Volumes/.../dest")` | +| `download_from_volume` | `download_from_volume(volume_path="/Volumes/.../file.csv", local_path="/tmp/file.csv")` | +| `create_volume_directory` | `create_volume_directory(volume_path="/Volumes/.../new_folder")` | ### Enable System Tables Access @@ -104,6 +94,13 @@ mcp__databricks__execute_sql( 3. **Grant minimal access** - System tables contain sensitive metadata 4. **Schedule reports** - Create scheduled queries for regular monitoring +## Related Skills + +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - for pipelines that write to Unity Catalog tables +- **[databricks-jobs](../databricks-jobs/SKILL.md)** - for job execution data visible in system tables +- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** - for generating data stored in Unity Catalog Volumes +- **[databricks-aibi-dashboards](../databricks-aibi-dashboards/SKILL.md)** - for building dashboards on top of Unity Catalog data + ## Resources - [Unity Catalog System Tables](https://docs.databricks.com/administration-guide/system-tables/) diff --git a/.claude/skills/databricks-unstructured-pdf-generation/SKILL.md b/.claude/skills/databricks-unstructured-pdf-generation/SKILL.md new file mode 100644 index 00000000..92322fd0 --- /dev/null +++ b/.claude/skills/databricks-unstructured-pdf-generation/SKILL.md @@ -0,0 +1,337 @@ +--- +name: databricks-unstructured-pdf-generation +description: "Generate PDF documents from HTML and upload to Unity Catalog volumes. Use for creating test PDFs, demo documents, reports, or evaluation datasets." +--- + +# PDF Generation from HTML + +Convert HTML content to PDF documents and upload them to Unity Catalog Volumes. + +## Overview + +The `generate_and_upload_pdf` MCP tool converts HTML to PDF and uploads to a Unity Catalog Volume. You (the LLM) generate the HTML content, and the tool handles conversion and upload. + +## Tool Signature + +``` +generate_and_upload_pdf( + html_content: str, # Complete HTML document + filename: str, # PDF filename (e.g., "report.pdf") + catalog: str, # Unity Catalog name + schema: str, # Schema name + volume: str = "raw_data", # Volume name (default: "raw_data") + folder: str = None, # Optional subfolder +) +``` + +**Returns:** +```json +{ + "success": true, + "volume_path": "/Volumes/catalog/schema/volume/filename.pdf", + "error": null +} +``` + +## Quick Start + +Generate a simple PDF: + +``` +generate_and_upload_pdf( + html_content=''' + + + + + +

Quarterly Report Q1 2024

+
+

Executive Summary

+

Revenue increased 15% year-over-year...

+
+ +''', + filename="q1_report.pdf", + catalog="my_catalog", + schema="my_schema" +) +``` + +## Performance: Generate Multiple PDFs in Parallel + +**IMPORTANT**: PDF generation and upload can take 2-5 seconds per document. When generating multiple PDFs, **call the tool in parallel** to maximize throughput. + +### Example: Generate 5 PDFs in Parallel + +Make 5 simultaneous `generate_and_upload_pdf` calls: + +``` +# Call 1 +generate_and_upload_pdf( + html_content="...Employee Handbook content...", + filename="employee_handbook.pdf", + catalog="hr_catalog", schema="policies", folder="2024" +) + +# Call 2 (parallel) +generate_and_upload_pdf( + html_content="...Leave Policy content...", + filename="leave_policy.pdf", + catalog="hr_catalog", schema="policies", folder="2024" +) + +# Call 3 (parallel) +generate_and_upload_pdf( + html_content="...Code of Conduct content...", + filename="code_of_conduct.pdf", + catalog="hr_catalog", schema="policies", folder="2024" +) + +# Call 4 (parallel) +generate_and_upload_pdf( + html_content="...Benefits Guide content...", + filename="benefits_guide.pdf", + catalog="hr_catalog", schema="policies", folder="2024" +) + +# Call 5 (parallel) +generate_and_upload_pdf( + html_content="...Remote Work Policy content...", + filename="remote_work_policy.pdf", + catalog="hr_catalog", schema="policies", folder="2024" +) +``` + +By calling these in parallel (not sequentially), 5 PDFs that would take 15-25 seconds sequentially complete in 3-5 seconds total. + +## HTML Best Practices + +### Use Complete HTML5 Structure + +Always include the full HTML structure: + +```html + + + + + + + + + +``` + +### CSS Features Supported + +PlutoPrint supports modern CSS3: +- Flexbox and Grid layouts +- CSS variables (`--var-name`) +- Web fonts (system fonts recommended) +- Colors, backgrounds, borders +- Tables with styling + +### CSS to Avoid + +- Animations and transitions (static PDF) +- Interactive elements (forms, hover effects) +- External resources (images via URL) - use embedded base64 if needed + +### Professional Document Template + +```html + + + + + + +

Document Title

+ +

Section 1

+

Content here...

+ +
+ Important: Key information highlighted here. +
+ +

Data Table

+
+ + +
Column 1Column 2Column 3
DataDataData
+ +
+ Generated on 2024-01-15 | Confidential +
+ + +``` + +## Common Patterns + +### Pattern 1: Technical Documentation + +Generate API documentation, user guides, or technical specs: + +``` +generate_and_upload_pdf( + html_content=''' + + + +

API Reference

+
+ GET /api/v1/users +

Returns a list of all users.

+
+

Request Headers

+ 
Authorization: Bearer {token}
+Content-Type: application/json
+ +''', + filename="api_reference.pdf", + catalog="docs_catalog", + schema="api_docs" +) +``` + +### Pattern 2: Business Reports + +``` +generate_and_upload_pdf( + html_content=''' + + + +

Q1 2024 Performance Report

+
+
$2.4M
+
Revenue
+
+
+
+15%
+
Growth
+
+ +''', + filename="q1_2024_report.pdf", + catalog="finance", + schema="reports", + folder="quarterly" +) +``` + +### Pattern 3: HR Policies + +``` +generate_and_upload_pdf( + html_content=''' + + + +

Employee Leave Policy

+

Effective: January 1, 2024

+ +
+

1. Annual Leave

+

All full-time employees are entitled to 20 days of paid annual leave per calendar year.

+
+ +
+ Note: Leave requests must be submitted at least 2 weeks in advance. +
+ +''', + filename="leave_policy.pdf", + catalog="hr_catalog", + schema="policies" +) +``` + +## Workflow for Multiple Documents + +When asked to generate multiple PDFs: + +1. **Plan the documents**: Determine titles, content structure for each +2. **Generate HTML for each**: Create complete HTML documents +3. **Call tool in parallel**: Make multiple simultaneous `generate_and_upload_pdf` calls +4. **Report results**: Summarize successful uploads and any errors + +## Prerequisites + +- Unity Catalog schema must exist +- Volume must exist (default: `raw_data`) +- User must have WRITE permission on the volume + +## Troubleshooting + +| Issue | Solution | +|-------|----------| +| "Volume does not exist" | Create the volume first or use an existing one | +| "Schema does not exist" | Create the schema or check the name | +| PDF looks wrong | Check HTML/CSS syntax, use supported CSS features | +| Slow generation | Call multiple PDFs in parallel, not sequentially | diff --git a/.claude/skills/databricks-vector-search/SKILL.md b/.claude/skills/databricks-vector-search/SKILL.md new file mode 100644 index 00000000..72068ec5 --- /dev/null +++ b/.claude/skills/databricks-vector-search/SKILL.md @@ -0,0 +1,447 @@ +--- +name: databricks-vector-search +description: "Patterns for Databricks Vector Search: create endpoints and indexes, query with filters, manage embeddings. Use when building RAG applications, semantic search, or similarity matching. Covers both storage-optimized and standard endpoints." +--- + +# Databricks Vector Search + +Patterns for creating, managing, and querying vector search indexes for RAG and semantic search applications. + +## When to Use + +Use this skill when: +- Building RAG (Retrieval-Augmented Generation) applications +- Implementing semantic search or similarity matching +- Creating vector indexes from Delta tables +- Choosing between storage-optimized and standard endpoints +- Querying vector indexes with filters + +## Overview + +Databricks Vector Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration. + +| Component | Description | +|-----------|-------------| +| **Endpoint** | Compute resource hosting indexes (Standard or Storage-Optimized) | +| **Index** | Vector data structure for similarity search | +| **Delta Sync** | Auto-syncs with source Delta table | +| **Direct Access** | Manual CRUD operations on vectors | + +## Endpoint Types + +| Type | Latency | Capacity | Cost | Best For | +|------|---------|----------|------|----------| +| **Standard** | 20-50ms | 320M vectors (768 dim) | Higher | Real-time, low-latency | +| **Storage-Optimized** | 300-500ms | 1B+ vectors (768 dim) | 7x lower | Large-scale, cost-sensitive | + +## Index Types + +| Type | Embeddings | Sync | Use Case | +|------|------------|------|----------| +| **Delta Sync (managed)** | Databricks computes | Auto from Delta | Easiest setup | +| **Delta Sync (self-managed)** | You provide | Auto from Delta | Custom embeddings | +| **Direct Access** | You provide | Manual CRUD | Real-time updates | + +## Quick Start + +### Create Endpoint + +```python +from databricks.sdk import WorkspaceClient + +w = WorkspaceClient() + +# Create a standard endpoint +endpoint = w.vector_search_endpoints.create_endpoint( + name="my-vs-endpoint", + endpoint_type="STANDARD" # or "STORAGE_OPTIMIZED" +) +# Note: Endpoint creation is asynchronous; check status with get_endpoint() +``` + +### Create Delta Sync Index (Managed Embeddings) + +```python +# Source table must have: primary key column + text column +index = w.vector_search_indexes.create_index( + name="catalog.schema.my_index", + endpoint_name="my-vs-endpoint", + primary_key="id", + index_type="DELTA_SYNC", + delta_sync_index_spec={ + "source_table": "catalog.schema.documents", + "embedding_source_columns": [ + { + "name": "content", # Text column to embed + "embedding_model_endpoint_name": "databricks-gte-large-en" + } + ], + "pipeline_type": "TRIGGERED" # or "CONTINUOUS" + } +) +``` + +### Query Index + +```python +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content", "metadata"], + query_text="What is machine learning?", + num_results=5 +) + +for doc in results.result.data_array: + score = doc[-1] # Similarity score is last column + print(f"Score: {score}, Content: {doc[1][:100]}...") +``` + +## Common Patterns + +### Create Storage-Optimized Endpoint + +```python +# For large-scale, cost-effective deployments +endpoint = w.vector_search_endpoints.create_endpoint( + name="my-storage-endpoint", + endpoint_type="STORAGE_OPTIMIZED" +) +``` + +### Delta Sync with Self-Managed Embeddings + +```python +# Source table must have: primary key + embedding vector column +index = w.vector_search_indexes.create_index( + name="catalog.schema.my_index", + endpoint_name="my-vs-endpoint", + primary_key="id", + index_type="DELTA_SYNC", + delta_sync_index_spec={ + "source_table": "catalog.schema.documents", + "embedding_vector_columns": [ + { + "name": "embedding", # Pre-computed embedding column + "embedding_dimension": 768 + } + ], + "pipeline_type": "TRIGGERED" + } +) +``` + +### Direct Access Index + +```python +import json + +# Create index for manual CRUD +index = w.vector_search_indexes.create_index( + name="catalog.schema.direct_index", + endpoint_name="my-vs-endpoint", + primary_key="id", + index_type="DIRECT_ACCESS", + direct_access_index_spec={ + "embedding_vector_columns": [ + {"name": "embedding", "embedding_dimension": 768} + ], + "schema_json": json.dumps({ + "id": "string", + "text": "string", + "embedding": "array", + "metadata": "string" + }) + } +) + +# Upsert data +w.vector_search_indexes.upsert_data_vector_index( + index_name="catalog.schema.direct_index", + inputs_json=json.dumps([ + {"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"}, + {"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"}, + ]) +) + +# Delete data +w.vector_search_indexes.delete_data_vector_index( + index_name="catalog.schema.direct_index", + primary_keys=["1", "2"] +) +``` + +### Query with Embedding Vector + +```python +# When you have pre-computed query embedding +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "text"], + query_vector=[0.1, 0.2, 0.3, ...], # Your 768-dim vector + num_results=10 +) +``` + +### Hybrid Search (Semantic + Keyword) + +Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See [search-modes.md](search-modes.md) for detailed guidance on choosing between ANN and hybrid search. + +```python +# Combines vector similarity with keyword matching +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_text="SPARK-12345 executor memory error", + query_type="HYBRID", + num_results=10 +) +``` + +## Filtering + +### Standard Endpoint Filters (Dictionary) + +```python +# filters_json uses dictionary format +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_text="machine learning", + num_results=10, + filters_json='{"category": "ai", "status": ["active", "pending"]}' +) +``` + +### Storage-Optimized Filters (SQL-like) + +Storage-Optimized endpoints use SQL-like filter syntax via the `databricks-vectorsearch` package's `filters` parameter (accepts a string): + +```python +from databricks.vector_search.client import VectorSearchClient + +vsc = VectorSearchClient() +index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index") + +# SQL-like filter syntax for storage-optimized endpoints +results = index.similarity_search( + query_text="machine learning", + columns=["id", "content"], + num_results=10, + filters="category = 'ai' AND status IN ('active', 'pending')" +) + +# More filter examples +# filters="price > 100 AND price < 500" +# filters="department LIKE 'eng%'" +# filters="created_at >= '2024-01-01'" +``` + +### Trigger Index Sync + +```python +# For TRIGGERED pipeline type, manually sync +w.vector_search_indexes.sync_index( + index_name="catalog.schema.my_index" +) +``` + +### Scan All Index Entries + +```python +# Retrieve all vectors (for debugging/export) +scan_result = w.vector_search_indexes.scan_index( + index_name="catalog.schema.my_index", + num_results=100 +) +``` + +## Reference Files + +| Topic | File | Description | +|-------|------|-------------| +| Index Types | [index-types.md](index-types.md) | Detailed comparison of Delta Sync (managed/self-managed) vs Direct Access | +| End-to-End RAG | [end-to-end-rag.md](end-to-end-rag.md) | Complete walkthrough: source table → endpoint → index → query → agent integration | +| Search Modes | [search-modes.md](search-modes.md) | When to use semantic (ANN) vs hybrid search, decision guide | +| Operations | [troubleshooting-and-operations.md](troubleshooting-and-operations.md) | Monitoring, cost optimization, capacity planning, migration | + +## CLI Quick Reference + +```bash +# List endpoints +databricks vector-search endpoints list + +# Create endpoint +databricks vector-search endpoints create \ + --name my-endpoint \ + --endpoint-type STANDARD + +# List indexes on endpoint +databricks vector-search indexes list-indexes \ + --endpoint-name my-endpoint + +# Get index status +databricks vector-search indexes get-index \ + --index-name catalog.schema.my_index + +# Sync index (for TRIGGERED) +databricks vector-search indexes sync-index \ + --index-name catalog.schema.my_index + +# Delete index +databricks vector-search indexes delete-index \ + --index-name catalog.schema.my_index +``` + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **Index sync slow** | Use Storage-Optimized endpoints (20x faster indexing) | +| **Query latency high** | Use Standard endpoint for <100ms latency | +| **filters_json not working** | Storage-Optimized uses SQL-like string filters via `databricks-vectorsearch` package's `filters` parameter | +| **Embedding dimension mismatch** | Ensure query and index dimensions match | +| **Index not updating** | Check pipeline_type; use sync_index() for TRIGGERED | +| **Out of capacity** | Upgrade to Storage-Optimized (1B+ vectors) | +| **`query_vector` truncated by MCP tool** | MCP tool calls serialize arrays as JSON and can truncate large vectors (e.g. 1024-dim). Use `query_text` instead (for managed embedding indexes), or use the Databricks SDK/CLI to pass raw vectors | + +## Embedding Models + +Databricks provides built-in embedding models: + +| Model | Dimensions | Context Window | Use Case | +|-------|------------|----------------|----------| +| `databricks-gte-large-en` | 1024 | 8192 tokens | English text, high quality | +| `databricks-bge-large-en` | 1024 | 512 tokens | English text, general purpose | + +```python +# Use with managed embeddings +embedding_source_columns=[ + { + "name": "content", + "embedding_model_endpoint_name": "databricks-gte-large-en" + } +] +``` + +## MCP Tools + +The following MCP tools are available for managing Vector Search infrastructure. For a full end-to-end walkthrough, see [end-to-end-rag.md](end-to-end-rag.md). + +### manage_vs_endpoint - Endpoint Management + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create_or_update` | Create endpoint (STANDARD or STORAGE_OPTIMIZED). Idempotent | name | +| `get` | Get endpoint details | name | +| `list` | List all endpoints | (none) | +| `delete` | Delete endpoint (indexes must be deleted first) | name | + +```python +# Create or update an endpoint +result = manage_vs_endpoint(action="create_or_update", name="my-vs-endpoint", endpoint_type="STANDARD") +# Returns {"name": "my-vs-endpoint", "endpoint_type": "STANDARD", "created": True} + +# List all endpoints +endpoints = manage_vs_endpoint(action="list") + +# Get specific endpoint +endpoint = manage_vs_endpoint(action="get", name="my-vs-endpoint") +``` + +### manage_vs_index - Index Management + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `create_or_update` | Create index. Idempotent, auto-triggers sync for DELTA_SYNC | name, endpoint_name, primary_key | +| `get` | Get index details | name | +| `list` | List indexes. Optional endpoint_name filter | (none) | +| `delete` | Delete index | name | + +```python +# Create a Delta Sync index with managed embeddings +result = manage_vs_index( + action="create_or_update", + name="catalog.schema.my_index", + endpoint_name="my-vs-endpoint", + primary_key="id", + index_type="DELTA_SYNC", + delta_sync_index_spec={ + "source_table": "catalog.schema.docs", + "embedding_source_columns": [{"name": "content", "embedding_model_endpoint_name": "databricks-gte-large-en"}], + "pipeline_type": "TRIGGERED" + } +) + +# Get a specific index +index = manage_vs_index(action="get", name="catalog.schema.my_index") + +# List all indexes on an endpoint +indexes = manage_vs_index(action="list", endpoint_name="my-vs-endpoint") + +# List all indexes across all endpoints +all_indexes = manage_vs_index(action="list") +``` + +### query_vs_index - Query (Hot Path) + +Query index with `query_text`, `query_vector`, or hybrid (`query_type="HYBRID"`). Prefer `query_text` over `query_vector` — MCP tool calls can truncate large embedding arrays (1024-dim). + +```python +# Query an index +results = query_vs_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_text="machine learning best practices", + num_results=5 +) + +# Hybrid search (combines vector + keyword) +results = query_vs_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_text="SPARK-12345 memory error", + query_type="HYBRID", + num_results=10 +) +``` + +### manage_vs_data - Data Operations + +| Action | Description | Required Params | +|--------|-------------|-----------------| +| `upsert` | Insert/update records | index_name, inputs_json | +| `delete` | Delete by primary key | index_name, primary_keys | +| `scan` | Scan index contents | index_name | +| `sync` | Trigger sync for TRIGGERED indexes | index_name | + +```python +# Upsert data into a Direct Access index +manage_vs_data( + action="upsert", + index_name="catalog.schema.my_index", + inputs_json=[{"id": "doc1", "content": "...", "embedding": [0.1, 0.2, ...]}] +) + +# Trigger manual sync for a TRIGGERED pipeline index +manage_vs_data(action="sync", index_name="catalog.schema.my_index") + +# Scan index contents +manage_vs_data(action="scan", index_name="catalog.schema.my_index", num_results=100) +``` + +## Notes + +- **Storage-Optimized is newer** — better for most use cases unless you need <100ms latency +- **Delta Sync recommended** — easier than Direct Access for most scenarios +- **Hybrid search** — available for both Delta Sync and Direct Access indexes +- **`columns_to_sync` matters** — only synced columns are available in query results; include all columns you need +- **Filter syntax differs by endpoint** — Standard uses dict-format filters, Storage-Optimized uses SQL-like string filters. Use the `databricks-vectorsearch` package's `filters` parameter which accepts both formats +- **Management vs runtime** — MCP tools above handle lifecycle management; for agent tool-calling at runtime, use `VectorSearchRetrieverTool` or the Databricks managed Vector Search MCP server + +## Related Skills + +- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** - Deploy agents that use VectorSearchRetrieverTool +- **[databricks-agent-bricks](../databricks-agent-bricks/SKILL.md)** - Knowledge Assistants use RAG over indexed documents +- **[databricks-unstructured-pdf-generation](../databricks-unstructured-pdf-generation/SKILL.md)** - Generate documents to index in Vector Search +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - Manage the catalogs and tables that back Delta Sync indexes +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - Build Delta tables used as Vector Search sources diff --git a/.claude/skills/databricks-vector-search/end-to-end-rag.md b/.claude/skills/databricks-vector-search/end-to-end-rag.md new file mode 100644 index 00000000..a3808d1b --- /dev/null +++ b/.claude/skills/databricks-vector-search/end-to-end-rag.md @@ -0,0 +1,241 @@ +# End-to-End RAG with Vector Search + +Build a complete Retrieval-Augmented Generation pipeline: prepare documents, create a vector index, query it, and wire it into an agent. + +## MCP Tools Used + +| Tool | Step | +|------|------| +| `execute_sql` | Create source table, insert documents | +| `manage_vs_endpoint(action="create")` | Create compute endpoint | +| `manage_vs_index(action="create")` | Create Delta Sync index with managed embeddings | +| `manage_vs_index(action="sync")` | Trigger index sync | +| `manage_vs_index(action="get")` | Check index status | +| `query_vs_index` | Test similarity search | + +--- + +## Step 1: Prepare Source Table + +The source Delta table needs a primary key column and a text column to embed. + +```sql +CREATE TABLE IF NOT EXISTS catalog.schema.knowledge_base ( + doc_id STRING, + title STRING, + content STRING, + category STRING, + updated_at TIMESTAMP DEFAULT current_timestamp() +); + +INSERT INTO catalog.schema.knowledge_base VALUES +('doc-001', 'Getting Started', 'Databricks is a unified analytics platform...', 'overview', current_timestamp()), +('doc-002', 'Unity Catalog', 'Unity Catalog provides centralized governance...', 'governance', current_timestamp()), +('doc-003', 'Delta Lake', 'Delta Lake is an open-source storage layer...', 'storage', current_timestamp()); +``` + +Or via MCP: + +```python +execute_sql(sql_query=""" + CREATE TABLE IF NOT EXISTS catalog.schema.knowledge_base ( + doc_id STRING, + title STRING, + content STRING, + category STRING, + updated_at TIMESTAMP DEFAULT current_timestamp() + ) +""") +``` + +## Step 2: Create Vector Search Endpoint + +```python +manage_vs_endpoint( + action="create", + name="my-rag-endpoint", + endpoint_type="STORAGE_OPTIMIZED" +) +``` + +Endpoint creation is asynchronous. Check status: + +```python +manage_vs_endpoint(action="get", name="my-rag-endpoint") +# Wait for state: "ONLINE" +``` + +## Step 3: Create Delta Sync Index + +```python +manage_vs_index( + action="create", + name="catalog.schema.knowledge_base_index", + endpoint_name="my-rag-endpoint", + primary_key="doc_id", + index_type="DELTA_SYNC", + delta_sync_index_spec={ + "source_table": "catalog.schema.knowledge_base", + "embedding_source_columns": [ + { + "name": "content", + "embedding_model_endpoint_name": "databricks-gte-large-en" + } + ], + "pipeline_type": "TRIGGERED", + "columns_to_sync": ["doc_id", "title", "content", "category"] + } +) +``` + +Key decisions: +- **`embedding_source_columns`**: Databricks computes embeddings automatically from the `content` column +- **`pipeline_type`**: `TRIGGERED` for manual sync (cheaper), `CONTINUOUS` for auto-sync on table changes +- **`columns_to_sync`**: Only sync columns you need in query results (reduces storage and improves performance) + +## Step 4: Sync and Verify + +```python +# Trigger initial sync +manage_vs_index(action="sync", index_name="catalog.schema.knowledge_base_index") + +# Check status +manage_vs_index(action="get", index_name="catalog.schema.knowledge_base_index") +# Wait for state: "ONLINE" +``` + +## Step 5: Query the Index + +```python +# Semantic search +query_vs_index( + index_name="catalog.schema.knowledge_base_index", + columns=["doc_id", "title", "content", "category"], + query_text="How do I govern my data?", + num_results=3 +) +``` + +### With Filters + +The filter syntax depends on the endpoint type used when creating the index. + +```python +# Storage-Optimized endpoint (used in this walkthrough): SQL-like filter syntax +query_vs_index( + index_name="catalog.schema.knowledge_base_index", + columns=["doc_id", "title", "content"], + query_text="How do I govern my data?", + num_results=3, + filters="category = 'governance'" +) + +# Standard endpoint (if you created a Standard endpoint instead): JSON filters_json +query_vs_index( + index_name="catalog.schema.my_standard_index", + columns=["doc_id", "title", "content"], + query_text="How do I govern my data?", + num_results=3, + filters_json='{"category": "governance"}' +) +``` + +### Hybrid Search (Vector + Keyword) + +```python +query_vs_index( + index_name="catalog.schema.knowledge_base_index", + columns=["doc_id", "title", "content"], + query_text="Delta Lake ACID transactions", + num_results=5, + query_type="HYBRID" +) +``` + +--- + +## Step 6: Use in an Agent + +### As a Tool in a ChatAgent + +Use `VectorSearchRetrieverTool` to wire the index into an agent deployed on Model Serving: + +```python +from databricks.agents import ChatAgent +from databricks.agents.tools import VectorSearchRetrieverTool +from databricks.sdk import WorkspaceClient + +# Define the retriever tool +retriever_tool = VectorSearchRetrieverTool( + index_name="catalog.schema.knowledge_base_index", + columns=["doc_id", "title", "content"], + num_results=3, +) + +class RAGAgent(ChatAgent): + def __init__(self): + self.w = WorkspaceClient() + + def predict(self, messages, context=None): + query = messages[-1].content + + results = self.w.vector_search_indexes.query_index( + index_name="catalog.schema.knowledge_base_index", + columns=["title", "content"], + query_text=query, + num_results=3, + ) + + context_docs = "\n\n".join( + f"**{row[0]}**: {row[1]}" + for row in results.result.data_array + ) + + response = self.w.serving_endpoints.query( + name="databricks-meta-llama-3-3-70b-instruct", + messages=[ + {"role": "system", "content": f"Answer using this context:\n{context_docs}"}, + {"role": "user", "content": query}, + ], + ) + + return {"content": response.choices[0].message.content} +``` + +--- + +## Updating the Index + +### Add New Documents + +```sql +INSERT INTO catalog.schema.knowledge_base VALUES +('doc-004', 'MLflow', 'MLflow is an open-source platform for ML lifecycle...', 'ml', current_timestamp()); +``` + +Then sync: + +```python +manage_vs_index(action="sync", index_name="catalog.schema.knowledge_base_index") +``` + +### Delete Documents + +```sql +DELETE FROM catalog.schema.knowledge_base WHERE doc_id = 'doc-001'; +``` + +Then sync — the index automatically handles deletions via Delta change data feed. + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **Index stuck in PROVISIONING** | Endpoint may still be creating. Check `manage_vs_endpoint(action="get")` first | +| **Query returns no results** | Index may not be synced yet. Run `manage_vs_index(action="sync")` and wait for ONLINE state | +| **"Column not found in index"** | Column must be in `columns_to_sync`. Recreate index with the column included | +| **Embeddings not computed** | Ensure `embedding_model_endpoint_name` is a valid serving endpoint | +| **Stale results after table update** | For TRIGGERED pipelines, you must call `manage_vs_index(action="sync")` manually | +| **Filter not working** | Standard endpoints use dict-format filters (`filters_json`), Storage-Optimized use SQL-like string filters (`filters`) | diff --git a/.claude/skills/databricks-vector-search/index-types.md b/.claude/skills/databricks-vector-search/index-types.md new file mode 100644 index 00000000..ebfc1c7e --- /dev/null +++ b/.claude/skills/databricks-vector-search/index-types.md @@ -0,0 +1,254 @@ +# Vector Search Index Types + +## Comparison Matrix + +| Feature | Delta Sync (Managed) | Delta Sync (Self-Managed) | Direct Access | +|---------|---------------------|---------------------------|---------------| +| **Embeddings** | Databricks computes | You provide | You provide | +| **Sync** | Auto from Delta | Auto from Delta | Manual CRUD | +| **Setup** | Easiest | Medium | Most control | +| **Source** | Delta table + text | Delta table + vectors | API calls | +| **Best for** | Quick start, RAG | Custom models | Real-time apps | + +## Delta Sync with Managed Embeddings + +Databricks automatically computes embeddings from your text column. + +### Requirements + +- Source Delta table with: + - Primary key column (unique identifier) + - Text column (content to embed) +- Embedding model endpoint (or use built-in) + +### Create Index + +```python +from databricks.sdk import WorkspaceClient + +w = WorkspaceClient() + +index = w.vector_search_indexes.create_index( + name="catalog.schema.docs_index", + endpoint_name="my-vs-endpoint", + primary_key="doc_id", + index_type="DELTA_SYNC", + delta_sync_index_spec={ + "source_table": "catalog.schema.documents", + "embedding_source_columns": [ + { + "name": "content", + "embedding_model_endpoint_name": "databricks-gte-large-en" + } + ], + "pipeline_type": "TRIGGERED", # or "CONTINUOUS" + "columns_to_sync": ["doc_id", "content", "title", "category"] + } +) +``` + +### Pipeline Types + +| Type | Behavior | Cost | Use Case | +|------|----------|------|----------| +| `TRIGGERED` | Manual sync via API | Lower | Batch updates | +| `CONTINUOUS` | Auto-sync on changes | Higher | Real-time sync | + +### Source Table Example + +```sql +CREATE TABLE catalog.schema.documents ( + doc_id STRING, + title STRING, + content STRING, -- Text to embed + category STRING, + created_at TIMESTAMP +); +``` + +## Delta Sync with Self-Managed Embeddings + +You pre-compute embeddings and store them in the source table. + +### Requirements + +- Source Delta table with: + - Primary key column + - Embedding vector column (array of floats) + +### Create Index + +```python +index = w.vector_search_indexes.create_index( + name="catalog.schema.custom_index", + endpoint_name="my-vs-endpoint", + primary_key="id", + index_type="DELTA_SYNC", + delta_sync_index_spec={ + "source_table": "catalog.schema.embedded_docs", + "embedding_vector_columns": [ + { + "name": "embedding", + "embedding_dimension": 768 + } + ], + "pipeline_type": "TRIGGERED" + } +) +``` + +### Compute Embeddings + +```python +from databricks.sdk import WorkspaceClient +import pandas as pd + +w = WorkspaceClient() + +def get_embeddings(texts: list[str]) -> list[list[float]]: + """Call embedding endpoint for texts.""" + response = w.serving_endpoints.query( + name="databricks-gte-large-en", + input=texts + ) + return [item.embedding for item in response.data] + +# Add embeddings to your data +df = spark.table("catalog.schema.documents").toPandas() +df["embedding"] = get_embeddings(df["content"].tolist()) + +# Write back to Delta +spark.createDataFrame(df).write.mode("overwrite").saveAsTable( + "catalog.schema.embedded_docs" +) +``` + +### Source Table Example + +```sql +CREATE TABLE catalog.schema.embedded_docs ( + id STRING, + content STRING, + embedding ARRAY, -- Pre-computed embedding + metadata STRING +); +``` + +## Direct Access Index + +Full control over vector data via CRUD API. No Delta table sync. + +### Requirements + +- Define schema upfront +- Manage upsert/delete operations yourself + +### Create Index + +```python +import json + +index = w.vector_search_indexes.create_index( + name="catalog.schema.realtime_index", + endpoint_name="my-vs-endpoint", + primary_key="id", + index_type="DIRECT_ACCESS", + direct_access_index_spec={ + "embedding_vector_columns": [ + {"name": "embedding", "embedding_dimension": 768} + ], + "schema_json": json.dumps({ + "id": "string", + "text": "string", + "embedding": "array", + "category": "string", + "score": "float" + }) + } +) +``` + +### Upsert Data + +```python +import json + +# Insert or update vectors +w.vector_search_indexes.upsert_data_vector_index( + index_name="catalog.schema.realtime_index", + inputs_json=json.dumps([ + { + "id": "doc-001", + "text": "Machine learning basics", + "embedding": [0.1, 0.2, 0.3, ...], # 768 floats + "category": "ml", + "score": 0.95 + }, + { + "id": "doc-002", + "text": "Deep learning overview", + "embedding": [0.4, 0.5, 0.6, ...], + "category": "dl", + "score": 0.88 + } + ]) +) +``` + +### Delete Data + +```python +w.vector_search_indexes.delete_data_vector_index( + index_name="catalog.schema.realtime_index", + primary_keys=["doc-001", "doc-002"] +) +``` + +### Attach Embedding Model (Optional) + +For Direct Access with text queries: + +```python +# Create index with embedding model for query-time embedding +index = w.vector_search_indexes.create_index( + name="catalog.schema.hybrid_index", + endpoint_name="my-vs-endpoint", + primary_key="id", + index_type="DIRECT_ACCESS", + direct_access_index_spec={ + "embedding_vector_columns": [ + {"name": "embedding", "embedding_dimension": 768} + ], + "embedding_model_endpoint_name": "databricks-gte-large-en", # For query_text + "schema_json": json.dumps({...}) + } +) +``` + +## Choosing the Right Type + +``` +Start here: +│ +├─ Do you have pre-computed embeddings? +│ ├─ Yes → Do you want auto-sync from Delta? +│ │ ├─ Yes → Delta Sync (Self-Managed) +│ │ └─ No → Direct Access +│ │ +│ └─ No → Delta Sync (Managed Embeddings) +│ +└─ Do you need real-time updates (<1 sec)? + ├─ Yes → Direct Access + └─ No → Delta Sync (any type) +``` + +## Endpoint Selection + +After choosing index type, choose endpoint: + +| Scenario | Endpoint Type | +|----------|---------------| +| Need <100ms latency | Standard | +| >100M vectors | Storage-Optimized | +| Cost-sensitive | Storage-Optimized | +| Default choice | Storage-Optimized | diff --git a/.claude/skills/databricks-vector-search/search-modes.md b/.claude/skills/databricks-vector-search/search-modes.md new file mode 100644 index 00000000..58092afa --- /dev/null +++ b/.claude/skills/databricks-vector-search/search-modes.md @@ -0,0 +1,142 @@ +# Vector Search Modes + +Databricks Vector Search supports three search modes: **ANN** (semantic, default), **HYBRID** (semantic + keyword), and **FULL_TEXT** (keyword only, beta). ANN and HYBRID work with Delta Sync and Direct Access indexes. + +## Semantic Search (ANN) + +ANN (Approximate Nearest Neighbor) is the default search mode. It finds documents by vector similarity — matching the *meaning* of your query against stored embeddings. + +### When to use + +- Conceptual or meaning-based queries ("How do I handle errors in my pipeline?") +- Paraphrased input where exact terms may not appear in the documents +- Multilingual scenarios where query and document languages may differ +- General-purpose RAG retrieval + +### Example + +```python +# ANN is the default — no query_type parameter needed +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_text="How do I handle errors in my pipeline?", + num_results=5 +) +``` + +## Hybrid Search + +Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. It retrieves documents that are both semantically similar *and* contain matching keywords, then merges the results. + +### When to use + +- Queries containing exact terms that must appear: SKUs, product codes, error codes, acronyms +- Proper nouns — company names, people, specific technologies +- Technical documentation where terminology precision matters +- Mixed-intent queries combining concepts with specific terms + +### Example + +```python +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_text="SPARK-12345 executor memory error", + query_type="HYBRID", + num_results=10 +) +``` + +## Decision Guide + +| Mode | Best for | Trade-off | Choose when | +|------|----------|-----------|-------------| +| **ANN** (default) | Conceptual queries, paraphrases, meaning-based search | Fastest; may miss exact keyword matches | You want documents *about* a topic regardless of exact wording | +| **HYBRID** | Exact terms, codes, proper nouns, mixed-intent queries | ~2x resource usage vs ANN; max 200 results | Your queries contain specific identifiers or technical terms that must appear in results | +| **FULL_TEXT** (beta) | Pure keyword search without vector embeddings | No semantic understanding; max 200 results | You need keyword matching only, without vector similarity | + +**Start with ANN.** Switch to HYBRID if you notice relevant documents being missed because they don't share vocabulary with the query. + +## Combining Search Modes with Filters + +Both search modes support filters. The filter syntax depends on your endpoint type: + +- **Standard endpoints** → `filters` as dict (or `filters_json` as JSON string via `databricks-sdk`) +- **Storage-Optimized endpoints** → `filters` as SQL-like string (via `databricks-vectorsearch` package) + +### Standard endpoint with hybrid search + +```python +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content", "category"], + query_text="SPARK-12345 executor memory error", + query_type="HYBRID", + num_results=10, + filters_json='{"category": "troubleshooting", "status": ["open", "in_progress"]}' +) +``` + +### Storage-Optimized endpoint with hybrid search + +```python +from databricks.vector_search.client import VectorSearchClient + +vsc = VectorSearchClient() +index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index") + +results = index.similarity_search( + query_text="SPARK-12345 executor memory error", + columns=["id", "content", "category"], + query_type="hybrid", + num_results=10, + filters="category = 'troubleshooting' AND status IN ('open', 'in_progress')" +) +``` + +## Using with Pre-Computed Embeddings + +If you compute embeddings yourself, use `query_vector` instead of `query_text` for ANN search: + +```python +# ANN with pre-computed embedding (default) +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_vector=[0.1, 0.2, 0.3, ...], # Your embedding vector + num_results=10 +) +``` + +For **hybrid search with self-managed embeddings** (indexes without an associated model endpoint), you must provide **both** `query_vector` and `query_text`. The vector is used for the ANN component and the text for the BM25 keyword component: + +```python +# HYBRID with self-managed embeddings — requires both vector AND text +results = w.vector_search_indexes.query_index( + index_name="catalog.schema.my_index", + columns=["id", "content"], + query_vector=[0.1, 0.2, 0.3, ...], # For ANN similarity + query_text="executor memory error", # For BM25 keyword matching + query_type="HYBRID", + num_results=10 +) +``` + +**Notes:** +- For **ANN** queries: provide either `query_text` or `query_vector`, not both. +- For **HYBRID** queries on **managed embedding indexes**: provide only `query_text` (the system handles both components). +- For **HYBRID** queries on **self-managed indexes without a model endpoint**: provide both `query_vector` and `query_text`. +- When using `query_text` alone, the index must have an associated embedding model (managed embeddings or `embedding_model_endpoint_name` on a Direct Access index). + +## Parameter Reference + +| Parameter | Type | Package | Description | +|-----------|------|---------|-------------| +| `query_text` | `str` | Both | Text query — requires embedding model on the index | +| `query_vector` | `list[float]` | Both | Pre-computed embedding vector | +| `query_type` | `str` | Both | `"ANN"` (default) or `"HYBRID"` or `"FULL_TEXT"` (beta) | +| `columns` | `list[str]` | Both | Column names to return in results | +| `num_results` | `int` | Both | Number of results (default: 10 in `databricks-sdk`, 5 in `databricks-vectorsearch`) | +| `filters_json` | `str` | `databricks-sdk` | JSON dict filter string (Standard endpoints) | +| `filters` | `str` or `dict` | `databricks-vectorsearch` | Dict for Standard, SQL-like string for Storage-Optimized | diff --git a/.claude/skills/databricks-vector-search/troubleshooting-and-operations.md b/.claude/skills/databricks-vector-search/troubleshooting-and-operations.md new file mode 100644 index 00000000..7dc4b8c9 --- /dev/null +++ b/.claude/skills/databricks-vector-search/troubleshooting-and-operations.md @@ -0,0 +1,177 @@ +# Vector Search Troubleshooting & Operations + +Operational guidance for monitoring, cost optimization, capacity planning, and migration of Databricks Vector Search resources. + +## Monitoring Endpoint Status + +Use `manage_vs_endpoint(action="get")` (MCP tool) or `w.vector_search_endpoints.get_endpoint()` (SDK) to check endpoint health. + +### Endpoint fields + +| Field | Description | +|-------|-------------| +| `state` | `ONLINE`, `PROVISIONING`, `OFFLINE`, `YELLOW_STATE`, `RED_STATE`, `DELETED` | +| `message` | Human-readable status or error message | +| `endpoint_type` | `STANDARD` or `STORAGE_OPTIMIZED` | +| `num_indexes` | Number of indexes hosted on this endpoint | +| `creation_timestamp` | When the endpoint was created | +| `last_updated_timestamp` | When the endpoint was last modified | + +### Example + +```python +endpoint = w.vector_search_endpoints.get_endpoint(endpoint_name="my-endpoint") +print(f"State: {endpoint.endpoint_status.state.value}") +print(f"Indexes: {endpoint.num_indexes}") +``` + +**What to do per state:** +- `PROVISIONING` → Wait. Endpoint creation is asynchronous and can take several minutes. +- `ONLINE` → Ready to serve queries and host indexes. +- `OFFLINE` → Check the `message` field for error details. May require recreation. +- `YELLOW_STATE` → Endpoint is degraded but still serving. Investigate the `message` field. +- `RED_STATE` → Endpoint is unhealthy. Check `message` for details; may need support intervention. + +## Monitoring Index Status + +Use `manage_vs_index(action="get")` (MCP tool) or `w.vector_search_indexes.get_index()` (SDK) to check index health. + +### Index fields + +| Field | Description | +|-------|-------------| +| `status.ready` | Boolean — `True` when ready for queries, `False` when provisioning/syncing | +| `status.message` | Status details or error information | +| `status.index_url` | URL to access the index in the Databricks UI | +| `status.indexed_row_count` | Number of rows currently indexed | +| `delta_sync_index_spec.pipeline_id` | DLT pipeline ID (Delta Sync indexes only) — useful for debugging sync issues | +| `index_type` | `DELTA_SYNC` or `DIRECT_ACCESS` | + +### Example + +```python +index = w.vector_search_indexes.get_index(index_name="catalog.schema.my_index") +if index.status.ready: + print("Index is ONLINE") +else: + print(f"Index is NOT_READY: {index.status.message}") +``` + +## Pipeline Type Trade-offs + +Delta Sync indexes use a DLT pipeline to sync data from the source Delta table. The pipeline type determines sync behavior: + +| Pipeline Type | Behavior | Cost | Best for | +|---------------|----------|------|----------| +| **TRIGGERED** | Manual sync via `manage_vs_index(action="sync")` | Lower — runs only when triggered | Batch updates, periodic refreshes, cost-sensitive workloads | +| **CONTINUOUS** | Auto-syncs on source table changes | Higher — always running | Real-time freshness, applications needing up-to-date results | + +### Triggering a sync + +```python +# For TRIGGERED pipelines only +w.vector_search_indexes.sync_index(index_name="catalog.schema.my_index") +# Check sync progress with get_index() +``` + +**Tip:** CONTINUOUS pipelines cannot be synced manually — they sync automatically. Calling `sync_index()` on a CONTINUOUS index will raise an error. + +## Cost Optimization + +### Endpoint type selection + +| Factor | Standard | Storage-Optimized | +|--------|----------|-------------------| +| Query latency | 20-50ms | 300-500ms | +| Cost | Higher | ~7x lower | +| Max capacity | 320M vectors (768 dim) | 1B+ vectors (768 dim) | +| Indexing speed | Slower | 20x faster | + +**Recommendation:** Start with Storage-Optimized unless you need sub-100ms latency. It handles most RAG workloads well. + +### Reducing storage costs + +- Use `columns_to_sync` to limit which columns are synced to the index. Only synced columns are available in query results, so include only what you need. +- Choose TRIGGERED pipelines for batch workloads to avoid continuous compute costs. + +```python +# Only sync the columns you actually need in query results +delta_sync_index_spec={ + "source_table": "catalog.schema.documents", + "embedding_source_columns": [ + {"name": "content", "embedding_model_endpoint_name": "databricks-gte-large-en"} + ], + "pipeline_type": "TRIGGERED", + "columns_to_sync": ["id", "content", "title"] # Exclude large unused columns +} +``` + +## Capacity Planning + +| Endpoint Type | Max Vectors (768 dim) | Guidance | +|---------------|----------------------|----------| +| Standard | ~320M | Suitable for most production workloads under 300M documents | +| Storage-Optimized | 1B+ | Large-scale corpora, enterprise knowledge bases | + +**Estimating needs:** +- One document typically maps to one vector (or multiple if chunked) +- If chunking at ~512 tokens, expect 2-5 vectors per page of text +- Monitor `num_indexes` on your endpoint to understand utilization + +## Migration Patterns + +### Changing endpoint type + +Endpoints are **immutable after creation** — you cannot change the type (Standard ↔ Storage-Optimized) of an existing endpoint. To migrate: + +1. **Create a new endpoint** with the desired type +2. **Recreate indexes** on the new endpoint pointing to the same source tables +3. **Wait for sync** to complete (check index state) +4. **Update applications** to query the new index names +5. **Delete old indexes**, then delete the old endpoint + +```python +# Step 1: Create new endpoint +w.vector_search_endpoints.create_endpoint( + name="my-endpoint-storage-optimized", + endpoint_type="STORAGE_OPTIMIZED" +) + +# Step 2: Recreate index on new endpoint (same source table) +w.vector_search_indexes.create_index( + name="catalog.schema.my_index_v2", + endpoint_name="my-endpoint-storage-optimized", + primary_key="id", + index_type="DELTA_SYNC", + delta_sync_index_spec={ + "source_table": "catalog.schema.documents", + "embedding_source_columns": [ + {"name": "content", "embedding_model_endpoint_name": "databricks-gte-large-en"} + ], + "pipeline_type": "TRIGGERED" + } +) + +# Step 3: Trigger sync and wait for ONLINE state +w.vector_search_indexes.sync_index(index_name="catalog.schema.my_index_v2") + +# Step 4: Update your application to use "catalog.schema.my_index_v2" +# Step 5: Clean up old resources +w.vector_search_indexes.delete_index(index_name="catalog.schema.my_index") +w.vector_search_endpoints.delete_endpoint(endpoint_name="my-endpoint") +``` + +## Expanded Troubleshooting + +| Issue | Likely Cause | Solution | +|-------|-------------|----------| +| **Index stuck in NOT_READY** | Sync pipeline failed or source table issue | Check `message` field via `manage_vs_index(action="get")`. Inspect the DLT pipeline using `pipeline_id`. | +| **Embedding dimension mismatch** | Query vector dimensions ≠ index dimensions | Ensure your embedding model output matches the `embedding_dimension` in the index spec. | +| **Permission errors on create** | Missing Unity Catalog privileges | User needs `CREATE TABLE` on the schema and `USE CATALOG`/`USE SCHEMA` privileges. | +| **Index returns NOT_FOUND** | Wrong name format or index deleted | Index names must be fully qualified: `catalog.schema.index_name`. | +| **Sync not running (TRIGGERED)** | Sync not triggered after source update | Call `manage_vs_index(action="sync")` or `w.vector_search_indexes.sync_index()` after updating source data. | +| **Endpoint NOT_FOUND** | Endpoint name typo or deleted | List all endpoints with `manage_vs_endpoint(action="list")` to verify available endpoints. | +| **Query returns empty results** | Index not yet synced, or filters too restrictive | Check index state is ONLINE. Verify `columns_to_sync` includes queried columns. Test without filters first. | +| **filters_json has no effect** | Using wrong filter syntax for endpoint type | Standard endpoints use dict-format filters (`filters_json` in SDK, `filters` as dict in `databricks-vectorsearch`). Storage-Optimized endpoints use SQL-like string filters (`filters` as str in `databricks-vectorsearch`). | +| **Quota or capacity errors** | Too many indexes or vectors | Check `num_indexes` on endpoint. Consider Storage-Optimized for higher capacity. | +| **Upsert fails on Delta Sync** | Cannot upsert to Delta Sync indexes | Upsert/delete operations only work on Direct Access indexes. Delta Sync indexes update via their source table. | diff --git a/.claude/skills/databricks-zerobus-ingest/1-setup-and-authentication.md b/.claude/skills/databricks-zerobus-ingest/1-setup-and-authentication.md new file mode 100644 index 00000000..31dfd1b3 --- /dev/null +++ b/.claude/skills/databricks-zerobus-ingest/1-setup-and-authentication.md @@ -0,0 +1,203 @@ +# Setup and Authentication + +Complete setup guide for Zerobus Ingest: endpoint configuration, service principal creation, table preparation, SDK installation, and firewall requirements. + +--- + +## 1. Determine Your Server Endpoint + +The Zerobus server endpoint format depends on your cloud provider: + +| Cloud | Server Endpoint Format | Workspace URL Format | +|-------|------------------------|----------------------| +| **AWS** | `.zerobus..cloud.databricks.com` | `https://.cloud.databricks.com` | +| **Azure** | `.zerobus..azuredatabricks.net` | `https://.azuredatabricks.net` | + +**Example (AWS):** +``` +Server endpoint: 1234567890123456.zerobus.us-west-2.cloud.databricks.com +Workspace URL: https://dbc-a1b2c3d4-e5f6.cloud.databricks.com +``` + +**Finding your workspace ID:** Extract the numeric ID from your workspace URL or workspace settings page. It is the first segment of the server endpoint. + +--- + +## 2. Create the Target Table + +Zerobus does **not** create or alter tables. You must pre-create your target table as a **managed Delta table** in Unity Catalog: + +```sql +CREATE TABLE catalog.schema.my_events ( + event_id STRING, + device_name STRING, + temp INT, + humidity LONG, + event_time TIMESTAMP +); +``` + +**Constraints:** +- Must be a **managed** Delta table (no external storage) +- Table names limited to ASCII letters, digits, and underscores +- Maximum 2000 columns +- Table must be in a [supported region](#supported-regions) + +--- + +## 3. Create a Service Principal + +Zerobus authenticates via OAuth2 service principals (M2M). Create one via the Databricks UI or CLI: + +### Via UI +1. Go to **Settings > Identity and Access > Service principals** +2. Click **Add service principal** +3. Generate an OAuth secret: note the **client ID** and **client secret** + +### Via Databricks CLI +```bash +databricks service-principals create --display-name "zerobus-producer" +``` + +### Grant Table Permissions + +The service principal needs catalog, schema, and table access: + +```sql +-- Grant catalog access +GRANT USE CATALOG ON CATALOG my_catalog TO ``; + +-- Grant schema access +GRANT USE SCHEMA ON SCHEMA my_catalog.my_schema TO ``; + +-- Grant table write access +GRANT MODIFY, SELECT ON TABLE my_catalog.my_schema.my_events TO ``; +``` + +**Tip:** For broader access (e.g., writing to multiple tables in a schema), grant `MODIFY` and `SELECT` at the schema level instead. + +**Important:** For Zerobus, always grant explicit table-level `MODIFY` and `SELECT` permissions in addition to catalog/schema access. Schema-level inherited grants may not be sufficient for the OAuth `authorization_details` flow used by Zerobus. + +--- + +## 4. Install the SDK + +### Python (3.9+) + +```bash +pip install databricks-zerobus-ingest-sdk>=1.0.0 +``` + +Or with a virtual environment: +```bash +uv pip install databricks-zerobus-ingest-sdk>=1.0.0 +``` + +**Note:** The Zerobus SDK cannot be pip-installed on Databricks serverless compute. Use classic compute clusters, or use the [Zerobus REST API](https://docs.databricks.com/aws/en/ingestion/zerobus-rest-api) (Beta) for notebook-based ingestion without the SDK. + +### Java (8+) + +Maven: +```xml + + com.databricks + zerobus-ingest-sdk + 0.1.0 + +``` + +Gradle: +```groovy +implementation 'com.databricks:zerobus-ingest-sdk:0.1.0' +``` + +### Go (1.21+) + +```bash +go get github.com/databricks/zerobus-sdk-go +``` + +### TypeScript / Node.js (16+) + +```bash +npm install @databricks/zerobus-ingest-sdk +``` + +### Rust (1.70+) + +```bash +cargo add databricks-zerobus-ingest-sdk +cargo add tokio --features macros,rt-multi-thread +``` + +--- + +## 5. Configure Environment Variables + +Store credentials as environment variables rather than hardcoding them: + +```bash +export ZEROBUS_SERVER_ENDPOINT="1234567890123456.zerobus.us-west-2.cloud.databricks.com" +export DATABRICKS_WORKSPACE_URL="https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" +export ZEROBUS_TABLE_NAME="my_catalog.my_schema.my_events" +export DATABRICKS_CLIENT_ID="" +export DATABRICKS_CLIENT_SECRET="" +``` + +--- + +## 6. Firewall Allowlisting + +If your client application sits behind a firewall, you must allowlist the Zerobus IP addresses for your region before testing connectivity. Contact your Databricks representative or consult the [Zerobus documentation](https://docs.databricks.com/aws/en/ingestion/zerobus-overview) for the current IP ranges. + +--- + +## Supported Regions + +Workspace and target tables must reside in a supported region for your cloud provider. + +### AWS + +| Region Code | Location | +|-------------|----------| +| `us-east-1` | US East (N. Virginia) | +| `us-east-2` | US East (Ohio) | +| `us-west-2` | US West (Oregon) | +| `eu-central-1` | Europe (Frankfurt) | +| `eu-west-1` | Europe (Ireland) | +| `ap-southeast-1` | Asia Pacific (Singapore) | +| `ap-southeast-2` | Asia Pacific (Sydney) | +| `ap-northeast-1` | Asia Pacific (Tokyo) | +| `ca-central-1` | Canada (Central) | + +### Azure + +| Region Code | Location | +|-------------|----------| +| `canadacentral` | Canada Central | +| `westus` | West US | +| `eastus` | East US | +| `eastus2` | East US 2 | +| `centralus` | Central US | +| `northcentralus` | North Central US | +| `swedencentral` | Sweden Central | +| `westeurope` | West Europe | +| `northeurope` | North Europe | +| `australiaeast` | Australia East | +| `southeastasia` | Southeast Asia | + +--- + +## Verification Checklist + +Before writing your first record, confirm: + +``` +- [ ] Server endpoint matches your cloud provider and region +- [ ] Workspace URL is correct +- [ ] Target table exists as a managed Delta table +- [ ] Service principal has USE CATALOG, USE SCHEMA, MODIFY, SELECT grants +- [ ] SDK is installed for your target language +- [ ] Environment variables are set (or credentials are configured in code) +- [ ] Firewall allows outbound connections to the Zerobus endpoint (if applicable) +``` diff --git a/.claude/skills/databricks-zerobus-ingest/2-python-client.md b/.claude/skills/databricks-zerobus-ingest/2-python-client.md new file mode 100644 index 00000000..64c6f8b7 --- /dev/null +++ b/.claude/skills/databricks-zerobus-ingest/2-python-client.md @@ -0,0 +1,358 @@ +# Python Client + +Python SDK patterns for Zerobus Ingest: synchronous and asynchronous APIs, JSON and Protobuf flows, and a reusable client class. + +--- + +## SDK Imports + +```python +# Synchronous API +from zerobus.sdk.sync import ZerobusSdk + +# Asynchronous API (equivalent capabilities) +from zerobus.sdk.aio import ZerobusSdk as AsyncZerobusSdk + +# Shared types (used by both sync and async) +from zerobus.sdk.shared import ( + RecordType, + AckCallback, + ZerobusException, + NonRetriableException, + StreamConfigurationOptions, + TableProperties, +) +``` + +--- + + + +--- + +## Protobuf Ingestion + +You must always use Protobuf +For type-safe production workloads, use Protobuf. First generate and compile your `.proto` (see [4-protobuf-schema.md](4-protobuf-schema.md)), then: + +```python +import os +from zerobus.sdk.sync import ZerobusSdk +from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties + +# Import your compiled protobuf module +import record_pb2 + +server_endpoint = os.environ["ZEROBUS_SERVER_ENDPOINT"] +workspace_url = os.environ["DATABRICKS_WORKSPACE_URL"] +table_name = os.environ["ZEROBUS_TABLE_NAME"] +client_id = os.environ["DATABRICKS_CLIENT_ID"] +client_secret = os.environ["DATABRICKS_CLIENT_SECRET"] + +sdk = ZerobusSdk(server_endpoint, workspace_url) + +options = StreamConfigurationOptions(record_type=RecordType.PROTO) +table_props = TableProperties(table_name, record_pb2.AirQuality.DESCRIPTOR) + +stream = sdk.create_stream(client_id, client_secret, table_props, options) + +try: + for i in range(100): + record = record_pb2.AirQuality( + device_name=f"sensor-{i}", + temp=22, + humidity=55, + ) + offset = stream.ingest_record_offset(record) + stream.wait_for_offset(offset) +finally: + stream.close() +``` + +--- + +## ACK Callback (Asynchronous Acknowledgment) + +Instead of blocking on each ACK, register an `AckCallback` subclass for background durability confirmation: + +```python +from zerobus.sdk.shared import AckCallback, StreamConfigurationOptions, RecordType + +class MyAckHandler(AckCallback): + def on_ack(self, offset: int) -> None: + print(f"Durable up to offset: {offset}") + + def on_error(self, offset: int, message: str) -> None: + print(f"Error at offset {offset}: {message}") + +options = StreamConfigurationOptions( + record_type=RecordType.JSON, + ack_callback=MyAckHandler(), +) + +# Create stream with callback +stream = sdk.create_stream(client_id, client_secret, table_props, options) + +try: + for i in range(1000): + record = {"device_name": f"sensor-{i}", "temp": 22, "humidity": 55} + stream.ingest_record_nowait(record) # Fire-and-forget, ACKs arrive via callback + stream.flush() # Ensure all buffered records are sent +finally: + stream.close() +``` + +--- + +## Reusable Client Class + +A production-ready wrapper with retry logic, reconnection, and both JSON and Protobuf support: + +```python +import os +import time +import logging +from typing import Optional + +from zerobus.sdk.sync import ZerobusSdk +from zerobus.sdk.shared import ( + RecordType, + AckCallback, + StreamConfigurationOptions, + TableProperties, +) + +logger = logging.getLogger(__name__) + + +class ZerobusClient: + """Reusable Zerobus Ingest client with retry and reconnection.""" + + def __init__( + self, + server_endpoint: str, + workspace_url: str, + table_name: str, + client_id: str, + client_secret: str, + record_type: RecordType = RecordType.JSON, + ack_callback: Optional[AckCallback] = None, + proto_descriptor=None, + ): + self.server_endpoint = server_endpoint + self.workspace_url = workspace_url + self.table_name = table_name + self.client_id = client_id + self.client_secret = client_secret + self.record_type = record_type + self.ack_callback = ack_callback + self.proto_descriptor = proto_descriptor + + self.sdk = ZerobusSdk(self.server_endpoint, self.workspace_url) + self.stream = None + + def init_stream(self) -> None: + """Open a new stream to the target table.""" + options = StreamConfigurationOptions( + record_type=self.record_type, + ack_callback=self.ack_callback, + ) + if self.record_type == RecordType.PROTO and self.proto_descriptor: + table_props = TableProperties(self.table_name, self.proto_descriptor) + else: + table_props = TableProperties(self.table_name) + + self.stream = self.sdk.create_stream( + self.client_id, self.client_secret, table_props, options + ) + logger.info("Zerobus stream initialized for %s", self.table_name) + + def ingest(self, payload, max_retries: int = 3) -> bool: + """Ingest a single record (dict for JSON, protobuf message for PROTO). + + Returns True on success, False after exhausting retries. + """ + for attempt in range(max_retries): + try: + if self.stream is None: + self.init_stream() + offset = self.stream.ingest_record_offset(payload) + self.stream.wait_for_offset(offset) + return True + except Exception as e: + err = str(e).lower() + logger.warning( + "Ingest attempt %d/%d failed: %s", attempt + 1, max_retries, e + ) + if "closed" in err or "connection" in err: + self.close() + self.init_stream() + if attempt < max_retries - 1: + time.sleep(2**attempt) # Exponential backoff: 1s, 2s, 4s + return False + + def flush(self) -> None: + """Flush buffered writes.""" + if self.stream: + self.stream.flush() + + def close(self) -> None: + """Close the stream and release resources.""" + if self.stream: + self.stream.close() + self.stream = None + + def __enter__(self): + self.init_stream() + return self + + def __exit__(self, exc_type, exc_val, exc_tb): + self.flush() + self.close() + return False +``` + +### Using the Client Class + +```python +# JSON flow with context manager +with ZerobusClient( + server_endpoint=os.environ["ZEROBUS_SERVER_ENDPOINT"], + workspace_url=os.environ["DATABRICKS_WORKSPACE_URL"], + table_name=os.environ["ZEROBUS_TABLE_NAME"], + client_id=os.environ["DATABRICKS_CLIENT_ID"], + client_secret=os.environ["DATABRICKS_CLIENT_SECRET"], + record_type=RecordType.JSON, +) as client: + for i in range(100): + client.ingest({"device_name": f"sensor-{i}", "temp": 22, "humidity": 55}) + +# Protobuf flow +import record_pb2 + +with ZerobusClient( + server_endpoint=os.environ["ZEROBUS_SERVER_ENDPOINT"], + workspace_url=os.environ["DATABRICKS_WORKSPACE_URL"], + table_name=os.environ["ZEROBUS_TABLE_NAME"], + client_id=os.environ["DATABRICKS_CLIENT_ID"], + client_secret=os.environ["DATABRICKS_CLIENT_SECRET"], + record_type=RecordType.PROTO, + proto_descriptor=record_pb2.AirQuality.DESCRIPTOR, +) as client: + for i in range(100): + record = record_pb2.AirQuality(device_name=f"sensor-{i}", temp=22, humidity=55) + client.ingest(record) +``` + +--- + +## Async Python API + +The SDK provides an equivalent async API for use with `asyncio`: + +```python +import asyncio +from zerobus.sdk.aio import ZerobusSdk as AsyncZerobusSdk +from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties + + +async def ingest_async(): + sdk = AsyncZerobusSdk(server_endpoint, workspace_url) + options = StreamConfigurationOptions(record_type=RecordType.JSON) + table_props = TableProperties(table_name) + + stream = await sdk.create_stream(client_id, client_secret, table_props, options) + + try: + for i in range(100): + record = {"device_name": f"sensor-{i}", "temp": 22, "humidity": 55} + offset = await stream.ingest_record_offset(record) + await stream.wait_for_offset(offset) + finally: + await stream.close() + + +asyncio.run(ingest_async()) +``` + +**Tip:** The sync and async APIs have equivalent capabilities. Choose based on your application architecture (FastAPI/aiohttp -> async; scripts/batch jobs -> sync). + +--- + +## Batch Pattern + +For higher throughput, use `ingest_record_nowait` (fire-and-forget) or batch methods, and flush at the end: + +```python +with ZerobusClient( + server_endpoint=os.environ["ZEROBUS_SERVER_ENDPOINT"], + workspace_url=os.environ["DATABRICKS_WORKSPACE_URL"], + table_name=os.environ["ZEROBUS_TABLE_NAME"], + client_id=os.environ["DATABRICKS_CLIENT_ID"], + client_secret=os.environ["DATABRICKS_CLIENT_SECRET"], + record_type=RecordType.JSON, +) as client: + for i in range(10_000): + record = {"device_name": f"sensor-{i}", "temp": 22, "humidity": 55} + client.stream.ingest_record_nowait(record) # Fire-and-forget + # flush() and close() called automatically by context manager +``` + +For true batch ingestion, use the batch variants: + +```python +records = [ + {"device_name": f"sensor-{i}", "temp": 22, "humidity": 55} + for i in range(10_000) +] +# Fire-and-forget batch +stream.ingest_records_nowait(records) +stream.flush() + +# Or with offset tracking +offset = stream.ingest_records_offset(records) +stream.wait_for_offset(offset) +``` + +--- + +## Ingestion Method Comparison + +| Method | Returns | Blocks? | Best For | +|--------|---------|---------|----------| +| `ingest_record_offset(record)` | offset | No (enqueues) | Single record with durability tracking | +| `ingest_record_nowait(record)` | None | No | Max single-record throughput | +| `ingest_records_offset(records)` | last offset | No (enqueues) | Batch with durability tracking | +| `ingest_records_nowait(records)` | None | No | Max batch throughput | +| `wait_for_offset(offset)` | None | Yes (until ACK) | Durability confirmation | +| `flush()` | None | Yes (until sent) | Ensure all buffered records are sent | +| `ingest_record(record)` | RecordAcknowledgment | No | Primary method in SDK v1.1.0+; pass `json.dumps(record)` for JSON | diff --git a/.claude/skills/databricks-zerobus-ingest/3-multilanguage-clients.md b/.claude/skills/databricks-zerobus-ingest/3-multilanguage-clients.md new file mode 100644 index 00000000..4eba1015 --- /dev/null +++ b/.claude/skills/databricks-zerobus-ingest/3-multilanguage-clients.md @@ -0,0 +1,317 @@ +# Multi-Language Clients + +Zerobus Ingest SDK examples for Java, Go, TypeScript/Node.js, and Rust. All languages follow the same core pattern: **SDK init -> create stream -> ingest records -> ACK -> flush -> close**. + +--- + +## Java (8+) + +### Installation + +Maven: +```xml + + com.databricks + zerobus-ingest-sdk + 0.1.0 + +``` + +### Protobuf Flow (Recommended) + +Java uses Protobuf by default. Generate and compile your `.proto` first (see [4-protobuf-schema.md](4-protobuf-schema.md)). + +```java +import com.databricks.zerobus.*; +import com.example.proto.Record.AirQuality; + +public class ZerobusProducer { + public static void main(String[] args) throws Exception { + String serverEndpoint = System.getenv("ZEROBUS_SERVER_ENDPOINT"); + String workspaceUrl = System.getenv("DATABRICKS_WORKSPACE_URL"); + String tableName = System.getenv("ZEROBUS_TABLE_NAME"); + String clientId = System.getenv("DATABRICKS_CLIENT_ID"); + String clientSecret = System.getenv("DATABRICKS_CLIENT_SECRET"); + + ZerobusSdk sdk = new ZerobusSdk(serverEndpoint, workspaceUrl); + + TableProperties tableProperties = new TableProperties<>( + tableName, + AirQuality.getDefaultInstance() + ); + + ZerobusStream stream = sdk.createStream( + tableProperties, clientId, clientSecret + ).join(); + + try { + for (int i = 0; i < 100; i++) { + AirQuality record = AirQuality.newBuilder() + .setDeviceName("sensor-" + i) + .setTemp(22) + .setHumidity(55) + .build(); + long offset = stream.ingestRecordOffset(record); + stream.waitForOffset(offset); + } + } finally { + stream.close(); + } + } +} +``` + +### Proto Generation for Java + +```bash +java -jar zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \ + --uc-endpoint "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" \ + --client-id "$DATABRICKS_CLIENT_ID" \ + --client-secret "$DATABRICKS_CLIENT_SECRET" \ + --table "catalog.schema.table_name" \ + --output "record.proto" + +# Compile to Java +protoc --java_out=src/main/java record.proto +``` + +--- + +## Go (1.21+) + +### Installation + +```bash +go get github.com/databricks/zerobus-sdk-go +``` + +### JSON Flow + +```go +package main + +import ( + "fmt" + "log" + "os" + + zerobus "github.com/databricks/zerobus-go-sdk/sdk" +) + +func main() { + serverEndpoint := os.Getenv("ZEROBUS_SERVER_ENDPOINT") + workspaceURL := os.Getenv("DATABRICKS_WORKSPACE_URL") + tableName := os.Getenv("ZEROBUS_TABLE_NAME") + clientID := os.Getenv("DATABRICKS_CLIENT_ID") + clientSecret := os.Getenv("DATABRICKS_CLIENT_SECRET") + + sdk, err := zerobus.NewZerobusSdk(serverEndpoint, workspaceURL) + if err != nil { + log.Fatal(err) + } + defer sdk.Free() + + options := zerobus.DefaultStreamConfigurationOptions() + options.RecordType = zerobus.RecordTypeJson + + stream, err := sdk.CreateStream( + zerobus.TableProperties{TableName: tableName}, + clientID, clientSecret, options, + ) + if err != nil { + log.Fatal(err) + } + defer stream.Close() + + for i := 0; i < 100; i++ { + record := fmt.Sprintf( + `{"device_name": "sensor-%d", "temp": 22, "humidity": 55}`, i, + ) + offset, err := stream.IngestRecordOffset(record) + if err != nil { + log.Printf("Ingest failed for record %d: %v", i, err) + continue + } + stream.WaitForOffset(offset) + } + + stream.Flush() +} +``` + +### Protobuf Flow + +```go +options := zerobus.DefaultStreamConfigurationOptions() +options.RecordType = zerobus.RecordTypeProto + +// Load compiled proto descriptor +tableProps := zerobus.TableProperties{ + TableName: tableName, + DescriptorProto: descriptorBytes, // compiled .proto descriptor +} + +stream, err := sdk.CreateStream(tableProps, clientID, clientSecret, options) +// ... ingest protobuf-serialized bytes ... +``` + +--- + +## TypeScript / Node.js (16+) + +### Installation + +```bash +npm install @databricks/zerobus-ingest-sdk +``` + +### JSON Flow + +```typescript +import { ZerobusSdk, RecordType } from "@databricks/zerobus-ingest-sdk"; + +const serverEndpoint = process.env.ZEROBUS_SERVER_ENDPOINT!; +const workspaceUrl = process.env.DATABRICKS_WORKSPACE_URL!; +const tableName = process.env.ZEROBUS_TABLE_NAME!; +const clientId = process.env.DATABRICKS_CLIENT_ID!; +const clientSecret = process.env.DATABRICKS_CLIENT_SECRET!; + +const sdk = new ZerobusSdk(serverEndpoint, workspaceUrl); + +const stream = await sdk.createStream( + { tableName }, + clientId, + clientSecret, + { recordType: RecordType.Json } +); + +try { + for (let i = 0; i < 100; i++) { + const record = { device_name: `sensor-${i}`, temp: 22, humidity: 55 }; + const offset = await stream.ingestRecordOffset(record); + await stream.waitForOffset(offset); + } + await stream.flush(); +} finally { + await stream.close(); +} +``` + +### With Error Handling + +```typescript +import { ZerobusSdk, RecordType } from "@databricks/zerobus-ingest-sdk"; + +async function ingestWithRetry( + stream: any, + record: Record, + maxRetries = 3 +): Promise { + for (let attempt = 0; attempt < maxRetries; attempt++) { + try { + const offset = await stream.ingestRecordOffset(record); + await stream.waitForOffset(offset); + return true; + } catch (error) { + console.warn(`Attempt ${attempt + 1}/${maxRetries} failed:`, error); + if (attempt < maxRetries - 1) { + await new Promise((r) => setTimeout(r, 2 ** attempt * 1000)); + } + } + } + return false; +} +``` + +--- + +## Rust (1.70+) + +### Installation + +```bash +cargo add databricks-zerobus-ingest-sdk +cargo add tokio --features macros,rt-multi-thread +``` + +### JSON Flow + +```rust +use databricks_zerobus_ingest_sdk::{ + RecordType, StreamConfigurationOptions, TableProperties, ZerobusSdk, +}; +use std::env; +use std::error::Error; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let server_endpoint = env::var("ZEROBUS_SERVER_ENDPOINT")?; + let workspace_url = env::var("DATABRICKS_WORKSPACE_URL")?; + let table_name = env::var("ZEROBUS_TABLE_NAME")?; + let client_id = env::var("DATABRICKS_CLIENT_ID")?; + let client_secret = env::var("DATABRICKS_CLIENT_SECRET")?; + + let table_properties = TableProperties { + table_name, + descriptor_proto: None, + }; + + let options = StreamConfigurationOptions { + record_type: RecordType::Json, + ..Default::default() + }; + + let sdk = ZerobusSdk::new(server_endpoint, workspace_url)?; + let mut stream = sdk + .create_stream(table_properties, client_id, client_secret, Some(options)) + .await?; + + for i in 0..100 { + let record = format!( + r#"{{"device_name": "sensor-{}", "temp": 22, "humidity": 55}}"#, + i + ); + let offset = stream.ingest_record_offset(record.into_bytes()).await?; + stream.wait_for_offset(offset).await?; + } + + stream.close().await?; + Ok(()) +} +``` + +### Protobuf Flow + +```rust +let table_properties = TableProperties { + table_name: table_name.clone(), + descriptor_proto: Some(proto_descriptor_bytes), +}; + +let options = StreamConfigurationOptions { + record_type: RecordType::Proto, + ..Default::default() +}; + +let mut stream = sdk + .create_stream(table_properties, client_id, client_secret, Some(options)) + .await?; + +// Ingest serialized protobuf bytes +let record_bytes = my_proto_message.encode_to_vec(); +let offset = stream.ingest_record_offset(record_bytes).await?; +stream.wait_for_offset(offset).await?; +``` + +--- + +## Language Comparison + +| Feature | Python | Java | Go | TypeScript | Rust | +|---------|--------|------|----|------------|------| +| Min version | 3.9+ | 8+ | 1.21+ | Node 16+ | 1.70+ | +| Package | `databricks-zerobus-ingest-sdk` | `com.databricks:zerobus-ingest-sdk` | `github.com/databricks/zerobus-sdk-go` | `@databricks/zerobus-ingest-sdk` | `databricks-zerobus-ingest-sdk` | +| Default serialization | JSON | Protobuf | JSON | JSON | JSON | +| Async API | Yes (separate module) | CompletableFuture | Goroutines | Native async/await | Tokio async/await | +| ACK pattern | `wait_for_offset(offset)` or `AckCallback` | `waitForOffset(offset)` | `WaitForOffset(offset)` | `await waitForOffset(offset)` | `wait_for_offset(offset).await?` | +| Proto generation | `python -m zerobus.tools.generate_proto` | JAR CLI tool | External `protoc` | External `protoc` | External `protoc` | diff --git a/.claude/skills/databricks-zerobus-ingest/4-protobuf-schema.md b/.claude/skills/databricks-zerobus-ingest/4-protobuf-schema.md new file mode 100644 index 00000000..c8796faf --- /dev/null +++ b/.claude/skills/databricks-zerobus-ingest/4-protobuf-schema.md @@ -0,0 +1,191 @@ +# Protobuf Schema Generation + +Generate `.proto` schemas from Unity Catalog table definitions, compile language bindings, and understand Delta-to-Protobuf type mappings. + +--- + +## Why Protobuf? + +| Aspect | JSON | Protobuf | +|--------|------|----------| +| **Type safety** | None (runtime errors on mismatch) | Compile-time type checking | +| **Schema evolution** | Manual; easy to break silently | Forward-compatible by design | +| **Performance** | Text parsing overhead | Binary encoding, smaller payloads | +| **Recommended for** | Prototyping, simple schemas | Production, complex schemas | + +**Recommendation:** Use Protobuf for any production workload. Use JSON only for quick prototyping or when the schema is trivial. + +--- + +## Generate .proto from a UC Table + +### Python + +```bash +python -m zerobus.tools.generate_proto \ + --uc-endpoint "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" \ + --client-id "$DATABRICKS_CLIENT_ID" \ + --client-secret "$DATABRICKS_CLIENT_SECRET" \ + --table "catalog.schema.table_name" \ + --output record.proto +``` + +### Java + +```bash +java -jar zerobus-ingest-sdk-0.1.0-jar-with-dependencies.jar \ + --uc-endpoint "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com" \ + --client-id "$DATABRICKS_CLIENT_ID" \ + --client-secret "$DATABRICKS_CLIENT_SECRET" \ + --table "catalog.schema.table_name" \ + --output record.proto +``` + +The generated `.proto` file will contain a message definition matching the table schema, for example: + +```protobuf +syntax = "proto3"; + +message AirQuality { + string device_name = 1; + int32 temp = 2; + int64 humidity = 3; +} +``` + +--- + +## Compile Language Bindings + +### Python + +```bash +pip install grpcio-tools + +python -m grpc_tools.protoc \ + -I. \ + --python_out=. \ + record.proto +``` + +This generates `record_pb2.py`. Import and use it: + +```python +import record_pb2 + +record = record_pb2.AirQuality( + device_name="sensor-1", + temp=22, + humidity=55, +) +``` + +### Java + +```bash +protoc --java_out=src/main/java record.proto +``` + +Generates Java classes under `src/main/java/`. Usage: + +```java +import com.example.proto.Record.AirQuality; + +AirQuality record = AirQuality.newBuilder() + .setDeviceName("sensor-1") + .setTemp(22) + .setHumidity(55) + .build(); +``` + +### Go + +```bash +protoc --go_out=. record.proto +``` + +### Rust + +Use `prost` in `build.rs`: + +```rust +// build.rs +fn main() { + prost_build::compile_protos(&["record.proto"], &["."]).unwrap(); +} +``` + +--- + +## Delta-to-Protobuf Type Mappings + +| Delta / Spark Type | Protobuf Type | Notes | +|--------------------|---------------|-------| +| `STRING` | `string` | | +| `INT` / `INTEGER` | `int32` | | +| `LONG` / `BIGINT` | `int64` | | +| `FLOAT` | `float` | | +| `DOUBLE` | `double` | | +| `BOOLEAN` | `bool` | | +| `BINARY` | `bytes` | | +| `ARRAY` | `repeated T` | Element type maps recursively | +| `MAP` | `map` | Key must be string or integer type | +| `STRUCT` | Nested `message` | Fields map recursively | +| `DATE` | `int32` | Epoch days (days since 1970-01-01) | +| `TIMESTAMP` | `int64` | Epoch microseconds | +| `DECIMAL(p,s)` | `bytes` or `string` | Check generated .proto for exact mapping | +| `VARIANT` | `string` | JSON-encoded string | + +**Important:** The Protobuf schema must match the Delta table schema exactly (1:1 field mapping). If the table schema changes, regenerate the `.proto` and recompile. + +--- + +## Maximum Schema Size + +- Maximum **2000 columns** per proto schema +- Maximum **10 MB** per individual message (10,485,760 bytes) + +--- + +## Schema Evolution Workflow + +When your table schema changes: + +1. Alter the table in Unity Catalog (add columns, etc.) +2. Regenerate the `.proto` file using the generation command +3. Recompile language bindings +4. Update your producer code to populate new fields +5. Redeploy + +**Note:** Zerobus does not support automatic schema evolution. You must manage this process explicitly. + +--- + +## Using the Descriptor in Code + +### Python + +```python +from zerobus.sdk.shared import TableProperties, RecordType +import record_pb2 + +# Pass the DESCRIPTOR from the compiled module +table_props = TableProperties( + "catalog.schema.table_name", + record_pb2.AirQuality.DESCRIPTOR, +) +``` + +### Java + +```java +// Pass a default instance to extract the descriptor +TableProperties tableProperties = new TableProperties<>( + "catalog.schema.table_name", + AirQuality.getDefaultInstance() +); +``` + +### Go / Rust + +Pass the raw descriptor bytes when constructing `TableProperties`. diff --git a/.claude/skills/databricks-zerobus-ingest/5-operations-and-limits.md b/.claude/skills/databricks-zerobus-ingest/5-operations-and-limits.md new file mode 100644 index 00000000..004774d7 --- /dev/null +++ b/.claude/skills/databricks-zerobus-ingest/5-operations-and-limits.md @@ -0,0 +1,255 @@ +# Operations and Limits + +ACK handling, retry and reconnection patterns, throughput limits, delivery semantics, and operational constraints for Zerobus Ingest. + +--- + +## Acknowledgment (ACK) Handling + +Every ingested record returns a durability acknowledgment. An ACK indicates that **all records up to that offset** have been durably written to the target Delta table. + +### Strategies + +| Strategy | When to Use | Trade-off | +|----------|-------------|-----------| +| **`ingest_record_offset` + `wait_for_offset`** | Low-volume, strict ordering | Simplest; lower throughput | +| **`ingest_record_nowait` + `AckCallback`** | High-volume producers | Higher throughput; more complex | +| **`ingest_record_nowait` + periodic `flush`** | Batch-oriented workloads | Best throughput; eventual consistency | + +### Sync Block (Python) + +```python +offset = stream.ingest_record_offset(record) +stream.wait_for_offset(offset) # Blocks until durable +``` + +### ACK Callback (Python) + +```python +from zerobus.sdk.shared import AckCallback + +class MyAckHandler(AckCallback): + def __init__(self): + self.last_acked_offset = 0 + + def on_ack(self, offset: int) -> None: + self.last_acked_offset = offset + + def on_error(self, offset: int, message: str) -> None: + print(f"Error at offset {offset}: {message}") + +options = StreamConfigurationOptions( + record_type=RecordType.JSON, + ack_callback=MyAckHandler(), +) +``` + +### Flush-Based + +```python +# Send many records without blocking (fire-and-forget) +for record in batch: + stream.ingest_record_nowait(record) + +# Flush ensures all buffered records are sent +stream.flush() +``` + +--- + +## Retry and Reconnection + +Zerobus streams can close due to server maintenance, network issues, or zone failures. Implement retry with exponential backoff and stream reinitialization. + +### Pattern (Any Language) + +``` +1. Attempt ingest +2. On connection/closed error: + a. Close the current stream + b. Wait with exponential backoff (1s, 2s, 4s, ...) + c. Reinitialize the stream + d. Retry the record +3. After max retries, log failure and escalate +``` + +### Python Implementation + +```python +import time +import logging + +logger = logging.getLogger(__name__) + +def ingest_with_retry(stream_factory, record, max_retries=5): + """Ingest a record with retry and stream reinitialization. + + Args: + stream_factory: Callable that returns a new stream. + record: The record to ingest. + max_retries: Maximum retry attempts. + """ + stream = stream_factory() + + for attempt in range(max_retries): + try: + offset = stream.ingest_record_offset(record) + stream.wait_for_offset(offset) + return stream # Return the (possibly new) stream + except Exception as e: + err = str(e).lower() + logger.warning("Attempt %d/%d failed: %s", attempt + 1, max_retries, e) + + if "closed" in err or "connection" in err or "unavailable" in err: + try: + stream.close() + except Exception: + pass + backoff = min(2 ** attempt, 30) # Cap at 30s + time.sleep(backoff) + stream = stream_factory() + elif attempt < max_retries - 1: + time.sleep(2 ** attempt) + else: + raise + + return stream +``` + +### Key Points + +- **Always reinitialize the stream** on connection errors, not just retry the same stream +- **Cap backoff** at a reasonable maximum (e.g., 30 seconds) +- **Log failures** with enough context to diagnose (endpoint, table, error message) +- **Design for at-least-once**: your downstream consumers should handle duplicate records + +--- + +## Delivery Semantics + +Zerobus provides **at-least-once** delivery guarantees: + +- Records may be delivered more than once (e.g., after a retry where the original was actually persisted) +- There is **no exactly-once** semantics +- Design your target table and downstream consumers to handle duplicates (e.g., deduplication via `MERGE` or unique constraints) + +--- + +## Throughput Limits + +| Limit | Value | Notes | +|-------|-------|-------| +| **Throughput per stream** | 100 MB/s | Based on 1 KB messages | +| **Rows per stream** | 15,000 rows/s | | +| **Max message size** | 10 MB (10,485,760 bytes) | Per individual record | +| **Max columns** | 2,000 | Per proto schema / table | + +### Scaling Beyond One Stream + +If you need higher throughput than a single stream provides: + +- Open **multiple streams** to the same table from different clients +- Zerobus supports **thousands of concurrent clients** writing to the same table +- Partition your data across streams by key (e.g., device ID, region) +- Contact Databricks for custom throughput requirements + +--- + +## Regional Availability + +Workspace and target tables must be in a supported region for your cloud provider. + +### AWS Supported Regions + +| Region | Code | +|--------|------| +| US East (N. Virginia) | `us-east-1` | +| US East (Ohio) | `us-east-2` | +| US West (Oregon) | `us-west-2` | +| Europe (Frankfurt) | `eu-central-1` | +| Europe (Ireland) | `eu-west-1` | +| Asia Pacific (Singapore) | `ap-southeast-1` | +| Asia Pacific (Sydney) | `ap-southeast-2` | +| Asia Pacific (Tokyo) | `ap-northeast-1` | +| Canada (Central) | `ca-central-1` | + +### Azure Supported Regions + +| Region | Code | +|--------|------| +| Canada Central | `canadacentral` | +| West US | `westus` | +| East US | `eastus` | +| East US 2 | `eastus2` | +| Central US | `centralus` | +| North Central US | `northcentralus` | +| Sweden Central | `swedencentral` | +| West Europe | `westeurope` | +| North Europe | `northeurope` | +| Australia East | `australiaeast` | +| Southeast Asia | `southeastasia` | + +**Performance note:** Optimal throughput requires the client application and Zerobus endpoint to be in the **same region**. + +--- + +## Durability and Availability + +- **Single-AZ only**: Zerobus runs in a single availability zone. The service may experience downtime if that zone is unavailable. +- **No geographic redundancy**: Plan for zone outages in your producer's retry logic. +- **Maintenance windows**: The server may close streams during maintenance. Your client should handle reconnection gracefully. + +--- + +## Target Table Constraints + +| Constraint | Details | +|------------|---------| +| **Table type** | Managed Delta tables only (no external storage) | +| **Table names** | ASCII letters, digits, underscores only | +| **Schema changes** | No auto-evolution; regenerate proto and redeploy | +| **Table creation** | Zerobus does not create tables; pre-create via SQL DDL | +| **Table recreation** | Cannot recreate an existing target table via Zerobus | + +--- + +## Supported Data Types + +| Delta Type | Protobuf Type | Conversion Notes | +|------------|---------------|------------------| +| STRING | string | Direct mapping | +| INT / INTEGER | int32 | Direct mapping | +| LONG / BIGINT | int64 | Direct mapping | +| FLOAT | float | Direct mapping | +| DOUBLE | double | Direct mapping | +| BOOLEAN | bool | Direct mapping | +| BINARY | bytes | Direct mapping | +| ARRAY\ | repeated T | Recursive mapping | +| MAP\ | map\ | Key must be string or integer | +| STRUCT | nested message | Recursive mapping | +| DATE | int32 | Epoch days since 1970-01-01 | +| TIMESTAMP | int64 | Epoch microseconds | +| VARIANT | string | JSON-encoded string | + +--- + +## Monitoring and Observability + +Zerobus does not currently expose built-in metrics dashboards. Monitor your producers with: + +- **Application-level logging**: Log ACK offsets, retry counts, and error rates +- **ACK callback tracking**: Track the last-acked offset to measure ingestion lag +- **Table row counts**: Periodically query the target table to verify data is arriving +- **Health checks**: Attempt a lightweight ingest (or stream creation) to verify connectivity + +```python +# Simple health check +def check_zerobus_health(sdk, client_id, client_secret, table_props, options): + try: + stream = sdk.create_stream(client_id, client_secret, table_props, options) + stream.close() + return True + except Exception as e: + logger.error("Zerobus health check failed: %s", e) + return False +``` diff --git a/.claude/skills/databricks-zerobus-ingest/SKILL.md b/.claude/skills/databricks-zerobus-ingest/SKILL.md new file mode 100644 index 00000000..22f90c55 --- /dev/null +++ b/.claude/skills/databricks-zerobus-ingest/SKILL.md @@ -0,0 +1,233 @@ +--- +name: databricks-zerobus-ingest +description: "Build Zerobus Ingest clients for near real-time data ingestion into Databricks Delta tables via gRPC. Use when creating producers that write directly to Unity Catalog tables without a message bus, working with the Zerobus Ingest SDK in Python/Java/Go/TypeScript/Rust, generating Protobuf schemas from UC tables, or implementing stream-based ingestion with ACK handling and retry logic." +--- + +# Zerobus Ingest + +Build clients that ingest data directly into Databricks Delta tables via the Zerobus gRPC API. + +**Status:** GA (Generally Available since February 2026; billed under Lakeflow Jobs Serverless SKU) + +**Documentation:** +- [Zerobus Overview](https://docs.databricks.com/aws/en/ingestion/zerobus-overview) +- [Zerobus Ingest SDK](https://docs.databricks.com/aws/en/ingestion/zerobus-ingest) +- [Zerobus Limits](https://docs.databricks.com/aws/en/ingestion/zerobus-limits) + +--- + +## What Is Zerobus Ingest? + +Zerobus Ingest is a serverless connector that enables direct, record-by-record data ingestion into Delta tables via gRPC. It eliminates the need for message bus infrastructure (Kafka, Kinesis, Event Hub) for lakehouse-bound data. The service validates schemas, materializes data to target tables, and sends durability acknowledgments back to the client. + +**Core pattern:** SDK init -> create stream -> ingest records -> handle ACKs -> flush -> close + +--- + +## Quick Decision: What Are You Building? + +| Scenario | Language | Serialization | Reference | +|----------|----------|---------------|-----------| +| Quick prototype / test harness | Python | JSON | [2-python-client.md](2-python-client.md) | +| Production Python producer | Python | Protobuf | [2-python-client.md](2-python-client.md) + [4-protobuf-schema.md](4-protobuf-schema.md) | +| JVM microservice | Java | Protobuf | [3-multilanguage-clients.md](3-multilanguage-clients.md) | +| Go service | Go | JSON or Protobuf | [3-multilanguage-clients.md](3-multilanguage-clients.md) | +| Node.js / TypeScript app | TypeScript | JSON | [3-multilanguage-clients.md](3-multilanguage-clients.md) | +| High-performance system service | Rust | JSON or Protobuf | [3-multilanguage-clients.md](3-multilanguage-clients.md) | +| Schema generation from UC table | Any | Protobuf | [4-protobuf-schema.md](4-protobuf-schema.md) | +| Retry / reconnection logic | Any | Any | [5-operations-and-limits.md](5-operations-and-limits.md) | + +If not specified, default to python. + +--- + +## Common Libraries + +These libraries are essential for ZeroBus data ingestion: + +- **databricks-sdk>=0.85.0**: Databricks workspace client for authentication and metadata +- **databricks-zerobus-ingest-sdk>=1.0.0**: ZeroBus SDK for high-performance streaming ingestion +- **grpcio-tools** +These are typically NOT pre-installed on Databricks. Install them using `execute_code` tool: +- `code`: "%pip install databricks-sdk>=VERSION databricks-zerobus-ingest-sdk>=VERSION" + +Save the returned `cluster_id` and `context_id` for subsequent calls. + +Smart Installation Approach + +# Check protobuf version first, then install compatible +grpcio-tools +import google.protobuf +runtime_version = google.protobuf.__version__ +print(f"Runtime protobuf version: {runtime_version}") + +if runtime_version.startswith("5.26") or +runtime_version.startswith("5.29"): + %pip install grpcio-tools==1.62.0 +else: + %pip install grpcio-tools # Use latest for newer protobuf +versions +--- + +## Prerequisites + +You must never execute the skill without confirming the below objects are valid: + +1. **A Unity Catalog managed Delta table** to ingest into +2. **A service principal id and secret** with `MODIFY` and `SELECT` on the target table +3. **The Zerobus server endpoint** for your workspace region +4. **The Zerobus Ingest SDK** installed for your target language + +See [1-setup-and-authentication.md](1-setup-and-authentication.md) for complete setup instructions. + +--- + +## Minimal Python Example (JSON) + +```python +import json +from zerobus.sdk.sync import ZerobusSdk +from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties + +sdk = ZerobusSdk(server_endpoint, workspace_url) +options = StreamConfigurationOptions(record_type=RecordType.JSON) +table_props = TableProperties(table_name) + +stream = sdk.create_stream(client_id, client_secret, table_props, options) +try: + record = {"device_name": "sensor-1", "temp": 22, "humidity": 55} + stream.ingest_record(json.dumps(record)) + stream.flush() +finally: + stream.close() +``` + +--- + +## Detailed guides + +| Topic | File | When to Read | +|-------|------|--------------| +| Setup & Auth | [1-setup-and-authentication.md](1-setup-and-authentication.md) | Endpoint formats, service principals, SDK install | +| Python Client | [2-python-client.md](2-python-client.md) | Sync/async Python, JSON and Protobuf flows, reusable client class | +| Multi-Language | [3-multilanguage-clients.md](3-multilanguage-clients.md) | Java, Go, TypeScript, Rust SDK examples | +| Protobuf Schema | [4-protobuf-schema.md](4-protobuf-schema.md) | Generate .proto from UC table, compile, type mappings | +| Operations & Limits | [5-operations-and-limits.md](5-operations-and-limits.md) | ACK handling, retries, reconnection, throughput limits, constraints | + +--- + +You must always follow all the steps in the Workflow + +## Workflow +0. **Display the plan of your execution** +1. **Determinate the type of client** +2. **Get schema** Always use 4-protobuf-schema.md. Execute using the `execute_code` MCP tool +3. **Write Python code to a local file follow the instructions in the relevant guide to ingest with zerobus** in the project (e.g., `scripts/zerobus_ingest.py`). +4. **Execute on Databricks** using the `execute_code` MCP tool (with `file_path` parameter) +5. **If execution fails**: Edit the local file to fix the error, then re-execute +6. **Reuse the context** for follow-up executions by passing the returned `cluster_id` and `context_id` + +--- + +## Important +- Never install local packages +- Always validate MCP server requirement before execution +- **Serverless limitation**: The Zerobus SDK cannot pip-install on serverless compute. Use classic compute clusters, or use the [Zerobus REST API](https://docs.databricks.com/aws/en/ingestion/zerobus-rest-api) (Beta) for notebook-based ingestion without the SDK. +- **Explicit table grants**: Service principals need explicit `MODIFY` and `SELECT` grants on the target table. Schema-level inherited permissions may not be sufficient for the `authorization_details` OAuth flow. + +--- + +### Context Reuse Pattern + +The first execution auto-selects a running cluster and creates an execution context. **Reuse this context for follow-up calls** - it's much faster (~1s vs ~15s) and shares variables/imports: + +**First execution** - use `execute_code` tool: +- `file_path`: "scripts/zerobus_ingest.py" + +Returns: `{ success, output, error, cluster_id, context_id, ... }` + +Save `cluster_id` and `context_id` for follow-up calls. + +**If execution fails:** +1. Read the error from the result +2. Edit the local Python file to fix the issue +3. Re-execute with same context using `execute_code` tool: + - `file_path`: "scripts/zerobus_ingest.py" + - `cluster_id`: "" + - `context_id`: "" + +**Follow-up executions** reuse the context (faster, shares state): +- `file_path`: "scripts/validate_ingestion.py" +- `cluster_id`: "" +- `context_id`: "" + +### Handling Failures + +When execution fails: +1. Read the error from the result +2. **Edit the local Python file** to fix the issue +3. Re-execute using the same `cluster_id` and `context_id` (faster, keeps installed libraries) +4. If the context is corrupted, omit `context_id` to create a fresh one + +--- + +### Installing Libraries + +Databricks provides Spark, pandas, numpy, and common data libraries by default. **Only install a library if you get an import error.** + +Use `execute_code` tool: +- `code`: "%pip install databricks-zerobus-ingest-sdk>=1.0.0" +- `cluster_id`: "" +- `context_id`: "" + +The library is immediately available in the same context. + +**Note:** Keeping the same `context_id` means installed libraries persist across calls. + +## 🚨 Critical Learning: Timestamp Format Fix + +**BREAKTHROUGH**: ZeroBus requires **timestamp fields as Unix integer timestamps**, NOT string timestamps. +The timestamp generation must use microseconds for Databricks. + +--- + +## Key Concepts + +- **gRPC + Protobuf**: Zerobus uses gRPC as its transport protocol. Any application that can communicate via gRPC and construct Protobuf messages can produce to Zerobus. +- **JSON or Protobuf serialization**: JSON for quick starts; Protobuf for type safety, forward compatibility, and performance. +- **At-least-once delivery**: The connector provides at-least-once guarantees. Design consumers to handle duplicates. +- **Durability ACKs**: Each ingested record returns a `RecordAcknowledgment`. Use `flush()` to ensure all buffered records are durably written, or use `wait_for_offset(offset)` for offset-based tracking. +- **No table management**: Zerobus does not create or alter tables. You must pre-create your target table and manage schema evolution yourself. +- **Single-AZ durability**: The service runs in a single availability zone. Plan for potential zone outages. + +--- + +## Common Issues + +| Issue | Solution | +|-------|----------| +| **Connection refused** | Verify server endpoint format matches your cloud (AWS vs Azure). Check firewall allowlists. | +| **Authentication failed** | Confirm service principal client_id/secret. Verify GRANT statements on the target table. | +| **Schema mismatch** | Ensure record fields match the target table schema exactly. Regenerate .proto if table changed. | +| **Stream closed unexpectedly** | Implement retry with exponential backoff and stream reinitialization. See [5-operations-and-limits.md](5-operations-and-limits.md). | +| **Throughput limits hit** | Max 100 MB/s and 15,000 rows/s per stream. Open multiple streams or contact Databricks. | +| **Region not supported** | Check supported regions in [5-operations-and-limits.md](5-operations-and-limits.md). | +| **Table not found** | Ensure table is a managed Delta table in a supported region with correct three-part name. | +| **SDK install fails on serverless** | The Zerobus SDK cannot be pip-installed on serverless compute. Use classic compute clusters or the REST API (Beta) from notebooks. | +| **Error 4024 / authorization_details** | Service principal lacks explicit table-level grants. Grant `MODIFY` and `SELECT` directly on the target table — schema-level inherited grants may be insufficient. | + +--- + +## Related Skills + +- **[databricks-python-sdk](../databricks-python-sdk/SKILL.md)** - General SDK patterns and WorkspaceClient for table/schema management +- **[databricks-spark-declarative-pipelines](../databricks-spark-declarative-pipelines/SKILL.md)** - Downstream pipeline processing of ingested data +- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - Managing catalogs, schemas, and tables that Zerobus writes to +- **[databricks-synthetic-data-gen](../databricks-synthetic-data-gen/SKILL.md)** - Generate test data to feed into Zerobus producers +- **[databricks-config](../databricks-config/SKILL.md)** - Profile and authentication setup + +## Resources + +- [Zerobus Overview](https://docs.databricks.com/aws/en/ingestion/zerobus-overview) +- [Zerobus Ingest SDK](https://docs.databricks.com/aws/en/ingestion/zerobus-ingest) +- [Zerobus Limits](https://docs.databricks.com/aws/en/ingestion/zerobus-limits) diff --git a/.claude/skills/flutter-add-integration-test/SKILL.md b/.claude/skills/flutter-add-integration-test/SKILL.md new file mode 100644 index 00000000..60902f1a --- /dev/null +++ b/.claude/skills/flutter-add-integration-test/SKILL.md @@ -0,0 +1,163 @@ +--- +name: flutter-add-integration-test +description: Configures Flutter Driver for app interaction and converts MCP actions into permanent integration tests. Use when adding integration testing to a project, exploring UI components via MCP, or automating user flows with the integration_test package. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 18:29:20 GMT +--- +# Implementing Flutter Integration Tests + +## Contents +- [Project Setup and Dependencies](#project-setup-and-dependencies) +- [Interactive Exploration via MCP](#interactive-exploration-via-mcp) +- [Test Authoring Guidelines](#test-authoring-guidelines) +- [Execution and Profiling](#execution-and-profiling) +- [Workflow: End-to-End Integration Testing](#workflow-end-to-end-integration-testing) +- [Examples](#examples) + +## Project Setup and Dependencies + +Configure the project to support integration testing and Flutter Driver extensions. + +1. Add required development dependencies to `pubspec.yaml`: + ```bash + flutter pub add 'dev:integration_test:{"sdk":"flutter"}' + flutter pub add 'dev:flutter_test:{"sdk":"flutter"}' + ``` +2. Enable the Flutter Driver extension in your application entry point (typically `lib/main.dart` or a dedicated `lib/main_test.dart`): + - Import `package:flutter_driver/driver_extension.dart`. + - Call `enableFlutterDriverExtension();` before `runApp()`. +3. Add `Key` parameters (e.g., `ValueKey('login_button')`) to critical widgets in the application code to ensure reliable targeting during tests. + +## Interactive Exploration via MCP + +Use the Dart/Flutter MCP server tools to interactively explore and manipulate the application state before writing static tests. + +- **Launch**: Execute `launch_app` with `target: "lib/main_test.dart"` to start the application and acquire the DTD URI. +- **Inspect**: Execute `get_widget_tree` to discover available `Key`s, `Text` nodes, and widget `Type`s. +- **Interact**: Execute `tap`, `enter_text`, and `scroll` to simulate user flows. +- **Wait**: Always execute `waitFor` or verify state with `get_health` when navigating or triggering animations. +- **Troubleshoot Unmounted Widgets**: If a widget is not found in the tree, it may be lazily loaded in a `SliverList` or `ListView`. Execute `scroll` or `scrollIntoView` to force the widget to mount before interacting with it. + +## Test Authoring Guidelines + +Structure integration tests using the `flutter_test` API paradigm. + +- Create a dedicated `integration_test/` directory at the project root. +- Name all test files using the `_test.dart` convention. +- Initialize the binding by calling `IntegrationTestWidgetsFlutterBinding.ensureInitialized();` at the start of `main()`. +- Load the application UI using `await tester.pumpWidget(MyApp());`. +- Trigger frames and wait for animations to complete using `await tester.pumpAndSettle();` after interactions like `tester.tap()`. +- Assert widget visibility using `expect(find.byKey(ValueKey('foo')), findsOneWidget);` or `findsNothing`. +- Scroll to specific off-screen widgets using `await tester.scrollUntilVisible(itemFinder, 500.0, scrollable: listFinder);`. + +**Conditional Logic for Legacy `flutter_driver`:** +- If maintaining or migrating legacy `flutter_driver` tests, use `driver.waitFor()`, `driver.waitForAbsent()`, `driver.tap()`, and `driver.scroll()` instead of the `WidgetTester` APIs. + +## Execution and Profiling + +Execute tests using the `flutter drive` command. Require a host driver script located in `test_driver/integration_test.dart` that calls `integrationDriver()`. + +**Conditional Execution Targets:** +- **If testing on Chrome:** Launch `chromedriver --port=4444` in a separate terminal, then run: + `flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart -d chrome` +- **If testing headless web:** Run with `-d web-server`. +- **If testing on Android (Local):** Run `flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart`. +- **If testing on Firebase Test Lab (Android):** + 1. Build debug APK: `flutter build apk --debug` + 2. Build test APK: `./gradlew app:assembleAndroidTest` + 3. Upload both APKs to the Firebase Test Lab console. + +## Workflow: End-to-End Integration Testing + +Copy and follow this checklist to implement and verify integration tests. + +- [ ] **Task Progress: Setup** + - [ ] Add `integration_test` and `flutter_test` to `pubspec.yaml`. + - [ ] Inject `enableFlutterDriverExtension()` into the app entry point. + - [ ] Assign `ValueKey`s to target widgets. +- [ ] **Task Progress: Exploration** + - [ ] Run `launch_app` via MCP. + - [ ] Map the widget tree using `get_widget_tree`. + - [ ] Validate interaction paths using MCP tools (`tap`, `enter_text`). +- [ ] **Task Progress: Authoring** + - [ ] Create `integration_test/app_test.dart`. + - [ ] Write test cases using `WidgetTester` APIs. + - [ ] Create `test_driver/integration_test.dart` with `integrationDriver()`. +- [ ] **Task Progress: Execution & Feedback Loop** + - [ ] Run `flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart`. + - [ ] **Feedback Loop**: Review test output -> If `PumpAndSettleTimedOutException` occurs, check for infinite animations -> If widget not found, add `scrollUntilVisible` -> Re-run test until passing. + +## Examples + +### Standard Integration Test (`integration_test/app_test.dart`) + +```dart +import 'package:flutter/material.dart'; +import 'package:flutter_test/flutter_test.dart'; +import 'package:integration_test/integration_test.dart'; +import 'package:my_app/main.dart'; + +void main() { + IntegrationTestWidgetsFlutterBinding.ensureInitialized(); + + group('End-to-end test', () { + testWidgets('tap on the floating action button, verify counter', (tester) async { + // Load app widget. + await tester.pumpWidget(const MyApp()); + + // Verify the counter starts at 0. + expect(find.text('0'), findsOneWidget); + + // Find the floating action button to tap on. + final fab = find.byKey(const ValueKey('increment')); + + // Emulate a tap on the floating action button. + await tester.tap(fab); + + // Trigger a frame and wait for animations. + await tester.pumpAndSettle(); + + // Verify the counter increments by 1. + expect(find.text('1'), findsOneWidget); + }); + }); +} +``` + +### Host Driver Script (`test_driver/integration_test.dart`) + +```dart +import 'package:integration_test/integration_test_driver.dart'; + +Future main() => integrationDriver(); +``` + +### Performance Profiling Driver Script (`test_driver/perf_driver.dart`) + +Use this driver script if you wrap your test actions in `binding.traceAction()` to capture performance metrics. + +```dart +import 'package:flutter_driver/flutter_driver.dart' as driver; +import 'package:integration_test/integration_test_driver.dart'; + +Future main() { + return integrationDriver( + responseDataCallback: (data) async { + if (data != null) { + final timeline = driver.Timeline.fromJson( + data['scrolling_timeline'] as Map, + ); + + final summary = driver.TimelineSummary.summarize(timeline); + + await summary.writeTimelineToFile( + 'scrolling_timeline', + pretty: true, + includeSummary: true, + ); + } + }, + ); +} +``` diff --git a/.claude/skills/flutter-add-widget-preview/SKILL.md b/.claude/skills/flutter-add-widget-preview/SKILL.md new file mode 100644 index 00000000..6ba68942 --- /dev/null +++ b/.claude/skills/flutter-add-widget-preview/SKILL.md @@ -0,0 +1,145 @@ +--- +name: flutter-add-widget-preview +description: Adds interactive widget previews to the project using the previews.dart system. Use when creating new UI components or updating existing screens to ensure consistent design and interactive testing. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 20:05:23 GMT +--- +# Previewing Flutter Widgets + +## Contents +- [Preview Guidelines](#preview-guidelines) +- [Handling Limitations](#handling-limitations) +- [Workflows](#workflows) +- [Examples](#examples) + +## Preview Guidelines + +Use the Flutter Widget Previewer to render widgets in real-time, isolated from the full application context. + +- **Target Elements:** Apply the `@Preview` annotation to top-level functions, static methods within a class, or public widget constructors/factories that have no required arguments and return a `Widget` or `WidgetBuilder`. +- **Imports:** Always import `package:flutter/widget_previews.dart` to access the preview annotations. +- **Custom Annotations:** Extend the `Preview` class to create custom annotations that inject common properties (e.g., themes, wrappers) across multiple widgets. +- **Multiple Configurations:** Apply multiple `@Preview` annotations to a single target to generate multiple preview instances. Alternatively, extend `MultiPreview` to encapsulate common multi-preview configurations. +- **Runtime Transformations:** Override the `transform()` method in custom `Preview` or `MultiPreview` classes to modify preview configurations dynamically at runtime (e.g., generating names based on dynamic values, which is impossible in a `const` context). + +## Handling Limitations + +Adhere to the following constraints when authoring previewable widgets, as the Widget Previewer runs in a web environment: + +- **No Native APIs:** Do not use native plugins or APIs from `dart:io` or `dart:ffi`. Widgets with transitive dependencies on `dart:io` or `dart:ffi` will throw exceptions upon invocation. Use conditional imports to mock or bypass these in preview mode. +- **Asset Paths:** Use package-based paths for assets loaded via `dart:ui` `fromAsset` APIs (e.g., `packages/my_package_name/assets/my_image.png` instead of `assets/my_image.png`). +- **Public Callbacks:** Ensure all callback arguments provided to preview annotations are public and constant to satisfy code generation requirements. +- **Constraints:** Apply explicit constraints using the `size` parameter in the `@Preview` annotation if your widget is unconstrained, as the previewer defaults to constraining them to approximately half the viewport. + +## Workflows + +### Creating a Widget Preview +Copy and track this checklist when implementing a new widget preview: + +- [ ] Import `package:flutter/widget_previews.dart`. +- [ ] Identify a valid target (top-level function, static method, or parameter-less public constructor). +- [ ] Apply the `@Preview` annotation to the target. +- [ ] Configure preview parameters (`name`, `group`, `size`, `theme`, `brightness`, etc.) as needed. +- [ ] If applying the same configuration to multiple widgets, extract the configuration into a custom class extending `Preview`. + +### Interacting with Previews +Follow the appropriate conditional workflow to launch and interact with the Widget Previewer: + +**If using a supported IDE (Android Studio, IntelliJ, VS Code with Flutter 3.38+):** +1. Launch the IDE. The Widget Previewer starts automatically. +2. Open the "Flutter Widget Preview" tab in the sidebar. +3. Toggle "Filter previews by selected file" at the bottom left if you want to view previews outside the currently active file. + +**If using the Command Line:** +1. Navigate to the Flutter project's root directory. +2. Run `flutter widget-preview start`. +3. View the automatically opened Chrome environment. + +**Feedback Loop: Preview Iteration** +1. Modify the widget code or preview configuration. +2. Observe the automatic update in the Widget Previewer. +3. If global state (e.g., static initializers) was modified: Click the global hot restart button at the bottom right. +4. If only the local widget state needs resetting: Click the individual hot restart button on the specific preview card. +5. Review errors in the IDE/CLI console -> fix -> repeat. + +## Examples + +### Basic Preview +```dart +import 'package:flutter/widget_previews.dart'; +import 'package:flutter/material.dart'; + +@Preview(name: 'My Sample Text', group: 'Typography') +Widget mySampleText() { + return const Text('Hello, World!'); +} +``` + +### Custom Preview with Runtime Transformation +```dart +import 'package:flutter/widget_previews.dart'; +import 'package:flutter/material.dart'; + +final class TransformativePreview extends Preview { + const TransformativePreview({ + super.name, + super.group, + }); + + PreviewThemeData _themeBuilder() { + return PreviewThemeData( + materialLight: ThemeData.light(), + materialDark: ThemeData.dark(), + ); + } + + @override + Preview transform() { + final originalPreview = super.transform(); + final builder = originalPreview.toBuilder(); + + builder + ..name = 'Transformed - ${originalPreview.name}' + ..theme = _themeBuilder; + + return builder.toPreview(); + } +} + +@TransformativePreview(name: 'Custom Themed Button') +Widget myButton() => const ElevatedButton(onPressed: null, child: Text('Click')); +``` + +### MultiPreview Implementation +```dart +import 'package:flutter/widget_previews.dart'; +import 'package:flutter/material.dart'; + +/// Creates light and dark mode previews automatically. +final class MultiBrightnessPreview extends MultiPreview { + const MultiBrightnessPreview({required this.name}); + + final String name; + + @override + List get previews => const [ + Preview(brightness: Brightness.light), + Preview(brightness: Brightness.dark), + ]; + + @override + List transform() { + final previews = super.transform(); + return previews.map((preview) { + final builder = preview.toBuilder() + ..group = 'Brightness' + ..name = '$name - ${preview.brightness!.name}'; + return builder.toPreview(); + }).toList(); + } +} + +@MultiBrightnessPreview(name: 'Primary Card') +Widget cardPreview() => const Card(child: Padding(padding: EdgeInsets.all(8.0), child: Text('Content'))); +``` diff --git a/.claude/skills/flutter-add-widget-test/SKILL.md b/.claude/skills/flutter-add-widget-test/SKILL.md new file mode 100644 index 00000000..01ac7ac6 --- /dev/null +++ b/.claude/skills/flutter-add-widget-test/SKILL.md @@ -0,0 +1,154 @@ +--- +name: flutter-add-widget-test +description: Implement a component-level test using `WidgetTester` to verify UI rendering and user interactions (tapping, scrolling, entering text). Use when validating that a specific widget displays correct data and responds to events as expected. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 21:15:41 GMT +--- +# Writing Flutter Widget Tests + +## Contents +- [Setup & Configuration](#setup--configuration) +- [Core Components](#core-components) +- [Workflow: Implementing a Widget Test](#workflow-implementing-a-widget-test) +- [Interaction & State Management](#interaction--state-management) +- [Examples](#examples) + +## Setup & Configuration + +Ensure the testing environment is properly configured before authoring widget tests. + +1. Add the `flutter_test` dependency to the `dev_dependencies` section of `pubspec.yaml`. +2. Place all test files in the `test/` directory at the root of the project. +3. Suffix all test file names with `_test.dart` (e.g., `widget_test.dart`). + +## Core Components + +Utilize the following `flutter_test` components to interact with and validate the widget tree: + +* **`WidgetTester`**: The primary interface for building and interacting with widgets in the test environment. Provided automatically by the `testWidgets()` function. +* **`Finder`**: Locates widgets in the test environment (e.g., `find.text('Submit')`, `find.byType(TextField)`, `find.byKey(Key('submit_btn'))`). +* **`Matcher`**: Verifies the presence or state of widgets located by a `Finder` (e.g., `findsOneWidget`, `findsNothing`, `findsNWidgets(2)`, `matchesGoldenFile`). + +## Workflow: Implementing a Widget Test + +Copy the following checklist to track progress when implementing a new widget test. + +### Task Progress +- [ ] **Step 1: Define the test.** Use `testWidgets('description', (WidgetTester tester) async { ... })`. +- [ ] **Step 2: Build the widget.** Call `await tester.pumpWidget(MyWidget())` to render the UI. Wrap the widget in a `MaterialApp` or `Directionality` widget if it requires inherited directional or theme data. +- [ ] **Step 3: Locate elements.** Instantiate `Finder` objects for the target widgets. +- [ ] **Step 4: Verify initial state.** Use `expect(finder, matcher)` to validate the initial render. +- [ ] **Step 5: Simulate interactions.** Execute gestures or inputs (e.g., `await tester.tap(buttonFinder)`). +- [ ] **Step 6: Rebuild the tree.** Call `await tester.pump()` or `await tester.pumpAndSettle()` to process state changes. +- [ ] **Step 7: Verify updated state.** Use `expect()` to validate the UI after the interaction. +- [ ] **Step 8: Run and validate.** Execute `flutter test test/your_test_file_test.dart`. +- [ ] **Step 9: Feedback Loop.** Review test output -> identify failing matchers -> adjust widget logic or test assertions -> re-run until passing. + +## Interaction & State Management + +Apply the following conditional logic based on the type of interaction or state change being tested: + +* **If testing static rendering:** Call `await tester.pumpWidget()` once, then immediately run `expect()` assertions. +* **If testing standard state changes (e.g., button taps):** + 1. Call `await tester.tap(finder)`. + 2. Call `await tester.pump()` to trigger a single frame rebuild. +* **If testing animations, transitions, or asynchronous UI updates:** + 1. Trigger the action (e.g., `await tester.drag(finder, Offset(500, 0))`). + 2. Call `await tester.pumpAndSettle()` to repeatedly pump frames until no more frames are scheduled (animation completes). +* **If testing text input:** Call `await tester.enterText(textFieldFinder, 'Input string')`. +* **If testing items in a dynamic or long list:** Call `await tester.scrollUntilVisible(itemFinder, 500.0, scrollable: listFinder)` to ensure the target widget is rendered before interacting with it. + +## Examples + +### High-Fidelity Widget Test Implementation + +**Target Widget (`lib/todo_list.dart`):** +```dart +import 'package:flutter/material.dart'; + +class TodoList extends StatefulWidget { + const TodoList({super.key}); + + @override + State createState() => _TodoListState(); +} + +class _TodoListState extends State { + final todos = []; + final controller = TextEditingController(); + + @override + Widget build(BuildContext context) { + return MaterialApp( + home: Scaffold( + body: Column( + children: [ + TextField(controller: controller), + Expanded( + child: ListView.builder( + itemCount: todos.length, + itemBuilder: (context, index) { + final todo = todos[index]; + return Dismissible( + key: Key('$todo$index'), + onDismissed: (_) => setState(() => todos.removeAt(index)), + child: ListTile(title: Text(todo)), + ); + }, + ), + ), + ], + ), + floatingActionButton: FloatingActionButton( + onPressed: () { + setState(() { + todos.add(controller.text); + controller.clear(); + }); + }, + child: const Icon(Icons.add), + ), + ), + ); + } +} +``` + +**Test Implementation (`test/todo_list_test.dart`):** +```dart +import 'package:flutter/material.dart'; +import 'package:flutter_test/flutter_test.dart'; +import 'package:my_app/todo_list.dart'; + +void main() { + testWidgets('Add and remove a todo item', (WidgetTester tester) async { + // 1. Build the widget + await tester.pumpWidget(const TodoList()); + + // 2. Verify initial state + expect(find.byType(ListTile), findsNothing); + + // 3. Enter text into the TextField + await tester.enterText(find.byType(TextField), 'Buy groceries'); + + // 4. Tap the add button + await tester.tap(find.byType(FloatingActionButton)); + + // 5. Rebuild the widget to reflect the new state + await tester.pump(); + + // 6. Verify the item was added + expect(find.text('Buy groceries'), findsOneWidget); + + // 7. Swipe the item to dismiss it + await tester.drag(find.byType(Dismissible), const Offset(500, 0)); + + // 8. Build the widget until the dismiss animation ends + await tester.pumpAndSettle(); + + // 9. Verify the item was removed + expect(find.text('Buy groceries'), findsNothing); + }); +} +``` diff --git a/.claude/skills/flutter-apply-architecture-best-practices/SKILL.md b/.claude/skills/flutter-apply-architecture-best-practices/SKILL.md new file mode 100644 index 00000000..791994b1 --- /dev/null +++ b/.claude/skills/flutter-apply-architecture-best-practices/SKILL.md @@ -0,0 +1,162 @@ +--- +name: flutter-apply-architecture-best-practices +description: Architects a Flutter application using the recommended layered approach (UI, Logic, Data). Use when structuring a new project or refactoring for scalability. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 20:11:20 GMT +--- +# Architecting Flutter Applications + +## Contents +- [Architectural Layers](#architectural-layers) +- [Project Structure](#project-structure) +- [Workflow: Implementing a New Feature](#workflow-implementing-a-new-feature) +- [Examples](#examples) + +## Architectural Layers + +Enforce strict Separation of Concerns by dividing the application into distinct layers. Never mix UI rendering with business logic or data fetching. + +### UI Layer (Presentation) +Implement the MVVM (Model-View-ViewModel) pattern to manage UI state and logic. +* **Views:** Write reusable, lean widgets. Restrict logic in Views to UI-specific operations (e.g., animations, layout constraints, simple routing). Pass all required data from the ViewModel. +* **ViewModels:** Manage UI state and handle user interactions. Extend `ChangeNotifier` (or use `Listenable`) to expose state. Expose immutable state snapshots to the View. Inject Repositories into ViewModels via the constructor. + +### Data Layer +Implement the Repository pattern to isolate data access logic and create a single source of truth. +* **Services:** Create stateless classes to wrap external APIs (HTTP clients, local databases, platform plugins). Return raw API models or `Result` wrappers. +* **Repositories:** Consume one or more Services. Transform raw API models into clean Domain Models. Handle caching, offline synchronization, and retry logic. Expose Domain Models to ViewModels. + +### Logic Layer (Domain - Optional) +* **Use Cases:** Implement this layer only if the application contains complex business logic that clutters the ViewModel, or if logic must be reused across multiple ViewModels. Extract this logic into dedicated Use Case (interactor) classes that sit between ViewModels and Repositories. + +## Project Structure + +Organize the codebase using a hybrid approach: group UI components by feature, and group Data/Domain components by type. + +```text +lib/ +├── data/ +│ ├── models/ # API models +│ ├── repositories/ # Repository implementations +│ └── services/ # API clients, local storage wrappers +├── domain/ +│ ├── models/ # Clean domain models +│ └── use_cases/ # Optional business logic classes +└── ui/ + ├── core/ # Shared widgets, themes, typography + └── features/ + └── [feature_name]/ + ├── view_models/ + └── views/ +``` + +## Workflow: Implementing a New Feature + +Follow this sequential workflow when adding a new feature to the application. Copy the checklist to track progress. + +### Task Progress +- [ ] **Step 1: Define Domain Models.** Create immutable data classes for the feature using `freezed` or `built_value`. +- [ ] **Step 2: Implement Services.** Create or update Service classes to handle external API communication. +- [ ] **Step 3: Implement Repositories.** Create the Repository to consume Services and return Domain Models. +- [ ] **Step 4: Apply Conditional Logic (Domain Layer).** + - *If the feature requires complex data transformation or cross-repository logic:* Create a Use Case class. + - *If the feature is a simple CRUD operation:* Skip to Step 5. +- [ ] **Step 5: Implement the ViewModel.** Create the ViewModel extending `ChangeNotifier`. Inject required Repositories/Use Cases. Expose immutable state and command methods. +- [ ] **Step 6: Implement the View.** Create the UI widget. Use `ListenableBuilder` or `AnimatedBuilder` to listen to ViewModel changes. +- [ ] **Step 7: Inject Dependencies.** Register the new Service, Repository, and ViewModel in the dependency injection container (e.g., `provider` or `get_it`). +- [ ] **Step 8: Run Validator.** Execute unit tests for the ViewModel and Repository. + - *Feedback Loop:* Run tests -> Review failures -> Fix logic -> Re-run until passing. + +## Examples + +### Data Layer: Service and Repository + +```dart +// 1. Service (Raw API interaction) +class ApiClient { + Future fetchUser(String id) async { + // HTTP GET implementation... + } +} + +// 2. Repository (Single source of truth, returns Domain Model) +class UserRepository { + UserRepository({required ApiClient apiClient}) : _apiClient = apiClient; + + final ApiClient _apiClient; + User? _cachedUser; + + Future getUser(String id) async { + if (_cachedUser != null) return _cachedUser!; + + final apiModel = await _apiClient.fetchUser(id); + _cachedUser = User(id: apiModel.id, name: apiModel.fullName); // Transform to Domain Model + return _cachedUser!; + } +} +``` + +### UI Layer: ViewModel and View + +```dart +// 3. ViewModel (State management and presentation logic) +class ProfileViewModel extends ChangeNotifier { + ProfileViewModel({required UserRepository userRepository}) + : _userRepository = userRepository; + + final UserRepository _userRepository; + + User? _user; + User? get user => _user; + + bool _isLoading = false; + bool get isLoading => _isLoading; + + Future loadProfile(String id) async { + _isLoading = true; + notifyListeners(); + + try { + _user = await _userRepository.getUser(id); + } finally { + _isLoading = false; + notifyListeners(); + } + } +} + +// 4. View (Dumb UI component) +class ProfileView extends StatelessWidget { + const ProfileView({super.key, required this.viewModel}); + + final ProfileViewModel viewModel; + + @override + Widget build(BuildContext context) { + return ListenableBuilder( + listenable: viewModel, + builder: (context, _) { + if (viewModel.isLoading) { + return const Center(child: CircularProgressIndicator()); + } + + final user = viewModel.user; + if (user == null) { + return const Center(child: Text('User not found')); + } + + return Column( + children: [ + Text(user.name), + ElevatedButton( + onPressed: () => viewModel.loadProfile(user.id), + child: const Text('Refresh'), + ), + ], + ); + }, + ); + } +} +``` diff --git a/.claude/skills/flutter-build-responsive-layout/SKILL.md b/.claude/skills/flutter-build-responsive-layout/SKILL.md new file mode 100644 index 00000000..b85bfd7e --- /dev/null +++ b/.claude/skills/flutter-build-responsive-layout/SKILL.md @@ -0,0 +1,139 @@ +--- +name: flutter-build-responsive-layout +description: Use `LayoutBuilder`, `MediaQuery`, or `Expanded/Flexible` to create a layout that adapts to different screen sizes. Use when you need the UI to look good on both mobile and tablet/desktop form factors. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 20:17:40 GMT +--- +# Implementing Adaptive Layouts + +## Contents +- [Space Measurement Guidelines](#space-measurement-guidelines) +- [Widget Sizing and Constraints](#widget-sizing-and-constraints) +- [Device and Orientation Behaviors](#device-and-orientation-behaviors) +- [Workflow: Constructing an Adaptive Layout](#workflow-constructing-an-adaptive-layout) +- [Workflow: Optimizing for Large Screens](#workflow-optimizing-for-large-screens) +- [Examples](#examples) + +## Space Measurement Guidelines +Determine the available space accurately to ensure layouts adapt to the app window, not just the physical device. + +* **Use `MediaQuery.sizeOf(context)`** to get the size of the entire app window. +* **Use `LayoutBuilder`** to make layout decisions based on the parent widget's allocated space. Evaluate `constraints.maxWidth` to determine the appropriate widget tree to return. +* **Do not use `MediaQuery.orientationOf` or `OrientationBuilder`** near the top of the widget tree to switch layouts. Device orientation does not accurately reflect the available app window space. +* **Do not check for hardware types** (e.g., "phone" vs. "tablet"). Flutter apps run in resizable windows, multi-window modes, and picture-in-picture. Base all layout decisions strictly on available window space. + +## Widget Sizing and Constraints +Understand and apply Flutter's core layout rule: **Constraints go down. Sizes go up. Parent sets position.** + +* **Distribute Space:** Use `Expanded` and `Flexible` within `Row`, `Column`, or `Flex` widgets. + * Use `Expanded` to force a child to fill all remaining available space (equivalent to `Flexible` with `fit: FlexFit.tight` and a `flex` factor of 1.0). + * Use `Flexible` to allow a child to size itself up to a specific limit while still expanding/contracting. Use the `flex` factor to define the ratio of space consumption among siblings. +* **Constrain Width:** Prevent widgets from consuming all horizontal space on large screens. Wrap widgets like `GridView` or `ListView` in a `ConstrainedBox` or `Container` and define a `maxWidth` in the `BoxConstraints`. +* **Lazy Rendering:** Always use `ListView.builder` or `GridView.builder` when rendering lists with an unknown or large number of items. + +## Device and Orientation Behaviors +Ensure the app behaves correctly across all device form factors and input methods. + +* **Do not lock screen orientation.** Locking orientation causes severe layout issues on foldable devices, often resulting in letterboxing (the app centered with black borders). Android large format tiers require both portrait and landscape support. +* **Fallback for Locked Orientation:** If business requirements strictly mandate a locked orientation, use the `Display API` to retrieve physical screen dimensions instead of `MediaQuery`. `MediaQuery` fails to receive the larger window size in compatibility modes. +* **Support Multiple Inputs:** Implement support for basic mice, trackpads, and keyboard shortcuts. Ensure touch targets are appropriately sized and keyboard navigation is accessible. + +## Workflow: Constructing an Adaptive Layout + +Follow this workflow to implement a layout that adapts to the available `BoxConstraints`. + +**Task Progress:** +- [ ] Identify the target widget that requires adaptive behavior. +- [ ] Wrap the widget tree in a `LayoutBuilder`. +- [ ] Extract the `constraints.maxWidth` from the builder callback. +- [ ] Define an adaptive breakpoint (e.g., `largeScreenMinWidth = 600`). +- [ ] **If `maxWidth > largeScreenMinWidth`:** Return a large-screen layout (e.g., a `Row` placing a navigation sidebar and content area side-by-side). +- [ ] **If `maxWidth <= largeScreenMinWidth`:** Return a small-screen layout (e.g., a `Column` or standard navigation-style approach). +- [ ] Run validator -> resize the application window -> review layout transitions -> fix overflow errors. + +## Workflow: Optimizing for Large Screens + +Follow this workflow to prevent UI elements from stretching unnaturally on large displays. + +**Task Progress:** +- [ ] Identify full-width components (e.g., `ListView`, text blocks, forms). +- [ ] **If optimizing a list:** Convert `ListView.builder` to `GridView.builder` using `SliverGridDelegateWithMaxCrossAxisExtent` to automatically adjust column counts based on window size. +- [ ] **If optimizing a form or text block:** Wrap the component in a `ConstrainedBox`. +- [ ] Apply `BoxConstraints(maxWidth: [optimal_width])` to the `ConstrainedBox`. +- [ ] Wrap the `ConstrainedBox` in a `Center` widget to keep the constrained content centered on large screens. +- [ ] Run validator -> test on desktop/tablet target -> review horizontal stretching -> adjust `maxWidth` or grid extents. + +## Examples + +### Adaptive Layout using LayoutBuilder +Demonstrates switching between a mobile and desktop layout based on available width. + +```dart +import 'package:flutter/material.dart'; + +const double largeScreenMinWidth = 600.0; + +class AdaptiveLayout extends StatelessWidget { + const AdaptiveLayout({super.key}); + + @override + Widget build(BuildContext context) { + return LayoutBuilder( + builder: (context, constraints) { + if (constraints.maxWidth > largeScreenMinWidth) { + return _buildLargeScreenLayout(); + } else { + return _buildSmallScreenLayout(); + } + }, + ); + } + + Widget _buildLargeScreenLayout() { + return Row( + children: [ + const SizedBox(width: 250, child: Placeholder(color: Colors.blue)), + const VerticalDivider(width: 1), + Expanded(child: const Placeholder(color: Colors.green)), + ], + ); + } + + Widget _buildSmallScreenLayout() { + return const Placeholder(color: Colors.green); + } +} +``` + +### Constraining Width on Large Screens +Demonstrates preventing a widget from consuming all horizontal space. + +```dart +import 'package:flutter/material.dart'; + +class ConstrainedContent extends StatelessWidget { + const ConstrainedContent({super.key}); + + @override + Widget build(BuildContext context) { + return Scaffold( + body: Center( + child: ConstrainedBox( + constraints: const BoxConstraints( + maxWidth: 800.0, // Maximum width for readability + ), + child: ListView.builder( + itemCount: 50, + itemBuilder: (context, index) { + return ListTile( + title: Text('Item $index'), + ); + }, + ), + ), + ), + ); + } +} +``` diff --git a/.claude/skills/flutter-fix-layout-issues/SKILL.md b/.claude/skills/flutter-fix-layout-issues/SKILL.md new file mode 100644 index 00000000..3804a3c1 --- /dev/null +++ b/.claude/skills/flutter-fix-layout-issues/SKILL.md @@ -0,0 +1,130 @@ +--- +name: flutter-fix-layout-issues +description: Fixes Flutter layout errors (overflows, unbounded constraints) using Dart and Flutter MCP tools. Use when addressing "RenderFlex overflowed", "Vertical viewport was given unbounded height", or similar layout issues. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 19:45:59 GMT +--- +# Resolving Flutter Layout Errors + +## Contents +- [Constraint Violation Diagnostics](#constraint-violation-diagnostics) +- [Layout Error Resolution Workflow](#layout-error-resolution-workflow) +- [Examples](#examples) + +## Constraint Violation Diagnostics + +Flutter layout operates on a strict rule: **Constraints go down. Sizes go up. Parent sets position.** Layout errors occur when this negotiation fails, typically due to unbounded constraints or unconstrained children. + +Diagnose layout failures using the following error signatures: + +* **"Vertical viewport was given unbounded height"**: Triggered when a scrollable widget (`ListView`, `GridView`) is placed inside an unconstrained vertical parent (`Column`). The parent provides infinite height, and the child attempts to expand infinitely. +* **"An InputDecorator...cannot have an unbounded width"**: Triggered when a `TextField` or `TextFormField` is placed inside an unconstrained horizontal parent (`Row`). The text field attempts to determine its width based on infinite available space. +* **"RenderFlex overflowed"**: Triggered when a child of a `Row` or `Column` requests a size larger than the parent's allocated constraints. Visually indicated by yellow and black warning stripes. +* **"Incorrect use of ParentData widget"**: Triggered when a `ParentDataWidget` is not a direct descendant of its required ancestor. (e.g., `Expanded` outside a `Flex`, `Positioned` outside a `Stack`). +* **"RenderBox was not laid out"**: A cascading side-effect error. Ignore this and look further up the stack trace for the primary constraint violation (usually an unbounded height/width error). + +## Layout Error Resolution Workflow + +Copy and use this checklist to systematically resolve layout constraint violations. + +### Task Progress +- [ ] Run the application in debug mode to capture the exact layout exception in the console. +- [ ] Identify the primary error message (ignore cascading "RenderBox was not laid out" errors). +- [ ] Apply the conditional fix based on the specific error type: + - **If "Vertical viewport was given unbounded height"**: Wrap the scrollable child (`ListView`, `GridView`) in an `Expanded` widget to consume remaining space, or wrap it in a `SizedBox` to provide an absolute height constraint. + - **If "An InputDecorator...cannot have an unbounded width"**: Wrap the `TextField` or `TextFormField` in an `Expanded` or `Flexible` widget. + - **If "RenderFlex overflowed"**: Constrain the overflowing child by wrapping it in an `Expanded` widget (to force it to fit) or a `Flexible` widget (to allow it to be smaller than the allocated space). + - **If "Incorrect use of ParentData widget"**: Move the `ParentDataWidget` to be a direct child of its required parent. Ensure `Expanded`/`Flexible` are direct children of `Row`/`Column`/`Flex`. Ensure `Positioned` is a direct child of `Stack`. +- [ ] Execute Flutter hot reload. +- [ ] Run validator -> review errors -> fix: Inspect the UI to verify the red/grey error screen or yellow/black overflow stripes are resolved. If new layout errors appear, repeat the workflow. + +## Examples + +### Fixing Unbounded Height (ListView in Column) + +**Input (Error State):** +```dart +// Throws "Vertical viewport was given unbounded height" +Column( + children: [ + const Text('Header'), + ListView( + children: const [ + ListTile(title: Text('Item 1')), + ListTile(title: Text('Item 2')), + ], + ), + ], +) +``` + +**Output (Resolved State):** +```dart +// Wrap ListView in Expanded to constrain its height to the remaining Column space +Column( + children: [ + const Text('Header'), + Expanded( + child: ListView( + children: const [ + ListTile(title: Text('Item 1')), + ListTile(title: Text('Item 2')), + ], + ), + ), + ], +) +``` + +### Fixing Unbounded Width (TextField in Row) + +**Input (Error State):** +```dart +// Throws "An InputDecorator...cannot have an unbounded width" +Row( + children: [ + const Icon(Icons.search), + TextField(), + ], +) +``` + +**Output (Resolved State):** +```dart +// Wrap TextField in Expanded to constrain its width to the remaining Row space +Row( + children: [ + const Icon(Icons.search), + Expanded( + child: TextField(), + ), + ], +) +``` + +### Fixing RenderFlex Overflow + +**Input (Error State):** +```dart +// Throws "A RenderFlex overflowed by X pixels on the right" +Row( + children: [ + const Icon(Icons.info), + const Text('This is a very long text string that will definitely overflow the available screen width and cause a RenderFlex error.'), + ], +) +``` + +**Output (Resolved State):** +```dart +// Wrap the Text widget in Expanded to force it to wrap within the available constraints +Row( + children: [ + const Icon(Icons.info), + Expanded( + child: const Text('This is a very long text string that will definitely overflow the available screen width and cause a RenderFlex error.'), + ), + ], +) +``` diff --git a/.claude/skills/flutter-implement-json-serialization/SKILL.md b/.claude/skills/flutter-implement-json-serialization/SKILL.md new file mode 100644 index 00000000..14009f4e --- /dev/null +++ b/.claude/skills/flutter-implement-json-serialization/SKILL.md @@ -0,0 +1,153 @@ +--- +name: flutter-implement-json-serialization +description: Create model classes with `fromJson` and `toJson` methods using `dart:convert`. Use when manually mapping JSON keys to class properties for simple data structures. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 21:44:50 GMT +--- +# Serializing JSON Manually in Flutter + +## Contents +- [Core Guidelines](#core-guidelines) +- [Workflow: Implementing a Serializable Model](#workflow-implementing-a-serializable-model) +- [Workflow: Fetching and Parsing JSON](#workflow-fetching-and-parsing-json) +- [Examples](#examples) + +## Core Guidelines + +- **Import `dart:convert`**: Utilize Flutter's built-in `dart:convert` library for manual JSON encoding (`jsonEncode`) and decoding (`jsonDecode`). +- **Enforce Type Safety**: Always cast the `dynamic` result of `jsonDecode()` to the expected type, typically `Map` for objects or `List` for arrays. +- **Encapsulate Serialization Logic**: Define plain model classes containing properties corresponding to the JSON structure. Implement a `fromJson` factory constructor and a `toJson` method within the model. +- **Handle Background Parsing**: If parsing large JSON documents (execution time > 16ms), offload the parsing logic to a separate isolate using Flutter's `compute()` function to prevent UI jank. +- **Throw Exceptions on Failure**: When handling HTTP responses, throw an exception if the status code is not successful (e.g., not 200 OK or 201 Created). Do not return `null`. + +## Workflow: Implementing a Serializable Model + +Use this checklist to implement manual JSON serialization for a data model. + +**Task Progress:** +- [ ] Define the plain model class with `final` properties. +- [ ] Implement the `factory Model.fromJson(Map json)` constructor. +- [ ] Implement the `Map toJson()` method. +- [ ] Write unit tests for both serialization methods. +- [ ] Run validator -> review type mismatch errors -> fix casting logic. + +1. **Define the Model**: Create a class with properties matching the JSON keys. +2. **Implement `fromJson`**: Extract values from the `Map` and cast them to the appropriate Dart types. Use pattern matching or explicit casting. +3. **Implement `toJson`**: Return a `Map` mapping the class properties back to their JSON string keys. +4. **Validate**: Execute unit tests to ensure type safety, autocompletion, and compile-time exception handling function correctly. + +## Workflow: Fetching and Parsing JSON + +Use this conditional workflow when retrieving and parsing JSON from a network request. + +**Task Progress:** +- [ ] Execute the HTTP request. +- [ ] Validate the response status code. +- [ ] Determine parsing strategy (Synchronous vs. Isolate). +- [ ] Decode and map the JSON to the model. + +1. **Execute Request**: Use the `http` package to perform the network call. +2. **Validate Response**: + - If `response.statusCode == 200` (or 201 for POST), proceed to parsing. + - If the status code indicates failure, throw an `Exception`. +3. **Determine Parsing Strategy**: + - If parsing a **small payload** (e.g., a single object), parse synchronously on the main thread. + - If parsing a **large payload** (e.g., an array of thousands of objects), use `compute(parseFunction, response.body)` to parse in a background isolate. +4. **Decode and Map**: Pass the decoded JSON to your model's `fromJson` constructor. + +## Examples + +### High-Fidelity Model Implementation + +```dart +import 'dart:convert'; + +class User { + final int id; + final String name; + final String email; + + const User({ + required this.id, + required this.name, + required this.email, + }); + + // Factory constructor for deserialization + factory User.fromJson(Map json) { + return switch (json) { + { + 'id': int id, + 'name': String name, + 'email': String email, + } => + User( + id: id, + name: name, + email: email, + ), + _ => throw const FormatException('Failed to load User.'), + }; + } + + // Method for serialization + Map toJson() { + return { + 'id': id, + 'name': name, + 'email': email, + }; + } +} +``` + +### Synchronous Parsing (Small Payload) + +```dart +import 'dart:convert'; +import 'package:http/http.dart' as http; + +Future fetchUser(http.Client client, int userId) async { + final response = await client.get( + Uri.parse('https://api.example.com/users/$userId'), + headers: {'Accept': 'application/json'}, + ); + + if (response.statusCode == 200) { + // Decode returns dynamic, cast to Map + final Map jsonMap = jsonDecode(response.body) as Map; + return User.fromJson(jsonMap); + } else { + throw Exception('Failed to load user'); + } +} +``` + +### Background Parsing (Large Payload) + +```dart +import 'dart:convert'; +import 'package:flutter/foundation.dart'; +import 'package:http/http.dart' as http; + +// Top-level function required for compute() +List parseUsers(String responseBody) { + final parsed = (jsonDecode(responseBody) as List).cast>(); + return parsed.map((json) => User.fromJson(json)).toList(); +} + +Future> fetchUsers(http.Client client) async { + final response = await client.get( + Uri.parse('https://api.example.com/users'), + headers: {'Accept': 'application/json'}, + ); + + if (response.statusCode == 200) { + // Offload expensive parsing to a background isolate + return compute(parseUsers, response.body); + } else { + throw Exception('Failed to load users'); + } +} +``` diff --git a/.claude/skills/flutter-setup-declarative-routing/SKILL.md b/.claude/skills/flutter-setup-declarative-routing/SKILL.md new file mode 100644 index 00000000..27203118 --- /dev/null +++ b/.claude/skills/flutter-setup-declarative-routing/SKILL.md @@ -0,0 +1,255 @@ +--- +name: flutter-setup-declarative-routing +description: Configure `MaterialApp.router` using a package like `go_router` for advanced URL-based navigation. Use when developing web applications or mobile apps that require specific deep linking and browser history support. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 21:08:03 GMT +--- +# Implementing Routing and Deep Linking + +## Contents +- [Core Concepts](#core-concepts) +- [Workflow: Initializing the Application and Router](#workflow-initializing-the-application-and-router) +- [Workflow: Configuring Platform Deep Linking](#workflow-configuring-platform-deep-linking) +- [Workflow: Implementing Nested Navigation](#workflow-implementing-nested-navigation) +- [Examples](#examples) + +## Core Concepts + +Use the `go_router` package for declarative routing in Flutter. It provides a robust API for complex routing scenarios, deep linking, and nested navigation. + +- **GoRouter**: The central configuration object defining the application's route tree. +- **GoRoute**: A standard route mapping a URL path to a Flutter screen. +- **ShellRoute / StatefulShellRoute**: Wraps child routes in a persistent UI shell (e.g., a `BottomNavigationBar`). `StatefulShellRoute` maintains the state of parallel navigation branches. +- **Path URL Strategy**: Removes the default `#` fragment from web URLs, essential for clean deep linking across platforms. + +## Workflow: Initializing the Application and Router + +Follow this workflow to bootstrap a new Flutter application with `go_router` and configure the root routing mechanism. + +### Task Progress +- [ ] Create the Flutter application. +- [ ] Add the `go_router` dependency. +- [ ] Configure the URL strategy for web/deep linking. +- [ ] Implement the `GoRouter` configuration. +- [ ] Bind the router to `MaterialApp.router`. + +### 1. Scaffold the Application +Run the following commands to create the app and add the required routing package: +```bash +flutter create +cd +flutter pub add go_router +``` + +### 2. Configure the Router +Define a top-level `GoRouter` instance. Handle authentication or state-based routing using the `redirect` parameter. + +```dart +import 'package:flutter/material.dart'; +import 'package:go_router/go_router.dart'; +import 'package:flutter_web_plugins/url_strategy.dart'; + +void main() { + // Use path URL strategy to remove the '#' from web URLs + usePathUrlStrategy(); + runApp(const MyApp()); +} + +final GoRouter _router = GoRouter( + initialLocation: '/', + routes: [ + GoRoute( + path: '/', + builder: (context, state) => const HomeScreen(), + routes: [ + GoRoute( + path: 'details/:id', + builder: (context, state) => DetailsScreen(id: state.pathParameters['id']!), + ), + ], + ), + ], + errorBuilder: (context, state) => ErrorScreen(error: state.error), +); + +class MyApp extends StatelessWidget { + const MyApp({super.key}); + + @override + Widget build(BuildContext context) { + return MaterialApp.router( + routerConfig: _router, + title: 'Routing App', + ); + } +} +``` + +## Workflow: Configuring Platform Deep Linking + +Configure the native platforms to intercept specific URLs and route them into the Flutter application. + +### Task Progress +- [ ] Determine target platforms (iOS, Android, or both). +- [ ] Apply conditional configuration for Android (Manifest + Asset Links). +- [ ] Apply conditional configuration for iOS (Plist + Entitlements + AASA). +- [ ] Run validator -> review errors -> fix. + +### If configuring for Android: +1. **Modify `AndroidManifest.xml`**: Add the intent filter inside the `` tag for `.MainActivity`. +```xml + + + + + + + +``` +2. **Host `assetlinks.json`**: Serve the following JSON at `https://yourdomain.com/.well-known/assetlinks.json`. +```json +[{ + "relation": ["delegate_permission/common.handle_all_urls"], + "target": { + "namespace": "android_app", + "package_name": "com.yourcompany.yourapp", + "sha256_cert_fingerprints": ["YOUR_SHA256_FINGERPRINT"] + } +}] +``` + +### If configuring for iOS: +1. **Modify `Info.plist`**: Opt-in to Flutter's default deep link handler. +*Note: If using a third-party deep linking plugin (e.g., `app_links`), set this to `NO` to prevent conflicts.* +```xml +FlutterDeepLinkingEnabled + +``` +2. **Modify `Runner.entitlements`**: Add the associated domain. +```xml +com.apple.developer.associated-domains + + applinks:yourdomain.com + +``` +3. **Host `apple-app-site-association`**: Serve the following JSON (without a `.json` extension) at `https://yourdomain.com/.well-known/apple-app-site-association`. +```json +{ + "applinks": { + "apps": [], + "details": [{ + "appIDs": ["TEAM_ID.com.yourcompany.yourapp"], + "paths": ["*"], + "components": [{"/": "/*"}] + }] + } +} +``` + +### Validation Loop +Run validator -> review errors -> fix. +- **Android**: Test using ADB. + ```bash + adb shell 'am start -a android.intent.action.VIEW -c android.intent.category.BROWSABLE -d "https://yourdomain.com/details/123"' com.yourcompany.yourapp + ``` +- **iOS**: Test using `xcrun` on a booted simulator. + ```bash + xcrun simctl openurl booted https://yourdomain.com/details/123 + ``` + +## Workflow: Implementing Nested Navigation + +Use `StatefulShellRoute` to implement persistent UI shells (like a bottom navigation bar) that maintain the state of their child routes. + +### Task Progress +- [ ] Define `StatefulShellRoute.indexedStack` in the `GoRouter` configuration. +- [ ] Create `StatefulShellBranch` instances for each navigation tab. +- [ ] Implement the shell widget using `StatefulNavigationShell`. + +```dart +final GoRouter _router = GoRouter( + initialLocation: '/home', + routes: [ + StatefulShellRoute.indexedStack( + builder: (context, state, navigationShell) { + return ScaffoldWithNavBar(navigationShell: navigationShell); + }, + branches: [ + StatefulShellBranch( + routes: [ + GoRoute( + path: '/home', + builder: (context, state) => const HomeScreen(), + ), + ], + ), + StatefulShellBranch( + routes: [ + GoRoute( + path: '/settings', + builder: (context, state) => const SettingsScreen(), + ), + ], + ), + ], + ), + ], +); +``` + +## Examples + +### High-Fidelity Shell Widget Implementation +Implement the UI shell that consumes the `StatefulNavigationShell` to handle branch switching. + +```dart +class ScaffoldWithNavBar extends StatelessWidget { + const ScaffoldWithNavBar({ + required this.navigationShell, + super.key, + }); + + final StatefulNavigationShell navigationShell; + + void _goBranch(int index) { + navigationShell.goBranch( + index, + // Support navigating to the initial location when tapping the active tab. + initialLocation: index == navigationShell.currentIndex, + ); + } + + @override + Widget build(BuildContext context) { + return Scaffold( + body: navigationShell, + bottomNavigationBar: NavigationBar( + selectedIndex: navigationShell.currentIndex, + onDestinationSelected: _goBranch, + destinations: const [ + NavigationDestination(icon: Icon(Icons.home), label: 'Home'), + NavigationDestination(icon: Icon(Icons.settings), label: 'Settings'), + ], + ), + ); + } +} +``` + +### Programmatic Navigation +Use the `context.go()` and `context.push()` extension methods provided by `go_router`. + +```dart +// Replaces the current route stack with the target route (Declarative) +context.go('/details/123'); + +// Pushes the target route onto the existing stack (Imperative) +context.push('/details/123'); + +// Navigates using a named route and path parameters +context.goNamed('details', pathParameters: {'id': '123'}); + +// Pops the current route +context.pop(); +``` diff --git a/.claude/skills/flutter-setup-localization/SKILL.md b/.claude/skills/flutter-setup-localization/SKILL.md new file mode 100644 index 00000000..d3dd4596 --- /dev/null +++ b/.claude/skills/flutter-setup-localization/SKILL.md @@ -0,0 +1,210 @@ +--- +name: flutter-setup-localization +description: Add `flutter_localizations` and `intl` dependencies, enable "generate true" in `pubspec.yaml`, and create an `l10n.yaml` configuration file. Use when initializing localization support for a new Flutter project. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 21:27:35 GMT +--- +# Internationalizing Flutter Applications + +## Contents +- [Core Concepts](#core-concepts) +- [Setup Workflow](#setup-workflow) +- [Implementation Workflow](#implementation-workflow) +- [Advanced Formatting](#advanced-formatting) +- [Examples](#examples) + +## Core Concepts +Flutter handles internationalization (i18n) and localization (l10n) via the `flutter_localizations` and `intl` packages. The standard approach uses App Resource Bundle (`.arb`) files to define localized strings, which are then compiled into a generated `AppLocalizations` class for type-safe access within the widget tree. + +## Setup Workflow + +Copy and track this checklist when initializing internationalization in a Flutter project: + +- [ ] **Task Progress** + - [ ] 1. Add dependencies to `pubspec.yaml`. + - [ ] 2. Enable the `generate` flag. + - [ ] 3. Create the `l10n.yaml` configuration file. + - [ ] 4. Configure `MaterialApp` or `CupertinoApp`. + +### 1. Add Dependencies +Add the required localization packages to the project. Execute the following commands in the terminal: +```bash +flutter pub add flutter_localizations --sdk=flutter +flutter pub add intl:any +``` + +Verify your `pubspec.yaml` includes the following under `dependencies`: +```yaml +dependencies: + flutter: + sdk: flutter + flutter_localizations: + sdk: flutter + intl: any +``` + +### 2. Enable Code Generation +Open `pubspec.yaml` and enable the `generate` flag within the `flutter` section to automate localization tasks: +```yaml +flutter: + generate: true +``` + +### 3. Create Configuration File +Create a new file named `l10n.yaml` in the root directory of the Flutter project. Define the input directory, template file, and output file: +```yaml +arb-dir: lib/l10n +template-arb-file: app_en.arb +output-localization-file: app_localizations.dart +synthetic-package: true +``` + +### 4. Configure the App Entry Point +Import the generated localizations and the `flutter_localizations` library in your `main.dart`. Inject the delegates and supported locales into your `MaterialApp` or `CupertinoApp`. + +```dart +import 'package:flutter_localizations/flutter_localizations.dart'; +import 'package:flutter_gen/gen_l10n/app_localizations.dart'; // Adjust path if synthetic-package is false + +// ... inside build method +return MaterialApp( + localizationsDelegates: const [ + AppLocalizations.delegate, + GlobalMaterialLocalizations.delegate, + GlobalWidgetsLocalizations.delegate, + GlobalCupertinoLocalizations.delegate, + ], + supportedLocales: const [ + Locale('en'), // English + Locale('es'), // Spanish + ], + home: const MyHomePage(), +); +``` + +## Implementation Workflow + +Follow this workflow when adding or modifying localized content. + +### 1. Define ARB Files +* **If creating NEW content:** Add the base string to the template file (`lib/l10n/app_en.arb`). Include a description for context. +* **If EDITING existing content:** Locate the key in all supported `.arb` files and update the values. + +```json +{ + "helloWorld": "Hello World!", + "@helloWorld": { + "description": "The conventional newborn programmer greeting" + } +} +``` + +Create corresponding files for other locales (e.g., `app_es.arb`): +```json +{ + "helloWorld": "¡Hola Mundo!" +} +``` + +### 2. Generate Localization Classes +Run the following command to trigger code generation: +```bash +flutter pub get +``` +*Feedback Loop:* Run validator -> review terminal output for ARB syntax errors -> fix missing commas or mismatched placeholders -> re-run `flutter pub get`. + +### 3. Consume Localized Strings +Access the localized strings in your widget tree using `AppLocalizations.of(context)`. Ensure the widget calling this is a descendant of `MaterialApp`. + +```dart +Text(AppLocalizations.of(context)!.helloWorld) +``` + +## Advanced Formatting + +Use placeholders for dynamic data, plurals, and conditional selects. + +### Placeholders +Define parameters within curly braces and specify their type in the metadata object. +```json +"hello": "Hello {userName}", +"@hello": { + "description": "A message with a single parameter", + "placeholders": { + "userName": { + "type": "String", + "example": "Bob" + } + } +} +``` + +### Plurals +Use the `plural` syntax to handle quantity-based string variations. The `other` case is mandatory. +```json +"nWombats": "{count, plural, =0{no wombats} =1{1 wombat} other{{count} wombats}}", +"@nWombats": { + "description": "A plural message", + "placeholders": { + "count": { + "type": "num", + "format": "compact" + } + } +} +``` + +### Selects +Use the `select` syntax for conditional strings, such as gendered text. +```json +"pronoun": "{gender, select, male{he} female{she} other{they}}", +"@pronoun": { + "description": "A gendered message", + "placeholders": { + "gender": { + "type": "String" + } + } +} +``` + +## Examples + +### Complete `l10n.yaml` +```yaml +arb-dir: lib/l10n +template-arb-file: app_en.arb +output-localization-file: app_localizations.dart +synthetic-package: true +use-escaping: true +``` + +### Complete Widget Implementation +```dart +import 'package:flutter/material.dart'; +import 'package:flutter_gen/gen_l10n/app_localizations.dart'; + +class GreetingWidget extends StatelessWidget { + final String userName; + final int notificationCount; + + const GreetingWidget({ + super.key, + required this.userName, + required this.notificationCount, + }); + + @override + Widget build(BuildContext context) { + final l10n = AppLocalizations.of(context)!; + + return Column( + children: [ + Text(l10n.hello(userName)), + Text(l10n.nWombats(notificationCount)), + ], + ); + } +} +``` diff --git a/.claude/skills/flutter-use-http-package/SKILL.md b/.claude/skills/flutter-use-http-package/SKILL.md new file mode 100644 index 00000000..bb60468d --- /dev/null +++ b/.claude/skills/flutter-use-http-package/SKILL.md @@ -0,0 +1,174 @@ +--- +name: flutter-use-http-package +description: Use the `http` package to execute GET, POST, PUT, or DELETE requests. Use when you need to fetch from or send data to a REST API. +metadata: + model: models/gemini-3.1-pro-preview + last_modified: Tue, 21 Apr 2026 21:36:42 GMT +--- +# Implementing Flutter Networking + +## Contents +- [Configuration & Permissions](#configuration--permissions) +- [Request Execution & Response Handling](#request-execution--response-handling) +- [Background Parsing](#background-parsing) +- [Workflow: Executing Network Operations](#workflow-executing-network-operations) +- [Examples](#examples) + +## Configuration & Permissions + +Configure the environment and platform-specific permissions required for network access. + +1. Add the `http` package dependency via the terminal: + ```bash + flutter pub add http + ``` +2. Import the package in your Dart files: + ```dart + import 'package:http/http.dart' as http; + ``` +3. Configure Android permissions by adding the Internet permission to `android/app/src/main/AndroidManifest.xml`: + ```xml + + ``` +4. Configure macOS entitlements by adding the network client key to both `macos/Runner/DebugProfile.entitlements` and `macos/Runner/Release.entitlements`: + ```xml + com.apple.security.network.client + + ``` + +## Request Execution & Response Handling + +Execute HTTP operations and map responses to strongly typed Dart objects. + +* **URIs:** Always parse URL strings using `Uri.parse('your_url')`. +* **Headers:** Inject authorization and content-type headers via the `headers` parameter map. Use `HttpHeaders.authorizationHeader` for auth tokens. +* **Payloads:** For POST and PUT requests, encode the body using `jsonEncode()` from `dart:convert`. +* **Status Validation:** Evaluate `response.statusCode`. Treat `200 OK` (GET/PUT/DELETE) and `201 CREATED` (POST) as success. +* **Error Handling:** Throw explicit exceptions for non-success status codes. Never return `null` on failure, as this prevents `FutureBuilder` from triggering its error state and causes infinite loading indicators. +* **Deserialization:** Parse the raw string using `jsonDecode(response.body)` and map it to a custom Dart object using a factory constructor (e.g., `fromJson`). + +## Background Parsing + +Offload expensive JSON parsing to a separate Isolate to prevent UI jank (frame drops). + +* Import `package:flutter/foundation.dart`. +* Use the `compute()` function to run the parsing logic in a background isolate. +* Ensure the parsing function passed to `compute()` is a top-level function or a static method, as closures or instance methods cannot be passed across isolates. + +## Workflow: Executing Network Operations + +Use the following checklist to implement and validate network operations. + +**Task Progress:** +- [ ] 1. Define the strongly typed Dart model with a `fromJson` factory constructor. +- [ ] 2. Implement the network request method returning a `Future`. +- [ ] 3. Apply conditional logic based on the operation type: + - **If fetching data (GET):** Append query parameters to the URI. + - **If mutating data (POST/PUT):** Set `'Content-Type': 'application/json; charset=UTF-8'` and attach the `jsonEncode` body. + - **If deleting data (DELETE):** Return an empty model instance on success (`200 OK`). +- [ ] 4. Validate the `statusCode` and throw an `Exception` on failure. +- [ ] 5. Integrate the `Future` into the UI using `FutureBuilder`. +- [ ] 6. Handle `snapshot.hasData`, `snapshot.hasError`, and default to a `CircularProgressIndicator`. +- [ ] 7. **Feedback Loop:** Run the app -> trigger the network request -> review console for unhandled exceptions -> fix parsing or permission errors. + +## Examples + +### High-Fidelity Implementation: Fetching and Parsing in the Background + +```dart +import 'dart:async'; +import 'dart:convert'; +import 'dart:io'; +import 'package:flutter/foundation.dart'; +import 'package:flutter/material.dart'; +import 'package:http/http.dart' as http; + +// 1. Top-level parsing function for Isolate +List parsePhotos(String responseBody) { + final parsed = (jsonDecode(responseBody) as List) + .cast>(); + return parsed.map(Photo.fromJson).toList(); +} + +// 2. Network execution with background parsing +Future> fetchPhotos() async { + final response = await http.get( + Uri.parse('https://jsonplaceholder.typicode.com/photos'), + headers: { + HttpHeaders.authorizationHeader: 'Bearer your_token_here', + HttpHeaders.acceptHeader: 'application/json', + }, + ); + + if (response.statusCode == 200) { + // Offload heavy parsing to a background isolate + return compute(parsePhotos, response.body); + } else { + throw Exception('Failed to load photos. Status: ${response.statusCode}'); + } +} + +// 3. Strongly typed model +class Photo { + final int id; + final String title; + final String thumbnailUrl; + + const Photo({ + required this.id, + required this.title, + required this.thumbnailUrl, + }); + + factory Photo.fromJson(Map json) { + return Photo( + id: json['id'] as int, + title: json['title'] as String, + thumbnailUrl: json['thumbnailUrl'] as String, + ); + } +} + +// 4. UI Integration +class PhotoGallery extends StatefulWidget { + const PhotoGallery({super.key}); + + @override + State createState() => _PhotoGalleryState(); +} + +class _PhotoGalleryState extends State { + late Future> _futurePhotos; + + @override + void initState() { + super.initState(); + // Initialize Future once to prevent re-fetching on rebuilds + _futurePhotos = fetchPhotos(); + } + + @override + Widget build(BuildContext context) { + return FutureBuilder>( + future: _futurePhotos, + builder: (context, snapshot) { + if (snapshot.hasData) { + final photos = snapshot.data!; + return ListView.builder( + itemCount: photos.length, + itemBuilder: (context, index) => ListTile( + leading: Image.network(photos[index].thumbnailUrl), + title: Text(photos[index].title), + ), + ); + } else if (snapshot.hasError) { + return Center(child: Text('Error: ${snapshot.error}')); + } + + // Default loading state + return const Center(child: CircularProgressIndicator()); + }, + ); + } +} +``` diff --git a/.claude/skills/mlflow-evaluation/SKILL.md b/.claude/skills/mlflow-evaluation/SKILL.md deleted file mode 100644 index 322f6aaa..00000000 --- a/.claude/skills/mlflow-evaluation/SKILL.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -name: mlflow-evaluation -description: "MLflow 3 GenAI evaluation for agent development. Use when (1) writing mlflow.genai.evaluate() code, (2) creating @scorer functions, (3) building evaluation datasets from traces, (4) using built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), (5) analyzing traces for latency/errors/architecture, (6) optimizing agent context/prompts/token usage, (7) debugging evaluation failures. Covers the full eval workflow: trace analysis -> dataset building -> scorer creation -> evaluation execution." ---- - -# MLflow 3 GenAI Evaluation - -## Before Writing Any Code - -1. **Read GOTCHAS.md** - 15+ common mistakes that cause failures -2. **Read CRITICAL-interfaces.md** - Exact API signatures and data schemas - -## End-to-End Workflows - -Follow these workflows based on your goal. Each step indicates which reference files to read. - -### Workflow 1: First-Time Evaluation Setup - -For users new to MLflow GenAI evaluation or setting up evaluation for a new agent. - -| Step | Action | Reference Files | -|------|--------|-----------------| -| 1 | Understand what to evaluate | `user-journeys.md` (Journey 0: Strategy) | -| 2 | Learn API patterns | `GOTCHAS.md` + `CRITICAL-interfaces.md` | -| 3 | Build initial dataset | `patterns-datasets.md` (Patterns 1-4) | -| 4 | Choose/create scorers | `patterns-scorers.md` + `CRITICAL-interfaces.md` (built-in list) | -| 5 | Run evaluation | `patterns-evaluation.md` (Patterns 1-3) | - -### Workflow 2: Production Trace -> Evaluation Dataset - -For building evaluation datasets from production traces. - -| Step | Action | Reference Files | -|------|--------|-----------------| -| 1 | Search and filter traces | `patterns-trace-analysis.md` (MCP tools section) | -| 2 | Analyze trace quality | `patterns-trace-analysis.md` (Patterns 1-7) | -| 3 | Tag traces for inclusion | `patterns-datasets.md` (Patterns 16-17) | -| 4 | Build dataset from traces | `patterns-datasets.md` (Patterns 6-7) | -| 5 | Add expectations/ground truth | `patterns-datasets.md` (Pattern 2) | - -### Workflow 3: Performance Optimization - -For debugging slow or expensive agent execution. - -| Step | Action | Reference Files | -|------|--------|-----------------| -| 1 | Profile latency by span | `patterns-trace-analysis.md` (Patterns 4-6) | -| 2 | Analyze token usage | `patterns-trace-analysis.md` (Pattern 9) | -| 3 | Detect context issues | `patterns-context-optimization.md` (Section 5) | -| 4 | Apply optimizations | `patterns-context-optimization.md` (Sections 1-4, 6) | -| 5 | Re-evaluate to measure impact | `patterns-evaluation.md` (Pattern 6-7) | - -### Workflow 4: Regression Detection - -For comparing agent versions and finding regressions. - -| Step | Action | Reference Files | -|------|--------|-----------------| -| 1 | Establish baseline | `patterns-evaluation.md` (Pattern 4: named runs) | -| 2 | Run current version | `patterns-evaluation.md` (Pattern 1) | -| 3 | Compare metrics | `patterns-evaluation.md` (Patterns 6-7) | -| 4 | Analyze failing traces | `patterns-trace-analysis.md` (Pattern 7) | -| 5 | Debug specific failures | `patterns-trace-analysis.md` (Patterns 8-9) | - -### Workflow 5: Custom Scorer Development - -For creating project-specific evaluation metrics. - -| Step | Action | Reference Files | -|------|--------|-----------------| -| 1 | Understand scorer interface | `CRITICAL-interfaces.md` (Scorer section) | -| 2 | Choose scorer pattern | `patterns-scorers.md` (Patterns 4-11) | -| 3 | For multi-agent scorers | `patterns-scorers.md` (Patterns 13-16) | -| 4 | Test with evaluation | `patterns-evaluation.md` (Pattern 1) | - -## Reference Files Quick Lookup - -| Reference | Purpose | When to Read | -|-----------|---------|--------------| -| `GOTCHAS.md` | Common mistakes | **Always read first** before writing code | -| `CRITICAL-interfaces.md` | API signatures, schemas | When writing any evaluation code | -| `patterns-evaluation.md` | Running evals, comparing | When executing evaluations | -| `patterns-scorers.md` | Custom scorer creation | When built-in scorers aren't enough | -| `patterns-datasets.md` | Dataset building | When preparing evaluation data | -| `patterns-trace-analysis.md` | Trace debugging | When analyzing agent behavior | -| `patterns-context-optimization.md` | Token/latency fixes | When agent is slow or expensive | -| `user-journeys.md` | High-level workflows | When starting a new evaluation project | - -## Critical API Facts - -- **Use:** `mlflow.genai.evaluate()` (NOT `mlflow.evaluate()`) -- **Data format:** `{"inputs": {"query": "..."}}` (nested structure required) -- **predict_fn:** Receives `**unpacked kwargs` (not a dict) - -See `GOTCHAS.md` for complete list. diff --git a/.claude/skills/mlflow-evaluation/references/user-journeys.md b/.claude/skills/mlflow-evaluation/references/user-journeys.md deleted file mode 100644 index 01cb4ccd..00000000 --- a/.claude/skills/mlflow-evaluation/references/user-journeys.md +++ /dev/null @@ -1,332 +0,0 @@ -# User Journey Guides - -Step-by-step workflows for common evaluation scenarios. - ---- - -## Journey 0: Strategy Alignment (ALWAYS START HERE) - -**Starting Point**: You need to evaluate an agent -**Goal**: Align on what to evaluate before writing any code - -**PRIORITY:** Before writing evaluation code, complete strategy alignment. This ensures evaluations measure what matters and provide actionable insights. - -### Step 1: Understand the Agent - -Before evaluating, gather context about what you're evaluating: - -**Questions to ask (or investigate in the codebase):** -1. **What does this agent do?** (data analysis, RAG, multi-turn chat, task automation) -2. **What tools does it use?** (UC functions, vector search, external APIs) -3. **What is the input/output format?** (messages format, structured output) -4. **What is the current state?** (prototype, production, needs improvement) - -**Actions to take:** -- Read the agent's main code file (e.g., `agent.py`) -- Review the config file for system prompts and tool definitions -- Check existing tests or evaluation scripts -- Look at CLAUDE.md or README for project context - -### Step 2: Align on What to Evaluate - -**Evaluation dimensions to consider:** - -| Dimension | When to Use | Example Scorer | -|-----------|-------------|----------------| -| **Safety** | Always (table stakes) | `Safety()` | -| **Correctness** | When ground truth exists | `Correctness()` | -| **Relevance** | When responses should address queries | `RelevanceToQuery()` | -| **Groundedness** | RAG systems with retrieved context | `RetrievalGroundedness()` | -| **Domain Guidelines** | Domain-specific requirements | `Guidelines(name="...", guidelines="...")` | -| **Format/Structure** | Structured output requirements | Custom scorer | -| **Tool Usage** | Agents with tool calls | Custom scorer checking tool selection | - -**Questions to ask the user:** -1. What are the **must-have** quality criteria? (safety, accuracy, relevance) -2. What are the **nice-to-have** criteria? (conciseness, tone, format) -3. Are there **specific failure modes** you've seen or worry about? -4. Do you have **ground truth** or expected answers for test cases? - -### Step 3: Define User Scenarios (Evaluation Dataset) - -**Types of test cases to include:** - -| Category | Purpose | Example | -|----------|---------|---------| -| **Happy Path** | Core functionality works | Typical user questions | -| **Edge Cases** | Boundary conditions | Empty inputs, very long queries | -| **Adversarial** | Robustness testing | Prompt injection, off-topic | -| **Multi-turn** | Conversation handling | Follow-up questions, context recall | -| **Domain-specific** | Business logic | Industry terminology, specific formats | - -**Questions to ask the user:** -1. What are the **most common** questions users ask? -2. What are **challenging** questions the agent should handle? -3. Are there questions it should **refuse** to answer? -4. Do you have **existing test cases** or production traces to start from? - -### Step 4: Establish Success Criteria - -**Define quality gates before running evaluation:** - -```python -QUALITY_GATES = { - "safety": 1.0, # 100% - non-negotiable - "correctness": 0.9, # 90% - high bar for accuracy - "relevance": 0.85, # 85% - good relevance - "concise": 0.8, # 80% - nice to have -} -``` - -**Questions to ask the user:** -1. What pass rates are **acceptable** for each dimension? -2. Which metrics are **blocking** vs **informational**? -3. How will evaluation results **inform decisions**? (ship/no-ship, iterate, investigate) - -### Strategy Alignment Checklist - -Before implementing evaluation, confirm: -- [ ] Agent purpose and architecture understood -- [ ] Evaluation dimensions agreed upon -- [ ] Test case categories identified -- [ ] Success criteria defined -- [ ] Data source identified (new, traces, existing dataset) - ---- - -## Journey 3: "Something Broke" - Regression Detection - -**Starting Point**: You made changes to your agent and suspect something regressed -**Goal**: Identify what broke and verify the fix - -### Steps - -1. **Establish baseline metrics** - ```bash - # Run evaluation on the previous version (or use saved baseline) - cd agents/tool_calling_dspy - python run_quick_eval.py - ``` - Record key metrics: `classifier_accuracy`, `tool_selection_accuracy`, `follows_instructions` - -2. **Run evaluation on current version** - ```bash - python run_quick_eval.py - ``` - -3. **Compare metrics** - ```python - from evaluation.optimization_history import OptimizationHistory - - history = OptimizationHistory() - print(history.compare_iterations(-2, -1)) # Compare last two - ``` - -4. **Identify regression source** - - If `classifier_accuracy` dropped → Check ClassifierSignature changes - - If `tool_selection_accuracy` dropped → Check tool descriptions, required_tools field - - If `follows_instructions` dropped → Check ExecutorSignature output format - -5. **Analyze failing traces** - ``` - /eval:analyze-traces [experiment-id] - ``` - Look for: - - Error patterns in specific test categories - - Tool call failures - - Unexpected outputs - -6. **Fix and re-evaluate** - - Revert problematic changes or apply targeted fix - - Re-run evaluation - - Verify metrics restored - -### Commands Used -- `python run_quick_eval.py` - Run evaluation -- `/eval:analyze-traces` - Deep trace analysis -- `OptimizationHistory.compare_iterations()` - Metric comparison - -### Success Indicators -- Metrics return to baseline or improve -- No new failing test cases -- Trace analysis shows expected behavior - ---- - -## Journey 7: "My Multi-Agent is Slow" - Performance Optimization - -**Starting Point**: Your agent responses are too slow -**Goal**: Identify bottlenecks and reduce latency - -### Steps - -1. **Run evaluation with latency scoring** - ```bash - cd agents/tool_calling_dspy - python run_quick_eval.py - ``` - Note the latency metrics: - - `classifier_latency_ms` - - `rewriter_latency_ms` - - `executor_latency_ms` - - `total_latency_ms` - -2. **Identify the bottleneck stage** - | Latency | Typical Range | If High, Check | - |---------|---------------|----------------| - | classifier_latency | <5s | ClassifierSignature verbosity | - | rewriter_latency | <10s | QueryRewriterSignature complexity | - | executor_latency | <30s | Tool call count, response generation | - -3. **Analyze traces for slow stages** - ``` - /eval:analyze-traces [experiment-id] - ``` - Focus on: - - Span durations by stage - - Number of LLM calls per stage - - Tool execution times - -4. **Run signature analysis** - ```bash - python -m evaluation.analyze_signatures - ``` - Look for: - - High total description chars (>2000) - - Verbose OutputField descriptions - - Missing examples (causes more retries) - -5. **Apply optimizations** - - **For high classifier latency:** - - Simplify ClassifierSignature docstring - - Add concrete examples to reduce ambiguity - - **For high executor latency:** - - Simplify ExecutorSignature.answer format - - Reduce output format requirements - - Consider caching repeated tool calls - - **For high total latency:** - - Review if all stages are necessary - - Consider parallel execution where possible - -6. **Re-evaluate and compare** - ```bash - python run_quick_eval.py - ``` - Use `OptimizationHistory.compare_iterations()` to verify improvement - -### Commands Used -- `python run_quick_eval.py` - Run evaluation with latency scoring -- `/eval:analyze-traces` - Trace analysis with timing breakdown -- `python -m evaluation.analyze_signatures` - Signature verbosity analysis - -### Success Indicators -- Target latencies: classifier <5s, executor <30s, total <60s -- No regression in accuracy metrics -- Consistent improvement across test categories - ---- - -## Journey 8: "Improve My Prompts" - Systematic Prompt Optimization - -**Starting Point**: Your agent works but could be more accurate -**Goal**: Systematically improve prompt quality through evaluation - -### Steps - -1. **Establish baseline** - ```bash - cd agents/tool_calling_dspy - python run_quick_eval.py - ``` - Record all metrics in `optimization_history.json` - -2. **Run signature analysis** - ```bash - python -m evaluation.analyze_signatures - ``` - Review the report for: - - Metric correlations (which signatures affect which metrics) - - Specific issues flagged per signature - -3. **Prioritize fixes by metric impact** - - | Metric | Primary Signature | Common Issues | - |--------|-------------------|---------------| - | follows_instructions | ExecutorSignature | Verbose answer format, unclear structure | - | tool_selection_accuracy | ClassifierSignature | No examples, ambiguous tool descriptions | - | classifier_accuracy | ClassifierSignature | Verbose docstring, unclear query_type mapping | - -4. **Apply ONE fix at a time** - - Make a single, targeted change - - Document the change in your commit message - - Track in optimization_history.json - -5. **Re-evaluate immediately** - ```bash - python run_quick_eval.py - ``` - - If improved → Keep change, move to next fix - - If regressed → Revert and try different approach - - If unchanged → Consider if fix was necessary - -6. **Iterate until targets met** - - | Metric | Target | - |--------|--------| - | classifier_accuracy | 95%+ | - | tool_selection_accuracy | 90%+ | - | follows_instructions | 80%+ | - -7. **Document successful optimizations** - ```python - from evaluation.optimization_history import OptimizationHistory - - history = OptimizationHistory() - print(history.summary()) - ``` - -### Commands Used -- `python run_quick_eval.py` - Run evaluation -- `python -m evaluation.analyze_signatures` - Identify prompt issues -- `/optimize:context --quick` - Full optimization loop (when endpoint available) - -### Success Indicators -- All target metrics met -- No regressions from baseline -- Clear documentation of what changed and why -- Optimization history shows positive trend - ---- - -## Quick Reference - -### Which Journey Am I On? - -| Symptom | Journey | -|---------|---------| -| "It was working before" | Journey 3 (Regression) | -| "It's too slow" | Journey 7 (Performance) | -| "It's not accurate enough" | Journey 8 (Prompt Optimization) | - -### Common Tools Across Journeys - -| Tool | Purpose | -|------|---------| -| `run_quick_eval.py` | Fast evaluation (8 test cases) | -| `run_full_eval.py` | Full evaluation (23 test cases) | -| `analyze_signatures.py` | Signature/prompt analysis | -| `OptimizationHistory` | Track iterations | -| `/eval:analyze-traces` | Deep trace analysis | -| `/optimize:context` | Full optimization loop | - -### Metric Targets - -| Metric | Target | Critical Threshold | -|--------|--------|-------------------| -| classifier_accuracy | 95%+ | <80% | -| tool_selection_accuracy | 90%+ | <70% | -| follows_instructions | 80%+ | <50% | -| executor_latency | <30s | >60s | diff --git a/.claude/skills/refresh-databricks-skills/SKILL.md b/.claude/skills/refresh-databricks-skills/SKILL.md new file mode 100644 index 00000000..47395dd4 --- /dev/null +++ b/.claude/skills/refresh-databricks-skills/SKILL.md @@ -0,0 +1,59 @@ +--- +name: refresh-databricks-skills +description: Use when Databricks skills need updating, user asks to refresh or sync skills from upstream, or skills seem outdated compared to the ai-dev-kit repo +--- + +# Refresh Databricks Skills + +## Overview + +Pulls the latest Databricks skills from the upstream source repo and replaces all existing Databricks skills in the project while preserving non-Databricks skills (e.g., superpowers workflow skills). + +**Source repo:** `https://github.com/databricks-solutions/ai-dev-kit` (path: `databricks-skills/`) + +## When to Use + +- User asks to update, refresh, or sync Databricks skills +- Skills seem outdated or missing newer Databricks features +- A new Databricks skill was added upstream that the project needs + +## Process + +1. **Clone the upstream repo** (shallow clone for speed): + ```bash + git clone --depth 1 https://github.com/databricks-solutions/ai-dev-kit.git $TMPDIR/ai-dev-kit + ``` + +2. **Identify non-Databricks skills to preserve.** These are the superpowers workflow skills that live alongside Databricks skills. List them by checking which directories in `.claude/skills/` do NOT have a matching folder in the upstream `databricks-skills/` directory. Common superpowers skills include: `brainstorming`, `dispatching-parallel-agents`, `executing-plans`, `finishing-a-development-branch`, `receiving-code-review`, `requesting-code-review`, `subagent-driven-development`, `systematic-debugging`, `test-driven-development`, `using-git-worktrees`, `using-superpowers`, `verification-before-completion`, `writing-plans`, `writing-skills`. Also preserve any other project-specific skills (like this one: `refresh-databricks-skills`). + +3. **Remove old Databricks skills** from `.claude/skills/`, keeping all non-Databricks skills identified above. + +4. **Copy new Databricks skills** from the cloned repo. Copy every directory under `databricks-skills/` except `TEMPLATE`: + ```bash + SKILLS_DIR=".claude/skills" + UPSTREAM="$TMPDIR/ai-dev-kit/databricks-skills" + for dir in "$UPSTREAM"/databricks-* "$UPSTREAM"/spark-*; do + [ -d "$dir" ] && cp -r "$dir" "$SKILLS_DIR/$(basename "$dir")" + done + ``` + +5. **Clean up** the cloned repo: + ```bash + rm -rf $TMPDIR/ai-dev-kit + ``` + +6. **Report** the count of skills added, removed, and updated. + +## After Refreshing + +If the project is deployed as a Databricks App, remind the user to sync the updated skills to the workspace and redeploy: +```bash +databricks workspace import-dir --overwrite --profile +databricks apps deploy --source-code-path --profile +``` + +## Common Mistakes + +- **Deleting non-Databricks skills:** Always identify and preserve superpowers and project-specific skills before removing anything. +- **Forgetting this skill itself:** `refresh-databricks-skills` must be preserved during the refresh. +- **Not using `--depth 1`:** Full clone is slow and unnecessary. Always shallow clone. diff --git a/.claude/skills/shadcn-ui-flutter/SKILL.md b/.claude/skills/shadcn-ui-flutter/SKILL.md new file mode 100644 index 00000000..3a9256a1 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/SKILL.md @@ -0,0 +1,165 @@ +--- +name: shadcn-ui-flutter +description: A comprehensive Flutter UI library inspired by shadcn/ui. Provides high-quality, customizable, and accessible components including Buttons, Cards, Forms, and more. Use this skill when building Flutter UIs, implementing design systems, or needing specific component usage examples. +--- + +# Shadcn UI for Flutter + +This skill provides documentation and examples for using the `shadcn_ui` package in Flutter. + +## Theming and Customization +Shadcn UI for Flutter provides a powerful theming system. You can use built-in color schemes (blue, gray, green, neutral, orange, red, rose, slate, stone, violet, yellow, zinc) or create your own. + +### Applying a Theme +Use `ShadThemeData` within `ShadApp` to define your light and dark themes. + +### Detailed Guides +- [Theming](guides/theming.md) +- [Typography](guides/typography.md) +- [Material & Cupertino Interop](guides/interop.md) +- [Responsive](guides/responsive.md) +- [Decorator](guides/decorator.md) + +## Components +| Name | Description | Reference | +| :--- | :--- | :--- | +| Accordion | A vertically stacked set of interactive headings that each reveal a section of content. | [accordion.md](components/accordion.md) | +| Alert | Displays a callout for user attention. | [alert.md](components/alert.md) | +| Avatar | An image element with a placeholder for representing the user. | [avatar.md](components/avatar.md) | +| Badge | Displays a badge or a component that looks like a badge. | [badge.md](components/badge.md) | +| Breadcrumb | Displays the path to the current resource using a hierarchy of links. | [breadcrumb.md](components/breadcrumb.md) | +| Button | Displays a button or a component that looks like a button. | [button.md](components/button.md) | +| Calendar | A date field component that allows users to enter and edit date. | [calendar.md](components/calendar.md) | +| Card | Displays a card with header, content, and footer. | [card.md](components/card.md) | +| Checkbox | A control that allows the user to toggle between checked and not checked. | [checkbox.md](components/checkbox.md) | +| Context Menu | Displays a menu to the user — such as a set of actions or functions — triggered by a mouse right-click. | [context-menu.md](components/context-menu.md) | +| Date Picker | A date picker component with range and presets. | [date-picker.md](components/date-picker.md) | +| Dialog | A modal dialog that interrupts the user. | [dialog.md](components/dialog.md) | +| Form | Builds a form with validation and easy access to form fields values. | [form.md](components/form.md) | +| IconButton | Displays an icon button or a component that looks like a button with an icon. | [icon-button.md](components/icon-button.md) | +| Input | Displays a form input field or a component that looks like an input field. | [input.md](components/input.md) | +| InputOTP | Accessible one-time password component with copy paste functionality. | [input-otp.md](components/input-otp.md) | +| Menubar | A visually persistent menu common in desktop applications that provides quick access to a consistent set of commands. | [menubar.md](components/menubar.md) | +| Popover | Displays rich content in a portal, triggered by a button. | [popover.md](components/popover.md) | +| Progress | Displays an indicator showing the completion progress of a task, typically displayed as a progress bar. | [progress.md](components/progress.md) | +| RadioGroup | A set of checkable buttons—known as radio buttons—where no more than one of the buttons can be checked at a time. | [radio-group.md](components/radio-group.md) | +| Resizable | Resizable panel groups and layouts. | [resizable.md](components/resizable.md) | +| Select | Displays a list of options for the user to pick from—triggered by a button. | [select.md](components/select.md) | +| Separator | Visually or semantically separates content. | [separator.md](components/separator.md) | +| Sheet | Extends the Dialog component to display content that complements the main content of the screen. | [sheet.md](components/sheet.md) | +| Slider | An input where the user selects a value from within a given range. | [slider.md](components/slider.md) | +| Sonner | An opinionated toast component. | [sonner.md](components/sonner.md) | +| Switch | A control that allows the user to toggle between checked and not checked. | [switch.md](components/switch.md) | +| Table | A responsive table component. | [table.md](components/table.md) | +| Tabs | A set of layered sections of content—known as tab panels—that are displayed one at a time. | [tabs.md](components/tabs.md) | +| Textarea | Displays a form textarea or a component that looks like a textarea. | [textarea.md](components/textarea.md) | +| Time Picker | A time picker component. | [time-picker.md](components/time-picker.md) | +| Toast | A succinct message that is displayed temporarily. | [toast.md](components/toast.md) | +| Tooltip | A popup that displays information related to an element when the element receives keyboard focus or the mouse hovers over it. | [tooltip.md](components/tooltip.md) | + +## Usage Examples +Examples are available at the bottom of each component page. + +### Basic Setup +Here is a complete example of a Counter App using `shadcn_ui`, including light and dark theme support. +```dart +import 'package:shadcn_ui/shadcn_ui.dart'; + +void main() { + runApp(const MyApp()); +} + +class MyApp extends StatelessWidget { + const MyApp({super.key}); + + @override + Widget build(BuildContext context) { + return ShadApp( + debugShowCheckedModeBanner: false, + theme: ShadThemeData( + brightness: Brightness.light, + colorScheme: const ShadZincColorScheme.light(), + ), + darkTheme: ShadThemeData( + brightness: Brightness.dark, + colorScheme: const ShadZincColorScheme.dark(), + ), + themeMode: ThemeMode.system, + home: const CounterPage(), + ); + } +} + +class CounterPage extends StatefulWidget { + const CounterPage({super.key}); + + @override + State createState() => _CounterPageState(); +} + +class _CounterPageState extends State { + int _counter = 0; + + void _incrementCounter() { + setState(() { + _counter++; + }); + } + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return Scaffold( + appBar: AppBar(title: const Text('Shadcn Counter')), + body: Center( + child: Column( + mainAxisAlignment: MainAxisAlignment.center, + children: [ + Text( + 'You have pushed the button this many times:', + style: theme.textTheme.muted, + ), + Text( + '$_counter', + style: theme.textTheme.h1, + ), + ], + ), + ), + floatingActionButton: ShadButton( + onPressed: _incrementCounter, + child: const Icon(LucideIcons.plus), + ), + ); + } +} +``` + +## Packages included in the library + +Flutter Shadcn UI consists of fantastic open-source libraries that are exported and you can use them without importing them into your project. + +### [flutter_animate](https://pub.dev/packages/flutter_animate) + +The flutter animate library is a very cool animations library extensively used in Shadcn UI Components. + +With flutter_animate animations can be easily customized from the user, because components will take a `List`. + +### [lucide_icons_flutter](https://pub.dev/packages/lucide_icons_flutter) + +A nice icon library that is used in Shadcn UI Components. +You can use Lucide icons with the `LucideIcons` class, for example `LucideIcons.activity`. + +You can browse all the icons [here](https://lucide.dev/icons/). + +### [two_dimensional_scrollables](https://pub.dev/packages/two_dimensional_scrollables) + +A nice raw table (very performant) implementation used by the [ShadTable](../components/table) component. + +### [intl](https://pub.dev/packages/intl) + +The intl package provides internationalization and localization facilities, including message translation. + +### [universal_image](https://pub.dev/packages/universal_image) + +Support multiple image formats. Used by the [ShadAvatar](../components/avatar) component. diff --git a/.claude/skills/shadcn-ui-flutter/components/accordion.md b/.claude/skills/shadcn-ui-flutter/components/accordion.md new file mode 100644 index 00000000..63e187cd --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/accordion.md @@ -0,0 +1,161 @@ +# Accordion + +A vertically stacked set of interactive headings that each reveal a section of content. + + + +```dart +final details = [ + ( + title: 'Is it acceptable?', + content: 'Yes. It adheres to the WAI-ARIA design pattern.', + ), + ( + title: 'Is it styled?', + content: + "Yes. It comes with default styles that matches the other components' aesthetic.", + ), + ( + title: 'Is it animated?', + content: + "Yes. It's animated by default, but you can disable it if you prefer.", + ), +]; + +@override +Widget build(BuildContext context) { + return ShadAccordion<({String content, String title})>( + children: details.map( + (detail) => ShadAccordionItem( + value: detail, + title: Text(detail.title), + child: Text(detail.content), + ), + ), + ); +} +``` + + + +## Multiple + + + +```dart +final details = [ + ( + title: 'Is it acceptable?', + content: 'Yes. It adheres to the WAI-ARIA design pattern.', + ), + ( + title: 'Is it styled?', + content: + "Yes. It comes with default styles that matches the other components' aesthetic.", + ), + ( + title: 'Is it animated?', + content: + "Yes. It's animated by default, but you can disable it if you prefer.", + ), +]; + +@override +Widget build(BuildContext context) { + return ShadAccordion<({String content, String title})>.multiple( + children: details.map( + (detail) => ShadAccordionItem( + value: detail, + title: Text(detail.title), + child: Text(detail.content), + ), + ), + ); +} +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +final details = [ + ( + title: 'Is it acceptable?', + content: 'Yes. It adheres to the WAI-ARIA design pattern.', + ), + ( + title: 'Is it styled?', + content: + "Yes. It comes with default styles that matches the other components' aesthetic.", + ), + ( + title: 'Is it animated?', + content: + "Yes. It's animated by default, but you can disable it if you prefer.", + ), +]; + +class AccordionPage extends StatefulWidget { + const AccordionPage({super.key}); + + @override + State createState() => _AccordionPageState(); +} + +class _AccordionPageState extends State { + var type = ShadAccordionVariant.single; + var underlineTitle = true; + + @override + Widget build(BuildContext context) { + final children = details.map( + (detail) { + return ShadAccordionItem( + value: detail, + title: Text(detail.title), + underlineTitleOnHover: underlineTitle, + child: Text(detail.content), + ); + }, + ); + return BaseScaffold( + appBarTitle: 'Accordion', + editable: [ + MyEnumProperty( + label: 'Type', + value: type, + values: ShadAccordionVariant.values, + onChanged: (value) { + if (value != null) { + setState(() => type = value); + } + }, + ), + MyBoolProperty( + label: 'Underline title', + value: underlineTitle, + onChanged: (v) => setState(() => underlineTitle = v), + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: type == ShadAccordionVariant.single + ? ShadAccordion<({String content, String title})>( + children: children, + ) + : ShadAccordion<({String content, String title})>.multiple( + children: children, + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/alert.md b/.claude/skills/shadcn-ui-flutter/components/alert.md new file mode 100644 index 00000000..e471f918 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/alert.md @@ -0,0 +1,69 @@ +# Alert + +Displays a callout for user attention. + + + +```dart +ShadAlert( + icon: Icon(LucideIcons.terminal), + title: Text('Heads up!'), + description: + Text('You can add components to your app using the cli.'), +), +``` + + + +## Destructive + + + +```dart +ShadAlert.destructive( + icon: Icon(LucideIcons.circleAlert), + title: Text('Error'), + description: + Text('Your session has expired. Please log in again.'), +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class AlertPage extends StatelessWidget { + const AlertPage({super.key}); + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Alert', + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: const ShadAlert( + icon: Icon(LucideIcons.terminal), + title: Text('Heads up!'), + description: Text( + 'You can add components to your app using the cli.', + ), + ), + ), + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: const ShadAlert.destructive( + icon: Icon(LucideIcons.circleAlert), + title: Text('Error'), + description: Text('Your session has expired. Please log in again.'), + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/avatar.md b/.claude/skills/shadcn-ui-flutter/components/avatar.md new file mode 100644 index 00000000..17660148 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/avatar.md @@ -0,0 +1,38 @@ +# Avatar + +An image element with a placeholder for representing the user. + + + +```dart +ShadAvatar( + 'https://app.requestly.io/delay/2000/avatars.githubusercontent.com/u/124599?v=4', + placeholder: Text('CN'), +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class AvatarPage extends StatelessWidget { + const AvatarPage({super.key}); + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Avatar', + children: [ + ShadAvatar( + 'https://app.requestly.io/delay/2000/avatars.githubusercontent.com/u/124599?v=4&t=${DateTime.now().millisecondsSinceEpoch}', + placeholder: const Text('CN'), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/badge.md b/.claude/skills/shadcn-ui-flutter/components/badge.md new file mode 100644 index 00000000..49b507a8 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/badge.md @@ -0,0 +1,74 @@ +# Badge + +Displays a badge or a component that looks like a badge. + +## Primary + + + +```dart +ShadBadge( + child: const Text('Primary'), +) +``` + + + +## Secondary + + + +```dart +ShadBadge.secondary( + child: const Text('Secondary'), +) +``` + + + +## Destructive + + + +```dart +ShadBadge.destructive( + child: const Text('Destructive'), +) +``` + + + +## Outline + + + +```dart +ShadBadge.outline( + child: const Text('Outline'), +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class BadgePage extends StatelessWidget { + const BadgePage({super.key}); + + @override + Widget build(BuildContext context) { + return const BaseScaffold( + appBarTitle: 'Badge', + children: [ + ShadBadge(child: Text('Primary')), + ShadBadge.secondary(child: Text('Secondary')), + ShadBadge.destructive(child: Text('Destructive')), + ShadBadge.outline(child: Text('Outline')), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/breadcrumb.md b/.claude/skills/shadcn-ui-flutter/components/breadcrumb.md new file mode 100644 index 00000000..d166efc6 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/breadcrumb.md @@ -0,0 +1,287 @@ +# Breadcrumb + +Displays the path to the current resource using a hierarchy of links. + + + + +```dart +class PrimaryBreadcrumb extends StatelessWidget { + const PrimaryBreadcrumb({super.key}); + + @override + Widget build(BuildContext context) { + return ShadBreadcrumb( + children: [ + ShadBreadcrumbLink( + onPressed: () => print('Navigating to Home'), + child: const Text('Home'), + ), + ShadBreadcrumbDropdown( + items: [ + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Documentation'), + child: const Text('Documentation'), + ), + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Themes'), + child: const Text('Themes'), + ), + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Github'), + child: const Text('Github'), + ), + ], + showDropdownArrow: false, + child: ShadBreadcrumbEllipsis(), + ), + Text('Components'), + Text('Breadcrumb'), + ], + ); + } +} +``` + + + +## Custom separator + +Use a custom `separator` to change the default `>` separator. + + + +```dart +class CustomSeparatorBreadcrumb extends StatelessWidget { + const CustomSeparatorBreadcrumb({super.key}); + + @override + Widget build(BuildContext context) { + return ShadBreadcrumb( + separator: const Icon(LucideIcons.slash), + children: [ + ShadBreadcrumbLink( + onPressed: () => print('Navigating to Home'), + child: const Text('Home'), + ), + ShadBreadcrumbLink( + onPressed: () => print('Navigating to Components'), + child: const Text('Components'), + ), + Text('Breadcrumb'), + ], + ); + } +} +``` + + + + +## Dropdown + +You can use `ShadBreadcrumbDropdown` to create a dropdown in the breadcrumb. + + + +```dart +class DropdownBreadcrumb extends StatelessWidget { + const DropdownBreadcrumb({super.key}); + + @override + Widget build(BuildContext context) { + return ShadBreadcrumb( + children: [ + ShadBreadcrumbLink( + onPressed: () => print('Navigating to Home'), + child: const Text('Home'), + ), + ShadBreadcrumbDropdown( + items: [ + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Documentation'), + child: const Text('Documentation'), + ), + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Themes'), + child: const Text('Themes'), + ), + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Github'), + child: const Text('Github'), + ), + ], + child: const Text('Components'), + ), + Text('Breadcrumb'), + ], + ); + } +} +``` + +## Example +```dart +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; +import '../common/base_scaffold.dart'; + +class BreadcrumbPage extends StatelessWidget { + const BreadcrumbPage({super.key}); + + void _navigateToHome() { + print('Navigating to Home'); + } + + void _navigateToComponents() { + print('Navigating to Components'); + } + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Breadcrumb', + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + const Text( + 'Simple Breadcrumb', + style: TextStyle(fontSize: 18, fontWeight: FontWeight.bold), + ), + const ShadBreadcrumb( + children: [ + Text('Home'), + Text('Library'), + Text('Data'), + ], + ), + const SizedBox( + height: 20, + ), + const Text( + 'Breadcrumb with Links', + style: TextStyle(fontSize: 18, fontWeight: FontWeight.bold), + ), + ShadBreadcrumb( + children: [ + ShadBreadcrumbLink( + onPressed: _navigateToHome, + child: const Text('Home'), + ), + ShadBreadcrumbLink( + onPressed: _navigateToComponents, + child: const Text('Components'), + ), + const Text('Breadcrumb'), + ], + ), + const SizedBox( + height: 20, + ), + const Text( + 'Breadcrumb with Ellipsis', + style: TextStyle(fontSize: 18, fontWeight: FontWeight.bold), + ), + ShadBreadcrumb( + children: [ + ShadBreadcrumbLink( + onPressed: _navigateToHome, + child: const Text('Home'), + ), + const ShadBreadcrumbEllipsis(), + ShadBreadcrumbLink( + onPressed: _navigateToComponents, + child: const Text('Components'), + ), + const Text('Breadcrumb'), + ], + ), + const SizedBox(height: 20), + const Text( + 'Custom Separator', + style: TextStyle(fontSize: 18, fontWeight: FontWeight.bold), + ), + ShadBreadcrumb( + separator: const Icon(LucideIcons.slash), + children: [ + ShadBreadcrumbLink( + onPressed: _navigateToHome, + child: const Text('Home'), + ), + ShadBreadcrumbLink( + onPressed: _navigateToComponents, + child: const Text('Components'), + ), + const Text('Breadcrumb'), + ], + ), + const SizedBox( + height: 20, + ), + const Text( + 'Breadcrumb with Dropdown', + style: TextStyle(fontSize: 18, fontWeight: FontWeight.bold), + ), + ShadBreadcrumb( + children: [ + ShadBreadcrumbLink( + onPressed: _navigateToHome, + child: const Text('Home'), + ), + ShadBreadcrumbDropdown( + items: [ + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Documentation'), + child: const Text('Documentation'), + ), + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Themes'), + child: const Text('Themes'), + ), + ShadBreadcrumbDropMenuItem( + onPressed: () => print('Navigating to Github'), + child: const Text('Github'), + ), + ], + child: const Text('Components'), + ), + Text('Breadcrumb'), + ], + ), + const SizedBox( + height: 20, + ), + const Text( + 'Long Breadcrumb', + style: TextStyle(fontSize: 18, fontWeight: FontWeight.bold), + ), + ShadBreadcrumb( + children: [ + ShadBreadcrumbLink( + onPressed: _navigateToHome, + child: const Text('Home'), + ), + ShadBreadcrumbLink( + onPressed: _navigateToComponents, + child: const Text('Component 1'), + ), + ShadBreadcrumbLink( + onPressed: _navigateToComponents, + child: const Text('Component 2'), + ), + ShadBreadcrumbLink( + onPressed: _navigateToComponents, + child: const Text('Component 3'), + ), + ShadBreadcrumbLink( + onPressed: _navigateToComponents, + child: const Text('Component 4'), + ), + Text('Breadcrumb'), + ], + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/button.md b/.claude/skills/shadcn-ui-flutter/components/button.md new file mode 100644 index 00000000..e54753f9 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/button.md @@ -0,0 +1,259 @@ +# Button + +Displays a button or a component that looks like a button. + +## Primary + + + +```dart +ShadButton( + child: const Text('Primary'), + onPressed: () {}, +) +``` + + + +## Secondary + + + +```dart +ShadButton.secondary( + child: const Text('Secondary'), + onPressed: () {}, +) +``` + + + +## Destructive + + + +```dart +ShadButton.destructive( + child: const Text('Destructive'), + onPressed: () {}, +) +``` + + + +## Outline + + + +```dart +ShadButton.outline( + child: const Text('Outline'), + onPressed: () {}, +) +``` + + + +## Ghost + + + +```dart +ShadButton.ghost( + child: const Text('Ghost'), + onPressed: () {}, +) +``` + + + +## Link + + + +```dart +ShadButton.link( + child: const Text('Link'), + onPressed: () {}, +) +``` + + + +## Text and Icon + + + +```dart +ShadButton( + onPressed: () {}, + leading: const Icon(LucideIcons.mail), + child: const Text('Login with Email'), +) +``` + + + +## Loading + + + +```dart +ShadButton( + onPressed: () {}, + leading: SizedBox.square( + dimension: 16, + child: CircularProgressIndicator( + strokeWidth: 2, + color: ShadTheme.of(context).colorScheme.primaryForeground, + ), + ), + child: const Text('Please wait'), +) +``` + + + +## Gradient and Shadow + + + +```dart +ShadButton( + onPressed: () {}, + gradient: const LinearGradient(colors: [ + Colors.cyan, + Colors.indigo, + ]), + shadows: [ + BoxShadow( + color: Colors.blue.withOpacity(.4), + spreadRadius: 4, + blurRadius: 10, + offset: const Offset(0, 2), + ), + ], + child: const Text('Gradient with Shadow'), +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class ButtonPage extends StatefulWidget { + const ButtonPage({super.key}); + + @override + State createState() => _ButtonPageState(); +} + +class _ButtonPageState extends State { + var size = ShadButtonSize.regular; + var enabled = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return FocusTraversalGroup( + policy: WidgetOrderTraversalPolicy(), + child: BaseScaffold( + appBarTitle: 'Button', + editable: [ + MyEnumProperty( + label: 'Size', + value: size, + values: ShadButtonSize.values, + onChanged: (value) { + if (value != null) { + setState(() => size = value); + } + }, + ), + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + ], + children: [ + ShadButton( + size: size, + enabled: enabled, + child: const Text('Primary'), + onPressed: () => print('Primary'), + ), + ShadButton.secondary( + size: size, + enabled: enabled, + child: const Text('Secondary'), + onPressed: () => print('Secondary'), + ), + ShadButton.destructive( + size: size, + enabled: enabled, + child: const Text('Destructive'), + ), + ShadButton.outline( + size: size, + enabled: enabled, + child: const Text('Outline'), + ), + ShadButton.ghost( + size: size, + enabled: enabled, + child: const Text('Ghost'), + ), + ShadButton.link( + size: size, + enabled: enabled, + child: const Text('Link'), + ), + ShadButton( + size: size, + enabled: enabled, + leading: const Icon(LucideIcons.mail), + child: const Text('Login with Email'), + ), + ShadButton( + size: size, + enabled: enabled, + leading: SizedBox.square( + dimension: 16, + child: CircularProgressIndicator( + strokeWidth: 2, + color: theme.colorScheme.primaryForeground, + ), + ), + child: const Text('Please wait'), + ), + ShadButton( + size: size, + enabled: enabled, + gradient: const LinearGradient( + colors: [ + Colors.cyan, + Colors.indigo, + ], + ), + shadows: [ + BoxShadow( + color: Colors.blue.withValues(alpha: .4), + spreadRadius: 4, + blurRadius: 10, + offset: const Offset(0, 2), + ), + ], + child: const Text('Gradient with Shadow'), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/calendar.md b/.claude/skills/shadcn-ui-flutter/components/calendar.md new file mode 100644 index 00000000..eb5e6bc6 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/calendar.md @@ -0,0 +1,326 @@ +# Calendar + +A date field component that allows users to enter and edit date. + + + +```dart +class SingleCalendar extends StatefulWidget { + const SingleCalendar({super.key}); + + @override + State createState() => _SingleCalendarState(); +} + +class _SingleCalendarState extends State { + final today = DateTime.now(); + + @override + Widget build(BuildContext context) { + return ShadCalendar( + selected: today, + fromMonth: DateTime(today.year - 1), + toMonth: DateTime(today.year, 12), + ); + } +} +``` + + + +## Multiple + + + +```dart +class MultipleCalendar extends StatefulWidget { + const MultipleCalendar({super.key}); + + @override + State createState() => _MultipleCalendarState(); +} + +class _MultipleCalendarState extends State { + final today = DateTime.now(); + + @override + Widget build(BuildContext context) { + return ShadCalendar.multiple( + numberOfMonths: 2, + fromMonth: DateTime(today.year), + toMonth: DateTime(today.year + 1, 12), + min: 5, + max: 10, + ); + } +} +``` + + + +## Range + + + +```dart +class RangeCalendar extends StatelessWidget { + const RangeCalendar({super.key}); + + @override + Widget build(BuildContext context) { + return const ShadCalendar.range( + min: 2, + max: 5, + ); + } +} +``` + + + +#### DropdownMonths + + + +```dart +ShadCalendar( + captionLayout: ShadCalendarCaptionLayout.dropdownMonths, +); +``` + + + +#### DropdownYears + + + +```dart +ShadCalendar( + captionLayout: ShadCalendarCaptionLayout.dropdownYears, +); +``` + + + +### Hide Navigation + + + +```dart +ShadCalendar( + hideNavigation: true, +); +``` + + + +### Show Week Numbers + + + +```dart +ShadCalendar( + showWeekNumbers: true, +); +``` + + + +### Show Outside Days (false) + + + +```dart +ShadCalendar( + showOutsideDays: false, +); +``` + + + +### Fixed Weeks + + + +```dart +ShadCalendar( + fixedWeeks: true, +); +``` + + + +### Hide Weekday Names + + + +```dart +ShadCalendar( + hideWeekdayNames: true, +); +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class CalendarPage extends StatefulWidget { + const CalendarPage({super.key}); + + @override + State createState() => _CalendarPageState(); +} + +class _CalendarPageState extends State { + DateTime? selected = DateTime.now(); + bool reverseMonths = false; + ShadCalendarCaptionLayout captionLayout = ShadCalendarCaptionLayout.label; + bool hideNavigation = false; + bool showWeekNumbers = false; + bool showOutsideDays = true; + bool fixedWeeks = false; + bool hideWeekdayNames = false; + bool allowDeselection = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'Calendar', + editable: [ + MyBoolProperty( + label: 'Reverse months', + value: reverseMonths, + onChanged: (value) { + setState(() { + reverseMonths = value; + }); + }, + ), + MyEnumProperty( + label: 'Caption layout', + value: captionLayout, + values: ShadCalendarCaptionLayout.values, + onChanged: (value) { + if (value != null) { + setState(() { + captionLayout = value; + }); + } + }, + ), + MyBoolProperty( + label: 'Hide navigation', + value: hideNavigation, + onChanged: (value) { + setState(() { + hideNavigation = value; + }); + }, + ), + MyBoolProperty( + label: 'Show week numbers', + value: showWeekNumbers, + onChanged: (value) { + setState(() { + showWeekNumbers = value; + }); + }, + ), + MyBoolProperty( + label: 'Show outside days', + value: showOutsideDays, + enabled: !fixedWeeks, + onChanged: (value) { + setState(() { + showOutsideDays = value; + }); + }, + ), + MyBoolProperty( + label: 'Fixed weeks', + value: fixedWeeks, + enabled: showOutsideDays, + onChanged: (value) { + setState(() { + fixedWeeks = value; + }); + }, + ), + MyBoolProperty( + label: 'Hide weekday names', + value: hideWeekdayNames, + onChanged: (value) { + setState(() { + hideWeekdayNames = value; + }); + }, + ), + MyBoolProperty( + label: 'Allow deselection', + value: allowDeselection, + onChanged: (value) { + setState(() { + allowDeselection = value; + }); + }, + ), + ], + children: [ + Text('Single', style: theme.textTheme.h4), + ShadCalendar( + selected: selected, + fromMonth: DateTime(2023), + toMonth: DateTime(2024, 12), + hideNavigation: hideNavigation, + captionLayout: captionLayout, + onMonthChanged: (date) { + print('month changed to ${date.month}'); + }, + showWeekNumbers: showWeekNumbers, + showOutsideDays: showOutsideDays, + fixedWeeks: fixedWeeks, + hideWeekdayNames: hideWeekdayNames, + allowDeselection: allowDeselection, + ), + const ShadSeparator.horizontal(), + Text('Multiple', style: theme.textTheme.h4), + ShadCalendar.multiple( + numberOfMonths: 2, + fromMonth: DateTime(2024), + toMonth: DateTime(2024, 12), + onChanged: (dates) {}, + min: 5, + max: 10, + reverseMonths: reverseMonths, + hideNavigation: hideNavigation, + captionLayout: captionLayout, + showWeekNumbers: showWeekNumbers, + showOutsideDays: showOutsideDays, + fixedWeeks: fixedWeeks, + hideWeekdayNames: hideWeekdayNames, + ), + const ShadSeparator.horizontal(), + Text('Range', style: theme.textTheme.h4), + ShadCalendar.range( + onChanged: print, + min: 2, + max: 4, + hideNavigation: hideNavigation, + captionLayout: captionLayout, + showWeekNumbers: showWeekNumbers, + showOutsideDays: showOutsideDays, + fixedWeeks: fixedWeeks, + hideWeekdayNames: hideWeekdayNames, + allowDeselection: allowDeselection, + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/card.md b/.claude/skills/shadcn-ui-flutter/components/card.md new file mode 100644 index 00000000..e64fbcfa --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/card.md @@ -0,0 +1,421 @@ +# Card + +Displays a card with header, content, and footer. + + + +```dart +const frameworks = { + 'next': 'Next.js', + 'react': 'React', + 'astro': 'Astro', + 'nuxt': 'Nuxt.js', +}; + +class CardProject extends StatelessWidget { + const CardProject({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadCard( + width: 350, + title: Text('Create project', style: theme.textTheme.h4), + description: const Text('Deploy your new project in one-click.'), + footer: Row( + mainAxisAlignment: MainAxisAlignment.spaceBetween, + children: [ + ShadButton.outline( + child: const Text('Cancel'), + onPressed: () {}, + ), + ShadButton( + child: const Text('Deploy'), + onPressed: () {}, + ), + ], + ), + child: Padding( + padding: const EdgeInsets.symmetric(vertical: 16), + child: Column( + mainAxisSize: MainAxisSize.min, + crossAxisAlignment: CrossAxisAlignment.stretch, + children: [ + const Text('Name'), + const SizedBox(height: 6), + const ShadInput(placeholder: Text('Name of your project')), + const SizedBox(height: 16), + const Text('Framework'), + const SizedBox(height: 6), + ShadSelect( + placeholder: const Text('Select'), + options: frameworks.entries + .map((e) => ShadOption(value: e.key, child: Text(e.value))) + .toList(), + selectedOptionBuilder: (context, value) { + return Text(frameworks[value]!); + }, + onChanged: (value) {}, + ), + ], + ), + ), + ); + } +} +``` + + + +## Notifications Example + + + +```dart + + + +const notifications = [ + ( + title: "Your call has been confirmed.", + description: "1 hour ago", + ), + ( + title: "You have a new message!", + description: "1 hour ago", + ), + ( + title: "Your subscription is expiring soon!", + description: "2 hours ago", + ), +]; + +class CardNotifications extends StatefulWidget { + const CardNotifications({super.key}); + + @override + State createState() => _CardNotificationsState(); +} + +class _CardNotificationsState extends State { + final pushNotifications = ValueNotifier(false); + + @override + void dispose() { + pushNotifications.dispose(); + super.dispose(); + } + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadCard( + width: 380, + title: const Text('Notifications'), + description: const Text('You have 3 unread messages.'), + footer: ShadButton( + width: double.infinity, + leading: const Padding( + padding: EdgeInsets.only(right: 8), + child: Icon(LucideIcons.check), + ), + onPressed: () {}, + child: const Text('Mark all as read'), + ), + child: Column( + mainAxisSize: MainAxisSize.min, + children: [ + const SizedBox(height: 16), + Container( + padding: const EdgeInsets.all(16), + decoration: BoxDecoration( + borderRadius: theme.radius, + border: Border.all(color: theme.colorScheme.border), + ), + child: Row( + children: [ + Icon( + LucideIcons.bellRing, + size: 24, + color: theme.colorScheme.foreground, + ), + Expanded( + child: Padding( + padding: const EdgeInsets.only(left: 16), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text( + 'Push Notifications', + style: theme.textTheme.small, + ), + const SizedBox(height: 4), + Text( + 'Send notifications to device.', + style: theme.textTheme.muted, + ) + ], + ), + ), + ), + ValueListenableBuilder( + valueListenable: pushNotifications, + builder: (context, value, child) { + return ShadSwitch( + value: value, + onChanged: (v) => pushNotifications.value = v, + ); + }, + ), + ], + ), + ), + const SizedBox(height: 16), + ...notifications + .map( + (n) => Row( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Container( + width: 8, + height: 8, + margin: const EdgeInsets.only(top: 4), + decoration: const BoxDecoration( + color: Color(0xFF0CA5E9), + shape: BoxShape.circle, + ), + ), + Expanded( + child: Padding( + padding: const EdgeInsets.only(left: 16), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('Your call has been confirmed.', + style: theme.textTheme.small), + const SizedBox(height: 4), + Text(n.description, style: theme.textTheme.muted), + ], + ), + ), + ) + ], + ), + ) + .separatedBy(const SizedBox(height: 16)), + const SizedBox(height: 16), + ], + ), + ); + } +} +``` + +## Example +```dart +import 'package:awesome_flutter_extensions/awesome_flutter_extensions.dart'; +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +const frameworks = { + 'next': 'Next.js', + 'react': 'React', + 'astro': 'Astro', + 'nuxt': 'Nuxt.js', +}; + +const notifications = [ + ( + title: "Your call has been confirmed.", + description: "1 hour ago", + ), + ( + title: "You have a new message!", + description: "1 hour ago", + ), + ( + title: "Your subscription is expiring soon!", + description: "2 hours ago", + ), +]; + +class CardPage extends StatefulWidget { + const CardPage({super.key}); + + @override + State createState() => _CardPageState(); +} + +class _CardPageState extends State { + final pushNotifications = ValueNotifier(false); + + @override + void dispose() { + pushNotifications.dispose(); + super.dispose(); + } + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'Card', + children: [ + ShadCard( + width: 350, + title: const Text('Create project'), + description: const Text('Deploy your new project in one-click.'), + footer: Row( + mainAxisAlignment: MainAxisAlignment.spaceBetween, + children: [ + ShadButton.outline( + child: const Text('Cancel'), + onPressed: () {}, + ), + ShadButton( + child: const Text('Deploy'), + onPressed: () {}, + ), + ], + ), + child: Padding( + padding: const EdgeInsets.symmetric(vertical: 16), + child: Column( + mainAxisSize: MainAxisSize.min, + crossAxisAlignment: CrossAxisAlignment.stretch, + children: [ + const Text('Name'), + const SizedBox(height: 6), + const ShadInput(placeholder: Text('Name of your project')), + const SizedBox(height: 16), + const Text('Framework'), + const SizedBox(height: 6), + ShadSelect( + placeholder: const Text('Select'), + options: frameworks.entries + .map( + (e) => ShadOption(value: e.key, child: Text(e.value)), + ) + .toList(), + selectedOptionBuilder: (context, value) { + return Text(frameworks[value]!); + }, + onChanged: (value) {}, + ), + ], + ), + ), + ), + const SizedBox(height: 40), + ShadCard( + width: 380, + title: const Text('Notifications'), + description: const Text('You have 3 unread messages.'), + footer: ShadButton( + width: double.infinity, + leading: const Padding( + padding: EdgeInsetsDirectional.only(end: 8), + child: Icon(LucideIcons.check), + ), + onPressed: () {}, + child: const Text('Mark all as read'), + ), + child: Column( + mainAxisSize: MainAxisSize.min, + children: [ + const SizedBox(height: 16), + Container( + padding: const EdgeInsets.all(16), + decoration: BoxDecoration( + borderRadius: theme.radius, + border: Border.all(color: theme.colorScheme.border), + ), + child: Row( + children: [ + Icon( + LucideIcons.bellRing, + size: 24, + color: theme.colorScheme.foreground, + ), + Expanded( + child: Padding( + padding: const EdgeInsetsDirectional.only(start: 16), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text( + 'Push Notifications', + style: theme.textTheme.small, + ), + const SizedBox(height: 4), + Text( + 'Send notifications to device.', + style: theme.textTheme.muted, + ), + ], + ), + ), + ), + ValueListenableBuilder( + valueListenable: pushNotifications, + builder: (context, value, child) { + return ShadSwitch( + value: value, + onChanged: (v) => pushNotifications.value = v, + ); + }, + ), + ], + ), + ), + const SizedBox(height: 16), + ...notifications + .map( + (n) => Row( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Container( + width: 8, + height: 8, + margin: const EdgeInsets.only(top: 4), + decoration: const BoxDecoration( + color: Color(0xFF0CA5E9), + shape: BoxShape.circle, + ), + ), + Expanded( + child: Padding( + padding: const EdgeInsetsDirectional.only( + start: 16, + ), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text( + 'Your call has been confirmed.', + style: theme.textTheme.small, + ), + const SizedBox(height: 4), + Text( + n.description, + style: theme.textTheme.muted, + ), + ], + ), + ), + ), + ], + ), + ) + .separatedBy(const SizedBox(height: 16)), + const SizedBox(height: 16), + ], + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/checkbox.md b/.claude/skills/shadcn-ui-flutter/components/checkbox.md new file mode 100644 index 00000000..2e686308 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/checkbox.md @@ -0,0 +1,244 @@ +# Checkbox + +A control that allows the user to toggle between checked and not checked. + + + +```dart +class CheckboxSample extends StatefulWidget { + const CheckboxSample({super.key}); + + @override + State createState() => _CheckboxSampleState(); +} + +class _CheckboxSampleState extends State { + bool value = false; + + @override + Widget build(BuildContext context) { + return ShadCheckbox( + value: value, + onChanged: (v) => setState(() => value = v), + label: const Text('Accept terms and conditions'), + sublabel: const Text( + 'You agree to our Terms of Service and Privacy Policy.', + ), + ); + } +} +``` + + + +## Form + + + +```dart +ShadCheckboxFormField( + id: 'terms', + initialValue: false, + inputLabel: + const Text('I accept the terms and conditions'), + onChanged: (v) {}, + inputSublabel: + const Text('You agree to our Terms and Conditions'), + validator: (v) { + if (!v) { + return 'You must accept the terms and conditions'; + } + return null; + }, +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class CheckboxPage extends StatefulWidget { + const CheckboxPage({super.key}); + + @override + State createState() => _CheckboxPageState(); +} + +class _CheckboxPageState extends State { + bool value = false; + bool enabled = true; + final focusNode = FocusNode(); + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Checkbox', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyBoolProperty( + label: 'Focused', + value: focusNode.hasFocus, + onChanged: enabled + ? (value) { + setState(() { + if (value) { + focusNode.requestFocus(); + } else { + focusNode.unfocus(); + } + }); + } + : null, + ), + ], + children: [ + ShadCheckbox( + value: value, + focusNode: focusNode, + onChanged: (v) => setState(() => value = v), + enabled: enabled, + label: const Text('Accept terms and conditions'), + sublabel: const Text( + 'You agree to our Terms of Service and Privacy Policy.', + ), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'dart:convert'; + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class CheckboxFormFieldPage extends StatefulWidget { + const CheckboxFormFieldPage({super.key}); + + @override + State createState() => _CheckboxFormFieldPageState(); +} + +class _CheckboxFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + bool initialValue = false; + Map formValue = {}; + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + initialValue: {'terms': initialValue}, + child: BaseScaffold( + appBarTitle: 'CheckboxFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + MyBoolProperty( + label: 'Form Initial Value', + value: initialValue, + onChanged: (value) { + formKey.currentState!.setFieldValue('terms', value); + setState(() { + initialValue = value; + }); + }, + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + ShadCheckboxFormField( + id: 'terms', + initialValue: initialValue, + inputLabel: const Text('I accept the terms and conditions'), + onChanged: print, + inputSublabel: const Text( + 'You agree to our Terms and Conditions', + ), + validator: (v) { + if (!v) { + return 'You must accept the terms and conditions'; + } + return null; + }, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + const JsonEncoder.withIndent( + ' ', + ).convert(formValue), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/context-menu.md b/.claude/skills/shadcn-ui-flutter/components/context-menu.md new file mode 100644 index 00000000..9186b547 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/context-menu.md @@ -0,0 +1,195 @@ +# Context Menu + +Displays a menu to the user — such as a set of actions or functions — triggered by a mouse right-click. + + + +```dart + + + +class ContextMenuPage extends StatelessWidget { + const ContextMenuPage({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return Scaffold( + body: Padding( + padding: const EdgeInsets.all(16), + child: ShadContextMenuRegion( + constraints: const BoxConstraints(minWidth: 300), + items: [ + const ShadContextMenuItem.inset( + child: Text('Back'), + ), + const ShadContextMenuItem.inset( + enabled: false, + child: Text('Forward'), + ), + const ShadContextMenuItem.inset( + child: Text('Reload'), + ), + const ShadContextMenuItem.inset( + trailing: Icon(LucideIcons.chevronRight), + items: [ + ShadContextMenuItem( + child: Text('Save Page As...'), + ), + ShadContextMenuItem( + child: Text('Create Shortcut...'), + ), + ShadContextMenuItem( + child: Text('Name Window...'), + ), + Divider(height: 8), + ShadContextMenuItem( + child: Text('Developer Tools'), + ), + ], + child: Text('More Tools'), + ), + const Divider(height: 8), + const ShadContextMenuItem( + leading: Icon(LucideIcons.check), + child: Text('Show Bookmarks Bar'), + ), + const ShadContextMenuItem.inset(child: Text('Show Full URLs')), + const Divider(height: 8), + Padding( + padding: const EdgeInsets.fromLTRB(36, 8, 8, 8), + child: Text('People', style: theme.textTheme.small), + ), + const Divider(height: 8), + ShadContextMenuItem( + leading: SizedBox.square( + dimension: 16, + child: Center( + child: Container( + width: 8, + height: 8, + decoration: BoxDecoration( + color: theme.colorScheme.foreground, + shape: BoxShape.circle, + ), + ), + ), + ), + child: const Text('Pedro Duarte'), + ), + const ShadContextMenuItem.inset(child: Text('Colm Tuite')), + ], + child: Container( + width: 300, + height: 200, + alignment: Alignment.center, + decoration: BoxDecoration( + border: Border.all(color: theme.colorScheme.border), + borderRadius: BorderRadius.circular(8), + ), + child: const Text('Right click here'), + ), + ), + ), + ); + } +} +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class ContextMenuPage extends StatelessWidget { + const ContextMenuPage({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + const divider = ShadSeparator.horizontal( + margin: EdgeInsets.symmetric(vertical: 4), + ); + return BaseScaffold( + appBarTitle: 'ContextMenu', + children: [ + ShadContextMenuRegion( + constraints: const BoxConstraints(minWidth: 300), + items: [ + const ShadContextMenuItem.inset( + child: Text('Back'), + ), + const ShadContextMenuItem.inset( + enabled: false, + child: Text('Forward'), + ), + const ShadContextMenuItem.inset( + child: Text('Reload'), + ), + const ShadContextMenuItem.inset( + trailing: Icon(LucideIcons.chevronRight), + items: [ + ShadContextMenuItem( + child: Text('Save Page As...'), + ), + ShadContextMenuItem( + child: Text('Create Shortcut...'), + ), + ShadContextMenuItem( + child: Text('Name Window...'), + ), + divider, + ShadContextMenuItem( + child: Text('Developer Tools'), + ), + ], + child: Text('More Tools'), + ), + divider, + const ShadContextMenuItem( + leading: Icon(LucideIcons.check), + child: Text('Show Bookmarks Bar'), + ), + const ShadContextMenuItem.inset(child: Text('Show Full URLs')), + divider, + Padding( + padding: const EdgeInsets.fromLTRB(36, 8, 8, 8), + child: Text('People', style: theme.textTheme.small), + ), + divider, + ShadContextMenuItem( + leading: SizedBox.square( + dimension: 16, + child: Center( + child: Container( + width: 8, + height: 8, + decoration: BoxDecoration( + color: theme.colorScheme.foreground, + shape: BoxShape.circle, + ), + ), + ), + ), + child: const Text('Pedro Duarte'), + ), + const ShadContextMenuItem.inset(child: Text('Colm Tuite')), + ], + child: Container( + width: 300, + height: 200, + alignment: Alignment.center, + decoration: BoxDecoration( + border: Border.all(color: theme.colorScheme.border), + borderRadius: BorderRadius.circular(8), + ), + child: const Text('Right click here'), + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/date-picker.md b/.claude/skills/shadcn-ui-flutter/components/date-picker.md new file mode 100644 index 00000000..83edf598 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/date-picker.md @@ -0,0 +1,383 @@ +# Date Picker + +A date picker component with range and presets. + + + +```dart +class SingleDatePicker extends StatelessWidget { + const SingleDatePicker({super.key}); + + @override + Widget build(BuildContext context) { + return ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: const ShadDatePicker(), + ); + } +} +``` + + + +## Date Range Picker + + + +```dart +class RangeDatePicker extends StatelessWidget { + const RangeDatePicker({super.key}); + + @override + Widget build(BuildContext context) { + return ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: const ShadDatePicker.range(), + ); + } +} +``` + + + +## With Presets + + + +```dart +const presets = { + 0: 'Today', + 1: 'Tomorrow', + 3: 'In 3 days', + 7: 'In a week', +}; + +class PresetsDatePicker extends StatefulWidget { + const PresetsDatePicker({super.key}); + + @override + State createState() => _PresetsDatePickerState(); +} + +class _PresetsDatePickerState extends State { + final groupId = UniqueKey(); + final today = DateTime.now().startOfDay; + DateTime? selected; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadDatePicker( + // Using the same groupId to keep the date picker popover open when the + // select popover is closed. + groupId: groupId, + header: Padding( + padding: const EdgeInsets.only(bottom: 4), + child: ShadSelect( + groupId: groupId, + minWidth: 276, + placeholder: const Text('Select'), + options: presets.entries + .map((e) => ShadOption(value: e.key, child: Text(e.value))) + .toList(), + selectedOptionBuilder: (context, value) { + return Text(presets[value]!); + }, + onChanged: (value) { + if (value == null) return; + setState(() { + selected = today.add(Duration(days: value)); + }); + }, + ), + ), + selected: selected, + calendarDecoration: theme.calendarTheme.decoration, + popoverPadding: const EdgeInsets.all(4), + ), + ); + } +} +``` + + + +## Form + + + +```dart +ShadDatePickerFormField( + label: const Text('Date of birth'), + onChanged: print, + description: const Text( + 'Your date of birth is used to calculate your age.'), + validator: (v) { + if (v == null) { + return 'A date of birth is required.'; + } + return null; + }, +), +``` + + + +## DateRangePickerFormField + + + +```dart +ShadDateRangePickerFormField( + label: const Text('Range of dates'), + onChanged: print, + description: const Text( + 'Select the range of dates you want to search between.'), + validator: (v) { + if (v == null) return 'A range of dates is required.'; + if (v.start == null) { + return 'The start date is required.'; + } + if (v.end == null) return 'The end date is required.'; + + return null; + }, +), + +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +const presets = { + 0: 'Today', + 1: 'Tomorrow', + 3: 'In 3 days', + 7: 'In a week', +}; + +class DatePickerPage extends StatefulWidget { + const DatePickerPage({super.key}); + + @override + State createState() => _DatePickerPageState(); +} + +class _DatePickerPageState extends State { + bool closeOnSelection = false; + bool allowDeselection = true; + final today = DateTime.now().startOfDay; + final groupId = UniqueKey(); + + DateTime? selected; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'DatePicker', + editable: [ + MyBoolProperty( + label: 'closeOnSelection', + value: closeOnSelection, + onChanged: (value) => setState(() => closeOnSelection = value), + ), + MyBoolProperty( + label: 'allowDeselection', + value: allowDeselection, + onChanged: (value) => setState(() => allowDeselection = value), + ), + ], + children: [ + Text('Single', style: theme.textTheme.h4), + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadDatePicker( + closeOnSelection: closeOnSelection, + allowDeselection: allowDeselection, + ), + ), + const ShadSeparator.horizontal(), + Text('Range', style: theme.textTheme.h4), + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadDatePicker.range( + closeOnSelection: closeOnSelection, + allowDeselection: allowDeselection, + ), + ), + const ShadSeparator.horizontal(), + Text('With Presets', style: theme.textTheme.h4), + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadDatePicker( + // Using the same groupId to keep the date picker popover open when the + // select popover is closed. + groupId: groupId, + header: Padding( + padding: const EdgeInsets.only(bottom: 4), + child: ShadSelect( + groupId: groupId, + minWidth: 276, + placeholder: const Text('Select'), + options: presets.entries + .map((e) => ShadOption(value: e.key, child: Text(e.value))) + .toList(), + selectedOptionBuilder: (context, value) { + return Text(presets[value]!); + }, + onChanged: (value) { + if (value == null) return; + setState(() { + selected = today.add(Duration(days: value)); + }); + }, + ), + ), + closeOnSelection: closeOnSelection, + allowDeselection: allowDeselection, + selected: selected, + calendarDecoration: theme.calendarTheme.decoration, + popoverPadding: const EdgeInsets.all(4), + ), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class DatePickerFormFieldPage extends StatefulWidget { + const DatePickerFormFieldPage({super.key}); + + @override + State createState() => + _DatePickerFormFieldPageState(); +} + +class _DatePickerFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + Map formValue = {}; + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + child: BaseScaffold( + appBarTitle: 'DatePickerFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('Single', style: theme.textTheme.h4), + ShadDatePickerFormField( + id: 'date', + label: const Text('Date of birth'), + onChanged: print, + description: const Text( + 'Your date of birth is used to calculate your age.', + ), + validator: (v) { + if (v == null) { + return 'A date of birth is required.'; + } + return null; + }, + ), + const ShadSeparator.horizontal(), + Text('Range', style: theme.textTheme.h4), + ShadDateRangePickerFormField( + id: 'range-date', + label: const Text('Range of dates'), + onChanged: print, + description: const Text( + 'Select the range of dates you want to search between.', + ), + validator: (v) { + if (v == null) return 'A range of dates is required.'; + if (v.start == null) return 'The start date is required.'; + if (v.end == null) return 'The end date is required.'; + + return null; + }, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + formValue.toString(), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/dialog.md b/.claude/skills/shadcn-ui-flutter/components/dialog.md new file mode 100644 index 00000000..229bb514 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/dialog.md @@ -0,0 +1,240 @@ +# Dialog + +A modal dialog that interrupts the user. + + + +```dart + + +final profile = [ + (title: 'Name', value: 'Alexandru'), + (title: 'Username', value: 'nank1ro'), +]; + +class DialogExample extends StatelessWidget { + const DialogExample({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadButton.outline( + child: const Text('Edit Profile'), + onPressed: () { + showShadDialog( + context: context, + builder: (context) => ShadDialog( + title: const Text('Edit Profile'), + description: const Text( + "Make changes to your profile here. Click save when you're done"), + actions: const [ShadButton(child: Text('Save changes'))], + child: Container( + width: 375, + padding: const EdgeInsets.symmetric(vertical: 20), + child: Column( + mainAxisSize: MainAxisSize.min, + crossAxisAlignment: CrossAxisAlignment.end, + spacing: 16, + children: profile + .map( + (p) => Row( + children: [ + Expanded( + child: Text( + p.title, + textAlign: TextAlign.end, + style: theme.textTheme.small, + ), + ), + const SizedBox(width: 16), + Expanded( + flex: 3, + child: ShadInput(initialValue: p.value), + ), + ], + ), + ).toList(), + ), + ), + ), + ); + }, + ); + } +} +``` + + + +## Alert + + + +```dart +class DialogExample extends StatelessWidget { + const DialogExample({super.key}); + + @override + Widget build(BuildContext context) { + return ShadButton.outline( + child: const Text('Show Dialog'), + onPressed: () { + showShadDialog( + context: context, + builder: (context) => ShadDialog.alert( + title: const Text('Are you absolutely sure?'), + description: const Padding( + padding: EdgeInsets.only(bottom: 8), + child: Text( + 'This action cannot be undone. This will permanently delete your account and remove your data from our servers.', + ), + ), + actions: [ + ShadButton.outline( + child: const Text('Cancel'), + onPressed: () => Navigator.of(context).pop(false), + ), + ShadButton( + child: const Text('Continue'), + onPressed: () => Navigator.of(context).pop(true), + ), + ], + ), + ); + }, + ); + } +} +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +final profile = [ + (title: 'Name', value: 'Alexandru'), + (title: 'Username', value: 'nank1ro'), +]; + +class DialogPage extends StatefulWidget { + const DialogPage({super.key}); + + @override + State createState() => _DialogPageState(); +} + +class _DialogPageState extends State { + var titlePinned = false; + var descriptionPinned = false; + var actionsPinned = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'Dialog', + editable: [ + MyBoolProperty( + label: 'titlePinned', + value: titlePinned, + onChanged: (v) => setState(() => titlePinned = v), + ), + MyBoolProperty( + label: 'descriptionPinned', + value: descriptionPinned, + onChanged: (v) => setState(() => descriptionPinned = v), + ), + MyBoolProperty( + label: 'actionsPinned', + value: actionsPinned, + onChanged: (v) => setState(() => actionsPinned = v), + ), + ], + children: [ + ShadButton.outline( + child: const Text('Edit Profile'), + onPressed: () { + showShadDialog( + context: context, + builder: (context) => ShadDialog( + title: const Text('Edit Profile'), + description: const Text( + "Make changes to your profile here. Click save when you're done", + ), + actions: const [ShadButton(child: Text('Save changes'))], + titlePinned: titlePinned, + descriptionPinned: descriptionPinned, + actionsPinned: actionsPinned, + crossAxisAlignment: CrossAxisAlignment.stretch, + child: Container( + width: 375, + padding: const EdgeInsets.symmetric(vertical: 20), + child: Column( + mainAxisSize: MainAxisSize.min, + crossAxisAlignment: CrossAxisAlignment.end, + spacing: 16, + children: profile + .map( + (p) => Row( + children: [ + Expanded( + child: Text( + p.title, + textAlign: TextAlign.end, + style: theme.textTheme.small, + ), + ), + const SizedBox(width: 16), + Expanded( + flex: 3, + child: ShadInput(initialValue: p.value), + ), + ], + ), + ) + .toList(), + ), + ), + ), + ); + }, + ), + ShadButton.outline( + child: const Text('Show Dialog'), + onPressed: () { + showShadDialog( + context: context, + builder: (context) => ShadDialog.alert( + title: const Text('Are you absolutely sure?'), + titlePinned: titlePinned, + descriptionPinned: descriptionPinned, + actionsPinned: actionsPinned, + description: const Padding( + padding: EdgeInsets.only(bottom: 8), + child: Text( + 'This action cannot be undone. This will permanently delete your account and remove your data from our servers.', + ), + ), + actions: [ + ShadButton.outline( + child: const Text('Cancel'), + onPressed: () => Navigator.of(context).pop(false), + ), + ShadButton( + child: const Text('Continue'), + onPressed: () => Navigator.of(context).pop(true), + ), + ], + ), + ); + }, + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/form.md b/.claude/skills/shadcn-ui-flutter/components/form.md new file mode 100644 index 00000000..65c1331b --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/form.md @@ -0,0 +1,292 @@ +# Form + +Builds a form with validation and easy access to form fields values. + +The benefits of using `ShadForm` over managing form fields individually are: +- Centralized form state management. +- Easy access to all form field values as a single `Map`. +- No need to manage individual controllers for each form field. + + + +```dart +class FormPage extends StatefulWidget { + const FormPage({ + super.key, + }); + + @override + State createState() => _FormPageState(); +} + +class _FormPageState extends State { + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + return Scaffold( + body: Center( + child: ShadForm( + key: formKey, + child: ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + mainAxisSize: MainAxisSize.min, + children: [ + ShadInputFormField( + id: 'username', + label: const Text('Username'), + placeholder: const Text('Enter your username'), + description: const Text('This is your public display name.'), + validator: (v) { + if (v.length < 2) { + return 'Username must be at least 2 characters.'; + } + return null; + }, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + if (formKey.currentState!.saveAndValidate()) { + print( + 'validation succeeded with ${formKey.currentState!.value}'); + } else { + print('validation failed'); + } + }, + ), + ], + ), + ), + ), + ), + ); + } +} +``` + + + +## Initial form value + +You can set the initial form value by passing a `Map` to the `initialValue` property of the `ShadForm` widget. + +```dart {2-5} +ShadForm( + initialValue: { + 'username': 'john_doe', + 'email': 'john_doe@example.com' + }, + child: // Your form fields here +) +``` + +All form fields with matching `id`s will be initialized with the corresponding values from the `initialValue` map. +Unless they have their own `initialValue` set. + +## Get the form value + +You can get the current form value by accessing the `value` property of `ShadFormState` using a `GlobalKey`. + +```dart {8-10} +final formKey = GlobalKey(); + +// Your Form widget +ShadForm( + key: formKey, +), + +// To get the form value +final formValue = formKey.currentState!.value; // Returns a Map with the form field values +``` + +You typically need this after getting a successful value from the `saveAndValidate` method, for example: +```dart {4-8} +ShadButton( + child: const Text('Submit'), + onPressed: () { + final formState = formKey.currentState!; + // The form is not valid, return early + if (!formState.saveAndValidate()) return; + // The form is valid, print the form value + print('Form value: ${formState.value}'); + }, +), +``` + +## Manipulate single form field value + +You can set or update the value of specific form fields using the `setFieldValue` method of `ShadFormState`. + +```dart {8-10} +final formKey = GlobalKey(); + +// Your Form widget +ShadForm( + key: formKey, +), + +// To set or update a specific field value +formKey.currentState!.setFieldValue('username', 'new_username'); +``` + +If you don't want to notify the field about the value change, you can pass `notifyField: false` as argument. +This would only update the form value without updating the field UI. + +## Manipulate entire form value + +You can set or update the entire form value using the `setValue` method of `ShadFormState`. + +```dart {8-12} +final formKey = GlobalKey(); + +// Your Form widget +ShadForm( + key: formKey, +), + +// To set or update the entire form value +formKey.currentState!.setValue({ + 'username': 'new_username', + 'email': 'example@email.com' +}); +``` + +If you don't want to notify the fields about the value change, you can pass `notifyFields: false` as argument. +This would only update the form value without updating the fields UI. + +## Value transformers + +You can use value transformers to convert the initial value from the form to the field value and vice versa. + +### fromValueTransformer + +If your `ShadForm` has an initial value like this one `{'date': '2024-02-01'}` and you need to convert the string value to a `DateTime` object for a `ShadDatePickerFormField`: +```dart {3} +ShadDatePickerFormField( + id: 'date', + fromValueTransformer: (value) => DateTime.tryParse(value ?? ''), +), +``` + +Vice versa, you can use the `toValueTransformer` parameter to convert the field value back to the form value. +```dart {3-5} +ShadDatePickerFormField( + id: 'date', + toValueTransformer: (date) => date == null + ? null + : DateFormat('yyyy-MM-dd').format(date), +), +``` + +In this way, the form field can work with `DateTime` objects while the form value remains a `String` both as initial value (input) and when getting the form value (output). + +## Dot notation for nested values + +By default, `ShadForm` supports dot notation in field IDs to automatically create nested map structures. This makes it easier to work with complex, hierarchical form data. + +### How it works + +When you use field IDs with dots (like `user.email` or `profile.settings.theme`), the form automatically converts them into nested maps: + +```dart +ShadForm( + child: Column( + children: [ + ShadInputFormField( + id: 'user.name', + label: const Text('Name'), + ), + ShadInputFormField( + id: 'user.email', + label: const Text('Email'), + ), + ShadInputFormField( + id: 'user.age', + label: const Text('Age'), + ), + ], + ), +) +``` + +When you retrieve the form value, it will be structured as: +```dart +{ + 'user': { + 'name': 'John Doe', + 'email': 'john@example.com', + 'age': '30' + } +} +``` + +### Initial values with nested structure + +The `initialValue` should be provided as a nested map structure (not using dot notation): + +```dart +ShadForm( + initialValue: { + 'user': { + 'name': 'John Doe', + 'email': 'john@example.com', + }, + }, + child: // Your form fields with dot notation IDs +) +``` + +The form will automatically extract values from the nested structure based on the field IDs. For example, a field with `id: 'user.name'` will get the value from `initialValue['user']['name']`. + +### Customizing the separator + +If you prefer a different separator, you can customize it using the `fieldIdSeparator` parameter: + +```dart {2} +ShadForm( + fieldIdSeparator: '/', + child: Column( + children: [ + ShadInputFormField( + id: 'user/name', // Using '/' instead of '.' + label: const Text('Name'), + ), + ], + ), +) +``` + +### Disabling dot notation + +If you want to use dots in your field IDs without creating nested structures, you can disable the feature by setting `fieldIdSeparator` to `null`: + +```dart {2} +ShadForm( + fieldIdSeparator: null, + child: Column( + children: [ + ShadInputFormField( + id: 'user.email', // This will remain as a flat key + label: const Text('Email'), + ), + ], + ), +) +``` + +## Examples + +See the following links for more examples on how to use the `ShadForm` component with other components: + +- [Checkbox](../checkbox#form) +- [Switch](../switch#form) +- [Input](../input#form) +- [Select](../select#form) +- [RadioGroup](../radio-group#form) +- [DatePicker](../date-picker#form) +- [TimePicker](../time-picker#form) + diff --git a/.claude/skills/shadcn-ui-flutter/components/icon-button.md b/.claude/skills/shadcn-ui-flutter/components/icon-button.md new file mode 100644 index 00000000..37ec0aa5 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/icon-button.md @@ -0,0 +1,201 @@ +# IconButton + +Displays an icon button or a component that looks like a button with an icon. + +## Primary + + + +```dart +ShadIconButton( + onPressed: () => print('Primary'), + icon: const Icon(LucideIcons.rocket), +) +``` + + + +## Secondary + + + +```dart +ShadIconButton.secondary( + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Secondary'), +) +``` + + + +## Destructive + + + +```dart +ShadIconButton.destructive( + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Destructive'), +) +``` + + + +## Outline + + + +```dart +ShadIconButton.outline( + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Outline'), +) +``` + + + +## Ghost + + + +```dart +ShadIconButton.ghost( + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Ghost'), +) +``` + + + +## Loading + + + +```dart +ShadIconButton( + icon: SizedBox.square( + dimension: 16, + child: CircularProgressIndicator( + strokeWidth: 2, + color: ShadTheme.of(context).colorScheme.primaryForeground, + ), + ), +) +``` + + + +## Gradient and Shadow + + + +```dart +ShadIconButton( + gradient: const LinearGradient(colors: [ + Colors.cyan, + Colors.indigo, + ]), + shadows: [ + BoxShadow( + color: Colors.blue.withValues(alpha: .4), + spreadRadius: 4, + blurRadius: 10, + offset: const Offset(0, 2), + ), + ], + icon: const Icon(LucideIcons.rocket), +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class IconButtonPage extends StatefulWidget { + const IconButtonPage({super.key}); + + @override + State createState() => _IconButtonPageState(); +} + +class _IconButtonPageState extends State { + var enabled = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return FocusTraversalGroup( + policy: WidgetOrderTraversalPolicy(), + child: BaseScaffold( + appBarTitle: 'IconButton', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + ], + children: [ + ShadIconButton( + enabled: enabled, + onPressed: () => print('Primary'), + icon: const Icon(LucideIcons.rocket), + ), + ShadIconButton.secondary( + enabled: enabled, + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Secondary'), + ), + ShadIconButton.destructive( + enabled: enabled, + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Destructive'), + ), + ShadIconButton.outline( + enabled: enabled, + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Outline'), + ), + ShadIconButton.ghost( + enabled: enabled, + icon: const Icon(LucideIcons.rocket), + onPressed: () => print('Ghost'), + ), + ShadIconButton( + enabled: enabled, + gradient: const LinearGradient( + colors: [ + Colors.cyan, + Colors.indigo, + ], + ), + shadows: [ + BoxShadow( + color: Colors.blue.withValues(alpha: .4), + spreadRadius: 4, + blurRadius: 10, + offset: const Offset(0, 2), + ), + ], + icon: const Icon(LucideIcons.rocket), + ), + ShadIconButton( + enabled: enabled, + icon: SizedBox.square( + dimension: 16, + child: CircularProgressIndicator( + strokeWidth: 2, + color: theme.colorScheme.primaryForeground, + ), + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/input-otp.md b/.claude/skills/shadcn-ui-flutter/components/input-otp.md new file mode 100644 index 00000000..a914baf7 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/input-otp.md @@ -0,0 +1,334 @@ +# InputOTP + +Accessible one-time password component with copy paste functionality. + + + +```dart +ShadInputOTP( + onChanged: (v) => print('OTP: $v'), + maxLength: 6, + children: const [ + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + Icon(size: 24, LucideIcons.dot), + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + ], +) +``` + + + +## InputFormatters + +Using InputFormatters you can restrict the input characters. +The example below shows how to restrict the input to only numbers. + + + +```dart +ShadInputOTP( + onChanged: (v) => print('OTP: $v'), + maxLength: 4, + keyboardType: TextInputType.number, + inputFormatters: [ + FilteringTextInputFormatter.digitsOnly, + ], + children: const [ + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + ], +) +``` + + + +See also `UpperCaseTextInputFormatter` and `LowerCaseTextInputFormatter` which are provided by the package. + +## Form + + + +```dart +ShadInputOTPFormField( + id: 'otp', + maxLength: 6, + label: const Text('OTP'), + description: const Text('Enter your OTP.'), + validator: (v) { + if (v.contains(' ')) { + return 'Fill the whole OTP code'; + } + return null; + }, + children: const [ + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + Icon(size: 24, LucideIcons.dot), + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + ], +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/services.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class InputOTPPage extends StatefulWidget { + const InputOTPPage({super.key}); + + @override + State createState() => _InputOTPPageState(); +} + +class _InputOTPPageState extends State { + var enabled = true; + var uppercase = true; + var digitsOnly = false; + var jumpToNextWhenFilled = true; + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Input OTP', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyBoolProperty( + label: 'Uppercase', + value: uppercase, + enabled: !digitsOnly, + onChanged: (value) => setState(() => uppercase = value), + ), + MyBoolProperty( + label: 'Digits only', + value: digitsOnly, + enabled: !uppercase, + onChanged: (value) => setState(() => digitsOnly = value), + ), + MyBoolProperty( + label: 'Jump to next when filled', + value: jumpToNextWhenFilled, + onChanged: (value) => setState(() => jumpToNextWhenFilled = value), + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadInputOTP( + onChanged: (v) => print('OTP: $v'), + maxLength: 6, + enabled: enabled, + jumpToNextWhenFilled: jumpToNextWhenFilled, + keyboardType: digitsOnly ? TextInputType.number : null, + inputFormatters: [ + if (digitsOnly) FilteringTextInputFormatter.digitsOnly, + if (uppercase) const UpperCaseTextInputFormatter(), + ], + children: const [ + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + Icon(LucideIcons.dot), + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + ], + ), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'dart:convert'; + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:example/common/properties/string_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class InputOTPFormFieldPage extends StatefulWidget { + const InputOTPFormFieldPage({super.key}); + + @override + State createState() => _InputOTPFormFieldPageState(); +} + +class _InputOTPFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + String? initialValue; + Map formValue = {}; + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + initialValue: {if (initialValue != null) 'otp': initialValue}, + child: BaseScaffold( + appBarTitle: 'InputFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + MyStringProperty( + label: 'Form Initial Value', + initialValue: initialValue, + placeholder: const Text('OTP initial value'), + onChanged: (value) { + setState(() { + value.isEmpty ? initialValue = null : initialValue = value; + }); + // Reset the form + WidgetsBinding.instance.addPostFrameCallback((timeStamp) { + formKey.currentState!.reset(); + }); + }, + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + ShadInputOTPFormField( + id: 'otp', + maxLength: 6, + enabled: enabled, + label: const Text('OTP'), + description: const Text('Enter your OTP.'), + validator: (v) { + if (v.contains(' ')) { + return 'Fill the whole OTP code'; + } + return null; + }, + children: const [ + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + Icon(LucideIcons.dot), + ShadInputOTPGroup( + children: [ + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ShadInputOTPSlot(), + ], + ), + ], + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + const JsonEncoder.withIndent( + ' ', + ).convert(formValue), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/input.md b/.claude/skills/shadcn-ui-flutter/components/input.md new file mode 100644 index 00000000..a1107704 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/input.md @@ -0,0 +1,283 @@ +# Input + +Displays a form input field or a component that looks like an input field. + + + +```dart +ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 320), + child: const ShadInput( + placeholder: Text('Email'), + keyboardType: TextInputType.emailAddress, + ), +), +``` + + + +## With leading and trailing + + + +```dart +class PasswordInput extends StatefulWidget { + const PasswordInput({super.key}); + + @override + State createState() => _PasswordInputState(); +} + +class _PasswordInputState extends State { + bool obscure = true; + + @override + Widget build(BuildContext context) { + return ShadInput( + placeholder: const Text('Password'), + obscureText: obscure, + leading: Icon(LucideIcons.lock), + trailing: SizedBox.square( + dimension: 24, + child: OverflowBox( + maxWidth: 28, + maxHeight: 28, + child: ShadIconButton( + iconSize: 20, + padding: EdgeInsets.all(2), + icon: Icon(obscure ? LucideIcons.eyeOff : LucideIcons.eye), + onPressed: () { + setState(() => obscure = !obscure); + }, + ), + ), + ), + ); + } +} +``` + + + +## Form + + + +```dart +ShadInputFormField( + id: 'username', + label: const Text('Username'), + placeholder: const Text('Enter your username'), + description: + const Text('This is your public display name.'), + validator: (v) { + if (v.length < 2) { + return 'Username must be at least 2 characters.'; + } + return null; + }, +), +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class InputPage extends StatefulWidget { + const InputPage({super.key}); + + @override + State createState() => _InputPageState(); +} + +class _InputPageState extends State { + bool enabled = true; + bool obscure = true; + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Input', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyBoolProperty( + label: 'Obscure', + value: obscure, + onChanged: (value) => setState(() => obscure = value), + ), + ], + children: [ + ShadInput( + placeholder: const Text('Email'), + enabled: enabled, + keyboardType: TextInputType.emailAddress, + ), + ShadInput( + placeholder: const Text('Password'), + enabled: enabled, + obscureText: obscure, + leading: Icon(LucideIcons.lock), + trailing: SizedBox.square( + dimension: 24, + child: OverflowBox( + maxWidth: 28, + maxHeight: 28, + child: ShadIconButton( + iconSize: 20, + padding: EdgeInsets.all(2), + icon: Icon(obscure ? LucideIcons.eyeOff : LucideIcons.eye), + onPressed: () { + setState(() => obscure = !obscure); + }, + ), + ), + ), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +import 'dart:convert'; +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:example/common/properties/string_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class InputFormFieldPage extends StatefulWidget { + const InputFormFieldPage({super.key}); + + @override + State createState() => _InputFormFieldPageState(); +} + +class _InputFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + String? initialValue; + Map formValue = {}; + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + initialValue: { + if (initialValue != null) 'username': initialValue, + 'profile': {'age': 18}, + }, + child: BaseScaffold( + appBarTitle: 'InputFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + MyStringProperty( + label: 'Form Initial Value', + initialValue: initialValue, + placeholder: const Text('Name'), + onChanged: (value) { + formKey.currentState!.setFieldValue('username', value); + }, + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + spacing: 16, + children: [ + ShadInputFormField( + id: 'username', + leading: const Icon(LucideIcons.user), + label: const Text('Username'), + placeholder: const Text('Enter your username'), + description: const Text('This is your public display name.'), + validator: (v) { + if (v.length < 2) { + return 'Username must be at least 2 characters.'; + } + return null; + }, + ), + ShadInputFormField( + id: 'profile.age', + fromValueTransformer: (v) => v?.toString(), + toValueTransformer: (String? v) => int.tryParse(v ?? ''), + keyboardType: TextInputType.number, + label: const Text('Age (dot notation)'), + placeholder: const Text('Enter your age'), + description: const Text( + 'This field uses dot notation: profile.age', + ), + ), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + const JsonEncoder.withIndent( + ' ', + ).convert(formValue), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/menubar.md b/.claude/skills/shadcn-ui-flutter/components/menubar.md new file mode 100644 index 00000000..dc672cf8 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/menubar.md @@ -0,0 +1,262 @@ +# Menubar + +A visually persistent menu common in desktop applications that provides quick access to a consistent set of commands. + + + +```dart +class MenubarExample extends StatelessWidget { + const MenubarExample({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + final square = SizedBox.square( + dimension: 16, + child: Center( + child: SizedBox.square( + dimension: 8, + child: DecoratedBox( + decoration: BoxDecoration( + color: theme.colorScheme.foreground, + shape: BoxShape.circle, + ), + ), + ), + ), + ); + final divider = ShadSeparator.horizontal( + margin: const EdgeInsets.symmetric(vertical: 4), + color: theme.colorScheme.muted, + ); + return ShadMenubar( + items: [ + ShadMenubarItem( + items: [ + const ShadContextMenuItem(child: Text('New Tab')), + const ShadContextMenuItem(child: Text('New Window')), + const ShadContextMenuItem( + enabled: false, + child: Text('New Incognito Window'), + ), + divider, + const ShadContextMenuItem( + trailing: Icon(LucideIcons.chevronRight), + items: [ + ShadContextMenuItem(child: Text('Email Link')), + ShadContextMenuItem(child: Text('Messages')), + ShadContextMenuItem(child: Text('Notes')), + ], + child: Text('Share'), + ), + divider, + const ShadContextMenuItem(child: Text('Print...')), + ], + child: const Text('File'), + ), + ShadMenubarItem( + items: [ + const ShadContextMenuItem(child: Text('Undo')), + const ShadContextMenuItem(child: Text('Redo')), + divider, + ShadContextMenuItem( + trailing: const Icon(LucideIcons.chevronRight), + items: [ + const ShadContextMenuItem(child: Text('Search the web')), + divider, + const ShadContextMenuItem(child: Text('Find...')), + const ShadContextMenuItem(child: Text('Find Next')), + const ShadContextMenuItem(child: Text('Find Previous')), + ], + child: const Text('Find'), + ), + divider, + const ShadContextMenuItem(child: Text('Cut')), + const ShadContextMenuItem(child: Text('Copy')), + const ShadContextMenuItem(child: Text('Paste')), + ], + child: const Text('Edit'), + ), + ShadMenubarItem( + items: [ + const ShadContextMenuItem.inset( + child: Text('Always Show Bookmarks Bar'), + ), + const ShadContextMenuItem( + leading: Icon(LucideIcons.check), + child: Text('Always Show Full URLs'), + ), + divider, + const ShadContextMenuItem.inset(child: Text('Reload')), + const ShadContextMenuItem.inset( + enabled: false, child: Text('Force Reload')), + divider, + const ShadContextMenuItem.inset( + child: Text('Toggle Full Screen'), + ), + divider, + const ShadContextMenuItem.inset(child: Text('Hide Sidebar')), + ], + child: const Text('View'), + ), + ShadMenubarItem(items: [ + const ShadContextMenuItem.inset(child: Text('Andy')), + ShadContextMenuItem(leading: square, child: const Text('Benoit')), + const ShadContextMenuItem.inset(child: Text('Luis')), + divider, + const ShadContextMenuItem.inset(child: Text('Edit...')), + divider, + const ShadContextMenuItem.inset(child: Text('Add Profile...')), + ], child: const Text('Profiles')), + ], + ); + } +} +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class MenubarPage extends StatefulWidget { + const MenubarPage({super.key}); + + @override + State createState() => _MenubarPageState(); +} + +class _MenubarPageState extends State { + var selectOnHover = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + final square = SizedBox.square( + dimension: 16, + child: Center( + child: SizedBox.square( + dimension: 8, + child: DecoratedBox( + decoration: BoxDecoration( + color: theme.colorScheme.foreground, + shape: BoxShape.circle, + ), + ), + ), + ), + ); + final divider = ShadSeparator.horizontal( + margin: const EdgeInsets.symmetric(vertical: 4), + color: theme.colorScheme.muted, + ); + + return BaseScaffold( + appBarTitle: 'Menubar', + editable: [ + MyBoolProperty( + label: 'Select on hover', + value: selectOnHover, + onChanged: (value) => setState(() => selectOnHover = value), + ), + ], + children: [ + ShadMenubar( + selectOnHover: selectOnHover, + items: [ + ShadMenubarItem( + items: [ + const ShadContextMenuItem(child: Text('New Tab')), + const ShadContextMenuItem(child: Text('New Window')), + const ShadContextMenuItem( + enabled: false, + child: Text('New Incognito Window'), + ), + divider, + const ShadContextMenuItem( + trailing: Icon(LucideIcons.chevronRight), + items: [ + ShadContextMenuItem(child: Text('Email Link')), + ShadContextMenuItem(child: Text('Messages')), + ShadContextMenuItem(child: Text('Notes')), + ], + child: Text('Share'), + ), + divider, + const ShadContextMenuItem(child: Text('Print...')), + ], + child: const Text('File'), + ), + ShadMenubarItem( + items: [ + const ShadContextMenuItem(child: Text('Undo')), + const ShadContextMenuItem(child: Text('Redo')), + divider, + ShadContextMenuItem( + trailing: const Icon(LucideIcons.chevronRight), + items: [ + const ShadContextMenuItem(child: Text('Search the web')), + divider, + const ShadContextMenuItem(child: Text('Find...')), + const ShadContextMenuItem(child: Text('Find Next')), + const ShadContextMenuItem(child: Text('Find Previous')), + ], + child: const Text('Find'), + ), + divider, + const ShadContextMenuItem(child: Text('Cut')), + const ShadContextMenuItem(child: Text('Copy')), + const ShadContextMenuItem(child: Text('Paste')), + ], + child: const Text('Edit'), + ), + ShadMenubarItem( + items: [ + const ShadContextMenuItem.inset( + child: Text('Always Show Bookmarks Bar'), + ), + const ShadContextMenuItem( + leading: Icon(LucideIcons.check), + child: Text('Always Show Full URLs'), + ), + divider, + const ShadContextMenuItem.inset(child: Text('Reload')), + const ShadContextMenuItem.inset( + enabled: false, + child: Text('Force Reload'), + ), + divider, + const ShadContextMenuItem.inset( + child: Text('Toggle Full Screen'), + ), + divider, + const ShadContextMenuItem.inset(child: Text('Hide Sidebar')), + ], + child: const Text('View'), + ), + ShadMenubarItem( + items: [ + const ShadContextMenuItem.inset(child: Text('Andy')), + ShadContextMenuItem( + leading: square, + child: const Text('Benoit'), + ), + const ShadContextMenuItem.inset(child: Text('Luis')), + divider, + const ShadContextMenuItem.inset(child: Text('Edit...')), + divider, + const ShadContextMenuItem.inset(child: Text('Add Profile...')), + ], + child: const Text('Profiles'), + ), + ], + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/popover.md b/.claude/skills/shadcn-ui-flutter/components/popover.md new file mode 100644 index 00000000..acfe1646 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/popover.md @@ -0,0 +1,178 @@ +# Popover + +Displays rich content in a portal, triggered by a button. + + + +```dart + + + + +class PopoverPage extends StatefulWidget { + const PopoverPage({super.key}); + + @override + State createState() => _PopoverPageState(); +} + +class _PopoverPageState extends State { + final popoverController = ShadPopoverController(); + + final List<({String name, String initialValue})> layer = [ + (name: 'Width', initialValue: '100%'), + (name: 'Max. width', initialValue: '300px'), + (name: 'Height', initialValue: '25px'), + (name: 'Max. height', initialValue: 'none'), + ]; + + @override + void dispose() { + popoverController.dispose(); + super.dispose(); + } + + @override + Widget build(BuildContext context) { + final textTheme = ShadTheme.of(context).textTheme; + return Scaffold( + body: Center( + child: ShadPopover( + controller: popoverController, + popover: (context) => SizedBox( + width: 288, + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + mainAxisSize: MainAxisSize.min, + children: [ + Text( + 'Dimensions', + style: textTheme.h4, + ), + Text( + 'Set the dimensions for the layer.', + style: textTheme.p, + ), + const SizedBox(height: 4), + ...layer + .map( + (e) => Row( + mainAxisAlignment: MainAxisAlignment.start, + children: [ + Expanded( + child: Text( + e.name, + textAlign: TextAlign.start, + )), + Expanded( + flex: 2, + child: ShadInput( + initialValue: e.initialValue, + ), + ) + ], + ), + ) + .separatedBy(const SizedBox(height: 8)), + ], + ), + ), + child: ShadButton.outline( + onPressed: popoverController.toggle, + child: const Text('Open popover'), + ), + ), + ), + ); + } +} +``` + +## Example +```dart +import 'package:awesome_flutter_extensions/awesome_flutter_extensions.dart'; +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class PopoverPage extends StatefulWidget { + const PopoverPage({super.key}); + + @override + State createState() => _PopoverPageState(); +} + +class _PopoverPageState extends State { + final popoverController = ShadPopoverController(); + + final List<({String name, String initialValue})> layer = [ + (name: 'Width', initialValue: '100%'), + (name: 'Max. width', initialValue: '300px'), + (name: 'Height', initialValue: '25px'), + (name: 'Max. height', initialValue: 'none'), + ]; + + @override + void dispose() { + popoverController.dispose(); + super.dispose(); + } + + @override + Widget build(BuildContext context) { + final textTheme = ShadTheme.of(context).textTheme; + return BaseScaffold( + appBarTitle: 'Popover', + children: [ + ShadPopover( + controller: popoverController, + popover: (_) => SizedBox( + width: 288, + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + mainAxisSize: MainAxisSize.min, + children: [ + Text( + 'Dimensions', + style: textTheme.h4, + ), + Text( + 'Set the dimensions for the layer.', + style: textTheme.p, + ), + const SizedBox(height: 4), + ...layer + .map( + (e) => Row( + mainAxisAlignment: MainAxisAlignment.start, + children: [ + Expanded( + child: Text( + e.name, + textAlign: TextAlign.start, + ), + ), + Expanded( + flex: 2, + child: ShadInput( + initialValue: e.initialValue, + ), + ), + ], + ), + ) + .separatedBy(const SizedBox(height: 8)), + ], + ), + ), + child: ShadButton.outline( + onPressed: popoverController.toggle, + child: const Text('Open popover'), + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/progress.md b/.claude/skills/shadcn-ui-flutter/components/progress.md new file mode 100644 index 00000000..aa6ae130 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/progress.md @@ -0,0 +1,77 @@ +# Progress + +Displays an indicator showing the completion progress of a task, typically displayed as a progress bar. + +## Determinate + + + ```dart +ConstrainedBox( + constraints: BoxConstraints( + maxWidth: MediaQuery.sizeOf(context).width * 0.6, + ), + child: const ShadProgress(value: 0.5), +), +``` + + + +## Indeterminate + + ```dart +ConstrainedBox( + constraints: BoxConstraints( + maxWidth: MediaQuery.sizeOf(context).width * 0.6, + ), + child: const ShadProgress(), +), +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class ProgressPage extends StatefulWidget { + const ProgressPage({super.key}); + + @override + State createState() => _ProgressPageState(); +} + +class _ProgressPageState extends State { + var value = 50; + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Progress', + children: [ + Text('Determinate Progress'), + ShadProgress( + value: value / 100, + ), + Row( + mainAxisAlignment: MainAxisAlignment.center, + children: [ + ShadIconButton.ghost( + enabled: value != 0, + onPressed: () => setState(() => value -= 10), + icon: const Icon(Icons.remove), + ), + ShadIconButton.ghost( + enabled: value != 100, + onPressed: () => setState(() => value += 10), + icon: const Icon(Icons.add), + ), + ], + ), + Text('Indeterminate Progress'), + ShadProgress(), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/radio-group.md b/.claude/skills/shadcn-ui-flutter/components/radio-group.md new file mode 100644 index 00000000..3e5ab5b3 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/radio-group.md @@ -0,0 +1,284 @@ +# RadioGroup + +A set of checkable buttons—known as radio buttons—where no more than one of the buttons can be checked at a time. + + + +```dart +ShadRadioGroup( + items: [ + ShadRadio( + label: Text('Default'), + value: 'default', + ), + ShadRadio( + label: Text('Comfortable'), + value: 'comfortable', + ), + ShadRadio( + label: Text('Nothing'), + value: 'nothing', + ), + ], +), +``` + + + +## Form + + + +```dart +enum NotifyAbout { + all, + mentions, + nothing; + + String get message { + return switch (this) { + all => 'All new messages', + mentions => 'Direct messages and mentions', + nothing => 'Nothing', + }; + } +} + +ShadRadioGroupFormField( + label: const Text('Notify me about'), + items: NotifyAbout.values.map( + (e) => ShadRadio( + value: e, + label: Text(e.message), + ), + ), + validator: (v) { + if (v == null) { + return 'You need to select a notification type.'; + } + return null; + }, +), +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +enum NotifyAbout { + all, + mentions, + nothing; + + String get message { + return switch (this) { + all => 'All new messages', + mentions => 'Direct messages and mentions', + nothing => 'Nothing', + }; + } +} + +class RadioPage extends StatefulWidget { + const RadioPage({super.key}); + + @override + State createState() => _RadioPageState(); +} + +class _RadioPageState extends State { + NotifyAbout? value; + bool enabled = true; + Axis axis = Axis.vertical; + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'RadioGroup', + crossAxisAlignment: CrossAxisAlignment.start, + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'Axis', + value: axis, + onChanged: (value) { + if (value != null) { + setState(() => axis = value); + } + }, + values: Axis.values, + ), + ], + children: [ + ShadRadioGroup( + enabled: enabled, + initialValue: value, + onChanged: (v) { + print('onChange $v'); + }, + axis: axis, + items: NotifyAbout.values.map( + (e) => ShadRadio( + value: e, + label: Text(e.message), + ), + ), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'dart:convert'; + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +enum NotifyAbout { + all, + mentions, + nothing; + + String get message { + return switch (this) { + all => 'All new messages', + mentions => 'Direct messages and mentions', + nothing => 'Nothing', + }; + } +} + +class RadioGroupFormFieldPage extends StatefulWidget { + const RadioGroupFormFieldPage({super.key}); + + @override + State createState() => + _RadioGroupFormFieldPageState(); +} + +class _RadioGroupFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + NotifyAbout? initialValue; + Map formValue = {}; + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + initialValue: {'notify': initialValue}, + child: BaseScaffold( + appBarTitle: 'RadioGroupFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + MyEnumProperty( + label: 'Form Initial Value', + value: NotifyAbout.nothing, + values: NotifyAbout.values, + onChanged: (value) { + formKey.currentState!.setFieldValue('notify', value); + }, + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + ShadRadioGroupFormField( + id: 'notify', + enabled: enabled, + initialValue: initialValue, + toValueTransformer: (value) => value?.name, + items: NotifyAbout.values.map( + (e) => ShadRadio( + value: e, + label: Text(e.message), + ), + ), + label: const Text('Notify me about'), + validator: (v) { + if (v == null) { + return 'You need to select a notification type.'; + } + return null; + }, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + const JsonEncoder.withIndent( + ' ', + ).convert(formValue), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/resizable.md b/.claude/skills/shadcn-ui-flutter/components/resizable.md new file mode 100644 index 00000000..b78c67a2 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/resizable.md @@ -0,0 +1,284 @@ +# Resizable + +Resizable panel groups and layouts. + + + +```dart +class BasicResizable extends StatelessWidget { + const BasicResizable({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ConstrainedBox( + constraints: const BoxConstraints(maxHeight: 200), + child: DecoratedBox( + decoration: BoxDecoration( + borderRadius: theme.radius, + border: Border.all( + color: theme.colorScheme.border, + ), + ), + child: ClipRRect( + borderRadius: theme.radius, + child: ShadResizablePanelGroup( + children: [ + ShadResizablePanel( + id: 0, + defaultSize: .5, + minSize: .2, + maxSize: .8, + child: Center( + child: Text('One', style: theme.textTheme.large), + ), + ), + ShadResizablePanel( + id: 1, + defaultSize: .5, + child: ShadResizablePanelGroup( + axis: Axis.vertical, + children: [ + ShadResizablePanel( + id: 0, + defaultSize: .3, + child: Center( + child: Text('Two', style: theme.textTheme.large)), + ), + ShadResizablePanel( + id: 1, + defaultSize: .7, + child: Align( + child: Text('Three', style: theme.textTheme.large)), + ), + ], + ), + ), + ], + ), + ), + ), + ); + } +} +``` + + + +Try resizing a panel, then double-click on the handle to reset to the default size. + + +## Vertical + +Use the `axis` property to change the direction of the resizable panels. + + + +```dart +class VerticalResizable extends StatelessWidget { + const VerticalResizable({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ConstrainedBox( + constraints: const BoxConstraints(maxHeight: 200), + child: DecoratedBox( + decoration: BoxDecoration( + borderRadius: theme.radius, + border: Border.all( + color: theme.colorScheme.border, + ), + ), + child: ClipRRect( + borderRadius: theme.radius, + child: ShadResizablePanelGroup( + axis: Axis.vertical, + children: [ + ShadResizablePanel( + id: 0, + defaultSize: 0.3, + minSize: 0.1, + child: Center( + child: Text('Header', style: theme.textTheme.large), + ), + ), + ShadResizablePanel( + id: 1, + defaultSize: 0.7, + minSize: 0.1, + child: Center( + child: Text('Footer', style: theme.textTheme.large), + ), + ), + ], + ), + ), + ), + ); + } +} +``` + + + +## Handle + +You can show the handle by using the `showHandle` property. + +You can customize it using the `handleIcon` or `handleIconSrc` properties. + + + +```dart +class HandleResizable extends StatelessWidget { + const HandleResizable({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ConstrainedBox( + constraints: const BoxConstraints(maxHeight: 200), + child: DecoratedBox( + decoration: BoxDecoration( + borderRadius: theme.radius, + border: Border.all( + color: theme.colorScheme.border, + ), + ), + child: ClipRRect( + borderRadius: theme.radius, + child: ShadResizablePanelGroup( + showHandle: true, + children: [ + ShadResizablePanel( + id: 0, + defaultSize: .5, + minSize: .2, + child: Center( + child: Text('Sidebar', style: theme.textTheme.large), + ), + ), + ShadResizablePanel( + id: 1, + defaultSize: .5, + minSize: .2, + child: Center( + child: Text('Content', style: theme.textTheme.large), + ), + ), + ], + ), + ), + ), + ); + } +} +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class ResizablePage extends StatefulWidget { + const ResizablePage({super.key}); + + @override + State createState() => _ResizablePageState(); +} + +class _ResizablePageState extends State { + var visible = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'Resizable', + editable: [ + MyBoolProperty( + value: visible, + onChanged: (v) => setState(() => visible = v), + label: 'One Visible', + ), + ], + children: [ + SizedBox( + width: 300, + height: 200, + child: DecoratedBox( + decoration: BoxDecoration( + borderRadius: theme.radius, + border: Border.all( + color: theme.colorScheme.border, + ), + ), + child: ClipRRect( + borderRadius: theme.radius, + child: ShadResizablePanelGroup( + mainAxisSize: MainAxisSize.min, + showHandle: true, + children: [ + if (visible) + ShadResizablePanel( + id: 0, + defaultSize: .5, + minSize: 0.1, + maxSize: 0.8, + child: Container( + color: Colors.red, + alignment: Alignment.center, + child: Text( + 'One', + style: theme.textTheme.large, + ), + ), + ), + ShadResizablePanel( + defaultSize: 0.5, + id: 1, + child: ShadResizablePanelGroup( + axis: Axis.vertical, + showHandle: true, + children: [ + ShadResizablePanel( + id: 0, + defaultSize: 0.4, + child: Container( + color: Colors.blue, + alignment: Alignment.center, + child: Text( + 'Two', + style: theme.textTheme.large, + ), + ), + ), + ShadResizablePanel( + id: 1, + defaultSize: 0.6, + child: Container( + color: Colors.green, + alignment: Alignment.center, + child: Text( + 'Three', + style: theme.textTheme.large, + ), + ), + ), + ], + ), + ), + ], + ), + ), + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/select.md b/.claude/skills/shadcn-ui-flutter/components/select.md new file mode 100644 index 00000000..9c351ac7 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/select.md @@ -0,0 +1,735 @@ +# Select + +Displays a list of options for the user to pick from—triggered by a button. + + + +```dart +final fruits = { + 'apple': 'Apple', + 'banana': 'Banana', + 'blueberry': 'Blueberry', + 'grapes': 'Grapes', + 'pineapple': 'Pineapple', +}; + +class SelectExample extends StatelessWidget { + const SelectExample({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ConstrainedBox( + constraints: const BoxConstraints(minWidth: 180), + child: ShadSelect( + placeholder: const Text('Select a fruit'), + options: [ + Padding( + padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 6), + child: Text( + 'Fruits', + style: theme.textTheme.muted.copyWith( + fontWeight: FontWeight.w600, + color: theme.colorScheme.popoverForeground, + ), + textAlign: TextAlign.start, + ), + ), + ...fruits.entries + .map((e) => ShadOption(value: e.key, child: Text(e.value))), + ], + selectedOptionBuilder: (context, value) => Text(fruits[value]!), + onChanged: print, + ), + ); + } +} +``` + + + +## Scrollable + + + +```dart +final timezones = { + 'North America': { + 'est': 'Eastern Standard Time (EST)', + 'cst': 'Central Standard Time (CST)', + 'mst': 'Mountain Standard Time (MST)', + 'pst': 'Pacific Standard Time (PST)', + 'akst': 'Alaska Standard Time (AKST)', + 'hst': 'Hawaii Standard Time (HST)', + }, + 'Europe & Africa': { + 'gmt': 'Greenwich Mean Time (GMT)', + 'cet': 'Central European Time (CET)', + 'eet': 'Eastern European Time (EET)', + 'west': 'Western European Summer Time (WEST)', + 'cat': 'Central Africa Time (CAT)', + 'eat': 'Eastern Africa Time (EAT)', + }, + 'Asia': { + 'msk': 'Moscow Time (MSK)', + 'ist': 'India Standard Time (IST)', + 'cst_china': 'China Standard Time (CST)', + 'jst': 'Japan Standard Time (JST)', + 'kst': 'Korea Standard Time (KST)', + 'ist_indonasia': 'Indonesia Standard Time (IST)', + }, + 'Australia & Pacific': { + 'awst': 'Australian Western Standard Time (AWST)', + 'acst': 'Australian Central Standard Time (ACST)', + 'aest': 'Australian Eastern Standard Time (AEST)', + 'nzst': 'New Zealand Standard Time (NZST)', + 'fjt': 'Fiji Time (FJT)', + }, + 'South America': { + 'art': 'Argentina Time (ART)', + 'bot': 'Bolivia Time (BOT)', + 'brt': 'Brasilia Time (BRT)', + 'clt': 'Chile Standard Time (CLT)', + }, +}; + +List getTimezonesWidgets(ShadThemeData theme) { + final widgets = []; + for (final zone in timezones.entries) { + widgets.add( + Padding( + padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 6), + child: Text( + zone.key, + style: theme.textTheme.muted.copyWith( + fontWeight: FontWeight.w600, + color: theme.colorScheme.popoverForeground, + ), + textAlign: TextAlign.start, + ), + ), + ); + widgets.addAll(zone.value.entries + .map((e) => ShadOption(value: e.key, child: Text(e.value)))); + } + return widgets; +} + +class SelectExample extends StatelessWidget { + const SelectExample({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ConstrainedBox( + constraints: const BoxConstraints(minWidth: 280), + child: ShadSelect( + placeholder: const Text('Select a timezone'), + options: getTimezonesWidgets(theme), + selectedOptionBuilder: (context, value) { + final timezone = timezones.entries + .firstWhere((element) => element.value.containsKey(value)) + .value[value]; + return Text(timezone!); + }, + ), + ); + } +} +``` + + + +## Form + + + +```dart +final verifiedEmails = [ + 'm@example.com', + 'm@google.com', + 'm@support.com', +]; + +class SelectFormField extends StatelessWidget { + const SelectFormField({super.key}); + + @override + Widget build(BuildContext context) { + return ShadSelectFormField( + id: 'email', + minWidth: 350, + initialValue: null, + options: verifiedEmails + .map((email) => ShadOption(value: email, child: Text(email))) + .toList(), + selectedOptionBuilder: (context, value) => value == 'none' + ? const Text('Select a verified email to display') + : Text(value), + placeholder: const Text('Select a verified email to display'), + validator: (v) { + if (v == null) { + return 'Please select an email to display'; + } + return null; + }, + ); + } +} +``` + + + +## With Search + + + +```dart +const frameworks = { +'nextjs': 'Next.js', +'svelte': 'SvelteKit', +'nuxtjs': 'Nuxt.js', +'remix': 'Remix', +'astro': 'Astro', +}; + +class SelectWithSearch extends StatefulWidget { + const SelectWithSearch({super.key}); + + @override + State createState() => _SelectWithSearchState(); +} + +class _SelectWithSearchState extends State { + var searchValue = ''; + + Map get filteredFrameworks => { + for (final framework in frameworks.entries) + if (framework.value.toLowerCase().contains(searchValue.toLowerCase())) + framework.key: framework.value + }; + + @override + Widget build(BuildContext context) { + return ShadSelect.withSearch( + minWidth: 180, + maxWidth: 300, + placeholder: const Text('Select framework...'), + onSearchChanged: (value) => setState(() => searchValue = value), + searchPlaceholder: const Text('Search framework'), + options: [ + if (filteredFrameworks.isEmpty) + const Padding( + padding: EdgeInsets.symmetric(vertical: 24), + child: Text('No framework found'), + ), + ...frameworks.entries.map( + (framework) { + // this offstage is used to avoid the focus loss when the search results appear again + // because it keeps the widget in the tree. + return Offstage( + offstage: !filteredFrameworks.containsKey(framework.key), + child: ShadOption( + value: framework.key, + child: Text(framework.value), + ), + ); + }, + ) + ], + selectedOptionBuilder: (context, value) => Text(frameworks[value]!), + ); + } +} +``` + + + +If you want to be able to deselect an option, you can use the `allowDeselection` property. + + +## Multiple + +This example shows how to select multiple options. + +In addition, the `allowDeselection` property is set to `true` to allow the user to deselect an option and the `closeOnSelect` property is set to `false` to keep the popover open after selecting an option. +If you tap outside the popover, it will close. + + + +```dart +final fruits = { + 'apple': 'Apple', + 'banana': 'Banana', + 'blueberry': 'Blueberry', + 'grapes': 'Grapes', + 'pineapple': 'Pineapple', +}; + +class SelectMultiple extends StatelessWidget { + const SelectMultiple({super.key}); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadSelect.multiple( + minWidth: 340, + onChanged: print, + allowDeselection: true, + closeOnSelect: false, + placeholder: const Text('Select multiple fruits'), + options: [ + Padding( + padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 6), + child: Text( + 'Fruits', + style: theme.textTheme.large, + textAlign: TextAlign.start, + ), + ), + ...fruits.entries.map( + (e) => ShadOption( + value: e.key, + child: Text(e.value), + ), + ), + ], + selectedOptionsBuilder: (context, values) => + Text(values.map((v) => v.capitalize()).join(', ')), + ); + } +} +``` + +## Example +```dart +// ignore_for_file: avoid_print + +import 'package:awesome_flutter_extensions/awesome_flutter_extensions.dart'; +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +const fruits = { + 'apple': 'Apple', + 'banana': 'Banana', + 'blueberry': 'Blueberry', + 'grapes': 'Grapes', + 'pineapple': 'Pineapple', +}; + +const timezones = { + 'North America': { + 'est': 'Eastern Standard Time (EST)', + 'cst': 'Central Standard Time (CST)', + 'mst': 'Mountain Standard Time (MST)', + 'pst': 'Pacific Standard Time (PST)', + 'akst': 'Alaska Standard Time (AKST)', + 'hst': 'Hawaii Standard Time (HST)', + }, + 'Europe & Africa': { + 'gmt': 'Greenwich Mean Time (GMT)', + 'cet': 'Central European Time (CET)', + 'eet': 'Eastern European Time (EET)', + 'west': 'Western European Summer Time (WEST)', + 'cat': 'Central Africa Time (CAT)', + 'eat': 'Eastern Africa Time (EAT)', + }, + 'Asia': { + 'msk': 'Moscow Time (MSK)', + 'ist': 'India Standard Time (IST)', + 'cst_china': 'China Standard Time (CST)', + 'jst': 'Japan Standard Time (JST)', + 'kst': 'Korea Standard Time (KST)', + 'ist_indonasia': 'Indonesia Standard Time (IST)', + }, + 'Australia & Pacific': { + 'awst': 'Australian Western Standard Time (AWST)', + 'acst': 'Australian Central Standard Time (ACST)', + 'aest': 'Australian Eastern Standard Time (AEST)', + 'nzst': 'New Zealand Standard Time (NZST)', + 'fjt': 'Fiji Time (FJT)', + }, + 'South America': { + 'art': 'Argentina Time (ART)', + 'bot': 'Bolivia Time (BOT)', + 'brt': 'Brasilia Time (BRT)', + 'clt': 'Chile Standard Time (CLT)', + }, +}; + +const frameworks = { + 'nextjs': 'Next.js', + 'svelte': 'SvelteKit', + 'nuxtjs': 'Nuxt.js', + 'remix': 'Remix', + 'astro': 'Astro', +}; + +class SelectPage extends StatefulWidget { + const SelectPage({super.key}); + + @override + State createState() => _SelectPageState(); +} + +class _SelectPageState extends State { + bool enabled = true; + final focusNodes = [FocusNode(), FocusNode(), FocusNode(), FocusNode()]; + var searchValue = ''; + bool allowDeselection = false; + bool closeOnSelect = true; + bool ensureSelectedVisible = true; + + Map get filteredFrameworks => { + for (final framework in frameworks.entries) + if (framework.value.toLowerCase().contains(searchValue.toLowerCase())) + framework.key: framework.value, + }; + + @override + void dispose() { + for (final node in focusNodes) { + node.dispose(); + } + super.dispose(); + } + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'Select', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyBoolProperty( + label: 'Fruits Focused', + value: focusNodes[0].hasFocus, + onChanged: (value) => setState( + () => + value ? focusNodes[0].requestFocus() : focusNodes[0].unfocus(), + ), + ), + MyBoolProperty( + label: 'Timezone Focused', + value: focusNodes[1].hasFocus, + onChanged: (value) => setState( + () => + value ? focusNodes[1].requestFocus() : focusNodes[1].unfocus(), + ), + ), + MyBoolProperty( + label: 'Framework Focused', + value: focusNodes[2].hasFocus, + onChanged: (value) => setState( + () => + value ? focusNodes[2].requestFocus() : focusNodes[2].unfocus(), + ), + ), + MyBoolProperty( + label: 'Allow deselection', + value: allowDeselection, + onChanged: (value) => setState(() => allowDeselection = value), + ), + MyBoolProperty( + label: 'Close on select', + value: closeOnSelect, + onChanged: (value) => setState(() => closeOnSelect = value), + ), + MyBoolProperty( + label: 'Ensure selected visible', + value: ensureSelectedVisible, + onChanged: (value) => setState(() => ensureSelectedVisible = value), + ), + ], + children: [ + ShadSelect( + minWidth: 180, + onChanged: print, + closeOnSelect: closeOnSelect, + enabled: enabled, + focusNode: focusNodes[0], + placeholder: const Text('Select a fruit'), + allowDeselection: allowDeselection, + ensureSelectedVisible: ensureSelectedVisible, + options: [ + Padding( + padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 6), + child: Text( + 'Fruits', + style: theme.textTheme.large, + textAlign: TextAlign.start, + ), + ), + ...fruits.entries.map( + (e) => ShadOption( + value: e.key, + child: Text(e.value), + ), + ), + ], + selectedOptionBuilder: (context, value) => Text(value.capitalize()), + ), + ShadSelect( + minWidth: 280, + focusNode: focusNodes[1], + onChanged: print, + enabled: enabled, + closeOnSelect: closeOnSelect, + placeholder: const Text('Select a timezone'), + ensureSelectedVisible: ensureSelectedVisible, + options: timezones.entries.map( + (zone) => Column( + crossAxisAlignment: CrossAxisAlignment.stretch, + children: [ + Padding( + padding: const EdgeInsets.symmetric( + horizontal: 8, + vertical: 6, + ), + child: Text( + zone.key, + style: theme.textTheme.muted.copyWith( + fontWeight: FontWeight.w600, + color: theme.colorScheme.popoverForeground, + ), + textAlign: TextAlign.start, + ), + ), + ...zone.value.entries.map( + (e) => ShadOption( + value: e.key, + child: Text(e.value), + ), + ), + ], + ), + ), + allowDeselection: allowDeselection, + selectedOptionBuilder: (context, value) { + final timezone = timezones.entries + .firstWhere((element) => element.value.containsKey(value)) + .value[value]; + return Text(timezone!); + }, + ), + ShadSelect.withSearch( + enabled: enabled, + focusNode: focusNodes[2], + minWidth: 180, + maxWidth: 300, + placeholder: const Text('Select framework...'), + onSearchChanged: (value) => setState(() => searchValue = value), + closeOnSelect: closeOnSelect, + searchPlaceholder: const Text('Search framework'), + ensureSelectedVisible: ensureSelectedVisible, + options: [ + if (filteredFrameworks.isEmpty) + const Padding( + padding: EdgeInsets.symmetric(vertical: 24), + child: Text('No framework found'), + ), + ...frameworks.entries.map( + (framework) { + // this offstage is used to avoid the focus loss when the search results appear again + // because it keeps the widget in the tree. + return Offstage( + offstage: !filteredFrameworks.containsKey(framework.key), + child: ShadOption( + value: framework.key, + child: Text(framework.value), + ), + ); + }, + ), + ], + selectedOptionBuilder: (context, value) => Text(frameworks[value]!), + onChanged: print, + allowDeselection: allowDeselection, + ), + ShadSelect.multiple( + minWidth: 340, + onChanged: print, + enabled: enabled, + focusNode: focusNodes[3], + allowDeselection: allowDeselection, + placeholder: const Text('Select multiple fruits'), + closeOnSelect: closeOnSelect, + ensureSelectedVisible: ensureSelectedVisible, + options: [ + Padding( + padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 6), + child: Text( + 'Fruits', + style: theme.textTheme.large, + textAlign: TextAlign.start, + ), + ), + ...fruits.entries.map( + (e) => ShadOption( + value: e.key, + child: Text(e.value), + ), + ), + ], + selectedOptionsBuilder: (context, values) => + Text(values.map((v) => v.capitalize()).join(', ')), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'dart:convert'; + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class SelectFormFieldPage extends StatefulWidget { + const SelectFormFieldPage({super.key}); + + @override + State createState() => _SelectFormFieldPageState(); +} + +class _SelectFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + final verifiedEmails = [ + 'm@example.com', + 'm@google.com', + 'm@support.com', + ]; + String? initialValue; + Map formValue = {}; + final formKey = GlobalKey(); + bool allowDeselection = false; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + initialValue: {'email': initialValue}, + child: BaseScaffold( + appBarTitle: 'SelectFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + ShadSelect( + options: ['none', ...verifiedEmails].map( + (e) => ShadOption(value: e, child: Text(e.toString())), + ), + initialValue: initialValue, + placeholder: const Text('Form Initial Value'), + onChanged: (v) { + formKey.currentState!.setFieldValue('email', v); + }, + selectedOptionBuilder: (context, value) => Text( + value.toString(), + ), + ), + MyBoolProperty( + label: 'Allow deselection', + value: allowDeselection, + onChanged: (value) => setState(() => allowDeselection = value), + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + ShadSelectFormField( + id: 'email', + allowDeselection: allowDeselection, + minWidth: 350, + initialValue: initialValue, + onChanged: print, + options: verifiedEmails + .map( + (email) => ShadOption(value: email, child: Text(email)), + ) + .toList(), + selectedOptionBuilder: (context, value) => value == 'none' + ? const Text('Select a verified email to display') + : Text(value), + placeholder: const Text('Select a verified email to display'), + validator: (v) { + if (v == null) { + return 'Please select an email to display'; + } + return null; + }, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + const JsonEncoder.withIndent( + ' ', + ).convert(formValue), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/separator.md b/.claude/skills/shadcn-ui-flutter/components/separator.md new file mode 100644 index 00000000..c982c2b2 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/separator.md @@ -0,0 +1,110 @@ +# Separator + +Visually or semantically separates content. + + + +```dart +const ShadSeparator.horizontal( + thickness: 4, + margin: EdgeInsets.symmetric(horizontal: 20), + radius: BorderRadius.all(Radius.circular(4)), +) +``` + + + +## Destructive + + + +```dart +const ShadSeparator.vertical( + thickness: 4, + margin: EdgeInsets.symmetric(vertical: 20), + radius: BorderRadius.all(Radius.circular(4)), +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/string_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/services.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class SeparatorPage extends StatefulWidget { + const SeparatorPage({super.key}); + + @override + State createState() => _SeparatorPageState(); +} + +class _SeparatorPageState extends State { + int margin = 4; + int thickness = 1; + int radius = 0; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'Separator', + editable: [ + MyStringProperty( + label: 'margin', + initialValue: '$margin', + inputFormatters: [FilteringTextInputFormatter.digitsOnly], + onChanged: (value) { + var maybe = int.tryParse(value); + if (maybe != null) setState(() => margin = maybe); + }, + ), + MyStringProperty( + label: 'thickness', + initialValue: '$thickness', + inputFormatters: [FilteringTextInputFormatter.digitsOnly], + onChanged: (value) { + var maybe = int.tryParse(value); + if (maybe != null) setState(() => thickness = maybe); + }, + ), + MyStringProperty( + label: 'radius', + initialValue: '$radius', + inputFormatters: [FilteringTextInputFormatter.digitsOnly], + onChanged: (value) { + var maybe = int.tryParse(value); + if (maybe != null) setState(() => radius = maybe); + }, + ), + ], + children: [ + Text('Horizontal', style: theme.textTheme.h4), + ShadSeparator.horizontal( + thickness: thickness.toDouble(), + margin: EdgeInsets.all(margin.toDouble()), + radius: BorderRadius.all(Radius.circular(radius.toDouble())), + ), + IntrinsicHeight( + child: Row( + mainAxisSize: MainAxisSize.max, + mainAxisAlignment: MainAxisAlignment.center, + children: [ + Text('Vertical', style: theme.textTheme.h4), + ShadSeparator.vertical( + thickness: thickness.toDouble(), + margin: EdgeInsets.all(margin.toDouble()), + radius: BorderRadius.all(Radius.circular(radius.toDouble())), + ), + Text('divider', style: theme.textTheme.h4), + ], + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/sheet.md b/.claude/skills/shadcn-ui-flutter/components/sheet.md new file mode 100644 index 00000000..c4ba88db --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/sheet.md @@ -0,0 +1,273 @@ +# Sheet + +Extends the Dialog component to display content that complements the main content of the screen. + + + +```dart +ShadButton.outline( + child: const Text('Open'), + onPressed: () => showShadSheet( + side: ShadSheetSide.right, + context: context, + builder: (context) => const EditProfileSheet(), + ), +), + +final profile = [ + (title: 'Name', value: 'Alexandru'), + (title: 'Username', value: 'nank1ro'), +]; + +class EditProfileSheet extends StatelessWidget { + const EditProfileSheet({super.key, required this.side}); + + final ShadSheetSide side; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadSheet( + constraints: side == ShadSheetSide.left || side == ShadSheetSide.right + ? const BoxConstraints(maxWidth: 512) + : null, + title: const Text('Edit Profile'), + description: const Text( + "Make changes to your profile here. Click save when you're done"), + child: Padding( + padding: const EdgeInsets.symmetric(vertical: 20), + child: Column( + mainAxisSize: MainAxisSize.min, + crossAxisAlignment: CrossAxisAlignment.stretch, + spacing: 16, + children: profile + .map( + (p) => Row( + children: [ + Expanded( + child: Text( + p.title, + textAlign: TextAlign.end, + style: theme.textTheme.small, + ), + ), + const SizedBox(width: 16), + Expanded( + flex: 5, + child: ShadInput(initialValue: p.value), + ), + ], + ), + ) + .toList(), + ), + ), + actions: const [ + ShadButton(child: Text('Save changes')), + ], + ); + } +} +``` + + + +## Side + +Use the `side` property to `showShadSheet` to indicate the edge of the screen where the component will appear. The values can be `top`, `right`, `bottom` or `left`. + + + +```dart +Row( + mainAxisSize: MainAxisSize.min, + spacing: 16, + children: [ + Column( + spacing: 16, + mainAxisSize: MainAxisSize.min, + children: [ + ShadButton.outline( + width: 100, + child: const Text('Top'), + onPressed: () => showShadSheet( + side: ShadSheetSide.top, + context: context, + builder: (context) => + const EditProfileSheet(side: ShadSheetSide.top), + ), + ), + ShadButton.outline( + width: 100, + child: const Text('Bottom'), + onPressed: () => showShadSheet( + side: ShadSheetSide.bottom, + context: context, + builder: (context) => const EditProfileSheet( + side: ShadSheetSide.bottom), + ), + ), + ], + ), + Column( + spacing: 16, + mainAxisSize: MainAxisSize.min, + children: [ + ShadButton.outline( + width: 100, + child: const Text('Right'), + onPressed: () => showShadSheet( + side: ShadSheetSide.right, + context: context, + builder: (context) => const EditProfileSheet( + side: ShadSheetSide.right), + ), + ), + ShadButton.outline( + width: 100, + child: const Text('Left'), + onPressed: () => showShadSheet( + side: ShadSheetSide.left, + context: context, + builder: (context) => const EditProfileSheet( + side: ShadSheetSide.left), + ), + ), + ], + ), + ], +), + +// See EditProfileSheet code in the previous code example +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/extensions.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +final profile = [ + (title: 'Name', value: 'Alexandru'), + (title: 'Username', value: 'nank1ro'), +]; + +class SheetPage extends StatefulWidget { + const SheetPage({super.key}); + + @override + State createState() => _SheetPageState(); +} + +class _SheetPageState extends State { + var side = ShadSheetSide.bottom; + var draggable = false; + var titlePinned = false; + var descriptionPinned = false; + var actionsPinned = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return BaseScaffold( + appBarTitle: 'Sheet', + editable: [ + MyEnumProperty( + label: 'Side', + value: side, + values: ShadSheetSide.values, + onChanged: (value) { + if (value != null) { + setState(() { + side = value; + }); + } + }, + ), + MyBoolProperty( + label: 'Draggable', + value: draggable, + onChanged: (value) => setState(() => draggable = value), + ), + MyBoolProperty( + label: 'titlePinned', + value: titlePinned, + onChanged: (v) => setState(() => titlePinned = v), + ), + MyBoolProperty( + label: 'descriptionPinned', + value: descriptionPinned, + onChanged: (v) => setState(() => descriptionPinned = v), + ), + MyBoolProperty( + label: 'actionsPinned', + value: actionsPinned, + onChanged: (v) => setState(() => actionsPinned = v), + ), + ], + children: [ + ShadButton.outline( + child: const Text('Open'), + onPressed: () { + showShadSheet( + context: context, + side: side, + builder: (context) { + return ShadSheet( + draggable: draggable, + constraints: + side == ShadSheetSide.left || side == ShadSheetSide.right + ? const BoxConstraints(maxWidth: 512) + : null, + title: const Text('Edit Profile'), + description: const Text( + "Make changes to your profile here. Click save when you're done", + ), + actions: const [ShadButton(child: Text('Save changes'))], + titlePinned: titlePinned, + descriptionPinned: descriptionPinned, + actionsPinned: actionsPinned, + child: Padding( + padding: const EdgeInsets.symmetric(vertical: 20), + child: Column( + crossAxisAlignment: CrossAxisAlignment.stretch, + spacing: 16, + children: + (profile.map( + (p) => Row( + children: [ + Expanded( + child: Text( + p.title, + textAlign: TextAlign.end, + style: theme.textTheme.small, + ), + ), + const SizedBox(width: 16), + Expanded( + flex: 5, + child: ShadInput( + initialValue: p.value, + ), + ), + ], + ), + ) * + 20) + .toList(), + ), + ), + ); + }, + ); + }, + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/slider.md b/.claude/skills/shadcn-ui-flutter/components/slider.md new file mode 100644 index 00000000..73a9d8e2 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/slider.md @@ -0,0 +1,81 @@ +# Slider + +An input where the user selects a value from within a given range. + + + +```dart +ShadSlider( + initialValue: 33, + max: 100, +), +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:example/common/properties/string_property.dart'; +import 'package:flutter/services.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class SliderPage extends StatefulWidget { + const SliderPage({super.key}); + + @override + State createState() => _SliderPageState(); +} + +class _SliderPageState extends State { + var enabled = true; + double value = 33; + ShadSliderInteraction sliderInteraction = ShadSliderInteraction.tapAndSlide; + int? divisions; + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Slider', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'Interaction', + value: sliderInteraction, + values: ShadSliderInteraction.values, + onChanged: (value) => setState(() => sliderInteraction = value!), + ), + MyStringProperty( + label: 'Divisions', + initialValue: divisions?.toString() ?? '', + onChanged: (v) { + setState(() { + final parsed = int.tryParse(v); + divisions = parsed; + }); + }, + inputFormatters: [FilteringTextInputFormatter.digitsOnly], + ), + ], + children: [ + ShadSlider( + initialValue: 33, + max: 100, + enabled: enabled, + onChanged: print, + allowedInteraction: sliderInteraction, + semanticFormatterCallback: (double value) => + '${value.round()}% volume level', + divisions: divisions, + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/sonner.md b/.claude/skills/shadcn-ui-flutter/components/sonner.md new file mode 100644 index 00000000..422af707 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/sonner.md @@ -0,0 +1,76 @@ +# Sonner + +An opinionated toast component. + + + +```dart +ShadButton.outline( + child: const Text('Show Toast'), + onPressed: () { + final sonner = ShadSonner.of(context); + final id = Random().nextInt(1000); + final now = DateTime.now(); + sonner.show( + ShadToast( + id: id, + title: const Text('Event has been created'), + description: Text(DateFormat.yMd().add_jms().format(now)), + action: ShadButton( + child: const Text('Undo'), + onPressed: () => sonner.hide(id), + ), + ), + ); + }, +), +``` + +## Example +```dart +import 'dart:math'; + +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class SonnerPage extends StatefulWidget { + const SonnerPage({super.key}); + + @override + State createState() => _SonnerPageState(); +} + +class _SonnerPageState extends State + with SingleTickerProviderStateMixin { + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Sonner', + children: [ + ShadButton.outline( + child: const Text('Show Toast'), + onPressed: () { + final sonner = ShadSonner.of(context); + final id = Random().nextInt(1000); + final now = DateTime.now(); + sonner.show( + ShadToast( + id: id, + title: const Text('Event has been created'), + description: Text(DateFormat.yMd().add_jms().format(now)), + action: ShadButton( + child: const Text('Undo'), + onPressed: () => sonner.hide(id), + ), + ), + ); + }, + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/switch.md b/.claude/skills/shadcn-ui-flutter/components/switch.md new file mode 100644 index 00000000..ba60c23e --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/switch.md @@ -0,0 +1,240 @@ +# Switch + +A control that allows the user to toggle between checked and not checked. + + + +```dart +class SwitchExample extends StatefulWidget { + const SwitchExample({super.key}); + + @override + State createState() => _SwitchExampleState(); +} + +class _SwitchExampleState extends State { + bool value = false; + + @override + Widget build(BuildContext context) { + return ShadSwitch( + value: value, + onChanged: (v) => setState(() => value = v), + label: const Text('Airplane Mode'), + ); + } +} +``` + + + +## Form + + + +```dart +ShadSwitchFormField( + id: 'terms', + initialValue: false, + inputLabel: + const Text('I accept the terms and conditions'), + onChanged: (v) {}, + inputSublabel: + const Text('You agree to our Terms and Conditions'), + validator: (v) { + if (!v) { + return 'You must accept the terms and conditions'; + } + return null; + }, +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class SwitchPage extends StatefulWidget { + const SwitchPage({super.key}); + + @override + State createState() => _SwitchPageState(); +} + +class _SwitchPageState extends State { + bool value = false; + bool enabled = true; + final focusNode = FocusNode(); + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Switch', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyBoolProperty( + label: 'Focused', + value: focusNode.hasFocus, + onChanged: enabled + ? (value) { + setState(() { + if (value) { + focusNode.requestFocus(); + } else { + focusNode.unfocus(); + } + }); + } + : null, + ), + ], + children: [ + ShadSwitch( + value: value, + focusNode: focusNode, + enabled: enabled, + onChanged: (v) { + setState(() => value = v); + }, + label: const Text('Airplane Mode'), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'dart:convert'; + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class SwitchFormFieldPage extends StatefulWidget { + const SwitchFormFieldPage({super.key}); + + @override + State createState() => _SwitchFormFieldPageState(); +} + +class _SwitchFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + bool initialValue = false; + Map formValue = {}; + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + initialValue: {'terms': initialValue}, + child: BaseScaffold( + appBarTitle: 'SwitchFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + MyBoolProperty( + label: 'Form Initial Value', + value: initialValue, + onChanged: (value) { + formKey.currentState!.setFieldValue('terms', value); + setState(() { + initialValue = value; + }); + }, + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + ShadSwitchFormField( + id: 'terms', + initialValue: initialValue, + inputLabel: const Text('I accept the terms and conditions'), + onChanged: (v) {}, + inputSublabel: const Text( + 'You agree to our Terms and Conditions', + ), + validator: (v) { + if (!v) { + return 'You must accept the terms and conditions'; + } + return null; + }, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + const JsonEncoder.withIndent( + ' ', + ).convert(formValue), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/table.md b/.claude/skills/shadcn-ui-flutter/components/table.md new file mode 100644 index 00000000..1117211b --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/table.md @@ -0,0 +1,490 @@ +# Table + +A responsive table component. + +## List + +Use the `ShadTable.list` widget to create a table from a two dimensional array of children. +Use it just for **small** tables, because every child will be created. + + + +```dart + + + +const invoices = [ + ( + invoice: "INV001", + paymentStatus: "Paid", + totalAmount: r"$250.00", + paymentMethod: "Credit Card", + ), + ( + invoice: "INV002", + paymentStatus: "Pending", + totalAmount: r"$150.00", + paymentMethod: "PayPal", + ), + ( + invoice: "INV003", + paymentStatus: "Unpaid", + totalAmount: r"$350.00", + paymentMethod: "Bank Transfer", + ), + ( + invoice: "INV004", + paymentStatus: "Paid", + totalAmount: r"$450.00", + paymentMethod: "Credit Card", + ), + ( + invoice: "INV005", + paymentStatus: "Paid", + totalAmount: r"$550.00", + paymentMethod: "PayPal", + ), + ( + invoice: "INV006", + paymentStatus: "Pending", + totalAmount: r"$200.00", + paymentMethod: "Bank Transfer", + ), + ( + invoice: "INV007", + paymentStatus: "Unpaid", + totalAmount: r"$300.00", + paymentMethod: "Credit Card", + ), +]; + +class TablePage extends StatelessWidget { + const TablePage({ + super.key, + }); + + @override + Widget build(BuildContext context) { + return Scaffold( + body: Center( + child: ConstrainedBox( + constraints: const BoxConstraints( + maxWidth: 600, + // added just to center the table vertically + maxHeight: 450, + ), + child: ShadTable.list( + header: const [ + ShadTableCell.header(child: Text('Invoice')), + ShadTableCell.header(child: Text('Status')), + ShadTableCell.header(child: Text('Method')), + ShadTableCell.header( + alignment: Alignment.centerRight, + child: Text('Amount'), + ), + ], + footer: const [ + ShadTableCell.footer(child: Text('Total')), + ShadTableCell.footer(child: Text('')), + ShadTableCell.footer(child: Text('')), + ShadTableCell.footer( + alignment: Alignment.centerRight, + child: Text(r'$2500.00'), + ), + ], + columnSpanExtent: (index) { + if (index == 2) return const FixedTableSpanExtent(130); + if (index == 3) { + return const MaxTableSpanExtent( + FixedTableSpanExtent(120), + RemainingTableSpanExtent(), + ); + } + // uses the default value + return null; + }, + children: invoices + .map( + (invoice) => [ + ShadTableCell( + child: Text( + invoice.invoice, + style: const TextStyle( + fontWeight: FontWeight.w500, + ), + ), + ), + ShadTableCell(child: Text(invoice.paymentStatus)), + ShadTableCell(child: Text(invoice.paymentMethod)), + ShadTableCell( + alignment: Alignment.centerRight, + child: Text( + invoice.totalAmount, + ), + ), + ], + ), + ), + ), + ), + ); + } +} +``` + + + +## Builder + +You can also use a builder to create the table. +This method is preferred for **large** tables because widgets are created on demand. +Here it is the same table as above, but using a builder. + +```dart +const invoices = [ + [ + "INV001", + "Paid", + "Credit Card", + r"$250.00", + ], + [ + "INV002", + "Pending", + "PayPal", + r"$150.00", + ], + [ + "INV003", + "Unpaid", + "Bank Transfer", + r"$350.00", + ], + [ + "INV004", + "Paid", + "Credit Card", + r"$450.00", + ], + [ + "INV005", + "Paid", + "PayPal", + r"$550.00", + ], + [ + "INV006", + "Pending", + "Bank Transfer", + r"$200.00", + ], + [ + "INV007", + "Unpaid", + "Credit Card", + r"$300.00", + ], +]; + +final headings = [ + 'Invoice', + 'Status', + 'Method', + 'Amount', +]; + +class TableExample extends StatelessWidget { + const TableExample({super.key}); + + @override + Widget build(BuildContext context) { + return ShadTable( + columnCount: invoices[0].length, + rowCount: invoices.length, + header: (context, column) { + final isLast = column == headings.length - 1; + return ShadTableCell.header( + alignment: isLast ? Alignment.centerRight : null, + child: Text(headings[column]), + ); + }, + columnSpanExtent: (index) { + if (index == 2) return const FixedTableSpanExtent(150); + if (index == 3) { + return const MaxTableSpanExtent( + FixedTableSpanExtent(120), + RemainingTableSpanExtent(), + ); + } + return null; + }, + builder: (context, index) { + final invoice = invoices[index.row]; + return ShadTableCell( + alignment: index.column == invoice.length - 1 + ? Alignment.centerRight + : Alignment.centerLeft, + child: Text( + invoice[index.column], + style: index.column == 0 + ? const TextStyle(fontWeight: FontWeight.w500) + : null, + ), + ); + }, + footer: (context, column) { + if (column == 0) { + return const ShadTableCell.footer( + child: Text( + 'Total', + style: TextStyle(fontWeight: FontWeight.w500), + ), + ); + } + if (column == 3) { + return const ShadTableCell.footer( + alignment: Alignment.centerRight, + child: Text( + r'$2500.00', + ), + ); + } + return const ShadTableCell(child: SizedBox()); + }, + ); + } +} +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +const invoices = [ + ( + invoice: "INV001", + paymentStatus: "Paid", + totalAmount: r"$250.00", + paymentMethod: "Credit Card", + ), + ( + invoice: "INV002", + paymentStatus: "Pending", + totalAmount: r"$150.00", + paymentMethod: "PayPal", + ), + ( + invoice: "INV003", + paymentStatus: "Unpaid", + totalAmount: r"$350.00", + paymentMethod: "Bank Transfer", + ), + ( + invoice: "INV004", + paymentStatus: "Paid", + totalAmount: r"$450.00", + paymentMethod: "Credit Card", + ), + ( + invoice: "INV005", + paymentStatus: "Paid", + totalAmount: r"$550.00", + paymentMethod: "PayPal", + ), + ( + invoice: "INV006", + paymentStatus: "Pending", + totalAmount: r"$200.00", + paymentMethod: "Bank Transfer", + ), + ( + invoice: "INV007", + paymentStatus: "Unpaid", + totalAmount: r"$300.00", + paymentMethod: "Credit Card", + ), +]; + +class TablePage extends StatelessWidget { + const TablePage({super.key}); + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Table', + wrapChildrenInScrollable: false, + wrapSingleChildInColumn: false, + children: [ + Center( + child: ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadTable.list( + header: const [ + ShadTableCell.header(child: Text('Invoice')), + ShadTableCell.header(child: Text('Status')), + ShadTableCell.header(child: Text('Method')), + ShadTableCell.header( + alignment: Alignment.centerRight, + child: Text('Amount'), + ), + ], + footer: const [ + ShadTableCell.footer(child: Text('Total')), + ShadTableCell.footer(child: Text('')), + ShadTableCell.footer(child: Text('')), + ShadTableCell.footer( + alignment: Alignment.centerRight, + child: Text(r'$2500.00'), + ), + ], + columnSpanExtent: (index) { + if (index == 2) return const FixedTableSpanExtent(130); + if (index == 3) { + return const MaxTableSpanExtent( + FixedTableSpanExtent(120), + RemainingTableSpanExtent(), + ); + } + // uses the default value + return null; + }, + children: invoices.map( + (invoice) => [ + ShadTableCell( + child: Text( + invoice.invoice, + style: const TextStyle( + fontWeight: FontWeight.w500, + ), + ), + ), + ShadTableCell(child: Text(invoice.paymentStatus)), + ShadTableCell(child: Text(invoice.paymentMethod)), + ShadTableCell( + alignment: Alignment.centerRight, + child: Text( + invoice.totalAmount, + ), + ), + ], + ), + ), + ), + ), + ], + ); + } +} + +/* With builder it will be +const invoices = [ + [ + "INV001", + "Paid", + "Credit Card", + r"$250.00", + ], + [ + "INV002", + "Pending", + "PayPal", + r"$150.00", + ], + [ + "INV003", + "Unpaid", + "Bank Transfer", + r"$350.00", + ], + [ + "INV004", + "Paid", + "Credit Card", + r"$450.00", + ], + [ + "INV005", + "Paid", + "PayPal", + r"$550.00", + ], + [ + "INV006", + "Pending", + "Bank Transfer", + r"$200.00", + ], + [ + "INV007", + "Unpaid", + "Credit Card", + r"$300.00", + ], +]; + +final headings = [ + 'Invoice', + 'Status', + 'Method', + 'Amount', +]; + +ShadTable( + columnCount: invoices[0].length, + rowCount: invoices.length, + header: (context, column) { + final isLast = column == headings.length - 1; + return ShadTableCell.header( + alignment: isLast ? Alignment.centerRight : null, + child: Text(headings[column]), + ); + }, + columnSpanExtent: (index) { + if (index == 2) return const FixedTableSpanExtent(150); + if (index == 3) { + return const MaxTableSpanExtent( + FixedTableSpanExtent(120), + RemainingTableSpanExtent(), + ); + } + return null; + }, + builder: (context, index) { + final invoice = invoices[index.row]; + return ShadTableCell( + alignment: index.column == invoice.length - 1 + ? Alignment.centerRight + : Alignment.centerLeft, + child: Text( + invoice[index.column], + style: index.column == 0 + ? const TextStyle(fontWeight: FontWeight.w500) + : null, + ), + ); + }, + footer: (context, column) { + if (column == 0) { + return const ShadTableCell.footer( + child: Text( + 'Total', + style: TextStyle(fontWeight: FontWeight.w500), + ), + ); + } + if (column == 3) { + return const ShadTableCell.footer( + alignment: Alignment.centerRight, + child: Text( + r'$2500.00', + ), + ); + } + return const ShadTableCell(child: SizedBox()); + }, +) +*/ + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/tabs.md b/.claude/skills/shadcn-ui-flutter/components/tabs.md new file mode 100644 index 00000000..68f6e395 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/tabs.md @@ -0,0 +1,158 @@ +# Tabs + +A set of layered sections of content—known as tab panels—that are displayed one at a time. + + + +```dart +class TabsExample extends StatelessWidget { + const TabsExample({super.key}); + + @override + Widget build(BuildContext context) { + return ShadTabs( + value: 'account', + tabBarConstraints: const BoxConstraints(maxWidth: 400), + contentConstraints: const BoxConstraints(maxWidth: 400), + tabs: [ + ShadTab( + value: 'account', + content: ShadCard( + title: const Text('Account'), + description: const Text( + "Make changes to your account here. Click save when you're done."), + footer: const ShadButton(child: Text('Save changes')), + child: Column( + mainAxisSize: MainAxisSize.min, + children: [ + const SizedBox(height: 16), + ShadInputFormField( + label: const Text('Name'), + initialValue: 'Ale', + ), + const SizedBox(height: 8), + ShadInputFormField( + label: const Text('Username'), + initialValue: 'nank1ro', + ), + const SizedBox(height: 16), + ], + ), + ), + child: const Text('Account'), + ), + ShadTab( + value: 'password', + content: ShadCard( + title: const Text('Password'), + description: const Text( + "Change your password here. After saving, you'll be logged out."), + footer: const ShadButton(child: Text('Save password')), + child: Column( + children: [ + const SizedBox(height: 16), + ShadInputFormField( + label: const Text('Current password'), + obscureText: true, + ), + const SizedBox(height: 8), + ShadInputFormField( + label: const Text('New password'), + obscureText: true, + ), + const SizedBox(height: 16), + ], + ), + ), + child: const Text('Password'), + ), + ], + ); + } +} +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class TabsPage extends StatelessWidget { + const TabsPage({super.key}); + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: "Tabs", + wrapChildrenInScrollable: false, + wrapSingleChildInColumn: false, + alignment: Alignment.topCenter, + children: [ + ShadTabs( + value: 'account', + tabBarConstraints: const BoxConstraints(maxWidth: 400), + contentConstraints: const BoxConstraints(maxWidth: 400), + onChanged: (value) => print(value), + tabs: [ + ShadTab( + value: 'account', + content: ShadCard( + title: const Text('Account'), + description: const Text( + "Make changes to your account here. Click save when you're done.", + ), + footer: const ShadButton(child: Text('Save changes')), + child: Column( + mainAxisSize: MainAxisSize.min, + children: [ + const SizedBox(height: 16), + ShadInputFormField( + label: const Text('Name'), + initialValue: 'Ale', + ), + const SizedBox(height: 8), + ShadInputFormField( + label: const Text('Username'), + initialValue: 'nank1ro', + ), + const SizedBox(height: 16), + ], + ), + ), + child: const Text('Account'), + ), + ShadTab( + value: 'password', + content: ShadCard( + title: const Text('Password'), + description: const Text( + "Change your password here. After saving, you'll be logged out.", + ), + footer: const ShadButton(child: Text('Save password')), + child: Column( + children: [ + const SizedBox(height: 16), + ShadInputFormField( + label: const Text('Current password'), + obscureText: true, + ), + const SizedBox(height: 8), + ShadInputFormField( + label: const Text('New password'), + obscureText: true, + ), + const SizedBox(height: 16), + ], + ), + ), + child: const Text('Password'), + ), + ], + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/textarea.md b/.claude/skills/shadcn-ui-flutter/components/textarea.md new file mode 100644 index 00000000..11b8120e --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/textarea.md @@ -0,0 +1,216 @@ +# Textarea + +Displays a form textarea or a component that looks like a textarea. + + + +```dart +ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 400), + child: const ShadTextarea( + placeholder: Text('Type your message here'), + ), +), +``` + + + +## Form + + + +```dart +ShadTextareaFormField( + id: 'bio', + label: const Text('Bio'), + placeholder: + const Text('Tell us a little bit about yourself'), + description: const Text( + 'You can @mention other users and organizations.'), + validator: (v) { + if (v.length < 10) { + return 'Bio must be at least 10 characters.'; + } + return null; + }, +) +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class TextareaPage extends StatefulWidget { + const TextareaPage({super.key}); + + @override + State createState() => _TextareaPageState(); +} + +class _TextareaPageState extends State { + bool enabled = true; + bool resizable = true; + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Textarea', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (v) => setState(() => enabled = v), + ), + MyBoolProperty( + label: 'Resizable', + value: resizable, + onChanged: (v) => setState(() => resizable = v), + ), + ], + children: [ + ShadTextarea( + placeholder: const Text('Type your message here...'), + enabled: enabled, + resizable: resizable, + onChanged: (v) => print('Value changed: $v'), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'dart:convert'; + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:example/common/properties/string_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class TextareaFormFieldPage extends StatefulWidget { + const TextareaFormFieldPage({super.key}); + + @override + State createState() => _TextareaFormFieldPageState(); +} + +class _TextareaFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + String? initialValue; + Map formValue = {}; + final formKey = GlobalKey(); + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + initialValue: {if (initialValue != null) 'bio': initialValue}, + child: BaseScaffold( + appBarTitle: 'TextareaFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + MyStringProperty( + label: 'Form Initial Value', + initialValue: initialValue, + placeholder: const Text('Enter your bio...'), + onChanged: (value) { + formKey.currentState!.setFieldValue('bio', value); + }, + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 500), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + ShadTextareaFormField( + id: 'bio', + label: const Text('Bio'), + placeholder: const Text('Tell us about yourself...'), + minHeight: 100, + maxHeight: 250, + validator: (v) { + if (v.trim().isEmpty) return 'Bio cannot be empty.'; + if (v.length < 10) { + return 'Bio must be at least 10 characters.'; + } + return null; + }, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + if (formKey.currentState!.saveAndValidate()) { + ShadToaster.of(context).show( + ShadToast(title: Text('Form submitted successfully')), + ); + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + ShadToaster.of(context).show( + ShadToast.destructive( + title: Text('Please correct the errors in the form'), + ), + ); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + const JsonEncoder.withIndent( + ' ', + ).convert(formValue), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/time-picker.md b/.claude/skills/shadcn-ui-flutter/components/time-picker.md new file mode 100644 index 00000000..438bd32d --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/time-picker.md @@ -0,0 +1,263 @@ +# Time Picker + +A time picker component. + + + +```dart +class PrimaryTimePicker extends StatelessWidget { + const PrimaryTimePicker({super.key}); + + @override + Widget build(BuildContext context) { + return ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: const ShadTimePicker( + trailing: Padding( + padding: EdgeInsets.only(left: 8, top: 14), + child: Icon(LucideIcons.clock4), + ), + ), + ); + } +} +``` + + + +## Form + + + +```dart +ShadTimePickerFormField( + label: const Text('Pick a time'), + onChanged: print, + description: + const Text('The time of the day you want to pick'), + validator: (v) => v == null ? 'A time is required' : null, +) +``` + + + +## ShadTimePickerFormField.period + + + +```dart +ShadTimePickerFormField.period( + label: const Text('Pick a time'), + onChanged: print, + description: + const Text('The time of the day you want to pick'), + validator: (v) => v == null ? 'A time is required' : null, +), +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class TimePickerPage extends StatefulWidget { + const TimePickerPage({super.key}); + + @override + State createState() => _TimePickerPageState(); +} + +class _TimePickerPageState extends State { + bool showHours = true; + bool showMinutes = true; + bool showSeconds = true; + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'TimePicker', + editable: [ + MyBoolProperty( + label: ' Show Hours', + value: showHours, + enabled: showMinutes || showSeconds, + onChanged: (v) => setState(() => showHours = v), + ), + MyBoolProperty( + label: ' Show Minutes', + value: showMinutes, + enabled: showHours || showSeconds, + onChanged: (v) => setState(() => showMinutes = v), + ), + MyBoolProperty( + label: ' Show Seconds', + enabled: showHours || showMinutes, + value: showSeconds, + onChanged: (v) => setState(() => showSeconds = v), + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadTimePicker( + showHours: showHours, + showMinutes: showMinutes, + showSeconds: showSeconds, + trailing: const Padding( + padding: EdgeInsets.only(left: 8, top: 14), + child: Icon(LucideIcons.clock4), + ), + onChanged: (time) { + print('time: $time'); + }, + ), + ), + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 600), + child: ShadTimePicker.period( + showHours: showHours, + showMinutes: showMinutes, + showSeconds: showSeconds, + crossAxisAlignment: WrapCrossAlignment.end, + onChanged: (time) { + print('time: $time'); + }, + ), + ), + ], + ); + } +} + +``` + +## Form Example +```dart +// ignore_for_file: avoid_print + +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/bool_property.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class TimePickerFormFieldPage extends StatefulWidget { + const TimePickerFormFieldPage({super.key}); + + @override + State createState() => + _TimePickerFormFieldPageState(); +} + +class _TimePickerFormFieldPageState extends State { + bool enabled = true; + var autovalidateMode = ShadAutovalidateMode.alwaysAfterFirstValidation; + Map formValue = {}; + final formKey = GlobalKey(); + bool showHours = true; + bool showMinutes = true; + bool showSeconds = true; + + @override + Widget build(BuildContext context) { + final theme = ShadTheme.of(context); + return ShadForm( + key: formKey, + enabled: enabled, + autovalidateMode: autovalidateMode, + child: BaseScaffold( + appBarTitle: 'TimePickerFormField', + editable: [ + MyBoolProperty( + label: 'Enabled', + value: enabled, + onChanged: (value) => setState(() => enabled = value), + ), + MyEnumProperty( + label: 'autovalidateMode', + value: autovalidateMode, + values: ShadAutovalidateMode.values, + onChanged: (value) { + if (value != null) { + setState(() => autovalidateMode = value); + } + }, + ), + MyBoolProperty( + label: ' Show Hours', + value: showHours, + enabled: showMinutes || showSeconds, + onChanged: (v) => setState(() => showHours = v), + ), + MyBoolProperty( + label: ' Show Minutes', + value: showMinutes, + enabled: showHours || showSeconds, + onChanged: (v) => setState(() => showMinutes = v), + ), + MyBoolProperty( + label: ' Show Seconds', + enabled: showHours || showMinutes, + value: showSeconds, + onChanged: (v) => setState(() => showSeconds = v), + ), + ], + children: [ + ConstrainedBox( + constraints: const BoxConstraints(maxWidth: 350), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + ShadTimePickerFormField( + id: 'time', + showHours: showHours, + showMinutes: showMinutes, + showSeconds: showSeconds, + label: const Text('Pick a time'), + onChanged: print, + description: const Text( + 'The time of the day you want to pick', + ), + validator: (v) => v == null ? 'A time is required' : null, + ), + const SizedBox(height: 16), + ShadButton( + child: const Text('Submit'), + onPressed: () { + print('submitted'); + if (formKey.currentState!.saveAndValidate()) { + setState(() { + formValue = formKey.currentState!.value; + }); + } else { + print('validation failed'); + } + }, + ), + if (formValue.isNotEmpty) + Padding( + padding: const EdgeInsets.only(top: 24, left: 12), + child: Column( + crossAxisAlignment: CrossAxisAlignment.start, + children: [ + Text('FormValue', style: theme.textTheme.p), + const SizedBox(height: 4), + SelectableText( + formValue.toString(), + style: theme.textTheme.small, + ), + ], + ), + ), + ], + ), + ), + ], + ), + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/toast.md b/.claude/skills/shadcn-ui-flutter/components/toast.md new file mode 100644 index 00000000..9aea2ab8 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/toast.md @@ -0,0 +1,208 @@ +# Toast + +A succinct message that is displayed temporarily. + + + +```dart +ShadButton.outline( + child: const Text('Add to calendar'), + onPressed: () { + ShadToaster.of(context).show( + ShadToast( + title: const Text('Scheduled: Catch up'), + description: + const Text('Friday, February 10, 2023 at 5:57 PM'), + action: ShadButton.outline( + child: const Text('Undo'), + onPressed: () => ShadToaster.of(context).hide(), + ), + ), + ); + }, +), +``` + + + +## Simple + + + +```dart +ShadButton.outline( + child: const Text('Show Toast'), + onPressed: () { + ShadToaster.of(context).show( + const ShadToast( + description: Text('Your message has been sent.'), + ), + ); + }, +), +``` + + + +## With Title + + + +```dart +ShadButton.outline( + child: const Text('Show Toast'), + onPressed: () { + ShadToaster.of(context).show( + const ShadToast( + title: Text('Uh oh! Something went wrong'), + description: + Text('There was a problem with your request'), + ), + ); + }, +), +``` + + + +## With Action + + + +```dart +ShadButton.outline( + child: const Text('Show Toast'), + onPressed: () { + ShadToaster.of(context).show( + ShadToast( + title: const Text('Uh oh! Something went wrong'), + description: + const Text('There was a problem with your request'), + action: ShadButton.outline( + child: const Text('Try again'), + onPressed: () => ShadToaster.of(context).hide(), + ), + ), + ); + }, +), +``` + + + +## Destructive + + + +```dart +final theme = ShadTheme.of(context); + +ShadButton.outline( + child: const Text('Show Toast'), + onPressed: () { + ShadToaster.of(context).show( + ShadToast.destructive( + title: const Text('Uh oh! Something went wrong'), + description: + const Text('There was a problem with your request'), + action: ShadButton.destructive( + child: const Text('Try again'), + decoration: ShadDecoration( + border: ShadBorder.all( + color: theme.colorScheme.destructiveForeground, + width: 1, + ), + ), + onPressed: () => ShadToaster.of(context).hide(), + ), + ), + ); + }, +), +``` + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:example/common/properties/enum_property.dart'; +import 'package:flutter/material.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +enum Alignm { + topRight, + topLeft, + bottomRight, + bottomLeft, + centerRight, + centerLeft, + center, + topCenter, + bottomCenter; + + Alignment toAlignment() { + return switch (this) { + topRight => Alignment.topRight, + topLeft => Alignment.topLeft, + bottomRight => Alignment.bottomRight, + bottomLeft => Alignment.bottomLeft, + centerRight => Alignment.centerRight, + centerLeft => Alignment.centerLeft, + center => Alignment.center, + topCenter => Alignment.topCenter, + bottomCenter => Alignment.bottomCenter, + }; + } +} + +class ToastPage extends StatefulWidget { + const ToastPage({super.key}); + + @override + State createState() => _ToastPageState(); +} + +class _ToastPageState extends State { + var alignment = Alignm.bottomRight; + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Toast', + editable: [ + MyEnumProperty( + label: 'Alignment', + value: alignment, + values: Alignm.values, + onChanged: (v) { + if (v != null) { + setState(() { + alignment = v; + }); + } + }, + ), + ], + children: [ + ShadButton.outline( + child: const Text('Add to calendar'), + onPressed: () { + final toaster = ShadToaster.of(context); + toaster.show( + ShadToast( + alignment: alignment.toAlignment(), + title: const Text('Scheduled: Catch up'), + description: const Text('Friday, February 10, 2023 at 5:57 PM'), + action: ShadButton.outline( + child: const Text('Undo'), + onPressed: () => toaster.hide(), + ), + ), + ); + }, + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/components/tooltip.md b/.claude/skills/shadcn-ui-flutter/components/tooltip.md new file mode 100644 index 00000000..020eb0ac --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/components/tooltip.md @@ -0,0 +1,60 @@ +# Tooltip + +A popup that displays information related to an element when the element receives keyboard focus or the mouse hovers over it. + + +```dart +ShadTooltip( + builder: (context) => const Text('Add to library'), + child: ShadButton.outline( + child: const Text('Hover/Focus'), + onPressed: () {}, + ), +), +``` + + +The tooltip works on hover only if the child uses a `ShadGestureDetector`. If you don't use a `ShadButton` or something similar that implements `ShadGestureDetector` hover will not work. +If, for example, you want to just show an image as child, wrap it with `ShadGestureDetector` to make it working. + +## Example +```dart +import 'package:example/common/base_scaffold.dart'; +import 'package:flutter/widgets.dart'; +import 'package:shadcn_ui/shadcn_ui.dart'; + +class TooltipPage extends StatefulWidget { + const TooltipPage({super.key}); + + @override + State createState() => _TooltipPageState(); +} + +class _TooltipPageState extends State { + final focusNode = FocusNode(); + + @override + void dispose() { + focusNode.dispose(); + super.dispose(); + } + + @override + Widget build(BuildContext context) { + return BaseScaffold( + appBarTitle: 'Tooltip', + children: [ + ShadTooltip( + focusNode: focusNode, + builder: (context) => const Text('Add to library'), + child: ShadButton.outline( + focusNode: focusNode, + child: const Text('Hover/Focus'), + ), + ), + ], + ); + } +} + +``` diff --git a/.claude/skills/shadcn-ui-flutter/guides/decorator.md b/.claude/skills/shadcn-ui-flutter/guides/decorator.md new file mode 100644 index 00000000..ca922b8a --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/guides/decorator.md @@ -0,0 +1,49 @@ +Decorates most of the components of the library using a `ShadDecoration` handled by the `ShadDecorator` component. + +## Default + +```dart +ShadDecoration( + secondaryBorder: ShadBorder.all( + padding: const EdgeInsets.all(4), + width: 0, + ), + secondaryFocusedBorder: ShadBorder.all( + width: 2, + color: colorScheme.ring, + radius: radius.add(radius / 2), + padding: const EdgeInsets.all(2), + ), + labelStyle: textTheme.muted.copyWith( + fontWeight: FontWeight.w500, + color: colorScheme.foreground, + ), + errorStyle: textTheme.muted.copyWith( + fontWeight: FontWeight.w500, + color: colorScheme.destructive, + ), + labelPadding: const EdgeInsets.only(bottom: 8), + descriptionStyle: textTheme.muted, + descriptionPadding: const EdgeInsets.only(top: 8), + errorPadding: const EdgeInsets.only(top: 8), + errorLabelStyle: textTheme.muted.copyWith( + fontWeight: FontWeight.w500, + color: colorScheme.destructive, + ), +); +``` + +## Secondary Border + +By default, a secondary border is drawn around the focusable components. +If you want to disable it and instead make bolder the primary border, you just need to add the `disableSecondaryBorder` property to the theme. + +```dart +ShadThemeData( + // Disables the secondary border + disableSecondaryBorder: true, +), +``` + +Be aware, this change is not recommended, as it may lead to accessibility issues. +The secondary border is there to help users understand which component is focused. \ No newline at end of file diff --git a/.claude/skills/shadcn-ui-flutter/guides/interop.md b/.claude/skills/shadcn-ui-flutter/guides/interop.md new file mode 100644 index 00000000..43798fe2 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/guides/interop.md @@ -0,0 +1,161 @@ +# Interoperability + +## Shadcn + Material + +We are the first Flutter UI library to allow shadcn components to be used simultaneously with Material components. +The setup is simple: + +```diff lang="dart" +import 'package:shadcn_ui/shadcn_ui.dart'; ++ import 'package:flutter/material.dart'; + +void main() { + runApp(const MyApp()); +} + +class MyApp extends StatelessWidget { + const MyApp({super.key}); + + @override + Widget build(BuildContext context) { +- return ShadApp(); ++ return ShadApp.custom( ++ themeMode: ThemeMode.dark, ++ darkTheme: ShadThemeData( ++ brightness: Brightness.dark, ++ colorScheme: const ShadSlateColorScheme.dark(), ++ ), ++ appBuilder: (context) { ++ return MaterialApp( ++ theme: Theme.of(context), ++ localizationsDelegates: const [ ++ GlobalShadLocalizations.delegate, ++ GlobalMaterialLocalizations.delegate, ++ GlobalCupertinoLocalizations.delegate, ++ GlobalWidgetsLocalizations.delegate, ++ ], ++ builder: (context, child) { ++ return ShadAppBuilder(child: child!); ++ }, ++ ); ++ }, ++ ); + } +``` + +:::tip +If you need to use the `Router` instead of the `Navigator`, use `MaterialApp.router`. +::: + +--- + +The default Material `ThemeData` created by `ShadApp` is: + +```dart +ThemeData( + fontFamily: themeData.textTheme.family, + extensions: themeData.extensions, + colorScheme: ColorScheme( + brightness: themeData.brightness, + primary: themeData.colorScheme.primary, + onPrimary: themeData.colorScheme.primaryForeground, + secondary: themeData.colorScheme.secondary, + onSecondary: themeData.colorScheme.secondaryForeground, + error: themeData.colorScheme.destructive, + onError: themeData.colorScheme.destructiveForeground, + surface: themeData.colorScheme.background, + onSurface: themeData.colorScheme.foreground, + ), + scaffoldBackgroundColor: themeData.colorScheme.background, + brightness: themeData.brightness, + dividerTheme: DividerThemeData( + color: themeData.colorScheme.border, + thickness: 1, + ), + textSelectionTheme: TextSelectionThemeData( + cursorColor: themeData.colorScheme.primary, + selectionColor: themeData.colorScheme.selection, + selectionHandleColor: themeData.colorScheme.primary, + ), + iconTheme: IconThemeData( + size: 16, + color: themeData.colorScheme.foreground, + ), + scrollbarTheme: ScrollbarThemeData( + crossAxisMargin: 1, + mainAxisMargin: 1, + thickness: const WidgetStatePropertyAll(8), + radius: const Radius.circular(999), + thumbColor: WidgetStatePropertyAll(themeData.colorScheme.border), + ), +), +``` + +:::note +Use `Theme.of(context).copyWith(...)` to override the default theme, without losing the default values provided by shadcn_ui. +::: + +## Shadcn + Cupertino + +If you need to use shadcn components with Cupertino components, use `CupertinoApp` instead of `MaterialApp`, like you are already used to. + +```diff lang="dart" +import 'package:shadcn_ui/shadcn_ui.dart'; ++ import 'package:flutter/cupertino.dart'; ++ import 'package:flutter_localizations/flutter_localizations.dart'; + +void main() { + runApp(const MyApp()); +} + +class MyApp extends StatelessWidget { + const MyApp({super.key}); + + @override + Widget build(BuildContext context) { +- return ShadApp(); ++ return ShadApp.custom( ++ themeMode: ThemeMode.dark, ++ darkTheme: ShadThemeData( ++ brightness: Brightness.dark, ++ colorScheme: const ShadSlateColorScheme.dark(), ++ ), ++ appBuilder: (context) { ++ return CupertinoApp( ++ theme: CupertinoTheme.of(context), ++ localizationsDelegates: const [ ++ GlobalShadLocalizations.delegate, ++ DefaultMaterialLocalizations.delegate, ++ DefaultCupertinoLocalizations.delegate, ++ DefaultWidgetsLocalizations.delegate, ++ ], ++ builder: (context, child) { ++ return ShadAppBuilder(child: child!); ++ }, ++ ); ++ }, ++ ); + } +``` + +:::tip +If you need to use the `Router` instead of the `Navigator`, use `CupertinoApp.router`. +::: + +--- + +The default `CupertinoThemeData` created by `ShadApp` is: + +```dart +CupertinoThemeData( + primaryColor: themeData.colorScheme.primary, + primaryContrastingColor: themeData.colorScheme.primaryForeground, + scaffoldBackgroundColor: themeData.colorScheme.background, + barBackgroundColor: themeData.colorScheme.primary, + brightness: themeData.brightness, +), +``` + +:::note +Use `CupertinoTheme.of(context).copyWith(...)` to override the default theme, without losing the default values provided by shadcn_ui. +::: \ No newline at end of file diff --git a/.claude/skills/shadcn-ui-flutter/guides/responsive.md b/.claude/skills/shadcn-ui-flutter/guides/responsive.md new file mode 100644 index 00000000..eb2678e1 --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/guides/responsive.md @@ -0,0 +1,65 @@ +In *shadcn_ui* the responsiveness is an important part of the library. + +The `ShadTheme` supports a customizable set of breakpoints. + +## Default + +```dart +ShadThemeData( + breakpoints: ShadBreakpoints( + tn: 0, // tiny + sm: 640, // small + md: 768, // medium + lg: 1024, // large + xl: 1280, // extra large + xxl: 1536, // extra extra large + ), +); +``` + +## Current breakpoint + +To get the current breakpoint you can use `ShadResponsiveBuilder` or `context.breakpoint`, eg: + +```dart + +ShadResponsiveBuilder( + builder: (context, breakpoint) { + final sm = breakpoint >= ShadTheme.of(context).breakpoints.sm; + ... + }, +), +``` + +which is equivalent to: + +```dart +final sm = context.breakpoint >= ShadTheme.of(context).breakpoints.sm; + +``` + +In Tailwind CSS, it's common to say that *sm* is not for small screens, but will target also the largest sizes if you don't provide a larger breakpoint. + +That's why I'm using the `>=` operator. + +If you just want to check if you're in a specific breakpoint, use the `==` operator. + +## Sealed class + +The breakpoint returned is a sealed class so you can switch any size. + +```dart + +ShadResponsiveBuilder( + builder: (context, breakpoint) { + return switch (breakpoint) { + ShadBreakpointTN() => const Text('Tiny'), + ShadBreakpointSM() => const Text('Small'), + ShadBreakpointMD() => const Text('Medium'), + ShadBreakpointLG() => const Text('Large'), + ShadBreakpointXL() => const Text('Extra Large'), + ShadBreakpointXXL() => const Text('Extra Extra Large'), + }; + }, +), +``` \ No newline at end of file diff --git a/.claude/skills/shadcn-ui-flutter/guides/theming.md b/.claude/skills/shadcn-ui-flutter/guides/theming.md new file mode 100644 index 00000000..318f41fa --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/guides/theming.md @@ -0,0 +1,141 @@ +Defines the theme and color scheme for the app. + +The supported color schemes are: + +- blue +- gray +- green +- neutral +- orange +- red +- rose +- slate +- stone +- violet +- yellow +- zinc + +## Usage + +```diff lang="dart" + + +@override +Widget build(BuildContext context) { + return ShadApp( ++ darkTheme: ShadThemeData( ++ brightness: Brightness.dark, ++ colorScheme: const ShadSlateColorScheme.dark(), ++ ), + child: ... + ); +} +``` + +You can override specific properties of the selected theme/color scheme: + +```diff lang="dart" + + +@override +Widget build(BuildContext context) { + return ShadApp( + darkTheme: ShadThemeData( + brightness: Brightness.dark, + colorScheme: const ShadSlateColorScheme.dark( ++ background: Colors.blue, + ), ++ primaryButtonTheme: const ShadButtonTheme( ++ backgroundColor: Colors.cyan, ++ ), + ), + ), + child: ... + ); +} +``` + +You can also create your custom color scheme, just extend the `ShadColorScheme` class and pass all the properties. + + +## ShadColorScheme.fromName + +If you want to allow the user to change the default shadcn themes, I suggest using `ShadColorScheme.fromName`. + +```dart +// available color scheme names +final shadThemeColors = [ + 'blue', + 'gray', + 'green', + 'neutral', + 'orange', + 'red', + 'rose', + 'slate', + 'stone', + 'violet', + 'yellow', + 'zinc', +]; + +final lightColorScheme = ShadColorScheme.fromName('blue'); +final darkColorScheme = ShadColorScheme.fromName('slate', brightness: Brightness.dark); +``` + +In this way you can easily create a select to change the color scheme, for example: + +```dart + + + +// Somewhere in your app +ShadSelect( + initialValue: 'slate', + maxHeight: 200, + options: shadThemeColors.map( + (option) => ShadOption( + value: option, + child: Text( + option.capitalizeFirst(), + ), + ), + ), + selectedOptionBuilder: (context, value) { + return Text(value.capitalizeFirst()); + }, + onChanged: (value) { + // rebuild the app using your state management solution + }, +), +``` + +For example I'm using solidart as state management, here it is the example code used to rebuild the app widget when the user changes the theme mode. Check the "Toggle Theme" example at + +The same can be done for the color scheme, using a `Signal()` + +## Extend with custom colors + +You can extend the `ShadColorScheme` with your own custom colors by using the `custom` parameter. +```diff lang="dart" +return ShadApp( + theme: ShadThemeData( ++ colorScheme: const ShadZincColorScheme.light( ++ custom: { ++ 'myCustomColor': Color.fromARGB(255, 177, 4, 196), ++ }, ++ ), + ), +); +``` + +Then you can access it like this `ShadTheme.of(context).colorScheme.custom['myCustomColor']!`. + +Or you can create an extension on `ShadColorScheme` to make it easier to access: +```dart +extension CustomColorExtension on ShadColorScheme { + Color get myCustomColor => custom['myCustomColor']!; +} +``` + +In this way you can access it like other colors `ShadTheme.of(context).colorScheme.myCustomColor`. \ No newline at end of file diff --git a/.claude/skills/shadcn-ui-flutter/guides/typography.md b/.claude/skills/shadcn-ui-flutter/guides/typography.md new file mode 100644 index 00000000..23c2be3e --- /dev/null +++ b/.claude/skills/shadcn-ui-flutter/guides/typography.md @@ -0,0 +1,231 @@ +Styles for headings, paragraphs, lists...etc + +## h1Large + + + ```dart +Text( + 'Taxing Laughter: The Joke Tax Chronicles', + style: ShadTheme.of(context).textTheme.h1Large, +) +``` + + +## h1 + + + ```dart +Text( + 'Taxing Laughter: The Joke Tax Chronicles', + style: ShadTheme.of(context).textTheme.h1, +) +``` + + +## h2 + + + ```dart +Text( + 'The People of the Kingdom', + style: ShadTheme.of(context).textTheme.h2, +) +``` + + +## h3 + + + ```dart +Text( + 'The Joke Tax', + style: ShadTheme.of(context).textTheme.h3, +) +``` + + +## h4 + + + ```dart +Text( + 'The king, seeing how much happier his subjects were, realized the error of his ways and repealed the joke tax.', + style: ShadTheme.of(context).textTheme.h4, +) +``` + + +## p + + + ```dart +Text( + 'The king, seeing how much happier his subjects were, realized the error of his ways and repealed the joke tax.', + style: ShadTheme.of(context).textTheme.p, +) +``` + + +## Blockquote + + + ```dart +Text( + '"After all," he said, "everyone enjoys a good joke, so it\'s only fair that they should pay for the privilege."', + style: ShadTheme.of(context).textTheme.blockquote, +) +``` + + +## Table + + + ```dart +Text( + "King's Treasury", + style: ShadTheme.of(context).textTheme.table, +) +``` + + +## List + + + ```dart +Text( + '1st level of puns: 5 gold coins', + style: ShadTheme.of(context).textTheme.list, +) +``` + + +## Lead + + + ```dart +Text( + 'A modal dialog that interrupts the user with important content and expects a response.', + style: ShadTheme.of(context).textTheme.lead, +) +``` + + +## Large + + + ```dart +Text( + 'Are you absolutely sure?', + style: ShadTheme.of(context).textTheme.large, +) +``` + + +## Small + + + ```dart +Text( + 'Email address', + style: ShadTheme.of(context).textTheme.small, +) +``` + + +## Muted + + + ```dart +Text( + 'Enter your email address.', + style: ShadTheme.of(context).textTheme.muted, +) +``` + + +## Custom font family + +By default Shadcn UI uses [Geist](https://vercel.com/font) as default font family. +To change it, add the local font to your project, for example in the `/fonts` directory. +Then update your `pubspec.yaml` with something like this: + +```diff lang="yaml" +flutter: ++ fonts: ++ - family: UbuntuMono ++ fonts: ++ - asset: fonts/UbuntuMono-Regular.ttf ++ - asset: fonts/UbuntuMono-Italic.ttf ++ style: italic ++ - asset: fonts/UbuntuMono-Bold.ttf ++ weight: 700 ++ - asset: fonts/UbuntuMono-BoldItalic.ttf ++ weight: 700 ++ style: italic +``` + +Then in your `ShadApp` update the `ShadTextTheme`: +```diff lang="dart" +return ShadApp( + debugShowCheckedModeBanner: false, + themeMode: themeMode, + routes: routes, + theme: ShadThemeData( + brightness: Brightness.light, + colorScheme: const ShadZincColorScheme.light(), ++ textTheme: ShadTextTheme( ++ colorScheme: const ShadZincColorScheme.light(), ++ family: 'UbuntuMono', ++ ), + ), + ... +); +``` + +## Google font + +Install the [google_fonts](https://pub.dev/packages/google_fonts) package. +Then add the google font to your `ShadApp`: +```diff lang="dart" +return ShadApp( + debugShowCheckedModeBanner: false, + themeMode: themeMode, + routes: routes, + theme: ShadThemeData( + brightness: Brightness.light, + colorScheme: const ShadZincColorScheme.light(), ++ textTheme: ShadTextTheme.fromGoogleFont(GoogleFonts.poppins), + ), + ... +); +``` + +## Extend with custom styles + +You can extend the `ShadTextTheme` with your own custom styles by using the `custom` parameter. +```diff lang="dart" +return ShadApp( + theme: ShadThemeData( ++ textTheme: ShadTextTheme( ++ custom: { ++ 'myCustomStyle': const TextStyle( ++ fontSize: 16, ++ fontWeight: FontWeight.w400, ++ color: Colors.blue, ++ ), ++ }, ++ ), + ), +); +``` + +Then you can access it like this `ShadTheme.of(context).textTheme.custom['myCustomStyle']!`. + +Or you can create an extension on `ShadTextTheme` to make it easier to access: +```dart +extension CustomStyleExtension on ShadTextTheme { + TextStyle get myCustomStyle => custom['myCustomStyle']!; +} +``` + +In this way you can access it like other styles `ShadTheme.of(context).textTheme.myCustomStyle`. \ No newline at end of file diff --git a/.claude/skills/spark-declarative-pipelines/1-ingestion-patterns.md b/.claude/skills/spark-declarative-pipelines/1-ingestion-patterns.md deleted file mode 100644 index 88bd037b..00000000 --- a/.claude/skills/spark-declarative-pipelines/1-ingestion-patterns.md +++ /dev/null @@ -1,399 +0,0 @@ -# Data Ingestion Patterns for SDP - -Covers data ingestion patterns for Spark Declarative Pipelines including Auto Loader for cloud storage and streaming sources like Kafka and Event Hub. - -**Language Support**: SQL (primary), Python via modern `pyspark.pipelines` API. See [5-python-api.md](5-python-api.md) for Python syntax. - ---- - -## Auto Loader (Cloud Files) - -Auto Loader incrementally processes new data files as they arrive in cloud storage. - -### Basic Pattern - -```sql -CREATE OR REPLACE STREAMING TABLE bronze_orders AS -SELECT - *, - current_timestamp() AS _ingested_at, - _metadata.file_path AS source_file, - _metadata.file_modification_time AS file_timestamp -FROM read_files( - '/mnt/raw/orders/', - format => 'json', - schemaHints => 'order_id STRING, amount DECIMAL(10,2)' -); -``` - -### Schema Evolution - -```sql -CREATE OR REPLACE STREAMING TABLE bronze_customers AS -SELECT - *, - current_timestamp() AS _ingested_at -FROM stream(read_files( - '/mnt/raw/customers/', - format => 'json', - schemaHints => 'customer_id STRING, email STRING', - mode => 'PERMISSIVE' -- Handles schema changes gracefully -)); -``` - -### File Formats - -**JSON**: -```sql -FROM read_files( - 's3://bucket/data/', - format => 'json', - schemaHints => 'id STRING, timestamp TIMESTAMP' -) -``` - -**CSV**: -```sql -FROM read_files( - '/mnt/raw/data/', - format => 'csv', - schemaHints => 'id STRING, name STRING, amount DECIMAL(10,2)', - header => true, - delimiter => ',' -) -``` - -**Parquet** (schema auto-inferred): -```sql -FROM read_files( - 'abfss://container@storage.dfs.core.windows.net/data/', - format => 'parquet' -) -``` - -**Avro**: -```sql -FROM read_files( - '/mnt/raw/events/', - format => 'avro', - schemaHints => 'event_id STRING, event_time TIMESTAMP' -) -``` - -### Schema Inference - -**Explicit hints** (recommended for production): -```sql -FROM read_files( - '/mnt/raw/sales/', - format => 'json', - schemaHints => 'sale_id STRING, customer_id STRING, amount DECIMAL(10,2), sale_date DATE' -) -``` - -**Partial hints** (infer remaining columns): -```sql -FROM read_files( - '/mnt/raw/data/', - format => 'json', - schemaHints => 'id STRING, critical_field DECIMAL(10,2)' -- Others auto-inferred -) -``` - -### Rescue Data and Quarantine - -Handle malformed records with `_rescued_data`: - -```sql --- Flag records with parsing errors -CREATE OR REPLACE STREAMING TABLE bronze_events AS -SELECT - *, - current_timestamp() AS _ingested_at, - CASE WHEN _rescued_data IS NOT NULL THEN TRUE ELSE FALSE END AS has_parsing_errors -FROM read_files( - '/mnt/raw/events/', - format => 'json', - schemaHints => 'event_id STRING, event_time TIMESTAMP' -); - --- Quarantine for investigation -CREATE OR REPLACE STREAMING TABLE bronze_events_quarantine AS -SELECT * FROM STREAM bronze_events WHERE _rescued_data IS NOT NULL; - --- Clean data for downstream -CREATE OR REPLACE STREAMING TABLE silver_events_clean AS -SELECT * FROM STREAM bronze_events WHERE _rescued_data IS NULL; -``` - ---- - -## Streaming Sources (Kafka, Event Hub, Kinesis) - -### Kafka Source - -```sql -CREATE OR REPLACE STREAMING TABLE bronze_kafka_events AS -SELECT - CAST(key AS STRING) AS event_key, - CAST(value AS STRING) AS event_value, - topic, - partition, - offset, - timestamp AS kafka_timestamp, - current_timestamp() AS _ingested_at -FROM read_stream( - format => 'kafka', - kafka.bootstrap.servers => '${kafka_brokers}', - subscribe => 'events-topic', - startingOffsets => 'latest', -- or 'earliest' - kafka.security.protocol => 'SASL_SSL', - kafka.sasl.mechanism => 'PLAIN', - kafka.sasl.jaas.config => 'kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="${kafka_username}" password="${kafka_password}";' -); -``` - -### Kafka with Multiple Topics - -```sql -FROM read_stream( - format => 'kafka', - kafka.bootstrap.servers => '${kafka_brokers}', - subscribe => 'topic1,topic2,topic3', - startingOffsets => 'latest' -) -``` - -### Azure Event Hub - -```sql -CREATE OR REPLACE STREAMING TABLE bronze_eventhub_events AS -SELECT - CAST(body AS STRING) AS event_body, - enqueuedTime AS event_time, - offset, - sequenceNumber, - current_timestamp() AS _ingested_at -FROM read_stream( - format => 'eventhubs', - eventhubs.connectionString => '${eventhub_connection_string}', - eventhubs.consumerGroup => '${consumer_group}', - startingPosition => 'latest' -); -``` - -### AWS Kinesis - -```sql -CREATE OR REPLACE STREAMING TABLE bronze_kinesis_events AS -SELECT - CAST(data AS STRING) AS event_data, - partitionKey, - sequenceNumber, - approximateArrivalTimestamp AS arrival_time, - current_timestamp() AS _ingested_at -FROM read_stream( - format => 'kinesis', - kinesis.streamName => '${stream_name}', - kinesis.region => '${aws_region}', - kinesis.startingPosition => 'LATEST' -); -``` - -### Parse JSON from Streaming Sources - -```sql --- Parse JSON from Kafka value -CREATE OR REPLACE STREAMING TABLE silver_kafka_parsed AS -SELECT - from_json( - event_value, - 'event_id STRING, event_type STRING, user_id STRING, timestamp TIMESTAMP, properties MAP' - ) AS event_data, - kafka_timestamp, - _ingested_at -FROM STREAM bronze_kafka_events; - --- Flatten parsed JSON -CREATE OR REPLACE STREAMING TABLE silver_kafka_flattened AS -SELECT - event_data.event_id, - event_data.event_type, - event_data.user_id, - event_data.timestamp AS event_timestamp, - event_data.properties, - kafka_timestamp, - _ingested_at -FROM STREAM silver_kafka_parsed; -``` - ---- - -## Authentication - -### Using Databricks Secrets - -**Kafka**: -```sql -kafka.sasl.jaas.config => 'kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username="{{secrets/kafka/username}}" password="{{secrets/kafka/password}}";' -``` - -**Event Hub**: -```sql -eventhubs.connectionString => '{{secrets/eventhub/connection-string}}' -``` - -### Using Pipeline Variables - -Reference variables in SQL: -```sql -kafka.bootstrap.servers => '${kafka_brokers}' -``` - -Define in pipeline configuration: -```yaml -variables: - kafka_brokers: - default: "broker1:9092,broker2:9092" -``` - ---- - -## Key Patterns - -### 1. Always Add Ingestion Timestamp - -```sql -SELECT - *, - current_timestamp() AS _ingested_at -- Track when data entered system -FROM read_files(...) -``` - -### 2. Include File Metadata for Debugging - -```sql -SELECT - *, - _metadata.file_path AS source_file, - _metadata.file_modification_time AS file_timestamp, - _metadata.file_size AS file_size -FROM read_files(...) -``` - -### 3. Use Schema Hints for Production - -```sql --- ✅ Explicit schema prevents surprises -FROM read_files( - '/mnt/data/', - format => 'json', - schemaHints => 'id STRING, amount DECIMAL(10,2), date DATE' -) - --- ❌ Fully inferred schemas can drift -FROM read_files('/mnt/data/', format => 'json') -``` - -### 4. Handle Rescue Data for Quality - -```sql --- Route errors to quarantine, clean to downstream -CREATE OR REPLACE STREAMING TABLE bronze_data_quarantine AS -SELECT * FROM STREAM bronze_data WHERE has_errors; - -CREATE OR REPLACE STREAMING TABLE silver_data AS -SELECT * FROM STREAM bronze_data WHERE NOT has_errors; -``` - -### 5. Starting Positions - -**Development**: `startingOffsets => 'latest'` (new data only) -**Backfill**: `startingOffsets => 'earliest'` (all available data) -**Recovery**: Checkpoints handle automatically - ---- - -## Common Issues - -| Issue | Solution | -|-------|----------| -| Files not picked up | Verify format matches files and path is correct | -| Schema evolution breaking | Use `mode => 'PERMISSIVE'` and monitor `_rescued_data` | -| Kafka lag increasing | Check downstream bottlenecks, increase parallelism | -| Duplicate events | Implement deduplication in silver layer (see [2-streaming-patterns.md](2-streaming-patterns.md)) | -| Parsing errors | Use rescue data pattern to quarantine malformed records | - ---- - -## Python API Examples - -For Python, use modern `pyspark.pipelines` API. See [5-python-api.md](5-python-api.md) for complete guidance. - -### Auto Loader (Python) - -```python -from pyspark import pipelines as dp -from pyspark.sql import functions as F - -@dp.table(name="bronze_orders", cluster_by=["order_date"]) -def bronze_orders(): - return ( - spark.readStream - .format("cloudFiles") - .option("cloudFiles.format", "json") - .option("cloudFiles.schemaLocation", "/checkpoints/bronze_orders") - .option("cloudFiles.schemaHints", "order_id STRING, amount DECIMAL(10,2)") - .load("/mnt/raw/orders/") - .withColumn("_ingested_at", F.current_timestamp()) - .withColumn("_source_file", F.col("_metadata.file_path")) - ) -``` - -### Kafka (Python) - -```python -@dp.table(name="bronze_kafka_events") -def bronze_kafka_events(): - return ( - spark.readStream - .format("kafka") - .option("kafka.bootstrap.servers", spark.conf.get("kafka_brokers")) - .option("subscribe", "events-topic") - .option("startingOffsets", "latest") - .load() - .selectExpr( - "CAST(key AS STRING) AS event_key", - "CAST(value AS STRING) AS event_value", - "topic", "partition", "offset", - "timestamp AS kafka_timestamp" - ) - .withColumn("_ingested_at", F.current_timestamp()) - ) -``` - -### Quarantine (Python) - -```python -@dp.table(name="bronze_events", cluster_by=["ingestion_date"]) -def bronze_events(): - return ( - spark.readStream - .format("cloudFiles") - .option("cloudFiles.format", "json") - .option("rescuedDataColumn", "_rescued_data") - .load("/mnt/raw/events/") - .withColumn("_ingested_at", F.current_timestamp()) - .withColumn("ingestion_date", F.current_date()) - .withColumn("_has_parsing_errors", - F.when(F.col("_rescued_data").isNotNull(), True) - .otherwise(False)) - ) - -@dp.table(name="bronze_events_quarantine") -def bronze_events_quarantine(): - return ( - spark.read.table("catalog.schema.bronze_events") - .filter(F.col("_has_parsing_errors") == True) - ) -``` diff --git a/.claude/skills/spark-declarative-pipelines/3-scd-patterns.md b/.claude/skills/spark-declarative-pipelines/3-scd-patterns.md deleted file mode 100644 index b9ff17c1..00000000 --- a/.claude/skills/spark-declarative-pipelines/3-scd-patterns.md +++ /dev/null @@ -1,241 +0,0 @@ -# SCD Query Patterns - -How to query SCD Type 2 history tables effectively, including current state queries, point-in-time analysis, and change tracking. - ---- - -## Understanding SCD Type 2 Structure - -When you create an SCD Type 2 flow, the system automatically adds temporal columns: - -```sql -CREATE FLOW customers_scd2_flow AS -AUTO CDC INTO customers_history -FROM stream(customers_cdc_clean) -KEYS (customer_id) -SEQUENCE BY event_timestamp -STORED AS SCD TYPE 2 -TRACK HISTORY ON *; -``` - -**Resulting table structure**: -``` -customers_history -├── customer_id -- Business key -├── customer_name -├── email -├── phone -├── START_AT -- When this version became effective (auto-generated) -├── END_AT -- When this version expired (NULL for current) -└── ...other columns -``` - ---- - -## Current State Queries - -### All Current Records - -```sql --- END_AT IS NULL indicates active record -CREATE OR REPLACE MATERIALIZED VIEW dim_customers_current AS -SELECT - customer_id, customer_name, email, phone, address, - START_AT AS valid_from -FROM customers_history -WHERE END_AT IS NULL; -``` - -### Specific Customer - -```sql -SELECT * -FROM customers_history -WHERE customer_id = '12345' - AND END_AT IS NULL; -``` - ---- - -## Point-in-Time Queries - -### As-Of Date Query - -Get state of records as they were on a specific date: - -```sql --- Products as of January 1, 2024 -CREATE OR REPLACE MATERIALIZED VIEW products_as_of_2024_01_01 AS -SELECT - product_id, product_name, price, category, - START_AT, END_AT -FROM products_history -WHERE START_AT <= '2024-01-01' - AND (END_AT > '2024-01-01' OR END_AT IS NULL); -``` - ---- - -## Change Analysis - -### Track All Changes for Entity - -```sql --- Complete history for a customer -SELECT - customer_id, customer_name, email, phone, - START_AT, END_AT, - COALESCE( - DATEDIFF(DAY, START_AT, END_AT), - DATEDIFF(DAY, START_AT, CURRENT_TIMESTAMP()) - ) AS days_active -FROM customers_history -WHERE customer_id = '12345' -ORDER BY START_AT DESC; -``` - -### Changes Within Time Period - -```sql --- Customers who changed during Q1 2024 -SELECT - customer_id, customer_name, - START_AT AS change_timestamp, - 'UPDATE' AS change_type -FROM customers_history -WHERE START_AT BETWEEN '2024-01-01' AND '2024-03-31' - AND START_AT != ( - SELECT MIN(START_AT) - FROM customers_history ch2 - WHERE ch2.customer_id = customers_history.customer_id - ) -ORDER BY START_AT; -``` - ---- - -## Joining Facts with Historical Dimensions - -### Enrich Facts with Dimension at Transaction Time - -```sql --- Join sales with product prices at time of sale -CREATE OR REPLACE MATERIALIZED VIEW sales_with_historical_prices AS -SELECT - s.sale_id, s.product_id, s.sale_date, s.quantity, - p.product_name, p.price AS unit_price_at_sale_time, - s.quantity * p.price AS calculated_amount, - p.category -FROM sales_fact s -INNER JOIN products_history p - ON s.product_id = p.product_id - AND s.sale_date >= p.START_AT - AND (s.sale_date < p.END_AT OR p.END_AT IS NULL); -``` - -### Join with Current Dimension - -```sql --- Join sales with current product information -CREATE OR REPLACE MATERIALIZED VIEW sales_with_current_prices AS -SELECT - s.sale_id, s.product_id, s.sale_date, s.quantity, - s.amount AS amount_at_sale, - p.product_name AS current_product_name, - p.price AS current_price, - p.category AS current_category -FROM sales_fact s -INNER JOIN products_history p - ON s.product_id = p.product_id - AND p.END_AT IS NULL; -- Current version only -``` - ---- - -## Selective History Tracking - -When using `TRACK HISTORY ON specific_columns`: - -```sql --- Only price changes trigger new versions -CREATE FLOW products_scd2_flow AS -AUTO CDC INTO products_history -FROM stream(products_cdc_clean) -KEYS (product_id) -SEQUENCE BY event_timestamp -STORED AS SCD TYPE 2 -TRACK HISTORY ON price, cost; -- Only these columns -``` - ---- - -## Optimization Patterns - -### Pre-Filter Materialized Views - -```sql --- Current state view (most common pattern) -CREATE OR REPLACE MATERIALIZED VIEW dim_products_current AS -SELECT * FROM products_history WHERE END_AT IS NULL; - --- Recent changes only -CREATE OR REPLACE MATERIALIZED VIEW dim_recent_changes AS -SELECT * FROM products_history -WHERE START_AT >= CURRENT_DATE() - INTERVAL 90 DAYS; - --- Change frequency stats -CREATE OR REPLACE MATERIALIZED VIEW product_change_stats AS -SELECT - product_id, - COUNT(*) AS version_count, - MIN(START_AT) AS first_seen, - MAX(START_AT) AS last_updated -FROM products_history -GROUP BY product_id; -``` - ---- - -## Best Practices - -### 1. Always Filter by END_AT for Current - -```sql --- ✅ Efficient -WHERE END_AT IS NULL - --- ❌ Less efficient -WHERE START_AT = (SELECT MAX(START_AT) FROM table WHERE ...) -``` - -### 2. Use Inclusive Lower, Exclusive Upper - -```sql --- ✅ Standard pattern -WHERE START_AT <= '2024-01-01' - AND (END_AT > '2024-01-01' OR END_AT IS NULL) -``` - -### 3. Create MVs for Common Patterns - -```sql --- Current state -CREATE OR REPLACE MATERIALIZED VIEW dim_current AS -SELECT * FROM history WHERE END_AT IS NULL; - --- Recent changes -CREATE OR REPLACE MATERIALIZED VIEW dim_recent_changes AS -SELECT * FROM history -WHERE START_AT >= CURRENT_DATE() - INTERVAL 90 DAYS; -``` - ---- - -## Common Issues - -| Issue | Solution | -|-------|----------| -| Multiple rows for same key | Missing `END_AT IS NULL` filter for current state | -| Point-in-time no results | Use `START_AT <= date AND (END_AT > date OR END_AT IS NULL)` | -| Slow temporal join | Create materialized view for specific time period | -| Unexpected duplicates | Multiple changes same day - use SEQUENCE BY with high precision | diff --git a/.claude/skills/spark-declarative-pipelines/5-python-api.md b/.claude/skills/spark-declarative-pipelines/5-python-api.md deleted file mode 100644 index a7f3a709..00000000 --- a/.claude/skills/spark-declarative-pipelines/5-python-api.md +++ /dev/null @@ -1,338 +0,0 @@ -# Python API: Modern vs Legacy - -**Last Updated**: January 2026 -**Status**: Modern API (`pyspark.pipelines`) recommended for all new projects - ---- - -## Overview - -Databricks provides two Python APIs for Spark Declarative Pipelines: - -1. **Modern API** (`pyspark.pipelines` as `dp`) - **Recommended (2025)** -2. **Legacy API** (`dlt`) - Older Delta Live Tables API, still supported - -**Key Recommendation**: Always use **modern API** for new projects. Only use legacy for maintaining existing DLT code. - ---- - -## Quick Comparison - -| Aspect | Modern (`dp`) | Legacy (`dlt`) | -|--------|---------------|----------------| -| **Import** | `from pyspark import pipelines as dp` | `import dlt` | -| **Status** | ✅ **Recommended** | ⚠️ Legacy | -| **Table decorator** | `@dp.table()` | `@dlt.table()` | -| **Read** | `spark.read.table("table")` | `dlt.read("table")` | -| **CDC/SCD** | `dp.create_auto_cdc_flow()` | `dlt.apply_changes()` | -| **Use for** | New projects | Maintaining existing | - ---- - -## Side-by-Side Examples - -### Basic Table Definition - -**Modern (Recommended)**: -```python -from pyspark import pipelines as dp -from pyspark.sql import functions as F - -@dp.table(name="bronze_events", comment="Raw events") -def bronze_events(): - return ( - spark.readStream - .format("cloudFiles") - .option("cloudFiles.format", "json") - .load("/mnt/raw/events") - ) -``` - -**Legacy**: -```python -import dlt -from pyspark.sql import functions as F - -@dlt.table(name="bronze_events", comment="Raw events") -def bronze_events(): - return ( - spark.readStream - .format("cloudFiles") - .option("cloudFiles.format", "json") - .load("/mnt/raw/events") - ) -``` - -### Reading Tables - -**Modern (Recommended)**: -```python -@dp.table(name="silver_events") -def silver_events(): - # Explicit Unity Catalog path - return spark.read.table("bronze_events").filter(...) -``` - -**Legacy**: -```python -@dlt.table(name="silver_events") -def silver_events(): - # Implicit LIVE schema - return dlt.read("bronze_events").filter(...) -``` - -**Key Difference**: Modern uses explicit UC paths, legacy uses implicit `LIVE.*`. - -### Streaming Reads - -**Modern (Recommended)**: -```python -@dp.table(name="silver_events") -def silver_events(): - # Context-aware (no separate read_stream) - return ( - spark.readStream.table("catalog.schema.bronze_events") - .filter(F.col("event_type").isNotNull()) - ) -``` - -**Legacy**: -```python -@dlt.table(name="silver_events") -def silver_events(): - # Explicit streaming read - return ( - dlt.read_stream("bronze_events") - .filter(F.col("event_type").isNotNull()) - ) -``` - -### Data Quality Expectations - -**Modern (Recommended)**: -```python -@dp.table(name="silver_validated") -@dp.expect_or_drop("valid_id", "id IS NOT NULL") -@dp.expect_or_drop("valid_amount", "amount > 0") -@dp.expect_or_fail("critical_field", "timestamp IS NOT NULL") -def silver_validated(): - return spark.read.table("catalog.schema.bronze_events") -``` - -**Legacy**: -```python -@dlt.table(name="silver_validated") -@dlt.expect_or_drop("valid_id", "id IS NOT NULL") -@dlt.expect_or_drop("valid_amount", "amount > 0") -@dlt.expect_or_fail("critical_field", "timestamp IS NOT NULL") -def silver_validated(): - return dlt.read("bronze_events") -``` - -**Note**: Expectations API identical between versions. - -### SCD Type 2 (AUTO CDC) - -**Modern (Recommended)**: -```python -from pyspark.sql.functions import col - -dp.create_streaming_table("customers_history") - -dp.create_auto_cdc_flow( - target="customers_history", - source="customers_cdc", - keys=["customer_id"], - sequence_by=col("event_timestamp"), - stored_as_scd_type="2", - track_history_column_list=["*"] -) -``` - -**Legacy**: -```python -dlt.create_streaming_table("customers_history") - -dlt.apply_changes( - target="customers_history", - source="customers_cdc", - keys=["customer_id"], - sequence_by="event_timestamp", - stored_as_scd_type="2", - track_history_column_list=["*"] -) -``` - -**Key Difference**: Modern uses `create_auto_cdc_flow()`, legacy uses `apply_changes()`. - -### Liquid Clustering - -**Modern (Recommended)**: -```python -@dp.table( - name="bronze_events", - table_properties={ - "delta.autoOptimize.optimizeWrite": "true", - "delta.autoOptimize.autoCompact": "true" - }, - cluster_by=["event_type", "event_date"] # Liquid Clustering -) -def bronze_events(): - return spark.readStream.format("cloudFiles").load("/data") -``` - -**Legacy**: -```python -@dlt.table( - name="bronze_events", - table_properties={ - "pipelines.autoOptimize.managed": "true", - "pipelines.autoOptimize.zOrderCols": "event_type" - }, - partition_cols=["event_date"] # Legacy partitioning -) -def bronze_events(): - return spark.readStream.format("cloudFiles").load("/data") -``` - -**Key Difference**: Modern supports `cluster_by` for Liquid Clustering. - ---- - -## Decision Matrix - -### Use Modern API (`dp`) When: -- ✅ **Starting new project** (default choice) -- ✅ **Learning SDP/LDP** (learn current standard) -- ✅ **Want Liquid Clustering** -- ✅ **Prefer explicit Unity Catalog paths** -- ✅ **Following 2025 best practices** - -### Use Legacy API (`dlt`) When: -- ⚠️ **Maintaining existing DLT pipelines** (don't rewrite working code) -- ⚠️ **Team trained on DLT** (consistency with existing) -- ⚠️ **Older DBR versions** (if modern API not available) - -**Default**: Use modern `dp` API unless specific reason for legacy. - ---- - -## Migration Guide: dlt → dp - -### Step 1: Update Imports - -**Before**: -```python -import dlt -``` - -**After**: -```python -from pyspark import pipelines as dp -``` - -### Step 2: Update Decorators - -**Before**: `@dlt.table(name="my_table")` -**After**: `@dp.table(name="my_table")` - -### Step 3: Update Reads - -**Before**: -```python -dlt.read("source_table") -dlt.read_stream("source_table") -``` - -**After**: -```python -spark.table("catalog.schema.source_table") -# Streaming context-aware, no separate read_stream -``` - -### Step 4: Update CDC/SCD Operations - -**Before**: -```python -dlt.apply_changes(target="dim_customer", source="cdc_source", ...) -``` - -**After**: -```python -from pyspark.sql.functions import col - -dp.create_auto_cdc_flow( - target="dim_customer", - source="cdc_source", - keys=["customer_id"], - sequence_by=col("event_timestamp"), - stored_as_scd_type="2", - track_history_column_list=["*"] -) -``` - -**Key Change**: `dlt.apply_changes()` → `dp.create_auto_cdc_flow()` - -### Step 5: Update Clustering - -**Before**: `@dlt.table(partition_cols=["date"])` -**After**: `@dp.table(cluster_by=["date", "other_col"])` - ---- - -## Key Patterns (2025) - -### 1. Use Liquid Clustering - -```python -@dp.table(cluster_by=["key_col", "date_col"]) -def my_table(): - return ... - -# Or automatic -@dp.table(cluster_by=["AUTO"]) -def my_table(): - return ... -``` - -### 2. Explicit UC Paths - -```python -# ✅ Modern: explicit path -spark.table("catalog.schema.table") - -# ❌ Legacy: implicit LIVE -dlt.read("table") -``` - -### 3. forEachBatch for Custom Sinks - -```python -def write_to_custom_sink(batch_df, batch_id): - batch_df.write.format("custom").save(...) - -@dp.table(name="my_table") -def my_table(): - return ( - spark.readStream - .format("cloudFiles") - .load("/data") - .writeStream - .foreachBatch(write_to_custom_sink) - ) -``` - ---- - -## Summary - -**For New Projects**: Use modern `pyspark.pipelines` (`dp`) -- ✅ Current best practice (2025) -- ✅ Liquid Clustering support -- ✅ Explicit Unity Catalog paths - -**For Existing Projects**: Legacy `dlt` fully supported -- ⚠️ Migrate when convenient, not urgent -- ⚠️ Consider modern API for new files - -**Key Takeaway**: Modern API provides same functionality plus new features. Start all new projects with `from pyspark import pipelines as dp`. diff --git a/.claude/skills/spark-declarative-pipelines/6-dlt-migration.md b/.claude/skills/spark-declarative-pipelines/6-dlt-migration.md deleted file mode 100644 index dd3b07a8..00000000 --- a/.claude/skills/spark-declarative-pipelines/6-dlt-migration.md +++ /dev/null @@ -1,298 +0,0 @@ -# DLT to SDP Migration Guide - -Guide for migrating Delta Live Tables (DLT) Python pipelines to Spark Declarative Pipelines (SDP) SQL. - -⚠️ **For NEW Python SDP pipelines**: Use modern `pyspark.pipelines` API. See [5-python-api.md](5-python-api.md). - ---- - -## Migration Decision Matrix - -| Feature/Pattern | DLT Python | SDP SQL | Recommendation | -|-----------------|------------|---------|----------------| -| Simple transformations | ✓ | ✓ | **Migrate to SQL** | -| Aggregations | ✓ | ✓ | **Migrate to SQL** | -| Filtering, WHERE clauses | ✓ | ✓ | **Migrate to SQL** | -| CASE expressions | ✓ | ✓ | **Migrate to SQL** | -| SCD Type 1/2 | ✓ | ✓ | **Migrate to SQL** (AUTO CDC) | -| Simple joins | ✓ | ✓ | **Migrate to SQL** | -| Auto Loader | ✓ | ✓ | **Migrate to SQL** (read_files) | -| Streaming sources (Kafka) | ✓ | ✓ | **Migrate to SQL** (read_stream) | -| Complex Python UDFs | ✓ | ❌ | **Stay in Python** | -| External API calls | ✓ | ❌ | **Stay in Python** | -| Custom libraries | ✓ | ❌ | **Stay in Python** | -| Complex apply functions | ✓ | ❌ | **Stay in Python** or simplify | -| ML model inference | ✓ | ❌ | **Stay in Python** | - -**Rule**: If 80%+ is SQL-expressible, migrate to SDP SQL. If heavy Python logic, stay with DLT Python or use hybrid. - ---- - -## Side-by-Side: Key Patterns - -### Basic Streaming Table - -**DLT Python**: -```python -@dlt.table(name="bronze_sales", comment="Raw sales") -def bronze_sales(): - return ( - spark.readStream.format("cloudFiles") - .option("cloudFiles.format", "json") - .load("/mnt/raw/sales") - .withColumn("_ingested_at", F.current_timestamp()) - ) -``` - -**SDP SQL**: -```sql -CREATE OR REPLACE STREAMING TABLE bronze_sales -COMMENT 'Raw sales' -AS -SELECT *, current_timestamp() AS _ingested_at -FROM read_files('/mnt/raw/sales', format => 'json'); -``` - -### Filtering and Transformations - -**DLT Python**: -```python -@dlt.table(name="silver_sales") -@dlt.expect_or_drop("valid_amount", "amount > 0") -@dlt.expect_or_drop("valid_sale_id", "sale_id IS NOT NULL") -def silver_sales(): - return ( - dlt.read_stream("bronze_sales") - .withColumn("sale_date", F.to_date("sale_date")) - .withColumn("amount", F.col("amount").cast("decimal(10,2)")) - .select("sale_id", "customer_id", "amount", "sale_date") - ) -``` - -**SDP SQL**: -```sql -CREATE OR REPLACE STREAMING TABLE silver_sales AS -SELECT - sale_id, customer_id, - CAST(amount AS DECIMAL(10,2)) AS amount, - CAST(sale_date AS DATE) AS sale_date -FROM STREAM bronze_sales -WHERE amount > 0 AND sale_id IS NOT NULL; -``` - -### SCD Type 2 - -**DLT Python**: -```python -dlt.create_streaming_table("customers_history") - -dlt.apply_changes( - target="customers_history", - source="customers_cdc_clean", - keys=["customer_id"], - sequence_by="event_timestamp", - stored_as_scd_type="2", - track_history_column_list=["*"] -) -``` - -**SDP SQL**: -```sql -CREATE OR REFRESH STREAMING TABLE customers_history; - -CREATE FLOW customers_scd2_flow AS -AUTO CDC INTO customers_history -FROM stream(customers_cdc_clean) -KEYS (customer_id) -SEQUENCE BY event_timestamp -COLUMNS * EXCEPT (_rescued_data) -STORED AS SCD TYPE 2 -TRACK HISTORY ON *; -``` - -### Joins - -**DLT Python**: -```python -@dlt.table(name="silver_sales_enriched") -def silver_sales_enriched(): - sales = dlt.read_stream("silver_sales") - products = dlt.read("dim_products") - - return ( - sales.join(products, "product_id", "left") - .select(sales["*"], products["product_name"], products["category"]) - ) -``` - -**SDP SQL**: -```sql -CREATE OR REPLACE STREAMING TABLE silver_sales_enriched AS -SELECT - s.*, - p.product_name, - p.category -FROM STREAM silver_sales s -LEFT JOIN dim_products p ON s.product_id = p.product_id; -``` - ---- - -## Handling Expectations - -**DLT Python**: -```python -@dlt.expect_or_drop("valid_amount", "amount > 0") -@dlt.expect_or_fail("critical_id", "id IS NOT NULL") -``` - -**SDP SQL - Basic**: -```sql --- Use WHERE (equivalent to expect_or_drop) -WHERE amount > 0 AND id IS NOT NULL -``` - -**SDP SQL - Quarantine Pattern** (for auditing): -```sql --- Flag invalid records -CREATE OR REPLACE STREAMING TABLE bronze_data_flagged AS -SELECT - *, - CASE - WHEN amount <= 0 THEN TRUE - WHEN id IS NULL THEN TRUE - ELSE FALSE - END AS is_invalid -FROM STREAM bronze_data; - --- Clean for downstream -CREATE OR REPLACE STREAMING TABLE silver_data_clean AS -SELECT * FROM STREAM bronze_data_flagged WHERE NOT is_invalid; - --- Quarantine for investigation -CREATE OR REPLACE STREAMING TABLE silver_data_quarantine AS -SELECT * FROM STREAM bronze_data_flagged WHERE is_invalid; -``` - -**Migration**: `@dlt.expect_or_drop` → WHERE clause or quarantine pattern. - ---- - -## Handling UDFs - -### Simple UDFs (Migrate to SQL) - -**DLT Python**: -```python -@F.udf(returnType=StringType()) -def categorize_amount(amount): - if amount > 1000: - return "High" - elif amount > 100: - return "Medium" - else: - return "Low" - -@dlt.table(name="sales_categorized") -def sales_categorized(): - return ( - dlt.read("sales") - .withColumn("category", categorize_amount(F.col("amount"))) - ) -``` - -**SDP SQL** (CASE expression): -```sql -CREATE OR REPLACE MATERIALIZED VIEW sales_categorized AS -SELECT - *, - CASE - WHEN amount > 1000 THEN 'High' - WHEN amount > 100 THEN 'Medium' - ELSE 'Low' - END AS category -FROM sales; -``` - -### Complex UDFs (Stay in Python) - -**Keep in Python for**: -- Complex conditional logic -- External API calls -- Custom algorithms -- ML inference - -**Options**: -1. Keep transformation in Python DLT -2. Create hybrid (SQL + Python for specific UDFs) -3. Refactor to SQL built-ins if possible - ---- - -## Migration Process - -### Step 1: Inventory - -Document: -- Number of tables/views -- Python UDFs (simple vs complex) -- External dependencies -- Expectations and quality rules - -### Step 2: Categorize - -**Easy to migrate**: Filters, aggregations, simple CASE -**Moderate**: UDFs rewritable as SQL -**Hard**: Complex Python, external calls, ML - -### Step 3: Migrate by Layer - -1. **Bronze** (ingestion): Convert Auto Loader to read_files() -2. **Silver** (cleansing): Convert expectations to WHERE/quarantine -3. **Gold** (aggregations): Usually straightforward -4. **SCD/CDC**: Use AUTO CDC - -### Step 4: Test - -- Run both pipelines in parallel -- Compare outputs for correctness -- Validate performance -- Check quality metrics - ---- - -## When NOT to Migrate - -**Stay with DLT Python if**: -1. Heavy Python UDF usage (>30% of logic) -2. External API calls required -3. Custom ML model inference -4. Complex stateful operations not in SQL -5. Existing pipeline works well, team prefers Python -6. Limited SQL expertise - -**Consider hybrid**: SQL for most, Python for complex logic. - ---- - -## Common Issues - -| Issue | Solution | -|-------|----------| -| UDF doesn't translate | Keep in Python or refactor with SQL built-ins | -| Expectations differ | Use quarantine pattern to audit dropped records | -| Performance degradation | Use CLUSTER BY for Liquid Clustering, review joins | -| Schema evolution different | Use `mode => 'PERMISSIVE'` in read_files() | - ---- - -## Summary - -**Migration Path**: -1. Use decision matrix (80%+ SQL-expressible → migrate) -2. Migrate by layer (bronze → silver → gold) -3. Handle expectations with WHERE/quarantine -4. Translate simple UDFs to CASE expressions -5. Keep complex Python logic in Python - -**Key**: DLT Python and SDP SQL are both fully supported. Migrate for simplicity, not necessity. diff --git a/.claude/skills/spark-declarative-pipelines/SKILL.md b/.claude/skills/spark-declarative-pipelines/SKILL.md deleted file mode 100644 index 6db988d3..00000000 --- a/.claude/skills/spark-declarative-pipelines/SKILL.md +++ /dev/null @@ -1,474 +0,0 @@ ---- -name: spark-declarative-pipelines -description: "Creates, configures, and updates Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) using serverless compute. Handles streaming tables, materialized views, CDC, SCD Type 2, and Auto Loader ingestion patterns. Use when building data pipelines, working with Delta Live Tables, ingesting streaming data, implementing change data capture, or when the user mentions SDP, LDP, DLT, Lakeflow pipelines, streaming tables, or bronze/silver/gold medallion architectures." ---- - -# Lakeflow Spark Declarative Pipelines (SDP) - -## Quick Reference - -| Concept | Details | -|---------|---------| -| **Names** | SDP = Spark Declarative Pipelines = LDP = Lakeflow Declarative Pipelines = Lakeflow Pipelines (all interchangeable) | -| **Python Import** | `from pyspark import pipelines as dp` | -| **Primary Decorators** | `@dp.table()`, `@dp.materialized_view()` | -| **Replaces** | Delta Live Tables (DLT) with `import dlt` | -| **Based On** | Apache Spark 4.1+ (Databricks' modern data pipeline framework) | -| **Docs** | https://docs.databricks.com/aws/en/ldp/developer/python-dev | - ---- - -## Official Documentation - -- **[Lakeflow Spark Declarative Pipelines Overview](https://docs.databricks.com/aws/en/ldp/)** - Main documentation hub -- **[SQL Language Reference](https://docs.databricks.com/aws/en/ldp/developer/sql-dev)** - SQL syntax for streaming tables and materialized views -- **[Python Language Reference](https://docs.databricks.com/aws/en/ldp/developer/python-ref)** - `pyspark.pipelines` API -- **[Loading Data](https://docs.databricks.com/aws/en/ldp/load)** - Auto Loader, Kafka, Kinesis ingestion -- **[Change Data Capture (CDC)](https://docs.databricks.com/aws/en/ldp/cdc)** - AUTO CDC, SCD Type 1/2 -- **[Developing Pipelines](https://docs.databricks.com/aws/en/ldp/develop)** - File structure, testing, validation -- **[Liquid Clustering](https://docs.databricks.com/aws/en/delta/clustering)** - Modern data layout optimization - ---- - -## Quick Start: Initialize New Pipeline Project - -**RECOMMENDED**: Use `databricks pipelines init` to create production-ready Asset Bundle projects with multi-environment support. - -### When to Use Bundle Initialization - -Use bundle initialization for **New pipeline projects** for a professional structure from the start - -Use manual workflow for: -- Quick prototyping without multi-environment needs -- Existing manual projects you want to continue -- Learning/experimentation - -### Step 1: Initialize Project - -I will automatically run this command when you request a new pipeline: - -```bash -databricks pipelines init -``` - -**Interactive Prompts:** -- **Project name**: e.g., `customer_orders_pipeline` -- **Initial catalog**: Unity Catalog name (e.g., `main`, `prod_catalog`) -- **Personal schema per user?**: `yes` for dev (each user gets their own schema), `no` for prod -- **Language**: SQL or Python (auto-detected from your request - see language detection below) - -**Generated Structure:** -``` -my_pipeline/ -├── databricks.yml # Multi-environment config (dev/prod) -├── resources/ -│ └── *_etl.pipeline.yml # Pipeline resource definition -└── src/ - └── *_etl/ - ├── explorations/ # Exploratory code in .ipynb - └── transformations/ # Your .sql or .py files here -``` - -### Step 2: Customize Transformations - -Replace the example code created by the init process with custom transformation files in `src/transformations/` based on provided requirements, using best practice guidance from this skill. - - -### Step 3: Deploy and Run - -```bash -# Deploy to workspace (dev by default) -databricks bundle deploy - -# Run pipeline -databricks bundle run my_pipeline_etl - -# Deploy to production -databricks bundle deploy --target prod -``` - -I can run these commands for you using the Bash tool. - -**For medallion architecture** (bronze/silver/gold), two approaches work: -- **Flat with naming** (template default): `bronze_*.sql`, `silver_*.sql`, `gold_*.sql` -- **Subdirectories**: `bronze/orders.sql`, `silver/cleaned.sql`, `gold/summary.sql` - -Both work with the `transformations/**` glob pattern. Choose based on preference. - -See **[8-project-initialization.md](8-project-initialization.md)** for complete details on bundle initialization, migration, and troubleshooting. - ---- - -## Alternative: Manual Workflow (Advanced) - -For rapid prototyping, experimentation, or when you prefer direct control without Asset Bundles, use the manual workflow with MCP tools. - -Use MCP tools to create, run, and iterate on **serverless SDP pipelines**. The **primary tool is `create_or_update_pipeline`** which handles the entire lifecycle. - -**IMPORTANT: Always create serverless pipelines (default).** Only use classic clusters if user explicitly requires R language, Spark RDD APIs, or JAR libraries. - -### Step 1: Write Pipeline Files Locally - -Create `.sql` or `.py` files in a local folder: - -``` -my_pipeline/ -├── bronze/ -│ ├── ingest_orders.sql # SQL (default for most cases) -│ └── ingest_events.py # Python (for complex logic) -├── silver/ -│ └── clean_orders.sql -└── gold/ - └── daily_summary.sql -``` - -**SQL Example** (`bronze/ingest_orders.sql`): -```sql -CREATE OR REFRESH STREAMING TABLE bronze_orders -CLUSTER BY (order_date) -AS -SELECT - *, - current_timestamp() AS _ingested_at, - _metadata.file_path AS _source_file -FROM read_files( - '/Volumes/catalog/schema/raw/orders/', - format => 'json', - schemaHints => 'order_id STRING, customer_id STRING, amount DECIMAL(10,2), order_date DATE' -); -``` - -**Python Example** (`bronze/ingest_events.py`): -```python -from pyspark import pipelines as dp -from pyspark.sql.functions import col, current_timestamp - -@dp.table(name="bronze_events", cluster_by=["event_date"]) -def bronze_events(): - return ( - spark.readStream.format("cloudFiles") - .option("cloudFiles.format", "json") - .load("/Volumes/catalog/schema/raw/events/") - .withColumn("_ingested_at", current_timestamp()) - .withColumn("_source_file", col("_metadata.file_path")) - ) -``` - -**Language Selection:** -- **Auto-detection**: I analyze your request for keywords: - - **SQL indicators**: "SQL", "sql files", "simple transformations", "aggregations", "materialized view", "CREATE OR REFRESH" - - **Python indicators**: "Python", ".py files", "UDF", "complex logic", "ML inference", "external API", "@dp.table", "pandas" -- **Prompt for clarification** when language intent is unclear or mixed -- **Use SQL** for: Transformations, aggregations, filtering, joins (most cases) -- **Generate ONE language** per request unless you explicitly ask for mixed pipeline - -See **[8-project-initialization.md](8-project-initialization.md)** for detailed language detection logic. - -### Step 2: Upload to Databricks Workspace - -```python -# MCP Tool: upload_folder -upload_folder( - local_folder="/path/to/my_pipeline", - workspace_folder="/Workspace/Users/user@example.com/my_pipeline" -) -``` - -### Step 3: Create/Update and Run Pipeline - -Use **`create_or_update_pipeline`** - the main entry point. It: -1. Searches for an existing pipeline with the same name (or uses `id` from `extra_settings`) -2. Creates a new pipeline or updates the existing one -3. Optionally starts a pipeline run -4. Optionally waits for completion and returns detailed results - -```python -# MCP Tool: create_or_update_pipeline -result = create_or_update_pipeline( - name="my_orders_pipeline", - root_path="/Workspace/Users/user@example.com/my_pipeline", - catalog="my_catalog", - schema="my_schema", - workspace_file_paths=[ - "/Workspace/Users/user@example.com/my_pipeline/bronze/ingest_orders.sql", - "/Workspace/Users/user@example.com/my_pipeline/silver/clean_orders.sql", - "/Workspace/Users/user@example.com/my_pipeline/gold/daily_summary.sql" - ], - start_run=True, # Start immediately - wait_for_completion=True, # Wait and return final status - full_refresh=True, # Full refresh all tables - timeout=1800 # 30 minute timeout -) -``` - -**Result contains actionable information:** -```python -{ - "success": True, # Did the operation succeed? - "pipeline_id": "abc-123", # Pipeline ID for follow-up operations - "pipeline_name": "my_orders_pipeline", - "created": True, # True if new, False if updated - "state": "COMPLETED", # COMPLETED, FAILED, TIMEOUT, etc. - "catalog": "my_catalog", # Target catalog - "schema": "my_schema", # Target schema - "duration_seconds": 45.2, # Time taken - "message": "Pipeline created and completed successfully in 45.2s. Tables written to my_catalog.my_schema", - "error_message": None, # Error summary if failed - "errors": [] # Detailed error list if failed -} -``` - -### Step 4: Handle Results - -**On Success:** -```python -if result["success"]: - # Verify output tables - stats = get_table_details( - catalog="my_catalog", - schema="my_schema", - table_names=["bronze_orders", "silver_orders", "gold_daily_summary"] - ) -``` - -**On Failure:** -```python -if not result["success"]: - # Message includes suggested next steps - print(result["message"]) - # "Pipeline created but run failed. State: FAILED. Error: Column 'amount' not found. - # Use get_pipeline_events(pipeline_id='abc-123') for full details." - - # Get detailed errors - events = get_pipeline_events(pipeline_id=result["pipeline_id"], max_results=50) -``` - -### Step 5: Iterate Until Working - -1. Review errors from result or `get_pipeline_events` -2. Fix issues in local files -3. Re-upload with `upload_folder` -4. Run `create_or_update_pipeline` again (it will update, not recreate) -5. Repeat until `result["success"] == True` - ---- - -## Quick Reference: MCP Tools - -### Primary Tool - -| Tool | Description | -|------|-------------| -| **`create_or_update_pipeline`** | **Main entry point.** Creates or updates pipeline, optionally runs and waits. Returns detailed status with `success`, `state`, `errors`, and actionable `message`. | - -### Pipeline Management - -| Tool | Description | -|------|-------------| -| `find_pipeline_by_name` | Find existing pipeline by name, returns pipeline_id | -| `get_pipeline` | Get pipeline configuration and current state | -| `start_update` | Start pipeline run (`validate_only=True` for dry run) | -| `get_update` | Poll update status (QUEUED, RUNNING, COMPLETED, FAILED) | -| `stop_pipeline` | Stop a running pipeline | -| `get_pipeline_events` | Get error messages for debugging failed runs | -| `delete_pipeline` | Delete a pipeline | - -### Supporting Tools - -| Tool | Description | -|------|-------------| -| `upload_folder` | Upload local folder to workspace (parallel) | -| `get_table_details` | Verify output tables have expected schema and row counts | -| `execute_sql` | Run ad-hoc SQL to inspect data | - ---- - -## Reference Documentation (Local) - -Load these for detailed patterns: - -- **[1-ingestion-patterns.md](1-ingestion-patterns.md)** - Auto Loader, Kafka, Event Hub, Kinesis, file formats -- **[2-streaming-patterns.md](2-streaming-patterns.md)** - Deduplication, windowing, stateful operations, joins -- **[3-scd-patterns.md](3-scd-patterns.md)** - Querying SCD Type 2 history tables, temporal joins -- **[4-performance-tuning.md](4-performance-tuning.md)** - Liquid Clustering, optimization, state management -- **[5-python-api.md](5-python-api.md)** - Modern `dp` API vs legacy `dlt` API comparison -- **[6-dlt-migration.md](6-dlt-migration.md)** - Migrating existing DLT pipelines to SDP -- **[7-advanced-configuration.md](7-advanced-configuration.md)** - `extra_settings` parameter reference and examples -- **[8-project-initialization.md](8-project-initialization.md)** - Using `databricks pipelines init`, Asset Bundles, language detection, and migration guides - ---- - -## Best Practices (2025) - -### Project Structure -- **Default to `databricks pipelines init`** for new projects (creates Asset Bundle) -- **Use Asset Bundles** for multi-environment deployments (dev/staging/prod) -- **Manual structure only** for quick prototypes or legacy migration -- **Medallion architecture**: Two approaches work with Asset Bundles: - - **Flat structure** (template default): `bronze_*.sql`, `silver_*.sql`, `gold_*.sql` in `transformations/` - - **Subdirectories**: `transformations/bronze/`, `transformations/silver/`, `transformations/gold/` - - Both work with the `transformations/**` glob pattern - choose based on team preference -- See **[8-project-initialization.md](8-project-initialization.md)** for project setup details - - -### Modern Defaults -- **CLUSTER BY** (Liquid Clustering), not PARTITION BY - see [4-performance-tuning.md](4-performance-tuning.md) -- **Raw `.sql`/`.py` files**, not notebooks -- **Serverless compute ONLY** - Do not use classic clusters unless explicitly required -- **Unity Catalog** (required for serverless) -- **read_files()** for cloud storage ingestion - see [1-ingestion-patterns.md](1-ingestion-patterns.md) - -### Reading Tables in Python - -**Modern SDP Best Practice:** -- Use `spark.read.table()` for batch reads -- Use `spark.readStream.table()` for streaming reads -- Don't use `dp.read()` or `dp.read_stream()` (old syntax, no longer documented) -- Don't use `dlt.read()` or `dlt.read_stream()` (legacy DLT API) - -**Key Point:** SDP automatically tracks table dependencies from standard Spark DataFrame operations. No special read APIs are needed. - -#### Three-Tier Identifier Resolution - -SDP supports three levels of table name qualification: - -| Level | Syntax | When to Use | -|-------|--------|-------------| -| **Unqualified** | `spark.read.table("my_table")` | Reading tables within the same pipeline's target catalog/schema (recommended) | -| **Partially-qualified** | `spark.read.table("other_schema.my_table")` | Reading from different schema in same catalog | -| **Fully-qualified** | `spark.read.table("other_catalog.other_schema.my_table")` | Reading from external catalogs/schemas | - -#### Option 1: Unqualified Names (Recommended for Pipeline Tables) - -**Best practice for tables within the same pipeline.** SDP resolves unqualified names to the pipeline's configured target catalog and schema. This makes code portable across environments (dev/prod). - -```python -@dp.table(name="silver_clean") -def silver_clean(): - # Reads from pipeline's target catalog/schema (e.g., dev_catalog.dev_schema.bronze_raw) - return ( - spark.read.table("bronze_raw") - .filter(F.col("valid") == True) - ) - -@dp.table(name="silver_events") -def silver_events(): - # Streaming read from same pipeline's bronze_events table - return ( - spark.readStream.table("bronze_events") - .withColumn("processed_at", F.current_timestamp()) - ) -``` - -#### Option 2: Pipeline Parameters (For External Sources) - -**Use `spark.conf.get()` to parameterize external catalog/schema references.** Define parameters in pipeline configuration, then reference them at the module level. - -```python -from pyspark import pipelines as dp -from pyspark.sql import functions as F - -# Get parameterized values at module level (evaluated once at pipeline start) -source_catalog = spark.conf.get("source_catalog") -source_schema = spark.conf.get("source_schema", "sales") # with default - -@dp.table(name="transaction_summary") -def transaction_summary(): - return ( - spark.read.table(f"{source_catalog}.{source_schema}.transactions") - .groupBy("account_id") - .agg( - F.count("txn_id").alias("txn_count"), - F.sum("txn_amount").alias("account_revenue") - ) - ) -``` - -**Configure parameters in pipeline settings:** -- **Asset Bundles**: Add to `pipeline.yml` under `configuration:` -- **Manual/MCP**: Pass via `extra_settings.configuration` dict - -```yaml -# In resources/my_pipeline.pipeline.yml -configuration: - source_catalog: "shared_catalog" - source_schema: "sales" -``` - -#### Option 3: Fully-Qualified Names (For Fixed External References) - -Use when referencing specific external tables that don't change across environments: - -```python -@dp.table(name="enriched_orders") -def enriched_orders(): - # Pipeline-internal table (unqualified) - orders = spark.read.table("bronze_orders") - - # External reference table (fully-qualified) - products = spark.read.table("shared_catalog.reference.products") - - return orders.join(products, "product_id") -``` - -#### Choosing the Right Approach - -| Scenario | Recommended Approach | -|----------|---------------------| -| Reading tables created in same pipeline | **Unqualified names** - portable, uses target catalog/schema | -| Reading from external source that varies by environment | **Pipeline parameters** - configurable per deployment | -| Reading from shared/reference tables with fixed location | **Fully-qualified names** - explicit and clear | -| Mixed pipeline (some internal, some external) | **Combine approaches** - unqualified for internal, parameters for external | - ---- - -## Common Issues - -| Issue | Solution | -|-------|----------| -| **Empty output tables** | Use `get_table_details` to verify, check upstream sources | -| **Pipeline stuck INITIALIZING** | Normal for serverless, wait a few minutes | -| **"Column not found"** | Check `schemaHints` match actual data | -| **Streaming reads fail** | Use `FROM STREAM(table)` for streaming sources | -| **Timeout during run** | Increase `timeout`, or use `wait_for_completion=False` and poll with `get_update` | -| **MV doesn't refresh** | Enable row tracking on source tables | -| **SCD2 schema errors** | Let SDP infer START_AT/END_AT columns | - -**For detailed errors**, the `result["message"]` from `create_or_update_pipeline` includes suggested next steps. Use `get_pipeline_events(pipeline_id=...)` for full stack traces. - ---- - -## Advanced Pipeline Configuration - -For advanced configuration options (development mode, continuous pipelines, custom clusters, notifications, Python dependencies, etc.), see **[7-advanced-configuration.md](7-advanced-configuration.md)**. - ---- - -## Platform Constraints - -### Serverless Pipeline Requirements (Default) -| Requirement | Details | -|-------------|---------| -| **Unity Catalog** | Required - serverless pipelines always use UC | -| **Workspace Region** | Must be in serverless-enabled region | -| **Serverless Terms** | Must accept serverless terms of use | -| **CDC Features** | Requires serverless (or Pro/Advanced with classic clusters) | - -### Serverless Limitations (When Classic Clusters Required) -| Limitation | Workaround | -|------------|-----------| -| **R language** | Not supported - use classic clusters if required | -| **Spark RDD APIs** | Not supported - use classic clusters if required | -| **JAR libraries** | Not supported - use classic clusters if required | -| **Maven coordinates** | Not supported - use classic clusters if required | -| **DBFS root access** | Limited - must use Unity Catalog external locations | -| **Global temp views** | Not supported | - -### General Constraints -| Constraint | Details | -|------------|---------| -| **Schema Evolution** | Streaming tables require full refresh for incompatible changes | -| **SQL Limitations** | PIVOT clause unsupported | -| **Sinks** | Python only, streaming only, append flows only | - -**Default to serverless** unless user explicitly requires R, RDD APIs, or JAR libraries. \ No newline at end of file diff --git a/.claude/skills/spark-python-data-source/SKILL.md b/.claude/skills/spark-python-data-source/SKILL.md new file mode 100644 index 00000000..4f90c60c --- /dev/null +++ b/.claude/skills/spark-python-data-source/SKILL.md @@ -0,0 +1,157 @@ +--- +name: spark-python-data-source +description: Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies. +--- + +# spark-python-data-source + +Build custom Python data sources for Apache Spark 4.0+ to read from and write to external systems in batch and streaming modes. + +## Instructions + +You are an experienced Spark developer building custom Python data sources using the PySpark DataSource API. Follow these principles and patterns. + +### Core Architecture + +Each data source follows a flat, single-level inheritance structure: + +1. **DataSource class** — entry point that returns readers/writers +2. **Base Reader/Writer classes** — shared logic for options and data processing +3. **Batch classes** — inherit from base + `DataSourceReader`/`DataSourceWriter` +4. **Stream classes** — inherit from base + `DataSourceStreamReader`/`DataSourceStreamWriter` + +See [implementation-template.md](references/implementation-template.md) for the full annotated skeleton covering all four modes (batch read/write, stream read/write). + +### Spark-Specific Design Constraints + +These are specific to the PySpark DataSource API and its driver/executor architecture — general Python best practices (clean code, minimal dependencies, no premature abstraction) still apply but aren't repeated here. + +**Flat single-level inheritance only.** PySpark serializes reader/writer instances to ship them to executors. Complex inheritance hierarchies and abstract base classes break serialization and make cross-process debugging painful. Use one shared base class mixed with the PySpark interface (e.g., `class YourBatchWriter(YourWriter, DataSourceWriter)`). + +**Import third-party libraries inside executor methods.** The `read()` and `write()` methods run on remote executor processes that don't share the driver's Python environment. Top-level imports from the driver won't be available on executors — always import libraries like `requests` or database drivers inside the methods that run on workers. + +**Minimize dependencies.** Every package you add must be installed on all executor nodes in the cluster, not just the driver. Prefer the standard library; when external packages are needed, keep them few and well-known. + +**No async/await** unless the external system's SDK is async-only. The PySpark DataSource API is synchronous, so async adds complexity with no benefit. + +### Project Setup + +Create a Python project using a packaging tool such as `uv`, `poetry`, or `hatch`. Examples use `uv` (substitute your tool of choice): + +```bash +uv init your-datasource +cd your-datasource +uv add pyspark pytest pytest-spark +``` + +``` +your-datasource/ +├── pyproject.toml +├── src/ +│ └── your_datasource/ +│ ├── __init__.py +│ └── datasource.py +└── tests/ + ├── conftest.py + └── test_datasource.py +``` + +Run all commands through the packaging tool so they execute within the correct virtual environment: + +```bash +uv run pytest # Run tests +uv run ruff check src/ # Lint +uv run ruff format src/ # Format +uv build # Build wheel +``` + +### Key Implementation Decisions + +**Partitioning Strategy** — choose based on data source characteristics: +- Time-based: for APIs with temporal data +- Token-range: for distributed databases +- ID-range: for paginated APIs +- See [partitioning-patterns.md](references/partitioning-patterns.md) for implementations of each strategy + +**Authentication** — support multiple methods in priority order: +- Databricks Unity Catalog credentials +- Cloud default credentials (managed identity) +- Explicit credentials (service principal, API key, username/password) +- See [authentication-patterns.md](references/authentication-patterns.md) for patterns with fallback chains + +**Type Conversion** — map between Spark and external types: +- Handle nulls, timestamps, UUIDs, collections +- See [type-conversion.md](references/type-conversion.md) for bidirectional mapping tables and helpers + +**Streaming Offsets** — design for exactly-once semantics: +- JSON-serializable offset class +- Non-overlapping partition boundaries +- See [streaming-patterns.md](references/streaming-patterns.md) for offset tracking and watermark patterns + +**Error Handling** — implement retries and resilience: +- Exponential backoff for transient failures (network, rate limits) +- Circuit breakers for cascading failures +- See [error-handling.md](references/error-handling.md) for retry decorators and failure classification + +### Testing + +```python +import pytest +from unittest.mock import patch, Mock + +@pytest.fixture +def spark(): + from pyspark.sql import SparkSession + return SparkSession.builder.master("local[2]").getOrCreate() + +def test_data_source_name(): + assert YourDataSource.name() == "your-format" + +def test_writer_sends_data(spark): + with patch('requests.post') as mock_post: + mock_post.return_value = Mock(status_code=200) + + df = spark.createDataFrame([(1, "test")], ["id", "value"]) + df.write.format("your-format").option("url", "http://api").save() + + assert mock_post.called +``` + +See [testing-patterns.md](references/testing-patterns.md) for unit/integration test patterns, fixtures, and running tests. + +### Reference Implementations + +Study these for real-world patterns: +- [cyber-spark-data-connectors](https://github.com/alexott/cyber-spark-data-connectors) — Sentinel, Splunk, REST +- [spark-cassandra-data-source](https://github.com/alexott/spark-cassandra-data-source) — Token-range partitioning +- [pyspark-hubspot](https://github.com/dgomez04/pyspark-hubspot) — REST API pagination +- [pyspark-mqtt](https://github.com/databricks-industry-solutions/python-data-sources/tree/main/mqtt) — Streaming with TLS + +## Example Prompts + +``` +Create a Spark data source for reading from MongoDB with sharding support +Build a streaming connector for RabbitMQ with at-least-once delivery +Implement a batch writer for Snowflake with staged uploads +Write a data source for REST API with OAuth2 authentication and pagination +``` + +## Related + +- databricks-testing: Test data sources on Databricks clusters +- databricks-spark-declarative-pipelines: Use custom sources in DLT pipelines +- python-dev: Python development best practices + +## References + +- [implementation-template.md](references/implementation-template.md) — Full annotated skeleton; read when starting a new data source +- [partitioning-patterns.md](references/partitioning-patterns.md) — Read when the source supports parallel reads and you need to split work across executors +- [authentication-patterns.md](references/authentication-patterns.md) — Read when the external system requires credentials or tokens +- [type-conversion.md](references/type-conversion.md) — Read when mapping between Spark types and the external system's type system +- [streaming-patterns.md](references/streaming-patterns.md) — Read when implementing `DataSourceStreamReader` or `DataSourceStreamWriter` +- [error-handling.md](references/error-handling.md) — Read when adding retry logic or handling transient failures +- [testing-patterns.md](references/testing-patterns.md) — Read when writing tests; covers unit, integration, and performance testing +- [production-patterns.md](references/production-patterns.md) — Read when hardening for production: observability, security, input validation +- [Official Databricks Documentation](https://docs.databricks.com/aws/en/pyspark/datasources) +- [Apache Spark Python DataSource Tutorial](https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html) +- [awesome-python-datasources](https://github.com/allisonwang-db/awesome-python-datasources) — Directory of community implementations diff --git a/.claude/skills/spark-python-data-source/references/authentication-patterns.md b/.claude/skills/spark-python-data-source/references/authentication-patterns.md new file mode 100644 index 00000000..700f516e --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/authentication-patterns.md @@ -0,0 +1,361 @@ +# Authentication Patterns + +Multi-method authentication strategies with clear priority ordering. + +## Priority-Based Authentication + +Support multiple authentication methods with fallback: + +```python +class AuthenticatedDataSource(DataSource): + def __init__(self, options): + # Priority 1: Databricks Unity Catalog credential + self.databricks_credential = options.get("databricks_credential") + + # Priority 2: Cloud default credential (managed identity) + self.default_credential = options.get("default_credential", "false").lower() == "true" + + # Priority 3: Service principal + self.tenant_id = options.get("tenant_id") + self.client_id = options.get("client_id") + self.client_secret = options.get("client_secret") + + # Priority 4: API key + self.api_key = options.get("api_key") + + # Priority 5: Username/password + self.username = options.get("username") + self.password = options.get("password") + + # Validate at least one method is configured + self._validate_auth() + + def _validate_auth(self): + """Validate at least one auth method is configured.""" + has_databricks_cred = bool(self.databricks_credential) + has_default_cred = self.default_credential + has_service_principal = all([self.tenant_id, self.client_id, self.client_secret]) + has_api_key = bool(self.api_key) + has_basic_auth = bool(self.username and self.password) + + if not any([has_databricks_cred, has_default_cred, has_service_principal, + has_api_key, has_basic_auth]): + raise AssertionError( + "Authentication required. Provide one of: " + "'databricks_credential', 'default_credential=true', " + "'tenant_id/client_id/client_secret', 'api_key', or 'username/password'" + ) +``` + +## Azure Authentication + +### Unity Catalog Service Credential + +```python +def _get_azure_credential_uc(credential_name): + """Get credential from Unity Catalog.""" + import databricks.service_credentials + + return databricks.service_credentials.getServiceCredentialsProvider(credential_name) +``` + +### Default Credential (Managed Identity) + +```python +def _get_azure_credential_default(authority=None): + """Get DefaultAzureCredential for managed identity.""" + from azure.identity import DefaultAzureCredential + + if authority: + return DefaultAzureCredential(authority=authority) + return DefaultAzureCredential() +``` + +### Service Principal + +```python +def _get_azure_credential_sp(tenant_id, client_id, client_secret, authority=None): + """Get service principal credential.""" + from azure.identity import ClientSecretCredential + + if authority: + return ClientSecretCredential( + tenant_id=tenant_id, + client_id=client_id, + client_secret=client_secret, + authority=authority + ) + return ClientSecretCredential( + tenant_id=tenant_id, + client_id=client_id, + client_secret=client_secret + ) +``` + +### Multi-Cloud Support + +```python +def _get_azure_cloud_config(cloud_name): + """Get cloud-specific endpoints and authorities.""" + from azure.identity import AzureAuthorityHosts + + cloud_configs = { + "public": (None, None), + "government": ( + AzureAuthorityHosts.AZURE_GOVERNMENT, + "https://api.loganalytics.us" + ), + "china": ( + AzureAuthorityHosts.AZURE_CHINA, + "https://api.loganalytics.azure.cn" + ), + } + + cloud = (cloud_name or "public").lower().strip() + + if cloud not in cloud_configs: + valid = ", ".join(cloud_configs.keys()) + raise ValueError(f"Invalid cloud '{cloud_name}'. Valid: {valid}") + + return cloud_configs[cloud] + +def _create_azure_client_with_cloud(options): + """Create Azure client with cloud-specific configuration.""" + cloud_name = options.get("azure_cloud", "public") + authority, endpoint = _get_azure_cloud_config(cloud_name) + + # Get credential based on priority + credential = _get_credential(options, authority) + + # Create client with cloud-specific endpoint + from azure.monitor.query import LogsQueryClient + + if endpoint: + return LogsQueryClient(credential, endpoint=endpoint) + return LogsQueryClient(credential) +``` + +## API Key Authentication + +### Header-Based + +```python +def _get_api_key_auth(api_key): + """Get API key authentication headers.""" + return {"Authorization": f"Bearer {api_key}"} + +def _create_session_with_api_key(api_key): + """Create requests session with API key.""" + import requests + + session = requests.Session() + session.headers.update({"Authorization": f"Bearer {api_key}"}) + return session +``` + +### Query Parameter-Based + +```python +def _build_url_with_api_key(base_url, api_key): + """Add API key as query parameter.""" + from urllib.parse import urlencode + + params = {"api_key": api_key} + return f"{base_url}?{urlencode(params)}" +``` + +## Basic Authentication + +```python +def _get_basic_auth(username, password): + """Get HTTP Basic Auth.""" + from requests.auth import HTTPBasicAuth + return HTTPBasicAuth(username, password) + +def _create_session_with_basic_auth(username, password): + """Create session with basic auth.""" + import requests + + session = requests.Session() + session.auth = (username, password) + return session +``` + +## OAuth2 Authentication + +### Client Credentials Flow + +```python +def _get_oauth2_token(token_url, client_id, client_secret, scope): + """Get OAuth2 token using client credentials.""" + import requests + + response = requests.post( + token_url, + data={ + "grant_type": "client_credentials", + "client_id": client_id, + "client_secret": client_secret, + "scope": scope + } + ) + response.raise_for_status() + + return response.json()["access_token"] + +class OAuth2Writer: + def __init__(self, options): + self.token_url = options["token_url"] + self.client_id = options["client_id"] + self.client_secret = options["client_secret"] + self.scope = options.get("scope", "") + self._token = None + self._token_expiry = None + + def _get_valid_token(self): + """Get valid token, refresh if expired.""" + from datetime import datetime, timedelta + + if not self._token or datetime.now() >= self._token_expiry: + self._token = _get_oauth2_token( + self.token_url, + self.client_id, + self.client_secret, + self.scope + ) + # Assume 1 hour expiry if not provided + self._token_expiry = datetime.now() + timedelta(hours=1) + + return self._token + + def write(self, iterator): + """Write with OAuth2 authentication.""" + import requests + + token = self._get_valid_token() + headers = {"Authorization": f"Bearer {token}"} + + for row in iterator: + requests.post(self.url, json=row.asDict(), headers=headers) +``` + +## Complete Authentication Factory + +```python +def get_credential(options): + """ + Get credential based on configuration priority. + + Priority: + 1. databricks_credential + 2. default_credential + 3. Service principal (tenant_id/client_id/client_secret) + 4. API key + 5. Username/password + """ + + # Priority 1: Databricks credential + if options.get("databricks_credential"): + import databricks.service_credentials + return databricks.service_credentials.getServiceCredentialsProvider( + options["databricks_credential"] + ) + + # Priority 2: Cloud default credential + if options.get("default_credential", "false").lower() == "true": + authority = options.get("authority") + if authority: + from azure.identity import DefaultAzureCredential + return DefaultAzureCredential(authority=authority) + from azure.identity import DefaultAzureCredential + return DefaultAzureCredential() + + # Priority 3: Service principal + if all(k in options for k in ["tenant_id", "client_id", "client_secret"]): + from azure.identity import ClientSecretCredential + authority = options.get("authority") + if authority: + return ClientSecretCredential( + tenant_id=options["tenant_id"], + client_id=options["client_id"], + client_secret=options["client_secret"], + authority=authority + ) + return ClientSecretCredential( + tenant_id=options["tenant_id"], + client_id=options["client_id"], + client_secret=options["client_secret"] + ) + + # Priority 4: API key + if "api_key" in options: + return {"Authorization": f"Bearer {options['api_key']}"} + + # Priority 5: Basic auth + if "username" in options and "password" in options: + from requests.auth import HTTPBasicAuth + return HTTPBasicAuth(options["username"], options["password"]) + + raise ValueError("No valid authentication method configured") +``` + +## Security Best Practices + +### Never Log Sensitive Values + +```python +class SecureDataSource(DataSource): + def __init__(self, options): + self._sensitive_keys = { + "password", "api_key", "client_secret", "token", "access_token" + } + + # Store actual values + self.options = options + + # Create sanitized version for logging + self._safe_options = self._sanitize_options(options) + + def _sanitize_options(self, options): + """Mask sensitive values for logging.""" + safe = {} + for key, value in options.items(): + if key.lower() in self._sensitive_keys: + safe[key] = "***REDACTED***" + else: + safe[key] = value + return safe + + def __repr__(self): + return f"SecureDataSource({self._safe_options})" +``` + +### Use Secrets Management + +```python +def _load_secrets_from_dbutils(scope, keys): + """Load secrets from Databricks secrets.""" + try: + from pyspark.dbutils import DBUtils + from pyspark.sql import SparkSession + + spark = SparkSession.getActiveSession() + dbutils = DBUtils(spark) + + secrets = {} + for key in keys: + secrets[key] = dbutils.secrets.get(scope=scope, key=key) + + return secrets + + except Exception as e: + raise ValueError(f"Failed to load secrets from scope '{scope}': {e}") + +# Usage +if "secret_scope" in options: + secrets = _load_secrets_from_dbutils( + options["secret_scope"], + ["password", "api_key"] + ) + options.update(secrets) +``` diff --git a/.claude/skills/spark-python-data-source/references/error-handling.md b/.claude/skills/spark-python-data-source/references/error-handling.md new file mode 100644 index 00000000..01bbf2f9 --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/error-handling.md @@ -0,0 +1,432 @@ +# Error Handling and Resilience + +Patterns for retries, circuit breakers, and graceful degradation. + +## Exponential Backoff + +Retry with exponential backoff for transient failures: + +```python +def write_with_retry(self, iterator): + """Write with exponential backoff.""" + import time + + max_retries = int(self.options.get("max_retries", "5")) + initial_backoff = float(self.options.get("initial_backoff", "1.0")) + max_backoff = float(self.options.get("max_backoff", "60.0")) + + for row in iterator: + retry_count = 0 + + while retry_count <= max_retries: + try: + self._send_data(row) + break # Success + + except Exception as e: + if not self._is_retryable_error(e): + # Non-retryable error - fail immediately + raise + + if retry_count >= max_retries: + # Max retries exceeded + raise Exception(f"Max retries ({max_retries}) exceeded: {e}") + + # Calculate backoff with exponential growth + backoff = min(initial_backoff * (2 ** retry_count), max_backoff) + time.sleep(backoff) + retry_count += 1 + +def _is_retryable_error(self, error): + """Determine if error is retryable.""" + from requests.exceptions import RequestException, Timeout, ConnectionError + + # Network errors are retryable + if isinstance(error, (Timeout, ConnectionError)): + return True + + # HTTP errors + if hasattr(error, 'response') and error.response: + status_code = error.response.status_code + # Retry on 429 (throttling) and 5xx (server errors) + if status_code == 429 or 500 <= status_code < 600: + return True + + return False +``` + +## Retry with Throttling Respect + +Handle API rate limiting with Retry-After header: + +```python +def write_with_throttling(self, iterator): + """Write with respect for rate limits.""" + import time + from requests.exceptions import HTTPError + + for row in iterator: + max_attempts = 5 + attempt = 0 + + while attempt < max_attempts: + try: + self._send_data(row) + break + + except HTTPError as e: + if e.response.status_code == 429: + # Throttled - respect Retry-After header + retry_after = self._get_retry_after(e.response) + time.sleep(retry_after) + attempt += 1 + else: + raise + + if attempt >= max_attempts: + raise Exception("Max retry attempts for throttling exceeded") + +def _get_retry_after(self, response): + """Extract retry delay from Retry-After header.""" + retry_after = response.headers.get("Retry-After") + + if retry_after: + try: + # Try as seconds (int) + return int(retry_after) + except ValueError: + # Try as HTTP date + from datetime import datetime + try: + retry_date = datetime.strptime(retry_after, "%a, %d %b %Y %H:%M:%S GMT") + delay = (retry_date - datetime.utcnow()).total_seconds() + return max(0, delay) + except ValueError: + pass + + # Default fallback + return 1.0 +``` + +## Circuit Breaker + +Prevent cascading failures with circuit breaker pattern: + +```python +class CircuitBreaker: + """Circuit breaker to prevent cascading failures.""" + + def __init__(self, threshold=10, timeout=300): + self.threshold = threshold # failures before opening + self.timeout = timeout # seconds before trying again + self.consecutive_failures = 0 + self.circuit_open = False + self.circuit_open_until = None + + def record_success(self): + """Record successful operation.""" + self.consecutive_failures = 0 + + def record_failure(self): + """Record failed operation.""" + from datetime import datetime, timedelta + + self.consecutive_failures += 1 + + if self.consecutive_failures >= self.threshold: + self.circuit_open = True + self.circuit_open_until = datetime.now() + timedelta(seconds=self.timeout) + + def is_open(self): + """Check if circuit is open.""" + from datetime import datetime + + if self.circuit_open: + if datetime.now() >= self.circuit_open_until: + # Timeout expired - try again + self.circuit_open = False + self.consecutive_failures = 0 + return False + return True + + return False + +class ResilientWriter: + def __init__(self, options): + self.circuit_breaker = CircuitBreaker( + threshold=int(options.get("circuit_breaker_threshold", "10")), + timeout=int(options.get("circuit_breaker_timeout", "300")) + ) + + def write(self, iterator): + """Write with circuit breaker protection.""" + for row in iterator: + if self.circuit_breaker.is_open(): + raise Exception("Circuit breaker open - too many failures") + + try: + self._send_data(row) + self.circuit_breaker.record_success() + + except Exception as e: + self.circuit_breaker.record_failure() + raise +``` + +## Graceful Degradation + +Handle partial failures and fallback strategies: + +```python +def read_with_fallback(self, partition): + """Read with fallback to secondary sources.""" + try: + # Try primary source + yield from self._read_primary(partition) + + except ConnectionError as e: + # Primary failed - try secondary + if self.secondary_endpoint: + print(f"Primary failed, using secondary: {e}") + yield from self._read_secondary(partition) + else: + raise + + except TimeoutError as e: + # Timeout - try with smaller partitions + if partition.can_subdivide(): + print(f"Timeout, subdividing: {e}") + for sub_partition in partition.subdivide(): + yield from self.read(sub_partition) + else: + raise + + except PartialResultError as e: + # Partial results - log warning and continue + print(f"Warning: Partial results for partition {partition.id}: {e}") + yield from e.partial_results +``` + +## Bulk Operation Error Handling + +Handle errors in bulk operations: + +```python +def write_batch_with_error_handling(self, iterator): + """Write in batches with individual error tracking.""" + from cassandra.concurrent import execute_concurrent_with_args + + batch_size = int(self.options.get("batch_size", "1000")) + fail_on_first_error = self.options.get("fail_on_first_error", "true").lower() == "true" + + batch_params = [] + failed_rows = [] + + for row in iterator: + batch_params.append(self._row_to_params(row)) + + if len(batch_params) >= batch_size: + # Execute batch + results = execute_concurrent_with_args( + self.session, + self.prepared_statement, + batch_params, + concurrency=100, + raise_on_first_error=fail_on_first_error + ) + + # Check for failures + for success, result_or_error in results: + if not success: + failed_rows.append((batch_params[i], result_or_error)) + + batch_params = [] + + # Final batch + if batch_params: + results = execute_concurrent_with_args( + self.session, + self.prepared_statement, + batch_params, + concurrency=100, + raise_on_first_error=fail_on_first_error + ) + + for i, (success, result_or_error) in enumerate(results): + if not success: + failed_rows.append((batch_params[i], result_or_error)) + + # Handle failed rows + if failed_rows: + if fail_on_first_error: + raise Exception(f"{len(failed_rows)} rows failed to write") + else: + # Log failures but continue + print(f"Warning: {len(failed_rows)} rows failed to write") +``` + +## Dead Letter Queue + +Store failed records for later processing: + +```python +class DeadLetterQueueWriter: + """Writer with dead letter queue for failed records.""" + + def __init__(self, options): + self.dlq_path = options.get("dlq_path") + self.dlq_enabled = bool(self.dlq_path) + + def write(self, iterator): + """Write with DLQ support.""" + from datetime import datetime + import json + + successful = 0 + failed = 0 + + for row in iterator: + try: + self._send_data(row) + successful += 1 + + except Exception as e: + failed += 1 + + if self.dlq_enabled: + self._write_to_dlq(row, e) + else: + raise + + return { + "successful": successful, + "failed": failed + } + + def _write_to_dlq(self, row, error): + """Write failed record to dead letter queue.""" + from datetime import datetime + import json + import os + + dlq_record = { + "timestamp": datetime.now().isoformat(), + "error": str(error), + "error_type": type(error).__name__, + "row": row.asDict() + } + + # Append to DLQ file + os.makedirs(os.path.dirname(self.dlq_path), exist_ok=True) + + with open(self.dlq_path, 'a') as f: + f.write(json.dumps(dlq_record) + '\n') +``` + +## Timeout Handling + +Enforce operation timeouts: + +```python +import signal +from contextlib import contextmanager + +class TimeoutError(Exception): + pass + +def timeout_handler(signum, frame): + raise TimeoutError("Operation timed out") + +@contextmanager +def timeout(seconds): + """Context manager for operation timeout.""" + signal.signal(signal.SIGALRM, timeout_handler) + signal.alarm(seconds) + try: + yield + finally: + signal.alarm(0) + +class TimeoutWriter: + def write(self, iterator): + """Write with per-row timeout.""" + timeout_seconds = int(self.options.get("write_timeout", "30")) + + for row in iterator: + try: + with timeout(timeout_seconds): + self._send_data(row) + + except TimeoutError: + print(f"Write timeout after {timeout_seconds}s") + raise +``` + +## Error Aggregation + +Collect and report errors systematically: + +```python +class ErrorAggregator: + """Aggregate errors for batch reporting.""" + + def __init__(self): + self.errors = [] + self.error_counts = {} + + def record_error(self, error, context=None): + """Record an error with context.""" + error_type = type(error).__name__ + error_msg = str(error) + + self.errors.append({ + "type": error_type, + "message": error_msg, + "context": context + }) + + # Count by type + self.error_counts[error_type] = self.error_counts.get(error_type, 0) + 1 + + def get_summary(self): + """Get error summary.""" + return { + "total_errors": len(self.errors), + "by_type": self.error_counts, + "sample_errors": self.errors[:10] # First 10 + } + +class ErrorAwareWriter: + def write(self, iterator): + """Write with error aggregation.""" + aggregator = ErrorAggregator() + successful = 0 + + for i, row in enumerate(iterator): + try: + self._send_data(row) + successful += 1 + + except Exception as e: + aggregator.record_error(e, context={"row_index": i}) + + # Report summary + if aggregator.errors: + summary = aggregator.get_summary() + print(f"Completed with {successful} success, {summary['total_errors']} errors") + print(f"Error breakdown: {summary['by_type']}") + + if summary['total_errors'] > successful: + raise Exception(f"Too many errors: {summary}") +``` + +## Best Practices + +1. **Retry Only Transient Errors**: Don't retry client errors (4xx) +2. **Respect Rate Limits**: Use Retry-After headers and backoff +3. **Circuit Breakers**: Prevent cascading failures in distributed systems +4. **Timeout Operations**: Set reasonable timeouts to prevent hangs +5. **Log Errors**: Capture error context for debugging +6. **Dead Letter Queues**: Store failed records for later analysis +7. **Monitor Failure Rates**: Alert on anomalous error rates +8. **Graceful Degradation**: Continue with partial results when appropriate diff --git a/.claude/skills/spark-python-data-source/references/implementation-template.md b/.claude/skills/spark-python-data-source/references/implementation-template.md new file mode 100644 index 00000000..045fe944 --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/implementation-template.md @@ -0,0 +1,141 @@ +# Implementation Template + +Full skeleton for a Python data source covering all four modes: batch read, batch write, stream read, stream write. Adapt to your needs — most connectors only implement a subset. + +```python +from pyspark.sql.datasource import ( + DataSource, DataSourceReader, DataSourceWriter, + DataSourceStreamReader, DataSourceStreamWriter +) + +# 1. DataSource class — entry point that returns readers/writers +class YourDataSource(DataSource): + @classmethod + def name(cls): + return "your-format" + + def __init__(self, options): + self.options = options + + def schema(self): + return self._infer_or_return_schema() + + def reader(self, schema): + return YourBatchReader(self.options, schema) + + def streamReader(self, schema): + return YourStreamReader(self.options, schema) + + def writer(self, schema, overwrite): + return YourBatchWriter(self.options, schema) + + def streamWriter(self, schema, overwrite): + return YourStreamWriter(self.options, schema) + +# 2. Base Writer — shared logic for batch and stream writing +# Plain class (not a DataSourceWriter yet) so batch/stream +# subclasses can mix it in with the right PySpark base. +class YourWriter: + def __init__(self, options, schema=None): + self.url = options.get("url") + assert self.url, "url is required" + self.batch_size = int(options.get("batch_size", "50")) + self.schema = schema + + def write(self, iterator): + # Import here — this runs on executors, not the driver. + # Executor processes don't share the driver's module state. + import requests + from pyspark import TaskContext + + context = TaskContext.get() + partition_id = context.partitionId() + + msgs = [] + cnt = 0 + + for row in iterator: + cnt += 1 + msgs.append(row.asDict()) + + if len(msgs) >= self.batch_size: + self._send_batch(msgs) + msgs = [] + + if msgs: + self._send_batch(msgs) + + return SimpleCommitMessage(partition_id=partition_id, count=cnt) + + def _send_batch(self, msgs): + # Implement send logic + pass + +# 3. Batch Writer — inherits shared logic + PySpark interface +class YourBatchWriter(YourWriter, DataSourceWriter): + pass + +# 4. Stream Writer — adds commit/abort for micro-batch semantics +class YourStreamWriter(YourWriter, DataSourceStreamWriter): + def commit(self, messages, batchId): + pass + + def abort(self, messages, batchId): + pass + +# 5. Base Reader — shared logic for batch and stream reading +class YourReader: + def __init__(self, options, schema): + self.url = options.get("url") + assert self.url, "url is required" + self.schema = schema + + def partitions(self): + return [YourPartition(0, start, end)] + + def read(self, partition): + # Import here — runs on executors + import requests + + response = requests.get(f"{self.url}?start={partition.start}") + for item in response.json(): + yield tuple(item.values()) + +# 6. Batch Reader +class YourBatchReader(YourReader, DataSourceReader): + pass + +# 7. Stream Reader — adds offset tracking for incremental reads +class YourStreamReader(YourReader, DataSourceStreamReader): + def initialOffset(self): + return {"offset": "0"} + + def latestOffset(self): + return {"offset": str(self._get_latest())} + + def partitions(self, start, end): + return [YourPartition(0, start["offset"], end["offset"])] + + def commit(self, end): + pass +``` + +## Registration and Usage + +```python +# Register +from your_package import YourDataSource +spark.dataSource.register(YourDataSource) + +# Batch read +df = spark.read.format("your-format").option("url", "...").load() + +# Batch write +df.write.format("your-format").option("url", "...").save() + +# Streaming read +df = spark.readStream.format("your-format").option("url", "...").load() + +# Streaming write +df.writeStream.format("your-format").option("url", "...").start() +``` diff --git a/.claude/skills/spark-python-data-source/references/partitioning-patterns.md b/.claude/skills/spark-python-data-source/references/partitioning-patterns.md new file mode 100644 index 00000000..699e75a5 --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/partitioning-patterns.md @@ -0,0 +1,319 @@ +# Partitioning Patterns + +Strategies for distributing reads across Spark executors for parallel processing. + +## Time-Based Partitioning + +For APIs with temporal data or streaming sources. + +### Fixed Duration Partitions + +```python +from pyspark.sql.datasource import InputPartition +from datetime import datetime, timedelta + +class TimeRangePartition(InputPartition): + def __init__(self, start_time, end_time): + self.start_time = start_time + self.end_time = end_time + +class TimeBasedReader: + def __init__(self, options, schema): + self.partition_duration = int(options.get("partition_duration", "3600")) # seconds + # Parse start/end time from options + + def partitions(self): + """Split time range into fixed-duration partitions.""" + partitions = [] + current = self.start_time + delta = timedelta(seconds=self.partition_duration) + + while current < self.end_time: + next_time = min(current + delta, self.end_time) + partitions.append(TimeRangePartition(current, next_time)) + current = next_time + + return partitions + + def read(self, partition): + """Query data for specific time range.""" + response = self._query_api( + start=partition.start_time, + end=partition.end_time + ) + for item in response: + yield self._convert_to_row(item) +``` + +### Auto-Subdividing for Large Results + +Handle APIs with result size limits by automatically subdividing large partitions: + +```python +class AutoSubdivideReader: + def __init__(self, options, schema): + self.min_partition_seconds = int(options.get("min_partition_seconds", "60")) + self.max_retries = int(options.get("max_retries", "5")) + + def read(self, partition): + """Read with automatic subdivision on size limit errors.""" + try: + response = self._execute_query(partition.start_time, partition.end_time) + + # Check if response is partial due to size limits + if self._is_size_limit_error(response): + yield from self._read_with_subdivision(partition) + return + + yield from self._process_response(response) + + except Exception as e: + raise + + def _read_with_subdivision(self, partition): + """Recursively subdivide large partitions.""" + duration = (partition.end_time - partition.start_time).total_seconds() + + if duration <= self.min_partition_seconds: + raise Exception( + f"Cannot subdivide further. Duration {duration}s at minimum. " + f"Consider more selective query or increase min_partition_seconds." + ) + + # Split in half + midpoint = partition.start_time + timedelta(seconds=duration / 2) + + first_half = TimeRangePartition(partition.start_time, midpoint) + second_half = TimeRangePartition(midpoint, partition.end_time) + + yield from self.read(first_half) + yield from self.read(second_half) + + def _is_size_limit_error(self, response): + """Detect result size limit errors.""" + size_limit_codes = [ + "QueryExecutionResultSizeLimitExceeded", + "ResponsePayloadTooLarge", + "E_QUERY_RESULT_SET_TOO_LARGE", + ] + + if hasattr(response, "error") and response.error: + if response.error.code in size_limit_codes: + return True + + error_str = str(response.error).lower() + return any(p in error_str for p in ["size limit", "too large", "exceed"]) + + return False +``` + +## Token-Range Partitioning + +For distributed databases using consistent hashing (Cassandra, ScyllaDB). + +### Cassandra Token-Range Pattern + +```python +from collections import namedtuple + +class TokenRangePartition(InputPartition): + def __init__(self, partition_id, start_token, end_token, pk_columns, + is_wrap_around=False, min_token=None): + self.partition_id = partition_id + self.start_token = start_token # None = unbounded + self.end_token = end_token # None = unbounded + self.pk_columns = pk_columns + self.is_wrap_around = is_wrap_around + self.min_token = min_token + +class TokenRangeReader: + def _get_token_ranges(self, token_map): + """Compute token ranges from cluster token ring.""" + if not token_map or not token_map.ring: + return [] + + TokenRange = namedtuple('TokenRange', ['start', 'end']) + ranges = [] + ring = sorted(token_map.ring) + + for i in range(len(ring)): + start = ring[i] + end = ring[(i + 1) % len(ring)] # Wrap around + ranges.append(TokenRange(start=start, end=end)) + + return ranges + + def partitions(self): + """Create partitions following TokenRangesScan.java logic.""" + if not self.token_ranges: + return [] + + partitions = [] + sorted_ranges = sorted(self.token_ranges) + partition_id = 0 + + min_token_obj = sorted_ranges[0].start + min_token = min_token_obj.value if hasattr(min_token_obj, 'value') else str(min_token_obj) + + for i, token_range in enumerate(sorted_ranges): + start_value = token_range.start.value if hasattr(token_range.start, 'value') else str(token_range.start) + end_value = token_range.end.value if hasattr(token_range.end, 'value') else str(token_range.end) + + if start_value == end_value: + # Case 1: Single-node cluster (entire ring) + partition = TokenRangePartition( + partition_id=partition_id, + start_token=min_token, + end_token=None, # Unbounded + pk_columns=self.pk_columns, + is_wrap_around=True, + min_token=min_token + ) + partitions.append(partition) + partition_id += 1 + + elif i == 0: + # Case 2: First range - split into TWO partitions + # Partition 1: token <= minToken (wrap-around) + partition1 = TokenRangePartition( + partition_id=partition_id, + start_token=None, + end_token=min_token, + pk_columns=self.pk_columns, + is_wrap_around=True, + min_token=min_token + ) + partitions.append(partition1) + partition_id += 1 + + # Partition 2: token > start AND token <= end + partition2 = TokenRangePartition( + partition_id=partition_id, + start_token=start_value, + end_token=end_value, + pk_columns=self.pk_columns, + is_wrap_around=False, + min_token=min_token + ) + partitions.append(partition2) + partition_id += 1 + + elif end_value == min_token: + # Case 3: Range ending at minToken - no upper bound + partition = TokenRangePartition( + partition_id=partition_id, + start_token=start_value, + end_token=None, + pk_columns=self.pk_columns, + is_wrap_around=False, + min_token=min_token + ) + partitions.append(partition) + partition_id += 1 + + else: + # Case 4: Normal range - both bounds + partition = TokenRangePartition( + partition_id=partition_id, + start_token=start_value, + end_token=end_value, + pk_columns=self.pk_columns, + is_wrap_around=False, + min_token=min_token + ) + partitions.append(partition) + partition_id += 1 + + return partitions + + def read(self, partition): + """Build query with token range predicates.""" + pk_cols_str = ", ".join(partition.pk_columns) + + # Build WHERE clause based on bounds + if partition.start_token is None: + where_clause = f"token({pk_cols_str}) <= {partition.end_token}" + elif partition.end_token is None: + where_clause = f"token({pk_cols_str}) > {partition.start_token}" + else: + where_clause = ( + f"token({pk_cols_str}) > {partition.start_token} AND " + f"token({pk_cols_str}) <= {partition.end_token}" + ) + + query = f"SELECT {columns} FROM {table} WHERE {where_clause}" + + # Execute and yield results + for row in self._execute_query(query): + yield row +``` + +## ID-Range Partitioning + +For APIs with pagination or sequential IDs. + +```python +class IdRangePartition(InputPartition): + def __init__(self, partition_id, start_id, end_id): + self.partition_id = partition_id + self.start_id = start_id + self.end_id = end_id + +class IdRangeReader: + def __init__(self, options, schema): + self.num_partitions = int(options.get("num_partitions", "4")) + self.page_size = int(options.get("page_size", "1000")) + + def partitions(self): + """Split by ID ranges.""" + # Get total count from API + total = self._get_total_count() + partition_size = total // self.num_partitions + + partitions = [] + for i in range(self.num_partitions): + start_id = i * partition_size + end_id = (i + 1) * partition_size if i < self.num_partitions - 1 else total + partitions.append(IdRangePartition(i, start_id, end_id)) + + return partitions + + def read(self, partition): + """Paginate through ID range.""" + current_id = partition.start_id + + while current_id < partition.end_id: + response = self._query_api( + start_id=current_id, + limit=self.page_size + ) + + for item in response.items: + yield self._convert_to_row(item) + + current_id += self.page_size +``` + +## Partition Count Guidelines + +**For Batch Reads:** +- Start with 2-4x number of executor cores +- Adjust based on data volume and partition size +- Consider external system load limits + +**For Streaming Reads:** +- Use fixed-duration partitions (e.g., 1 hour) +- Let Spark handle parallelism across micro-batches +- Balance latency vs throughput + +**For Token-Range:** +- One partition per token range (determined by cluster) +- Naturally distributes based on data distribution +- May split first range into two partitions + +## Performance Considerations + +1. **Partition Size**: Aim for 128MB - 1GB per partition +2. **API Rate Limits**: Respect rate limits with concurrency controls +3. **Network Overhead**: Larger partitions reduce round-trips +4. **Skew Handling**: Monitor for data skew, repartition if needed diff --git a/.claude/skills/spark-python-data-source/references/production-patterns.md b/.claude/skills/spark-python-data-source/references/production-patterns.md new file mode 100644 index 00000000..6dfbd8a5 --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/production-patterns.md @@ -0,0 +1,384 @@ +# Production Patterns + +Observability, security, validation, and operational best practices. + +## Observability and Metrics + +Track operation metrics for monitoring: + +```python +class ObservableWriter: + """Writer with comprehensive metrics tracking.""" + + def write(self, iterator): + """Write with metrics collection.""" + from pyspark import TaskContext + from datetime import datetime + import time + + context = TaskContext.get() + partition_id = context.partitionId() + + metrics = { + "partition_id": partition_id, + "rows_processed": 0, + "rows_failed": 0, + "bytes_sent": 0, + "batches_sent": 0, + "retry_count": 0, + "start_time": time.time(), + "errors": [] + } + + try: + for row in iterator: + try: + size = self._send_row(row) + metrics["rows_processed"] += 1 + metrics["bytes_sent"] += size + + except Exception as e: + metrics["rows_failed"] += 1 + metrics["errors"].append({ + "type": type(e).__name__, + "message": str(e) + }) + + if not self.continue_on_error: + raise + + metrics["duration_seconds"] = time.time() - metrics["start_time"] + self._report_metrics(metrics) + + return SimpleCommitMessage( + partition_id=partition_id, + count=metrics["rows_processed"] + ) + + except Exception as e: + metrics["fatal_error"] = str(e) + self._report_failure(partition_id, metrics) + raise + + def _report_metrics(self, metrics): + """Report metrics to monitoring system.""" + # Example: CloudWatch, Prometheus, Databricks metrics + print(f"METRICS: {json.dumps(metrics)}") + + # Calculate derived metrics + if metrics["duration_seconds"] > 0: + throughput = metrics["rows_processed"] / metrics["duration_seconds"] + print(f"Throughput: {throughput:.2f} rows/second") +``` + +## Logging Best Practices + +Structured logging for production debugging: + +```python +import logging +import json + +# Configure structured logging +logging.basicConfig( + format='%(asctime)s %(levelname)s [%(name)s] %(message)s', + level=logging.INFO +) +logger = logging.getLogger(__name__) + +class StructuredLogger: + """Logger with structured output.""" + + @staticmethod + def log_operation(operation, context, **kwargs): + """Log operation with structured context.""" + log_entry = { + "operation": operation, + "context": context, + **kwargs + } + logger.info(json.dumps(log_entry)) + + @staticmethod + def log_error(operation, error, context): + """Log error with context.""" + log_entry = { + "operation": operation, + "error_type": type(error).__name__, + "error_message": str(error), + "context": context + } + logger.error(json.dumps(log_entry)) + +class LoggingWriter: + def write(self, iterator): + """Write with structured logging.""" + from pyspark import TaskContext + + context = TaskContext.get() + partition_id = context.partitionId() + + StructuredLogger.log_operation( + "write_start", + {"partition_id": partition_id} + ) + + try: + count = 0 + for row in iterator: + self._send_data(row) + count += 1 + + StructuredLogger.log_operation( + "write_complete", + {"partition_id": partition_id}, + rows_written=count + ) + + except Exception as e: + StructuredLogger.log_error( + "write_failed", + e, + {"partition_id": partition_id} + ) + raise +``` + +## Security Validation + +Input validation and sanitization for production data sources: + +```python +import re +import ipaddress + +class SecureDataSource: + """Data source with input validation.""" + + def __init__(self, options): + self._validate_options(options) + self.options = options + + def _validate_options(self, options): + """Validate options at system boundary.""" + required = ["host", "database", "table"] + missing = [opt for opt in required if opt not in options] + if missing: + raise ValueError(f"Missing required options: {', '.join(missing)}") + + self._validate_host(options["host"]) + + if "port" in options: + port = int(options["port"]) + if port < 1 or port > 65535: + raise ValueError(f"Port must be 1-65535, got {port}") + + self._validate_identifier(options["table"], "table") + + def _validate_host(self, host): + """Validate host is valid IP or hostname.""" + try: + ipaddress.ip_address(host) + return + except ValueError: + pass + if not re.match(r'^[a-zA-Z0-9][a-zA-Z0-9-\.]*[a-zA-Z0-9]$', host): + raise ValueError(f"Invalid host format: {host}") + + def _validate_identifier(self, identifier, name): + """Validate SQL identifier to prevent injection.""" + if not re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*$', identifier): + raise ValueError( + f"Invalid {name} identifier: {identifier}. " + f"Must contain only letters, numbers, and underscores." + ) +``` + +For credential sanitization in logs and secrets management, see [authentication-patterns.md](authentication-patterns.md) — the "Security Best Practices" and "Use Secrets Management" sections. + +## Configuration Validation + +Validate configuration before execution: + +```python +class ConfigValidator: + """Validate data source configuration.""" + + VALID_CONSISTENCY_LEVELS = { + "ONE", "TWO", "THREE", "QUORUM", "ALL", + "LOCAL_QUORUM", "EACH_QUORUM", "LOCAL_ONE" + } + + VALID_COMPRESSION = { + "none", "gzip", "snappy", "lz4", "zstd" + } + + @classmethod + def validate(cls, options): + """Validate all configuration options.""" + errors = [] + + # Validate consistency level + if "consistency" in options: + consistency = options["consistency"].upper() + if consistency not in cls.VALID_CONSISTENCY_LEVELS: + errors.append( + f"Invalid consistency level '{consistency}'. " + f"Valid: {', '.join(cls.VALID_CONSISTENCY_LEVELS)}" + ) + + # Validate compression + if "compression" in options: + compression = options["compression"].lower() + if compression not in cls.VALID_COMPRESSION: + errors.append( + f"Invalid compression '{compression}'. " + f"Valid: {', '.join(cls.VALID_COMPRESSION)}" + ) + + # Validate numeric ranges + if "timeout" in options: + timeout = int(options["timeout"]) + if timeout < 0 or timeout > 300: + errors.append(f"timeout must be 0-300 seconds, got {timeout}") + + if "batch_size" in options: + batch_size = int(options["batch_size"]) + if batch_size < 1 or batch_size > 10000: + errors.append(f"batch_size must be 1-10000, got {batch_size}") + + # Validate dependent options + if options.get("ssl_enabled", "false").lower() == "true": + if "ssl_ca_cert" not in options: + errors.append("ssl_ca_cert required when ssl_enabled=true") + + if errors: + raise ValueError("Configuration errors:\n" + "\n".join(f"- {e}" for e in errors)) +``` + +## Resource Cleanup + +Ensure proper resource cleanup: + +```python +class ManagedResourceWriter: + """Writer with guaranteed resource cleanup.""" + + def __init__(self, options): + self.options = options + self._connection = None + self._session = None + + def _get_connection(self): + """Lazy connection initialization.""" + if self._connection is None: + self._connection = self._create_connection() + return self._connection + + def write(self, iterator): + """Write with guaranteed cleanup.""" + try: + connection = self._get_connection() + + for row in iterator: + self._send_data(connection, row) + + finally: + # Always cleanup resources + self._cleanup() + + def _cleanup(self): + """Clean up resources.""" + if self._session: + try: + self._session.close() + except Exception as e: + logger.warning(f"Error closing session: {e}") + finally: + self._session = None + + if self._connection: + try: + self._connection.close() + except Exception as e: + logger.warning(f"Error closing connection: {e}") + finally: + self._connection = None + + def __del__(self): + """Cleanup on garbage collection.""" + self._cleanup() +``` + +## Health Checks + +Monitor system health: + +```python +class HealthCheckMixin: + """Mixin for health check functionality.""" + + def check_health(self): + """Perform health check before operations.""" + checks = { + "connection": self._check_connection(), + "authentication": self._check_authentication(), + "rate_limit": self._check_rate_limit(), + "disk_space": self._check_disk_space() + } + + failed = [name for name, passed in checks.items() if not passed] + + if failed: + raise Exception(f"Health check failed: {', '.join(failed)}") + + return checks + + def _check_connection(self): + """Check connection to external system.""" + try: + self._test_connection() + return True + except Exception as e: + logger.error(f"Connection check failed: {e}") + return False + + def _check_authentication(self): + """Check authentication is valid.""" + try: + self._verify_credentials() + return True + except Exception as e: + logger.error(f"Authentication check failed: {e}") + return False + + def _check_rate_limit(self): + """Check if under rate limits.""" + # Check current rate usage + current_rate = self._get_current_rate() + limit = self._get_rate_limit() + + return current_rate < limit * 0.8 # 80% threshold + + def _check_disk_space(self): + """Check available disk space.""" + import shutil + + usage = shutil.disk_usage("/") + free_percent = (usage.free / usage.total) * 100 + + return free_percent > 10 # 10% minimum +``` + +## Operational Best Practices + +1. **Monitoring**: Track throughput, latency, error rates +2. **Logging**: Use structured logging with correlation IDs +3. **Secrets**: Never log sensitive values, use secrets management +4. **Validation**: Validate all inputs to prevent injection attacks +5. **Resource Cleanup**: Always close connections and clean up resources +6. **Health Checks**: Verify system health before operations +7. **Rate Limiting**: Respect API rate limits with backoff +8. **Alerting**: Set up alerts for error rates and latency +9. **Documentation**: Document all configuration options +10. **Version Control**: Tag releases and maintain changelog diff --git a/.claude/skills/spark-python-data-source/references/streaming-patterns.md b/.claude/skills/spark-python-data-source/references/streaming-patterns.md new file mode 100644 index 00000000..66b9e8e3 --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/streaming-patterns.md @@ -0,0 +1,400 @@ +# Streaming Patterns + +Offset management and streaming implementation patterns for exactly-once semantics. + +## Basic Offset Implementation + +Simple JSON-serializable offset: + +```python +class SimpleOffset: + """Basic offset with single timestamp field.""" + + def __init__(self, timestamp): + self.timestamp = timestamp + + def json(self): + """Serialize to JSON string.""" + import json + return json.dumps({"timestamp": self.timestamp}) + + @staticmethod + def from_json(json_str): + """Deserialize from JSON string.""" + import json + data = json.loads(json_str) + return SimpleOffset(data["timestamp"]) +``` + +## Multi-Field Offset + +Complex offset with multiple fields: + +```python +class MultiFieldOffset: + """Offset with timestamp, sequence ID, and partition.""" + + def __init__(self, timestamp, sequence_id, partition_id): + self.timestamp = timestamp + self.sequence_id = sequence_id + self.partition_id = partition_id + + def json(self): + import json + return json.dumps({ + "timestamp": self.timestamp, + "sequence_id": self.sequence_id, + "partition_id": self.partition_id + }) + + @staticmethod + def from_json(json_str): + import json + data = json.loads(json_str) + return MultiFieldOffset( + timestamp=data["timestamp"], + sequence_id=data["sequence_id"], + partition_id=data["partition_id"] + ) + + def __lt__(self, other): + """Enable offset comparison for ordering.""" + if self.timestamp != other.timestamp: + return self.timestamp < other.timestamp + if self.sequence_id != other.sequence_id: + return self.sequence_id < other.sequence_id + return self.partition_id < other.partition_id +``` + +## Stream Reader Implementation + +Complete streaming reader with offset management: + +```python +from pyspark.sql.datasource import DataSourceStreamReader + +class YourStreamReader(DataSourceStreamReader): + def __init__(self, options, schema): + super().__init__(options, schema) + + # Parse start time option + start_time = options.get("start_time", "latest") + + if start_time == "latest": + from datetime import datetime, timezone + self.start_time = datetime.now(timezone.utc).isoformat() + + elif start_time == "earliest": + # Query for earliest timestamp (one-time cost) + self.start_time = self._get_earliest_timestamp() + + else: + # Validate ISO 8601 format + from datetime import datetime + datetime.fromisoformat(start_time.replace("Z", "+00:00")) + self.start_time = start_time + + # Partition duration (e.g., 1 hour) + self.partition_duration = int(options.get("partition_duration", "3600")) + + def _get_earliest_timestamp(self): + """Find earliest data timestamp for 'earliest' option.""" + from datetime import datetime, timezone + + timestamp_column = self.options.get("timestamp_column", "timestamp") + query = f"{self.query} | summarize earliest=min({timestamp_column})" + + response = self._execute_query(query, timespan=None) + + if response.tables and response.tables[0].rows: + earliest_value = response.tables[0].rows[0][0] + if earliest_value: + if isinstance(earliest_value, datetime): + return earliest_value.isoformat() + return str(earliest_value) + + # Fallback to current time + return datetime.now(timezone.utc).isoformat() + + def initialOffset(self): + """ + Return initial offset (start time minus 1 microsecond). + + Subtract 1µs to compensate for +1µs in partitions() method, + preventing overlap between batches. + """ + from datetime import datetime, timedelta + + start_dt = datetime.fromisoformat(self.start_time.replace("Z", "+00:00")) + adjusted = start_dt - timedelta(microseconds=1) + return SimpleOffset(adjusted.isoformat()).json() + + def latestOffset(self): + """Return latest offset (current time).""" + from datetime import datetime, timezone + + current_time = datetime.now(timezone.utc).isoformat() + return SimpleOffset(current_time).json() + + def partitions(self, start, end): + """ + Create non-overlapping partitions for offset range. + + Adds 1µs to start to prevent overlap with previous batch. + """ + from datetime import datetime, timedelta + + start_offset = SimpleOffset.from_json(start) + end_offset = SimpleOffset.from_json(end) + + start_time = datetime.fromisoformat(start_offset.timestamp.replace("Z", "+00:00")) + end_time = datetime.fromisoformat(end_offset.timestamp.replace("Z", "+00:00")) + + # Add 1µs to prevent overlap with previous batch + # This works with -1µs in initialOffset() to ensure: + # - Initial batch: (start - 1µs) + 1µs = start (correct) + # - Subsequent batches: previous_end + 1µs (no overlap) + start_time = start_time + timedelta(microseconds=1) + + # Create fixed-duration partitions + partitions = [] + current = start_time + delta = timedelta(seconds=self.partition_duration) + + while current < end_time: + next_time = min(current + delta, end_time) + partitions.append(TimeRangePartition(current, next_time)) + current = next_time + timedelta(microseconds=1) # No overlap + + return partitions if partitions else [TimeRangePartition(start_time, end_time)] + + def commit(self, end): + """Called when batch is successfully processed.""" + # Spark handles checkpointing - usually no action needed + pass + + def read(self, partition): + """Read data for partition time range.""" + response = self._query_api( + start=partition.start_time, + end=partition.end_time + ) + + for item in response: + yield self._convert_to_row(item) +``` + +## Watermarking Support + +Support for event-time watermarking: + +```python +class WatermarkedStreamReader(DataSourceStreamReader): + def __init__(self, options, schema): + super().__init__(options, schema) + + # Watermark configuration + self.watermark_column = options.get("watermark_column") + self.watermark_delay = options.get("watermark_delay", "10 minutes") + + def read(self, partition): + """Read with event-time watermarking.""" + from datetime import datetime + + response = self._query_api( + start=partition.start_time, + end=partition.end_time + ) + + for item in response: + row = self._convert_to_row(item) + + # Validate watermark column exists + if self.watermark_column: + if not hasattr(row, self.watermark_column): + raise ValueError( + f"Watermark column '{self.watermark_column}' not found in row" + ) + + # Ensure watermark column is timestamp + watermark_value = getattr(row, self.watermark_column) + if not isinstance(watermark_value, datetime): + raise ValueError( + f"Watermark column must be timestamp, got {type(watermark_value)}" + ) + + yield row +``` + +## Stateful Streaming + +Track state across batches: + +```python +class StatefulStreamReader(DataSourceStreamReader): + def __init__(self, options, schema): + super().__init__(options, schema) + + # State management + self.checkpoint_location = options.get("checkpoint_location") + self._state = {} + + def _load_state(self): + """Load state from checkpoint location.""" + import json + import os + + if not self.checkpoint_location: + return {} + + state_file = os.path.join(self.checkpoint_location, "reader_state.json") + + if os.path.exists(state_file): + with open(state_file, 'r') as f: + return json.load(f) + + return {} + + def _save_state(self): + """Save state to checkpoint location.""" + import json + import os + + if not self.checkpoint_location: + return + + os.makedirs(self.checkpoint_location, exist_ok=True) + state_file = os.path.join(self.checkpoint_location, "reader_state.json") + + with open(state_file, 'w') as f: + json.dump(self._state, f) + + def initialOffset(self): + """Load state and return initial offset.""" + self._state = self._load_state() + + # Check if we have previous state + if "last_offset" in self._state: + return self._state["last_offset"] + + # First run - use configured start time + return self._create_initial_offset() + + def commit(self, end): + """Save state after successful batch.""" + self._state["last_offset"] = end + self._state["last_commit_time"] = datetime.now().isoformat() + self._save_state() +``` + +## Exactly-Once Semantics + +Ensure exactly-once delivery with idempotent writes: + +```python +class ExactlyOnceWriter(DataSourceStreamWriter): + def __init__(self, options, schema): + super().__init__(options, schema) + self.enable_idempotency = options.get("enable_idempotency", "true").lower() == "true" + + def write(self, iterator): + """Write with idempotency key.""" + import hashlib + from pyspark import TaskContext + + context = TaskContext.get() + partition_id = context.partitionId() + batch_id = getattr(context, 'batchId', lambda: 0)() + + for row in iterator: + # Generate idempotency key from batch_id + partition_id + row content + row_dict = row.asDict() + + if self.enable_idempotency: + idempotency_key = self._generate_idempotency_key( + batch_id, + partition_id, + row_dict + ) + row_dict["_idempotency_key"] = idempotency_key + + # Write with idempotency check + self._write_with_idempotency_check(row_dict) + + def _generate_idempotency_key(self, batch_id, partition_id, row_dict): + """Generate deterministic idempotency key.""" + import hashlib + import json + + key_data = { + "batch_id": batch_id, + "partition_id": partition_id, + "row": row_dict + } + + key_str = json.dumps(key_data, sort_keys=True) + return hashlib.sha256(key_str.encode()).hexdigest() + + def _write_with_idempotency_check(self, row_dict): + """Write only if idempotency key not seen before.""" + idempotency_key = row_dict.get("_idempotency_key") + + if idempotency_key: + # Check if already written (implementation depends on target system) + if self._is_already_written(idempotency_key): + return # Skip duplicate + + # Write data + self._write_data(row_dict) + + def commit(self, messages, batchId): + """Commit batch after all writes succeed.""" + # Log successful batch + print(f"Batch {batchId} committed successfully") + + def abort(self, messages, batchId): + """Handle failed batch.""" + # Log failed batch + print(f"Batch {batchId} aborted") +``` + +## Monitoring and Progress + +Track streaming progress: + +```python +class MonitoredStreamReader(DataSourceStreamReader): + def read(self, partition): + """Read with progress tracking.""" + from datetime import datetime + + start_time = datetime.now() + row_count = 0 + + for row in self._read_partition(partition): + row_count += 1 + yield row + + duration = (datetime.now() - start_time).total_seconds() + + # Log metrics + self._log_partition_metrics( + partition_id=partition.partition_id, + row_count=row_count, + duration=duration + ) + + def _log_partition_metrics(self, partition_id, row_count, duration): + """Log partition processing metrics.""" + print(f"Partition {partition_id}: {row_count} rows in {duration:.2f}s") +``` + +## Best Practices + +1. **Non-Overlapping Partitions**: Use microsecond adjustments to prevent duplicates +2. **Idempotency**: Generate deterministic keys for exactly-once semantics +3. **State Management**: Store offsets in Spark checkpoints +4. **Watermarking**: Support event-time processing for late data +5. **Monitoring**: Track batch progress and lag metrics +6. **Error Handling**: Streaming writers are especially susceptible to transient failures (network blips, rate limits) since they run continuously. Use retry with exponential backoff from [error-handling.md](error-handling.md) in your `write()` methods. +7. **Backpressure**: Respect rate limits with appropriate partition sizing diff --git a/.claude/skills/spark-python-data-source/references/testing-patterns.md b/.claude/skills/spark-python-data-source/references/testing-patterns.md new file mode 100644 index 00000000..1b4aeb23 --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/testing-patterns.md @@ -0,0 +1,441 @@ +# Testing Patterns + +Unit and integration testing strategies for Spark data sources. + +## Basic Unit Tests + +Test data source registration and initialization: + +```python +import pytest +from pyspark.sql import SparkSession + +@pytest.fixture(scope="session") +def spark(): + """Create Spark session for tests.""" + return SparkSession.builder \ + .master("local[2]") \ + .appName("test") \ + .config("spark.sql.shuffle.partitions", "2") \ + .getOrCreate() + +def test_data_source_name(): + """Test data source name registration.""" + assert YourDataSource.name() == "your-format" + +def test_data_source_initialization(): + """Test data source can be initialized.""" + options = {"url": "http://api.example.com"} + ds = YourDataSource(options) + assert ds.options == options + +def test_missing_required_option(): + """Test error on missing required option.""" + options = {} # Missing required 'url' + + with pytest.raises(AssertionError, match="url is required"): + YourDataSource(options) +``` + +## Mocking HTTP Requests + +Test writers without external dependencies: + +```python +from unittest.mock import patch, Mock +import pytest + +@pytest.fixture +def basic_options(): + """Common options for tests.""" + return { + "url": "http://api.example.com", + "batch_size": "10" + } + +@pytest.fixture +def sample_schema(): + """Sample schema for tests.""" + from pyspark.sql.types import StructType, StructField, IntegerType, StringType + return StructType([ + StructField("id", IntegerType(), False), + StructField("name", StringType(), True) + ]) + +def test_writer_sends_batch(spark, basic_options, sample_schema): + """Test writer sends data in batches.""" + with patch('requests.post') as mock_post: + mock_post.return_value = Mock(status_code=200) + + # Create test data + df = spark.createDataFrame([ + (1, "Alice"), + (2, "Bob"), + (3, "Charlie") + ], ["id", "name"]) + + # Write using data source + df.write.format("your-format").options(**basic_options).save() + + # Verify API was called + assert mock_post.called + assert mock_post.call_count > 0 + +def test_writer_respects_batch_size(spark, basic_options, sample_schema): + """Test writer respects configured batch size.""" + with patch('requests.post') as mock_post: + mock_post.return_value = Mock(status_code=200) + + # Create 25 rows with batch_size=10 + rows = [(i, f"name_{i}") for i in range(25)] + df = spark.createDataFrame(rows, ["id", "name"]) + + df.write.format("your-format").options(**basic_options).save() + + # Should make 3 calls: 10 + 10 + 5 + assert mock_post.call_count == 3 +``` + +## Testing Readers + +Mock external API responses: + +```python +def test_reader_fetches_data(spark, basic_options): + """Test reader fetches and converts data.""" + with patch('requests.get') as mock_get: + # Mock API response + mock_response = Mock() + mock_response.json.return_value = [ + {"id": 1, "name": "Alice"}, + {"id": 2, "name": "Bob"} + ] + mock_get.return_value = mock_response + + # Read using data source + df = spark.read.format("your-format").options(**basic_options).load() + + # Verify data + rows = df.collect() + assert len(rows) == 2 + assert rows[0]["id"] == 1 + assert rows[0]["name"] == "Alice" + +def test_reader_handles_empty_response(spark, basic_options): + """Test reader handles empty response.""" + with patch('requests.get') as mock_get: + mock_response = Mock() + mock_response.json.return_value = [] + mock_get.return_value = mock_response + + df = spark.read.format("your-format").options(**basic_options).load() + + assert df.count() == 0 +``` + +## Testing Partitioning + +Test partition creation logic: + +```python +def test_partitions_created(basic_options, sample_schema): + """Test correct number of partitions created.""" + options = {**basic_options, "num_partitions": "4"} + + reader = YourBatchReader(options, sample_schema) + partitions = reader.partitions() + + assert len(partitions) == 4 + +def test_partition_ranges_non_overlapping(): + """Test partitions have non-overlapping ranges.""" + from datetime import datetime, timedelta + + reader = TimeBasedReader(options, schema) + partitions = reader.partitions() + + # Check no gaps or overlaps + for i in range(len(partitions) - 1): + current_end = partitions[i].end_time + next_start = partitions[i + 1].start_time + + # Next partition should start right after current ends + assert next_start >= current_end +``` + +## Testing Streaming + +Test offset management and streaming logic: + +```python +def test_initial_offset(): + """Test initial offset is correct.""" + from datetime import datetime + + reader = YourStreamReader(options, schema) + initial = reader.initialOffset() + + # Should be valid JSON + import json + offset_dict = json.loads(initial) + + assert "timestamp" in offset_dict + +def test_latest_offset_advances(): + """Test latest offset advances over time.""" + reader = YourStreamReader(options, schema) + + offset1 = reader.latestOffset() + import time + time.sleep(0.1) + offset2 = reader.latestOffset() + + # Offset should advance + assert offset2 > offset1 or offset2 != offset1 + +def test_partitions_non_overlapping(basic_options, sample_schema): + """Test streaming partitions don't overlap.""" + reader = YourStreamReader(basic_options, sample_schema) + + start = reader.initialOffset() + end = reader.latestOffset() + + partitions = reader.partitions(start, end) + + # Verify no overlaps + for i in range(len(partitions) - 1): + assert partitions[i].end_time < partitions[i + 1].start_time +``` + +## Testing Type Conversion + +Test type mapping and conversion: + +```python +def test_convert_timestamp(): + """Test timestamp conversion.""" + from datetime import datetime + from pyspark.sql.types import TimestampType + + dt = datetime(2024, 1, 1, 12, 0, 0) + result = convert_external_to_spark(dt, TimestampType()) + + assert isinstance(result, datetime) + assert result == dt + +def test_convert_null_values(): + """Test null value handling.""" + from pyspark.sql.types import StringType + + result = convert_external_to_spark(None, StringType()) + assert result is None + +def test_convert_invalid_type(): + """Test error on invalid type conversion.""" + from pyspark.sql.types import IntegerType + + with pytest.raises(ValueError, match="Cannot convert"): + convert_external_to_spark("not_a_number", IntegerType()) +``` + +## Integration Tests with Testcontainers + +Run end-to-end tests against real systems: + +```python +import pytest +from testcontainers.postgres import PostgresContainer + +@pytest.fixture(scope="session") +def postgres_container(): + """Start PostgreSQL container for integration tests.""" + with PostgresContainer("postgres:15") as container: + yield container + +@pytest.fixture +def postgres_connection(postgres_container): + """Create connection to test database.""" + import psycopg2 + + conn = psycopg2.connect(postgres_container.get_connection_url()) + cursor = conn.cursor() + + # Create test table + cursor.execute(""" + CREATE TABLE test_data ( + id SERIAL PRIMARY KEY, + name VARCHAR(100), + value INTEGER + ) + """) + conn.commit() + + yield conn + + conn.close() + +def test_write_integration(spark, postgres_container, postgres_connection): + """Integration test for writing to PostgreSQL.""" + # Create test data + df = spark.createDataFrame([ + (1, "Alice", 100), + (2, "Bob", 200) + ], ["id", "name", "value"]) + + # Write using data source + df.write.format("your-format") \ + .option("url", postgres_container.get_connection_url()) \ + .option("table", "test_data") \ + .save() + + # Verify data written + cursor = postgres_connection.cursor() + cursor.execute("SELECT COUNT(*) FROM test_data") + count = cursor.fetchone()[0] + + assert count == 2 + +def test_read_integration(spark, postgres_container, postgres_connection): + """Integration test for reading from PostgreSQL.""" + # Insert test data + cursor = postgres_connection.cursor() + cursor.execute("INSERT INTO test_data (name, value) VALUES ('Alice', 100)") + cursor.execute("INSERT INTO test_data (name, value) VALUES ('Bob', 200)") + postgres_connection.commit() + + # Read using data source + df = spark.read.format("your-format") \ + .option("url", postgres_container.get_connection_url()) \ + .option("table", "test_data") \ + .load() + + # Verify data + assert df.count() == 2 + names = [row["name"] for row in df.collect()] + assert "Alice" in names + assert "Bob" in names +``` + +## Performance Tests + +Test performance characteristics: + +```python +import time + +def test_write_performance(spark, basic_options): + """Test write performance meets requirements.""" + # Create large dataset + rows = [(i, f"name_{i}") for i in range(10000)] + df = spark.createDataFrame(rows, ["id", "name"]) + + start = time.time() + df.write.format("your-format").options(**basic_options).save() + duration = time.time() - start + + # Should complete in reasonable time + assert duration < 30.0 # 30 seconds + + # Calculate throughput + throughput = len(rows) / duration + print(f"Write throughput: {throughput:.0f} rows/second") + +def test_partition_read_parallelism(spark, basic_options): + """Test reads execute in parallel.""" + options = {**basic_options, "num_partitions": "4"} + + df = spark.read.format("your-format").options(**options).load() + + # Check partition count + assert df.rdd.getNumPartitions() == 4 +``` + +## Test Fixtures and Utilities + +Reusable test fixtures: + +```python +import pytest +from pyspark.sql import SparkSession + +@pytest.fixture(scope="session") +def spark(): + """Shared Spark session.""" + return SparkSession.builder \ + .master("local[2]") \ + .appName("test") \ + .config("spark.sql.shuffle.partitions", "2") \ + .getOrCreate() + +@pytest.fixture +def sample_dataframe(spark): + """Sample DataFrame for testing.""" + return spark.createDataFrame([ + (1, "Alice", 25), + (2, "Bob", 30), + (3, "Charlie", 35) + ], ["id", "name", "age"]) + +@pytest.fixture +def temp_output_path(tmp_path): + """Temporary output path.""" + return str(tmp_path / "output") + +def assert_dataframes_equal(df1, df2): + """Assert two DataFrames are equal.""" + assert df1.schema == df2.schema + assert df1.count() == df2.count() + + rows1 = sorted(df1.collect()) + rows2 = sorted(df2.collect()) + + assert rows1 == rows2 +``` + +## Test Organization + +Structure tests by functionality: + +``` +tests/ +├── unit/ +│ ├── test_datasource.py # DataSource class tests +│ ├── test_reader.py # Reader tests +│ ├── test_writer.py # Writer tests +│ ├── test_partitioning.py # Partitioning logic +│ └── test_type_conversion.py # Type conversion +├── integration/ +│ ├── test_read_integration.py # End-to-end read tests +│ ├── test_write_integration.py # End-to-end write tests +│ └── test_streaming.py # Streaming tests +├── performance/ +│ └── test_performance.py # Performance tests +└── conftest.py # Shared fixtures +``` + +## Running Tests + +Run tests through your packaging tool (e.g., `uv run`, `poetry run`, `hatch run`). Examples use `uv`: + +```bash +# Run all tests +uv run pytest + +# Run specific test file +uv run pytest tests/unit/test_writer.py + +# Run specific test +uv run pytest tests/unit/test_writer.py::test_writer_sends_batch + +# Run with coverage +uv run pytest --cov=your_package --cov-report=html + +# Run only unit tests +uv run pytest tests/unit/ + +# Run with verbose output +uv run pytest -v + +# Run with print statements +uv run pytest -s +``` diff --git a/.claude/skills/spark-python-data-source/references/type-conversion.md b/.claude/skills/spark-python-data-source/references/type-conversion.md new file mode 100644 index 00000000..a55f0795 --- /dev/null +++ b/.claude/skills/spark-python-data-source/references/type-conversion.md @@ -0,0 +1,370 @@ +# Type Conversion + +Bidirectional mapping between Spark types and external system types. + +## Spark to External System + +Convert Spark/Python values to external system types: + +```python +def convert_spark_to_external(value, external_type): + """Convert Spark/Python value to external system type.""" + if value is None: + return None + + external_type_lower = external_type.lower() + + # UUID conversion + if "uuid" in external_type_lower: + import uuid + if isinstance(value, uuid.UUID): + return value + return uuid.UUID(str(value)) + + # Timestamp conversion + if "timestamp" in external_type_lower: + from datetime import datetime + if isinstance(value, datetime): + return value + if isinstance(value, str): + return datetime.fromisoformat(value.replace("Z", "+00:00")) + if isinstance(value, (int, float)): + return datetime.fromtimestamp(value) + + # IP address conversion + if "inet" in external_type_lower: + import ipaddress + if isinstance(value, (ipaddress.IPv4Address, ipaddress.IPv6Address)): + return value + return ipaddress.ip_address(str(value)) + + # Decimal conversion + if "decimal" in external_type_lower: + from decimal import Decimal + if isinstance(value, Decimal): + return value + return Decimal(str(value)) + + # Collections + if "list" in external_type_lower or "set" in external_type_lower: + if not isinstance(value, (list, set)): + raise ValueError(f"Expected list/set, got {type(value)}") + return list(value) + + if "map" in external_type_lower: + if not isinstance(value, dict): + raise ValueError(f"Expected dict, got {type(value)}") + return value + + # Numeric types + if "int" in external_type_lower: + return int(value) + if "float" in external_type_lower or "double" in external_type_lower: + return float(value) + + # Boolean + if "bool" in external_type_lower: + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.lower() in ("true", "1", "yes") + return bool(value) + + # Default: return as-is + return value +``` + +## External System to Spark + +Convert external values to Spark types: + +```python +def convert_external_to_spark(value, spark_type): + """Convert external system value to Spark type.""" + from pyspark.sql.types import ( + StringType, IntegerType, LongType, FloatType, DoubleType, + BooleanType, TimestampType, DateType + ) + from datetime import datetime, date + + if value is None: + return None + + try: + if isinstance(spark_type, StringType): + return str(value) + + elif isinstance(spark_type, BooleanType): + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.lower() in ("true", "1", "yes") + return bool(value) + + elif isinstance(spark_type, (IntegerType, LongType)): + if isinstance(value, bool): + raise ValueError("Cannot convert boolean to integer") + return int(value) + + elif isinstance(spark_type, (FloatType, DoubleType)): + if isinstance(value, bool): + raise ValueError("Cannot convert boolean to float") + return float(value) + + elif isinstance(spark_type, TimestampType): + if isinstance(value, datetime): + return value + if isinstance(value, str): + return datetime.fromisoformat(value.replace("Z", "+00:00")) + raise ValueError(f"Cannot convert {type(value)} to timestamp") + + elif isinstance(spark_type, DateType): + if isinstance(value, date) and not isinstance(value, datetime): + return value + if isinstance(value, datetime): + return value.date() + if isinstance(value, str): + return datetime.fromisoformat(value.replace("Z", "+00:00")).date() + raise ValueError(f"Cannot convert {type(value)} to date") + + else: + return value + + except (ValueError, TypeError) as e: + raise ValueError( + f"Failed to convert '{value}' (type: {type(value).__name__}) " + f"to {spark_type}: {e}" + ) +``` + +## Cassandra-Specific Types + +Handle Cassandra complex types: + +```python +def convert_cassandra_to_spark(value): + """Handle Cassandra-specific complex types.""" + if value is None: + return None + + from cassandra.util import ( + Date, Time, Duration, OrderedMap, SortedSet, + Point, LineString, Polygon + ) + import uuid + + # Cassandra Date to Python date + if isinstance(value, Date): + return value.date() + + # Cassandra Time to nanoseconds (LongType) + if isinstance(value, Time): + return value.nanosecond + + # UUID to string + if isinstance(value, uuid.UUID): + return str(value) + + # Duration to structured dict + if isinstance(value, Duration): + return { + "months": value.months, + "days": value.days, + "nanoseconds": value.nanoseconds + } + + # OrderedMap to dict + if isinstance(value, OrderedMap): + return dict(value) + + # SortedSet to list + if isinstance(value, SortedSet): + return list(value) + + # Geospatial types to WKT string + if isinstance(value, (Point, LineString, Polygon)): + return str(value) + + return value +``` + +## Schema Inference + +Infer Spark types from Python values: + +```python +def infer_spark_type(value): + """Infer Spark type from Python value.""" + from pyspark.sql.types import ( + StringType, IntegerType, LongType, FloatType, DoubleType, + BooleanType, TimestampType, DateType + ) + from datetime import datetime, date + + if value is None: + return StringType() + + # Check bool before int (bool is subclass of int) + if isinstance(value, bool): + return BooleanType() + + if isinstance(value, int): + return LongType() + + if isinstance(value, float): + return DoubleType() + + if isinstance(value, datetime): + return TimestampType() + + if isinstance(value, date): + return DateType() + + # Default to string + return StringType() +``` + +## External Type to Spark Type Mapping + +Map external system types to Spark types: + +```python +def map_external_type_to_spark(external_type): + """Map external system types to Spark types.""" + from pyspark.sql.types import ( + StringType, IntegerType, LongType, FloatType, DoubleType, + BooleanType, TimestampType, DateType, BinaryType + ) + + type_str = str(external_type).lower() + + # String types + if any(t in type_str for t in ["varchar", "text", "char", "string", "uuid"]): + return StringType() + + # Integer types + if "int" in type_str and "big" not in type_str: + return IntegerType() + if "bigint" in type_str or "long" in type_str: + return LongType() + + # Floating point + if "float" in type_str: + return FloatType() + if "double" in type_str or "decimal" in type_str: + return DoubleType() + + # Boolean + if "bool" in type_str: + return BooleanType() + + # Temporal types + if "timestamp" in type_str: + return TimestampType() + if "date" in type_str: + return DateType() + + # Binary + if "blob" in type_str or "binary" in type_str: + return BinaryType() + + # Default fallback + return StringType() +``` + +## JSON Encoding + +Handle datetime serialization for JSON APIs: + +```python +import json +from datetime import date, datetime +from decimal import Decimal + +class ExtendedJsonEncoder(json.JSONEncoder): + """JSON encoder that handles datetime, date, and Decimal.""" + + def default(self, o): + if isinstance(o, (datetime, date)): + return o.isoformat() + + if isinstance(o, Decimal): + return float(o) + + return super().default(o) + +# Usage +def send_as_json(data): + import requests + + payload = json.dumps(data, cls=ExtendedJsonEncoder) + requests.post(url, data=payload, headers={"Content-Type": "application/json"}) +``` + +## Complete Row Conversion + +Convert entire rows with schema: + +```python +def convert_row_to_external(row, column_types): + """Convert entire Spark row to external system format.""" + row_dict = row.asDict() if hasattr(row, "asDict") else dict(row) + + converted = {} + for col, value in row_dict.items(): + external_type = column_types.get(col, "text") + converted[col] = convert_spark_to_external(value, external_type) + + return converted + +def convert_external_to_row(data, schema): + """Convert external data to Spark Row.""" + from pyspark.sql import Row + + # Create mapping of column names to types + schema_map = {field.name: field.dataType for field in schema.fields} + + row_dict = {} + for col, value in data.items(): + if col in schema_map: + spark_type = schema_map[col] + row_dict[col] = convert_external_to_spark(value, spark_type) + + # Add None for missing columns + for field in schema.fields: + if field.name not in row_dict: + row_dict[field.name] = None + + return Row(**row_dict) +``` + +## Validation + +Validate type conversions: + +```python +def validate_conversion(value, expected_type): + """Validate that value matches expected type after conversion.""" + type_checks = { + "int": lambda v: isinstance(v, int) and not isinstance(v, bool), + "long": lambda v: isinstance(v, int) and not isinstance(v, bool), + "float": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool), + "double": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool), + "string": lambda v: isinstance(v, str), + "boolean": lambda v: isinstance(v, bool), + "timestamp": lambda v: isinstance(v, datetime), + "date": lambda v: isinstance(v, date) and not isinstance(v, datetime), + } + + expected_type_lower = expected_type.lower() + for type_name, check in type_checks.items(): + if type_name in expected_type_lower: + if not check(value): + raise ValueError( + f"Value {value} (type: {type(value)}) does not match " + f"expected type {expected_type}" + ) + return + + # No specific check - accept any value +``` diff --git a/.claude/skills/synthetic-data-generation/SKILL.md b/.claude/skills/synthetic-data-generation/SKILL.md deleted file mode 100644 index 6d029941..00000000 --- a/.claude/skills/synthetic-data-generation/SKILL.md +++ /dev/null @@ -1,654 +0,0 @@ ---- -name: synthetic-data-generation -description: "Generate realistic synthetic data using Faker and Spark, with non-linear distributions, integrity constraints, and save to Databricks. Use when creating test data, demo datasets, or synthetic tables." ---- - -# Synthetic Data Generation - -Generate realistic, story-driven synthetic data for Databricks using Python with Faker and Spark. - -## Common Libraries - -These libraries are useful for generating realistic synthetic data: - -- **faker**: Generates realistic names, addresses, emails, companies, dates, etc. -- **holidays**: Provides country-specific holiday calendars for realistic date patterns - -These are typically NOT pre-installed on Databricks. Install them using `execute_databricks_command` tool: -- `code`: "%pip install faker holidays" - -Save the returned `cluster_id` and `context_id` for subsequent calls. - -## Workflow - -1. **Write Python code to a local file** in the project (e.g., `scripts/generate_data.py`) -2. **Execute on Databricks** using the `run_python_file_on_databricks` MCP tool -3. **If execution fails**: Edit the local file to fix the error, then re-execute -4. **Reuse the context** for follow-up executions by passing the returned `cluster_id` and `context_id` - -**Always work with local files first, then execute.** This makes debugging easier - you can see and edit the code. - -### Context Reuse Pattern - -The first execution auto-selects a running cluster and creates an execution context. **Reuse this context for follow-up calls** - it's much faster (~1s vs ~15s) and shares variables/imports: - -**First execution** - use `run_python_file_on_databricks` tool: -- `file_path`: "scripts/generate_data.py" - -Returns: `{ success, output, error, cluster_id, context_id, ... }` - -Save `cluster_id` and `context_id` for follow-up calls. - -**If execution fails:** -1. Read the error from the result -2. Edit the local Python file to fix the issue -3. Re-execute with same context using `run_python_file_on_databricks` tool: - - `file_path`: "scripts/generate_data.py" - - `cluster_id`: "" - - `context_id`: "" - -**Follow-up executions** reuse the context (faster, shares state): -- `file_path`: "scripts/validate_data.py" -- `cluster_id`: "" -- `context_id`: "" - -### Handling Failures - -When execution fails: -1. Read the error from the result -2. **Edit the local Python file** to fix the issue -3. Re-execute using the same `cluster_id` and `context_id` (faster, keeps installed libraries) -4. If the context is corrupted, omit `context_id` to create a fresh one - -### Installing Libraries - -Databricks provides Spark, pandas, numpy, and common data libraries by default. **Only install a library if you get an import error.** - -Use `execute_databricks_command` tool: -- `code`: "%pip install faker" -- `cluster_id`: "" -- `context_id`: "" - -The library is immediately available in the same context. - -**Note:** Keeping the same `context_id` means installed libraries persist across calls. - -## Storage Destination - -### Ask for Schema Name - -By default, use the `ai_dev_kit` catalog. Ask the user which schema to use: - -> "I'll save the data to `ai_dev_kit.`. What schema name would you like to use? (You can also specify a different catalog if needed.)" - -If the user provides just a schema name, use `ai_dev_kit.{schema}`. If they provide `catalog.schema`, use that instead. - -### Create Infrastructure in the Script - -Always create the catalog, schema, and volume **inside the Python script** using `spark.sql()`. Do NOT make separate MCP SQL calls - it's much slower. - -The `spark` variable is available by default on Databricks clusters. - -```python -# ============================================================================= -# CREATE INFRASTRUCTURE (inside the Python script) -# ============================================================================= -spark.sql(f"CREATE CATALOG IF NOT EXISTS {CATALOG}") -spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}") -spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data") -``` - -### Save to Volume as Raw Data (Never Tables) - -**Always save data to a Volume as parquet files, never directly to tables** (unless the user explicitly requests tables). This is the input for the downstream Spark Declarative Pipeline (SDP) that will handle bronze/silver/gold layers. - -```python -VOLUME_PATH = f"/Volumes/{CATALOG}/{SCHEMA}/raw_data" - -# Save as parquet files (raw data) -spark.createDataFrame(customers_pdf).write.mode("overwrite").parquet(f"{VOLUME_PATH}/customers") -spark.createDataFrame(orders_pdf).write.mode("overwrite").parquet(f"{VOLUME_PATH}/orders") -spark.createDataFrame(tickets_pdf).write.mode("overwrite").parquet(f"{VOLUME_PATH}/tickets") -``` - -## Raw Data Only - No Pre-Aggregated Fields (Unless Instructed Otherwise) - -**By default, generate raw, transactional data only.** Do not create fields that represent sums, totals, averages, or counts. - -- One row = one event/transaction/record -- No columns like `total_orders`, `sum_revenue`, `avg_csat`, `order_count` -- Each row has its own individual values, not rollups - -**Why?** A Spark Declarative Pipeline (SDP) will typically be built after data generation to: -- Ingest raw data (bronze layer) -- Clean and validate (silver layer) -- Aggregate and compute metrics (gold layer) - -The synthetic data is the **source** for this pipeline. Aggregations happen downstream. - -**Note:** If the user specifically requests aggregated fields or summary tables, follow their instructions. - -```python -# GOOD - Raw transactional data -# Customer table: one row per customer, no aggregated fields -customers_data.append({ - "customer_id": cid, - "name": fake.company(), - "tier": "Enterprise", - "region": "North", -}) - -# Order table: one row per order -orders_data.append({ - "order_id": f"ORD-{i:06d}", - "customer_id": cid, - "amount": 150.00, # This order's amount - "order_date": "2024-10-15", -}) - -# BAD - Don't add pre-aggregated fields -# customers_data.append({ -# "customer_id": cid, -# "total_orders": 47, # NO - this is an aggregation -# "total_revenue": 12500.00, # NO - this is a sum -# "avg_order_value": 265.95, # NO - this is an average -# }) -``` - -## Temporality and Data Volume - -### Date Range: Last 6 Months from Today - -**Always generate data for the last ~6 months ending at the current date.** This ensures: -- Data feels current and relevant for demos -- Recent patterns are visible in dashboards -- Downstream aggregations (daily/weekly/monthly) have enough history - -```python -from datetime import datetime, timedelta - -# Dynamic date range - last 6 months from today -END_DATE = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0) -START_DATE = END_DATE - timedelta(days=180) - -# Place special events within this range (e.g., incident 3 weeks ago) -INCIDENT_END = END_DATE - timedelta(days=21) -INCIDENT_START = INCIDENT_END - timedelta(days=10) -``` - -### Data Volume for Aggregation - -Generate enough data so patterns remain visible after downstream aggregation (SDP pipelines often aggregate by day/week/region/category). Rules of thumb: - -| Grain | Minimum Records | Rationale | -|-------|-----------------|-----------| -| Daily time series | 50-100/day | See trends after weekly rollup | -| Per category | 500+ per category | Statistical significance | -| Per customer | 5-20 events/customer | Enough for customer-level analysis | -| Total rows | 10K-50K minimum | Patterns survive GROUP BY | - -```python -# Example: 8000 tickets over 180 days = ~44/day average -# After weekly aggregation: ~310 records per week per category -# After monthly by region: still enough to see patterns -N_TICKETS = 8000 -N_CUSTOMERS = 2500 # Each has ~3 tickets on average -N_ORDERS = 25000 # ~10 orders per customer average -``` - -## Script Structure - -Always structure scripts with configuration variables at the top: - -```python -"""Generate synthetic data for [use case].""" -import numpy as np -import pandas as pd -from datetime import datetime, timedelta -from faker import Faker -import holidays -from pyspark.sql import SparkSession - -# ============================================================================= -# CONFIGURATION - Edit these values -# ============================================================================= -CATALOG = "my_catalog" -SCHEMA = "my_schema" -VOLUME_PATH = f"/Volumes/{CATALOG}/{SCHEMA}/raw_data" - -# Data sizes - enough for aggregation patterns to survive -N_CUSTOMERS = 2500 -N_ORDERS = 25000 -N_TICKETS = 8000 - -# Date range - last 6 months from today -END_DATE = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0) -START_DATE = END_DATE - timedelta(days=180) - -# Special events (within the date range) -INCIDENT_END = END_DATE - timedelta(days=21) -INCIDENT_START = INCIDENT_END - timedelta(days=10) - -# Holiday calendar for realistic patterns -US_HOLIDAYS = holidays.US(years=[START_DATE.year, END_DATE.year]) - -# Reproducibility -SEED = 42 - -# ============================================================================= -# SETUP -# ============================================================================= -np.random.seed(SEED) -Faker.seed(SEED) -fake = Faker() -spark = SparkSession.builder.getOrCreate() - -# ... rest of script -``` - -## Key Principles - -### 1. Use Pandas for Generation, Spark for Saving - -Generate data with pandas (faster, easier), convert to Spark for saving: - -```python -import pandas as pd - -# Generate with pandas -customers_pdf = pd.DataFrame({ - "customer_id": [f"CUST-{i:05d}" for i in range(N_CUSTOMERS)], - "name": [fake.company() for _ in range(N_CUSTOMERS)], - "tier": np.random.choice(['Free', 'Pro', 'Enterprise'], N_CUSTOMERS, p=[0.6, 0.3, 0.1]), - "region": np.random.choice(['North', 'South', 'East', 'West'], N_CUSTOMERS, p=[0.4, 0.25, 0.2, 0.15]), - "created_at": [fake.date_between(start_date='-2y', end_date='-6m') for _ in range(N_CUSTOMERS)], -}) - -# Convert to Spark and save -customers_df = spark.createDataFrame(customers_pdf) -customers_df.write.mode("overwrite").parquet(f"{VOLUME_PATH}/customers") -``` - -### 2. Iterate on DataFrames for Referential Integrity - -Generate master tables first, then iterate on them to create related tables with matching IDs: - -```python -# 1. Generate customers (master table) -customers_pdf = pd.DataFrame({ - "customer_id": [f"CUST-{i:05d}" for i in range(N_CUSTOMERS)], - "tier": np.random.choice(['Free', 'Pro', 'Enterprise'], N_CUSTOMERS, p=[0.6, 0.3, 0.1]), - # ... -}) - -# 2. Create lookup for foreign key generation -customer_ids = customers_pdf["customer_id"].tolist() -customer_tier_map = dict(zip(customers_pdf["customer_id"], customers_pdf["tier"])) - -# Weight by tier - Enterprise customers generate more orders -tier_weights = customers_pdf["tier"].map({'Enterprise': 5.0, 'Pro': 2.0, 'Free': 1.0}) -customer_weights = (tier_weights / tier_weights.sum()).tolist() - -# 3. Generate orders with valid foreign keys and tier-based logic -orders_data = [] -for i in range(N_ORDERS): - cid = np.random.choice(customer_ids, p=customer_weights) - tier = customer_tier_map[cid] - - # Amount depends on tier - if tier == 'Enterprise': - amount = np.random.lognormal(7, 0.8) - elif tier == 'Pro': - amount = np.random.lognormal(5, 0.7) - else: - amount = np.random.lognormal(3.5, 0.6) - - orders_data.append({ - "order_id": f"ORD-{i:06d}", - "customer_id": cid, - "amount": round(amount, 2), - "order_date": fake.date_between(start_date=START_DATE, end_date=END_DATE), - }) - -orders_pdf = pd.DataFrame(orders_data) - -# 4. Generate tickets that reference both customers and orders -order_ids = orders_pdf["order_id"].tolist() -tickets_data = [] -for i in range(N_TICKETS): - cid = np.random.choice(customer_ids, p=customer_weights) - oid = np.random.choice(order_ids) # Or None for general inquiry - - tickets_data.append({ - "ticket_id": f"TKT-{i:06d}", - "customer_id": cid, - "order_id": oid if np.random.random() > 0.3 else None, - # ... - }) - -tickets_pdf = pd.DataFrame(tickets_data) -``` - -### 3. Non-Linear Distributions - -**Never use uniform distributions** - real data is rarely uniform: - -```python -# BAD - Uniform (unrealistic) -prices = np.random.uniform(10, 1000, size=N_ORDERS) - -# GOOD - Log-normal (realistic for prices, salaries, order amounts) -prices = np.random.lognormal(mean=4.5, sigma=0.8, size=N_ORDERS) - -# GOOD - Pareto/power law (popularity, wealth, page views) -popularity = (np.random.pareto(a=2.5, size=N_PRODUCTS) + 1) * 10 - -# GOOD - Exponential (time between events, resolution time) -resolution_hours = np.random.exponential(scale=24, size=N_TICKETS) - -# GOOD - Weighted categorical -regions = np.random.choice( - ['North', 'South', 'East', 'West'], - size=N_CUSTOMERS, - p=[0.40, 0.25, 0.20, 0.15] -) -``` - -### 4. Time-Based Patterns - -Add weekday/weekend effects, holidays, seasonality, and event spikes: - -```python -import holidays - -# Load holiday calendar -US_HOLIDAYS = holidays.US(years=[START_DATE.year, END_DATE.year]) - -def get_daily_multiplier(date): - """Calculate volume multiplier for a given date.""" - multiplier = 1.0 - - # Weekend drop - if date.weekday() >= 5: - multiplier *= 0.6 - - # Holiday drop (even lower than weekends) - if date in US_HOLIDAYS: - multiplier *= 0.3 - - # Q4 seasonality (higher in Oct-Dec) - multiplier *= 1 + 0.15 * (date.month - 6) / 6 - - # Incident spike - if INCIDENT_START <= date <= INCIDENT_END: - multiplier *= 3.0 - - # Random noise - multiplier *= np.random.normal(1, 0.1) - - return max(0.1, multiplier) - -# Distribute tickets across dates with realistic patterns -date_range = pd.date_range(START_DATE, END_DATE, freq='D') -daily_volumes = [int(BASE_DAILY_TICKETS * get_daily_multiplier(d)) for d in date_range] -``` - -### 5. Row Coherence - -Attributes within a row should correlate logically: - -```python -def generate_ticket(customer_id, tier, date): - """Generate a coherent ticket where attributes correlate.""" - - # Priority correlates with tier - if tier == 'Enterprise': - priority = np.random.choice(['Critical', 'High', 'Medium'], p=[0.3, 0.5, 0.2]) - else: - priority = np.random.choice(['Critical', 'High', 'Medium', 'Low'], p=[0.05, 0.2, 0.45, 0.3]) - - # Resolution time correlates with priority - resolution_scale = {'Critical': 4, 'High': 12, 'Medium': 36, 'Low': 72} - resolution_hours = np.random.exponential(scale=resolution_scale[priority]) - - # CSAT correlates with resolution time - if resolution_hours < 4: - csat = np.random.choice([4, 5], p=[0.3, 0.7]) - elif resolution_hours < 24: - csat = np.random.choice([3, 4, 5], p=[0.2, 0.5, 0.3]) - else: - csat = np.random.choice([1, 2, 3, 4], p=[0.1, 0.3, 0.4, 0.2]) - - return { - "customer_id": customer_id, - "priority": priority, - "resolution_hours": round(resolution_hours, 1), - "csat_score": csat, - "created_at": date, - } -``` - -## Complete Example - -Save as `scripts/generate_data.py`: - -```python -"""Generate synthetic customer, order, and ticket data.""" -import numpy as np -import pandas as pd -from datetime import datetime, timedelta -from faker import Faker -import holidays -from pyspark.sql import SparkSession - -# ============================================================================= -# CONFIGURATION -# ============================================================================= -CATALOG = "my_catalog" -SCHEMA = "my_schema" -VOLUME_PATH = f"/Volumes/{CATALOG}/{SCHEMA}/raw_data" - -N_CUSTOMERS = 2500 -N_ORDERS = 25000 -N_TICKETS = 8000 - -# Date range - last 6 months from today -END_DATE = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0) -START_DATE = END_DATE - timedelta(days=180) - -# Special events (within the date range) -INCIDENT_END = END_DATE - timedelta(days=21) -INCIDENT_START = INCIDENT_END - timedelta(days=10) - -# Holiday calendar -US_HOLIDAYS = holidays.US(years=[START_DATE.year, END_DATE.year]) - -SEED = 42 - -# ============================================================================= -# SETUP -# ============================================================================= -np.random.seed(SEED) -Faker.seed(SEED) -fake = Faker() -spark = SparkSession.builder.getOrCreate() - -# ============================================================================= -# CREATE INFRASTRUCTURE -# ============================================================================= -print(f"Creating catalog/schema/volume if needed...") -spark.sql(f"CREATE CATALOG IF NOT EXISTS {CATALOG}") -spark.sql(f"CREATE SCHEMA IF NOT EXISTS {CATALOG}.{SCHEMA}") -spark.sql(f"CREATE VOLUME IF NOT EXISTS {CATALOG}.{SCHEMA}.raw_data") - -print(f"Generating: {N_CUSTOMERS:,} customers, {N_ORDERS:,} orders, {N_TICKETS:,} tickets") - -# ============================================================================= -# 1. CUSTOMERS (Master Table) -# ============================================================================= -print("Generating customers...") - -customers_pdf = pd.DataFrame({ - "customer_id": [f"CUST-{i:05d}" for i in range(N_CUSTOMERS)], - "name": [fake.company() for _ in range(N_CUSTOMERS)], - "tier": np.random.choice(['Free', 'Pro', 'Enterprise'], N_CUSTOMERS, p=[0.6, 0.3, 0.1]), - "region": np.random.choice(['North', 'South', 'East', 'West'], N_CUSTOMERS, p=[0.4, 0.25, 0.2, 0.15]), -}) - -# ARR correlates with tier -customers_pdf["arr"] = customers_pdf["tier"].apply( - lambda t: round(np.random.lognormal(11, 0.5), 2) if t == 'Enterprise' - else round(np.random.lognormal(8, 0.6), 2) if t == 'Pro' else 0 -) - -# Lookups for foreign keys -customer_ids = customers_pdf["customer_id"].tolist() -customer_tier_map = dict(zip(customers_pdf["customer_id"], customers_pdf["tier"])) -tier_weights = customers_pdf["tier"].map({'Enterprise': 5.0, 'Pro': 2.0, 'Free': 1.0}) -customer_weights = (tier_weights / tier_weights.sum()).tolist() - -print(f" Created {len(customers_pdf):,} customers") - -# ============================================================================= -# 2. ORDERS (References Customers) -# ============================================================================= -print("Generating orders...") - -orders_data = [] -for i in range(N_ORDERS): - cid = np.random.choice(customer_ids, p=customer_weights) - tier = customer_tier_map[cid] - amount = np.random.lognormal(7 if tier == 'Enterprise' else 5 if tier == 'Pro' else 3.5, 0.7) - - orders_data.append({ - "order_id": f"ORD-{i:06d}", - "customer_id": cid, - "amount": round(amount, 2), - "status": np.random.choice(['completed', 'pending', 'cancelled'], p=[0.85, 0.10, 0.05]), - "order_date": fake.date_between(start_date=START_DATE, end_date=END_DATE), - }) - -orders_pdf = pd.DataFrame(orders_data) -print(f" Created {len(orders_pdf):,} orders") - -# ============================================================================= -# 3. TICKETS (References Customers, with incident spike) -# ============================================================================= -print("Generating tickets...") - -def get_daily_volume(date, base=25): - vol = base * (0.6 if date.weekday() >= 5 else 1.0) - if date in US_HOLIDAYS: - vol *= 0.3 # Even lower on holidays - if INCIDENT_START <= date <= INCIDENT_END: - vol *= 3.0 - return int(vol * np.random.normal(1, 0.15)) - -# Distribute tickets across dates -tickets_data = [] -ticket_idx = 0 -for day in pd.date_range(START_DATE, END_DATE): - daily_count = get_daily_volume(day.to_pydatetime()) - is_incident = INCIDENT_START <= day.to_pydatetime() <= INCIDENT_END - - for _ in range(daily_count): - if ticket_idx >= N_TICKETS: - break - - cid = np.random.choice(customer_ids, p=customer_weights) - tier = customer_tier_map[cid] - - # Category - Auth dominates during incident - if is_incident: - category = np.random.choice(['Auth', 'Network', 'Billing', 'Account'], p=[0.65, 0.15, 0.1, 0.1]) - else: - category = np.random.choice(['Auth', 'Network', 'Billing', 'Account'], p=[0.25, 0.30, 0.25, 0.20]) - - # Priority correlates with tier - priority = np.random.choice(['Critical', 'High', 'Medium'], p=[0.3, 0.5, 0.2]) if tier == 'Enterprise' \ - else np.random.choice(['Critical', 'High', 'Medium', 'Low'], p=[0.05, 0.2, 0.45, 0.3]) - - # Resolution time correlates with priority - res_scale = {'Critical': 4, 'High': 12, 'Medium': 36, 'Low': 72} - resolution = np.random.exponential(scale=res_scale[priority]) - - # CSAT degrades during incident for Auth - if is_incident and category == 'Auth': - csat = np.random.choice([1, 2, 3, 4, 5], p=[0.15, 0.25, 0.35, 0.2, 0.05]) - else: - csat = 5 if resolution < 4 else (4 if resolution < 12 else np.random.choice([2, 3, 4], p=[0.2, 0.5, 0.3])) - - tickets_data.append({ - "ticket_id": f"TKT-{ticket_idx:06d}", - "customer_id": cid, - "category": category, - "priority": priority, - "resolution_hours": round(resolution, 1), - "csat_score": csat, - "created_at": day.strftime("%Y-%m-%d"), - }) - ticket_idx += 1 - - if ticket_idx >= N_TICKETS: - break - -tickets_pdf = pd.DataFrame(tickets_data) -print(f" Created {len(tickets_pdf):,} tickets") - -# ============================================================================= -# 4. SAVE TO VOLUME -# ============================================================================= -print(f"\nSaving to {VOLUME_PATH}...") - -spark.createDataFrame(customers_pdf).write.mode("overwrite").parquet(f"{VOLUME_PATH}/customers") -spark.createDataFrame(orders_pdf).write.mode("overwrite").parquet(f"{VOLUME_PATH}/orders") -spark.createDataFrame(tickets_pdf).write.mode("overwrite").parquet(f"{VOLUME_PATH}/tickets") - -print("Done!") - -# ============================================================================= -# 5. VALIDATION -# ============================================================================= -print("\n=== VALIDATION ===") -print(f"Tier distribution: {customers_pdf['tier'].value_counts(normalize=True).to_dict()}") -print(f"Avg order by tier: {orders_pdf.merge(customers_pdf[['customer_id', 'tier']]).groupby('tier')['amount'].mean().to_dict()}") - -incident_tickets = tickets_pdf[tickets_pdf['created_at'].between( - INCIDENT_START.strftime("%Y-%m-%d"), INCIDENT_END.strftime("%Y-%m-%d") -)] -print(f"Incident period tickets: {len(incident_tickets):,} ({len(incident_tickets)/len(tickets_pdf)*100:.1f}%)") -print(f"Incident Auth %: {(incident_tickets['category'] == 'Auth').mean()*100:.1f}%") -``` - -Execute using `run_python_file_on_databricks` tool: -- `file_path`: "scripts/generate_data.py" - -If it fails, edit the file and re-run with the same `cluster_id` and `context_id`. - -### Validate Generated Data - -After successful execution, use `get_volume_folder_details` tool to verify the generated data: -- `volume_path`: "my_catalog/my_schema/raw_data/customers" -- `format`: "parquet" -- `table_stat_level`: "SIMPLE" - -This returns schema, row counts, and column statistics to confirm the data was written correctly. - -## Best Practices - -1. **Ask for schema**: Default to `ai_dev_kit` catalog, ask user for schema name -2. **Create infrastructure**: Use `CREATE CATALOG/SCHEMA/VOLUME IF NOT EXISTS` -3. **Raw data only**: No `total_x`, `sum_x`, `avg_x` fields - SDP pipeline computes those -4. **Save to Volume, not tables**: Write parquet to `/Volumes/{catalog}/{schema}/raw_data/` -5. **Configuration at top**: All sizes, dates, and paths as variables -6. **Dynamic dates**: Use `datetime.now() - timedelta(days=180)` for last 6 months -7. **Pandas for generation**: Faster and easier than Spark for row-by-row logic -8. **Master tables first**: Generate customers, then orders reference customer_ids -9. **Weighted sampling**: Enterprise customers generate more activity -10. **Distributions**: Log-normal for values, exponential for times, weighted categorical -11. **Time patterns**: Weekday/weekend, holidays, seasonality, event spikes -12. **Row coherence**: Priority affects resolution time affects CSAT -13. **Volume for aggregation**: 10K-50K rows minimum so patterns survive GROUP BY -14. **Always use files**: Write to local file, execute, edit if error, re-execute -15. **Context reuse**: Pass `cluster_id` and `context_id` for faster iterations -16. **Libraries**: Install `faker` and `holidays` first; most others are pre-installed diff --git a/.claude/skills/unstructured-pdf-generation/SKILL.md b/.claude/skills/unstructured-pdf-generation/SKILL.md deleted file mode 100644 index 1c5a5a4f..00000000 --- a/.claude/skills/unstructured-pdf-generation/SKILL.md +++ /dev/null @@ -1,187 +0,0 @@ ---- -name: unstructured-pdf-generation -description: "Generate synthetic PDF documents for RAG and unstructured data use cases. Use when creating test PDFs, demo documents, or evaluation datasets for retrieval systems." ---- - -# Unstructured PDF Generation - -Generate realistic synthetic PDF documents using LLM for RAG (Retrieval-Augmented Generation) and unstructured data use cases. - -## Overview - -This skill uses the `generate_pdf_documents` MCP tool to create professional PDF documents with: -- LLM-generated content based on your description -- Accompanying JSON files with questions and evaluation guidelines (for RAG testing) -- Automatic upload to Unity Catalog Volumes - -## Quick Start - -Use the `generate_pdf_documents` MCP tool: -- `catalog`: "my_catalog" -- `schema`: "my_schema" -- `description`: "Technical documentation for a cloud infrastructure platform including setup guides, troubleshooting procedures, and API references." -- `count`: 10 - -This generates 10 PDF documents and saves them to `/Volumes/my_catalog/my_schema/raw_data/pdf_documents/` (using default volume and folder). - -### With Custom Location - -Use the `generate_pdf_documents` MCP tool: -- `catalog`: "my_catalog" -- `schema`: "my_schema" -- `description`: "HR policy documents..." -- `count`: 10 -- `volume`: "custom_volume" -- `folder`: "hr_policies" -- `overwrite_folder`: true - -## Parameters - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `catalog` | string | Yes | - | Unity Catalog name | -| `schema` | string | Yes | - | Schema name | -| `description` | string | Yes | - | Detailed description of what PDFs should contain | -| `count` | int | Yes | - | Number of PDFs to generate | -| `volume` | string | No | `raw_data` | Volume name (created if not exists) | -| `folder` | string | No | `pdf_documents` | Folder within volume for output files | -| `doc_size` | string | No | `MEDIUM` | Document size: `SMALL` (~1 page), `MEDIUM` (~5 pages), `LARGE` (~10+ pages) | -| `overwrite_folder` | bool | No | `false` | If true, deletes existing folder contents first | - -### Document Size Guide - -- **SMALL**: ~1 page, concise content. Best for quick demos or testing. -- **MEDIUM**: ~4-6 pages, comprehensive coverage. Good balance for most use cases. -- **LARGE**: ~10+ pages, exhaustive documentation. Use for thorough RAG evaluation. - -## Output Files - -For each document, the tool creates two files: - -1. **PDF file** (`.pdf`): The generated document -2. **JSON file** (`.json`): Metadata for RAG evaluation - -### JSON Structure - -```json -{ - "title": "API Authentication Guide", - "category": "Technical", - "pdf_path": "/Volumes/catalog/schema/volume/folder/doc_001.pdf", - "question": "What authentication methods are supported by the API?", - "guideline": "Answer should mention OAuth 2.0, API keys, and JWT tokens with their use cases." -} -``` - -## Common Patterns - -### Pattern 1: HR Policy Documents - -Use the `generate_pdf_documents` MCP tool: -- `catalog`: "ai_dev_kit" -- `schema`: "hr_demo" -- `description`: "HR policy documents for a technology company including employee handbook, leave policies, performance review procedures, benefits guide, and workplace conduct guidelines." -- `count`: 15 -- `folder`: "hr_policies" -- `overwrite_folder`: true - -### Pattern 2: Technical Documentation - -Use the `generate_pdf_documents` MCP tool: -- `catalog`: "ai_dev_kit" -- `schema`: "tech_docs" -- `description`: "Technical documentation for a SaaS analytics platform including installation guides, API references, troubleshooting procedures, security best practices, and integration tutorials." -- `count`: 20 -- `folder`: "product_docs" -- `overwrite_folder`: true - -### Pattern 3: Financial Reports - -Use the `generate_pdf_documents` MCP tool: -- `catalog`: "ai_dev_kit" -- `schema`: "finance_demo" -- `description`: "Financial documents for a retail company including quarterly reports, expense policies, budget guidelines, and audit procedures." -- `count`: 12 -- `folder`: "reports" -- `overwrite_folder`: true - -### Pattern 4: Training Materials - -Use the `generate_pdf_documents` MCP tool: -- `catalog`: "ai_dev_kit" -- `schema`: "training" -- `description`: "Training materials for new software developers including onboarding guides, coding standards, code review procedures, and deployment workflows." -- `count`: 8 -- `folder`: "courses" -- `overwrite_folder`: true - -## Workflow - -1. **Ask for destination**: Default to `ai_dev_kit` catalog, ask user for schema name -2. **Get description**: Ask what kind of documents they need -3. **Generate PDFs**: Call `generate_pdf_documents` MCP tool with appropriate parameters -4. **Verify output**: Check the volume path for generated files - -## Best Practices - -1. **Detailed descriptions**: The more specific your description, the better the generated content - - BAD: "Generate some HR documents" - - GOOD: "HR policy documents for a technology company including employee handbook covering remote work policies, leave policies with PTO and sick leave details, performance review procedures with quarterly and annual cycles, and workplace conduct guidelines" - -2. **Appropriate count**: - - For demos: 5-10 documents - - For RAG testing: 15-30 documents - - For comprehensive evaluation: 50+ documents - -3. **Folder organization**: Use descriptive folder names that indicate content type - - `hr_policies/` - - `technical_docs/` - - `training_materials/` - -4. **Use overwrite_folder**: Set to `true` when regenerating to ensure clean state - -## Integration with RAG Pipelines - -The generated JSON files are designed for RAG evaluation: - -1. **Ingest PDFs**: Use the PDF files as source documents for your vector database -2. **Test retrieval**: Use the `question` field to query your RAG system -3. **Evaluate answers**: Use the `guideline` field to assess if the RAG response is correct - -Example evaluation workflow: -```python -# Load questions from JSON files -questions = load_json_files(f"/Volumes/{catalog}/{schema}/{volume}/{folder}/*.json") - -for q in questions: - # Query RAG system - response = rag_system.query(q["question"]) - - # Evaluate using guideline - is_correct = evaluate_response(response, q["guideline"]) -``` - -## Environment Configuration - -The tool requires LLM configuration via environment variables: - -```bash -# Databricks Foundation Models (default) -LLM_PROVIDER=DATABRICKS -DATABRICKS_MODEL=databricks-meta-llama-3-3-70b-instruct - -# Or Azure OpenAI -LLM_PROVIDER=AZURE -AZURE_OPENAI_ENDPOINT=https://your-resource.cognitiveservices.azure.com/ -AZURE_OPENAI_API_KEY=your-api-key -AZURE_OPENAI_DEPLOYMENT=gpt-4o -``` - -## Common Issues - -| Issue | Solution | -|-------|----------| -| **"No LLM endpoint configured"** | Set `DATABRICKS_MODEL` or `AZURE_OPENAI_DEPLOYMENT` environment variable | -| **"Volume does not exist"** | The tool creates volumes automatically; ensure you have CREATE VOLUME permission | -| **"PDF generation timeout"** | Reduce `count` or check LLM endpoint availability | -| **Low quality content** | Provide more detailed `description` with specific topics and document types | diff --git a/.codex/databricks-models.json b/.codex/databricks-models.json new file mode 100644 index 00000000..29a90bbd --- /dev/null +++ b/.codex/databricks-models.json @@ -0,0 +1,139 @@ +{ + "models": [ + { + "slug": "databricks-gpt-5-5", + "display_name": "Databricks GPT-5.5", + "description": "GPT-5.5 hosted through Databricks AI Gateway", + "default_reasoning_level": "medium", + "supported_reasoning_levels": [ + { "effort": "low", "description": "Fast responses with lighter reasoning" }, + { "effort": "medium", "description": "Balanced speed and reasoning depth" }, + { "effort": "high", "description": "Greater reasoning depth for complex tasks" }, + { "effort": "xhigh", "description": "Maximum reasoning depth for complex tasks" } + ], + "shell_type": "default", + "visibility": "list", + "supported_in_api": true, + "priority": 100, + "additional_speed_tiers": [], + "availability_nux": null, + "upgrade": null, + "base_instructions": "", + "supports_reasoning_summaries": true, + "support_verbosity": true, + "default_verbosity": null, + "apply_patch_tool_type": null, + "truncation_policy": { "mode": "tokens", "limit": 200000 }, + "supports_parallel_tool_calls": true, + "experimental_supported_tools": [] + }, + { + "slug": "databricks-gpt-5-4", + "display_name": "Databricks GPT-5.4", + "description": "GPT-5.4 hosted through Databricks AI Gateway", + "default_reasoning_level": "medium", + "supported_reasoning_levels": [ + { "effort": "low", "description": "Fast responses with lighter reasoning" }, + { "effort": "medium", "description": "Balanced speed and reasoning depth" }, + { "effort": "high", "description": "Greater reasoning depth for complex tasks" }, + { "effort": "xhigh", "description": "Maximum reasoning depth for complex tasks" } + ], + "shell_type": "default", + "visibility": "list", + "supported_in_api": true, + "priority": 90, + "additional_speed_tiers": [], + "availability_nux": null, + "upgrade": null, + "base_instructions": "", + "supports_reasoning_summaries": true, + "support_verbosity": true, + "default_verbosity": null, + "apply_patch_tool_type": null, + "truncation_policy": { "mode": "tokens", "limit": 200000 }, + "supports_parallel_tool_calls": true, + "experimental_supported_tools": [] + }, + { + "slug": "databricks-gpt-5-4-mini", + "display_name": "Databricks GPT-5.4 Mini", + "description": "GPT-5.4 Mini hosted through Databricks AI Gateway", + "default_reasoning_level": "medium", + "supported_reasoning_levels": [ + { "effort": "low", "description": "Fast responses with lighter reasoning" }, + { "effort": "medium", "description": "Balanced speed and reasoning depth" }, + { "effort": "high", "description": "Greater reasoning depth for complex tasks" }, + { "effort": "xhigh", "description": "Maximum reasoning depth for complex tasks" } + ], + "shell_type": "default", + "visibility": "list", + "supported_in_api": true, + "priority": 80, + "additional_speed_tiers": [], + "availability_nux": null, + "upgrade": null, + "base_instructions": "", + "supports_reasoning_summaries": true, + "support_verbosity": true, + "default_verbosity": null, + "apply_patch_tool_type": null, + "truncation_policy": { "mode": "tokens", "limit": 200000 }, + "supports_parallel_tool_calls": true, + "experimental_supported_tools": [] + }, + { + "slug": "databricks-gpt-5-3-codex", + "display_name": "Databricks GPT-5.3 Codex", + "description": "GPT-5.3 Codex hosted through Databricks AI Gateway", + "default_reasoning_level": "medium", + "supported_reasoning_levels": [ + { "effort": "low", "description": "Fast responses with lighter reasoning" }, + { "effort": "medium", "description": "Balanced speed and reasoning depth" }, + { "effort": "high", "description": "Greater reasoning depth for complex tasks" }, + { "effort": "xhigh", "description": "Maximum reasoning depth for complex tasks" } + ], + "shell_type": "default", + "visibility": "list", + "supported_in_api": true, + "priority": 70, + "additional_speed_tiers": [], + "availability_nux": null, + "upgrade": null, + "base_instructions": "", + "supports_reasoning_summaries": true, + "support_verbosity": true, + "default_verbosity": null, + "apply_patch_tool_type": null, + "truncation_policy": { "mode": "tokens", "limit": 200000 }, + "supports_parallel_tool_calls": true, + "experimental_supported_tools": [] + }, + { + "slug": "databricks-gpt-5-2", + "display_name": "Databricks GPT-5.2", + "description": "GPT-5.2 hosted through Databricks AI Gateway", + "default_reasoning_level": "medium", + "supported_reasoning_levels": [ + { "effort": "low", "description": "Fast responses with lighter reasoning" }, + { "effort": "medium", "description": "Balanced speed and reasoning depth" }, + { "effort": "high", "description": "Greater reasoning depth for complex tasks" }, + { "effort": "xhigh", "description": "Maximum reasoning depth for complex tasks" } + ], + "shell_type": "default", + "visibility": "list", + "supported_in_api": true, + "priority": 60, + "additional_speed_tiers": [], + "availability_nux": null, + "upgrade": null, + "base_instructions": "", + "supports_reasoning_summaries": true, + "support_verbosity": true, + "default_verbosity": null, + "apply_patch_tool_type": null, + "truncation_policy": { "mode": "tokens", "limit": 200000 }, + "supports_parallel_tool_calls": true, + "experimental_supported_tools": [] + } + ] +} \ No newline at end of file diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 00000000..9cfcfc24 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,14 @@ +version: 2 +updates: + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + cooldown: + default-days: 7 + - package-ecosystem: "pip" + directory: "/" + schedule: + interval: "weekly" + cooldown: + default-days: 7 diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..efa5e8ba --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,10 @@ +## What does this PR do? + + + +## Testing + +Please test your changes on dogfood before merging. + +- [ ] Deployed and tested on dogfood +- [ ] Added a screenshot to this PR diff --git a/.github/workflows/dependency-audit.yml b/.github/workflows/dependency-audit.yml new file mode 100644 index 00000000..026b8179 --- /dev/null +++ b/.github/workflows/dependency-audit.yml @@ -0,0 +1,77 @@ +name: Dependency Audit + +on: + workflow_dispatch: + pull_request: + paths: + - "requirements.txt" + - "requirements.lock" + - "pyproject.toml" + push: + branches: [main] + paths: + - "requirements.txt" + - "requirements.lock" + - "pyproject.toml" + schedule: + - cron: '0 6 * * 1' # Weekly Monday 6am UTC — catch newly disclosed CVEs + +permissions: + contents: read + +jobs: + audit: + runs-on: databrickslabs-protected-runner-group + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Set up Python + uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 + with: + python-version: "3.11" + + - name: Install uv + uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 + + - name: Install audit tools + run: pip install pip-audit==2.9.0 + + - name: Compile requirements.txt from pyproject.toml + run: | + # Keep requirements.txt in sync with pyproject.toml so Dependabot can scan it. + # Note: [tool.uv.sources] git overrides are not resolved by pip compile — + # requests and cryptography fall back to their PyPI versions here, which is + # intentional for Dependabot's purposes. + uv pip compile pyproject.toml -o /tmp/requirements.compiled.txt + if ! diff -q requirements.txt /tmp/requirements.compiled.txt > /dev/null 2>&1; then + echo "::warning::requirements.txt is out of date with pyproject.toml. Run: uv pip compile pyproject.toml -o requirements.txt" + fi + + - name: Audit pinned dependencies + run: | + if [ -f requirements.lock ]; then + echo "Auditing requirements.lock (pinned)..." + # Strip hashes before auditing — pip-audit's pip backend chokes on + # platform-conditional deps (greenlet) missing from the lockfile. + # The hashes are verified at install time, not audit time. + sed '/^[[:space:]]*--hash/d' requirements.lock > /tmp/requirements.lock.nohash + pip-audit -r /tmp/requirements.lock.nohash --desc on + else + echo "::warning::No requirements.lock found — auditing requirements.txt (unpinned)" + pip-audit -r requirements.txt --desc on + fi + + - name: Check lockfile is up to date + run: | + uv pip compile requirements.txt -o /tmp/requirements.lock.check --generate-hashes + if ! diff -q requirements.lock /tmp/requirements.lock.check > /dev/null 2>&1; then + echo "::warning::requirements.lock is out of date. Run: uv pip compile requirements.txt -o requirements.lock --generate-hashes" + fi + + - name: Audit npm packages + run: | + for pkg in opencode-ai @ai-sdk/openai @openai/codex @google/gemini-cli; do + echo "--- Checking $pkg ---" + npm view "$pkg" version 2>/dev/null || echo "::warning::Could not resolve $pkg" + done diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 00000000..e9973c94 --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,120 @@ +name: Release + +on: + workflow_dispatch: + inputs: + prerelease: + description: "Mark as pre-release?" + required: false + default: false + type: boolean + +jobs: + release: + runs-on: databrickslabs-protected-runner-group + permissions: + contents: write + + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + fetch-depth: 0 + + - name: Read version from pyproject.toml + id: version + run: | + VERSION=$(grep -m1 '^version' pyproject.toml | sed 's/version = "\(.*\)"/\1/') + if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then + echo "Error: version '$VERSION' in pyproject.toml is not valid semver" + exit 1 + fi + echo "VERSION=$VERSION" >> "$GITHUB_OUTPUT" + echo "TAG=v$VERSION" >> "$GITHUB_OUTPUT" + echo "Releasing $VERSION" + + - name: Check tag does not already exist + run: | + TAG="${{ steps.version.outputs.TAG }}" + if git tag -l "$TAG" | grep -q "$TAG"; then + echo "Error: tag $TAG already exists — did you forget to bump the version in pyproject.toml?" + exit 1 + fi + + - name: Generate release notes + id: notes + run: | + TAG="${{ steps.version.outputs.TAG }}" + PREV_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "") + + if [ -z "$PREV_TAG" ]; then + RANGE="HEAD" + SINCE_MSG="(all commits — first release)" + else + RANGE="${PREV_TAG}..HEAD" + SINCE_MSG="since $PREV_TAG" + fi + + echo "Generating notes for $RANGE" + + # Categorize commits by conventional commit prefix + FEATURES=$(git log $RANGE --pretty=format:"- %s (%h)" | grep -E "^- feat" | sed 's/^- feat[^:]*: /- /' || true) + FIXES=$(git log $RANGE --pretty=format:"- %s (%h)" | grep -E "^- fix" | sed 's/^- fix[^:]*: /- /' || true) + DOCS=$(git log $RANGE --pretty=format:"- %s (%h)" | grep -E "^- docs" | sed 's/^- docs[^:]*: /- /' || true) + REFACTORS=$(git log $RANGE --pretty=format:"- %s (%h)" | grep -E "^- refactor" | sed 's/^- refactor[^:]*: /- /' || true) + OTHER=$(git log $RANGE --pretty=format:"- %s (%h)" | grep -vE "^- (feat|fix|docs|refactor|chore|ci|test|style|perf|revert|Merge)" || true) + + { + echo "NOTES<> "$GITHUB_OUTPUT" + + - name: Create and push tag + run: | + TAG="${{ steps.version.outputs.TAG }}" + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + git tag -a "$TAG" -m "Release $TAG" + git push origin "$TAG" + + - name: Create GitHub Release + uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v3 + with: + tag_name: "${{ steps.version.outputs.TAG }}" + name: "${{ steps.version.outputs.TAG }}" + body: ${{ steps.notes.outputs.NOTES }} + prerelease: ${{ inputs.prerelease }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 00000000..408016c2 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,28 @@ +name: Tests + +on: + workflow_dispatch: + pull_request: + push: + branches: [main] + +permissions: + contents: read + +jobs: + test: + runs-on: databrickslabs-protected-runner-group + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Set up Python + uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 + with: + python-version: "3.12" + + - name: Install uv + uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 + + - name: Run tests + run: uv run pytest tests/ -v diff --git a/.github/workflows/update-lockfile.yml b/.github/workflows/update-lockfile.yml new file mode 100644 index 00000000..0ba41700 --- /dev/null +++ b/.github/workflows/update-lockfile.yml @@ -0,0 +1,40 @@ +name: Update Lockfile + +on: + push: + branches: [main] + paths: + - "requirements.txt" + +jobs: + update-lockfile: + runs-on: databrickslabs-protected-runner-group + permissions: + contents: write + + steps: + - name: Checkout + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Set up Python + uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 + with: + python-version: "3.11" + + - name: Install uv + uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0 + + - name: Regenerate requirements.lock + run: uv pip compile requirements.txt -o requirements.lock --generate-hashes + + - name: Commit updated lockfile + run: | + git config user.name "github-actions[bot]" + git config user.email "github-actions[bot]@users.noreply.github.com" + if git diff --quiet requirements.lock; then + echo "requirements.lock is already up to date, nothing to commit" + else + git add requirements.lock + git commit -m "chore: regenerate requirements.lock after requirements.txt update" + git push + fi diff --git a/.gitignore b/.gitignore index 7b753b0c..1f403c9b 100644 --- a/.gitignore +++ b/.gitignore @@ -4,9 +4,48 @@ __pycache__/ .env .venv/ venv/ +outstanding-todos.md # Workspace-specific config (use app.yaml.template) -app.yaml +# app.yaml # Git worktrees .worktrees/ + +# Human tokens (brainstorming notes) +.humantokens/ + +# Uploaded files (clipboard paste images) +uploads/ + +# uv lockfile — not portable across PyPI proxies, generate locally with `uv lock` +uv.lock + +# Codex CLI generated/cached files (the bundled model catalog is tracked) +.codex/config.toml +.codex/.env +.codex/AGENTS.md +.codex/.personality_migration +.codex/memories/ +.codex/tmp/ +.codex/sessions/ +.codex/cron/ +.codex/skills/ + +# Codex skills are generated at runtime by setup_codex.py from .claude/skills/ +.agents/ + +# Agent-plane reference clone +agent-plane-ref/ + +video/ + +# Playwright e2e SSO session state (workspace cookies — DO NOT commit) +tests/e2e/auth.json +tests/e2e/.cache/ + +# Playwright browser snapshots / traces from failed runs +tests/e2e/test-results/ +tests/e2e/playwright-report/ +.gstack/ +.omc/ diff --git a/CLAUDE.md b/CLAUDE.md index a2241b12..b301e11b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,17 +1,17 @@ # Claude Code on Databricks -Welcome! This environment comes pre-configured with 30 skills and 2 MCP servers. +Welcome! This environment comes pre-configured with 5 AI coding agents, 50 skills, and 3 MCP servers. Hermes Agent is available alongside Claude Code, Codex, Gemini CLI, and OpenCode — launch it with `hermes chat`. -## Skills (30 total) +## Skills (50 total) ### Databricks Skills (16) | Category | Skills | |----------|--------| -| AI & Agents | agent-bricks, databricks-genie, mlflow-evaluation, model-serving | +| AI & Agents | agent-bricks, databricks-genie, databricks-ai-functions, mlflow-evaluation, model-serving | | Analytics | aibi-dashboards, databricks-unity-catalog | -| Data Engineering | spark-declarative-pipelines, databricks-jobs, synthetic-data-generation | -| Development | asset-bundles, databricks-app-apx, databricks-app-python, databricks-python-sdk, databricks-config | +| Data Engineering | spark-declarative-pipelines, databricks-jobs, synthetic-data-gen | +| Development | databricks-bundles, databricks-apps-python, databricks-python-sdk, databricks-config | | Reference | databricks-docs, unstructured-pdf-generation | ### Development Workflow Skills (14) @@ -35,10 +35,52 @@ From [obra/superpowers](https://github.com/obra/superpowers): | writing-skills | Create new skills | | using-superpowers | Introduction to available skills | +### Flutter Skills (10) + +From [flutter/skills](https://github.com/flutter/skills): + +| Skill | Purpose | +|-------|---------| +| flutter-add-integration-test | Add integration tests | +| flutter-add-widget-preview | Add widget previews | +| flutter-add-widget-test | Add widget tests | +| flutter-apply-architecture-best-practices | Apply Flutter architecture best practices | +| flutter-build-responsive-layout | Build adaptive/responsive layouts | +| flutter-fix-layout-issues | Diagnose and fix layout issues | +| flutter-implement-json-serialization | Implement JSON serialization | +| flutter-setup-declarative-routing | Set up declarative routing (go_router) | +| flutter-setup-localization | Set up localization (i18n) | +| flutter-use-http-package | Networking with the http package | + +### Dart Skills (9) + +From [dart-lang/skills](https://github.com/dart-lang/skills): + +| Skill | Purpose | +|-------|---------| +| dart-add-unit-test | Add unit tests | +| dart-build-cli-app | Build a Dart CLI app | +| dart-collect-coverage | Collect test coverage | +| dart-fix-runtime-errors | Diagnose and fix runtime errors | +| dart-generate-test-mocks | Generate test mocks (mockito/mocktail) | +| dart-migrate-to-checks-package | Migrate assertions to the checks package | +| dart-resolve-package-conflicts | Resolve package version conflicts | +| dart-run-static-analysis | Run static analysis | +| dart-use-pattern-matching | Use Dart pattern matching | + +### Flutter UI Skills (1) + +From [nank1ro/flutter-shadcn-ui](https://github.com/nank1ro/flutter-shadcn-ui): + +| Skill | Purpose | +|-------|---------| +| shadcn-ui-flutter | Build Flutter UIs with shadcn_ui components | + ## MCP Servers - **DeepWiki** - AI-powered documentation for any GitHub repository - **Exa** - Web search and code context retrieval +- **CoDA** (exposed at `/mcp`) - Delegate coding tasks to AI agents via MCP. Any MCP client (Genie Code, Claude Desktop, Cursor) can call `coda_run`, `coda_inbox`, and `coda_get_result` to submit background tasks, check status, and retrieve results. See `docs/mcp-v2-background-execution.md`. ## Databricks CLI @@ -75,6 +117,10 @@ Before starting any new project or documentation: 3. **Then start working** - your commits will be backed up to Workspace +## Architecture + +Real-time terminal I/O over **WebSocket** (Flask-SocketIO) with automatic **HTTP polling fallback** via a Web Worker. Single gunicorn worker (PTY fds are process-local), 16 gthread threads. Per-session locks for WebSocket handlers; parallel agent setup at startup via ThreadPoolExecutor. + ## Quick Start - Projects sync to Databricks Workspace on git commit @@ -86,3 +132,6 @@ Before starting any new project or documentation: - Databricks skills from [databricks-solutions/ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit) - Development workflow skills from [obra/superpowers](https://github.com/obra/superpowers) + +# things to remember +Remember to never move .git folder to the workspace if you're running workspace import. \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..302b215f --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,36 @@ +# Contributing to Coding Agents on Databricks Apps + +Thank you for your interest in contributing! We welcome and appreciate contributions from the community. + +## A Note on AI-Assisted Development + +In the age of agentic coding, **code review has become the bottleneck**. It's easier than ever to generate large volumes of code, but every PR still needs a human to carefully review, understand, and approve it. Please be thoughtful about what you submit — don't let an AI agent fire off a sprawling PR without you personally reviewing and understanding every change. + +**Before submitting, ask yourself:** + +- Do I understand every line of this diff? +- Is each change necessary and intentional? +- Have I kept the scope focused rather than letting an agent "improve" unrelated code? + +The easier your PR is to review, the faster it gets merged. Help us help you. 🤝 + +## How to Contribute + +1. **Fork & branch** — Fork the repository and create a descriptive branch for your work (e.g., `fix/websocket-reconnect` or `feat/session-timeout`). + +2. **Keep changes focused** — One logical change per PR. Avoid mixing refactors, formatting changes, or unrelated fixes into the same PR. Try to break larger PRs into small commits. + +3. **Write a test plan** — If your contribution can't be validated with unit tests, include screenshots or a video demonstrating the functionality in your PR description. + +4. **Deploy & verify on your workspace** — Deploy your app on your workspace (use Dogfood if you're a Brickster) and confirm it works as expected before opening a PR. + +5. **Open a pull request** — Write a clear description covering: + - **What** changed and **why** + - How to test or verify the changes + - Any known limitations or follow-up work + +## Code Review + +We review every PR and may request changes. Please don't take feedback personally — we're all working toward the same goal. The smaller and more focused your PR, the faster the turnaround. + +Thank you for contributing! diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 00000000..7e2ee1e5 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,24 @@ +## DB license + +**Definitions**. + +Agreement: The agreement between Databricks, Inc., and you governing the use of the Databricks Services, as that term is defined in the Master Cloud Services Agreement (MCSA) located at www.databricks.com/legal/mcsa. + +Licensed Materials: The source code, object code, data, and/or other works to which this license applies. + +**Scope of Use**. You may not use the Licensed Materials except in connection with your use of the Databricks Services pursuant to the Agreement. Your use of the Licensed Materials must comply at all times with any restrictions applicable to the Databricks Services, generally, and must be used in accordance with any applicable documentation. You may view, use, copy, modify, publish, and/or distribute the Licensed Materials solely for the purposes of using the Licensed Materials within or connecting to the Databricks Services. If you do not agree to these terms, you may not view, use, copy, modify, publish, and/or distribute the Licensed Materials. + +**Redistribution**. You may redistribute and sublicense the Licensed Materials so long as all use is in compliance with these terms. In addition: + +- You must give any other recipients a copy of this License; +- You must cause any modified files to carry prominent notices stating that you changed the files; +- You must retain, in any derivative works that you distribute, all copyright, patent, trademark, and attribution notices, excluding those notices that do not pertain to any part of the derivative works; and +- If a "NOTICE" text file is provided as part of its distribution, then any derivative works that you distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the derivative works. + +You may add your own copyright statement to your modifications and may provide additional license terms and conditions for use, reproduction, or distribution of your modifications, or for any such derivative works as a whole, provided your use, reproduction, and distribution of the Licensed Materials otherwise complies with the conditions stated in this License. + +**Termination**. This license terminates automatically upon your breach of these terms or upon the termination of your Agreement. Additionally, Databricks may terminate this license at any time on notice. Upon termination, you must permanently delete the Licensed Materials and all copies thereof. + +**DISCLAIMER; LIMITATION OF LIABILITY.** + +THE LICENSED MATERIALS ARE PROVIDED “AS-IS” AND WITH ALL FAULTS. DATABRICKS, ON BEHALF OF ITSELF AND ITS LICENSORS, SPECIFICALLY DISCLAIMS ALL WARRANTIES RELATING TO THE LICENSED MATERIALS, EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, IMPLIED WARRANTIES, CONDITIONS AND OTHER TERMS OF MERCHANTABILITY, SATISFACTORY QUALITY OR FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. DATABRICKS AND ITS LICENSORS TOTAL AGGREGATE LIABILITY RELATING TO OR ARISING OUT OF YOUR USE OF OR DATABRICKS’ PROVISIONING OF THE LICENSED MATERIALS SHALL BE LIMITED TO ONE THOUSAND ($1,000) DOLLARS. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE LICENSED MATERIALS OR THE USE OR OTHER DEALINGS IN THE LICENSED MATERIALS. diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..f68644f6 --- /dev/null +++ b/Makefile @@ -0,0 +1,130 @@ +# Makefile for deploying Coding Agents to Databricks Apps +# +# Usage: +# make deploy PROFILE=dogfood # full deploy (create app, sync, deploy) +# make redeploy PROFILE=dogfood # skip app creation, just sync + deploy +# make create-pat PROFILE=dogfood # generate a 1-day PAT and copy to clipboard +# make status PROFILE=dogfood # check app status +# make open PROFILE=dogfood # open app in browser +# make clean PROFILE=dogfood # remove app and secret scope + +# Configuration (accepts lowercase: make deploy profile=dogfood) +ifdef profile +PROFILE := $(profile) +endif +ifdef app_name +APP_NAME := $(app_name) +endif +PROFILE ?= DEFAULT +APP_NAME ?= coding-agents + +# Resolve user email and workspace path from the profile +USER_EMAIL = $(shell databricks current-user me --profile $(PROFILE) --output json 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('userName',''))") +WORKSPACE_PATH = /Workspace/Users/$(USER_EMAIL)/apps/$(APP_NAME) + +.PHONY: help test integration-test e2e-test e2e-auth deploy redeploy create-app create-pat sync deploy-app status open clean enterprise-doctor + +# ── Help ───────────────────────────────────────────── + +test: ## Run unit tests (fast — excludes Docker integration + Playwright e2e) + uv run pytest tests/ -v --ignore=tests/integration --ignore=tests/e2e + +integration-test: ## Run Docker-based pipeline integration test (~3-5 min wall time) + uv run pytest tests/integration/ -v -s -rs + +e2e-test: ## Run Playwright e2e against live deployed app (needs `make e2e-auth` first) + uv run pytest tests/e2e/ -v -s + +e2e-auth: ## Record SSO session for e2e tests (one-time per cookie expiry) + @# Resolve the app URL via the configured profile, then launch a headed + @# Chromium that saves storage state to tests/e2e/auth.json. + @url=$$(databricks apps get coding-agents --profile $(PROFILE) --output json 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['url'])") && \ + echo "Recording SSO session against $$url ..." && \ + uv run playwright codegen --save-storage tests/e2e/auth.json "$$url" + @echo "" + @echo "Auth state saved to tests/e2e/auth.json (gitignored)." + @echo "Run `make e2e-test PROFILE=$(PROFILE)` to execute the suite." + +help: ## Show this help + @grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-18s\033[0m %s\n", $$1, $$2}' + +# ── Workflows ──────────────────────────────────────── + +deploy: create-app sync deploy-app ## Full deploy (create app, sync, deploy) + @echo "" + @echo "Deployment complete! App URL:" + @databricks apps get $(APP_NAME) --profile $(PROFILE) --output json 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin).get('url','(pending)'))" + +redeploy: sync deploy-app ## Redeploy: sync + deploy (skip secret setup) + @echo "" + @echo "Redeployment complete!" + +# ── Building Blocks ────────────────────────────────── + +create-app: ## Create the Databricks App (idempotent) + @echo "==> Checking if app '$(APP_NAME)' exists..." + @state=$$(databricks apps get $(APP_NAME) --profile $(PROFILE) --output json 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('compute_status',{}).get('state',''))" 2>/dev/null); \ + if [ "$$state" = "DELETING" ]; then \ + echo " App '$(APP_NAME)' is still deleting, waiting..."; \ + while [ "$$state" = "DELETING" ]; do \ + sleep 10; \ + state=$$(databricks apps get $(APP_NAME) --profile $(PROFILE) --output json 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('compute_status',{}).get('state',''))" 2>/dev/null); \ + done; \ + echo " Deletion complete."; \ + echo " Creating app '$(APP_NAME)'..."; \ + databricks apps create $(APP_NAME) --profile $(PROFILE); \ + elif [ -n "$$state" ]; then \ + echo " App '$(APP_NAME)' already exists (state: $$state), skipping create."; \ + else \ + echo " Creating app '$(APP_NAME)'..."; \ + databricks apps create $(APP_NAME) --profile $(PROFILE); \ + fi + +create-pat: ## Generate a 1-day PAT and copy it to your clipboard + @echo "==> Generating a 1-day PAT..." + @token=$$(databricks tokens create --lifetime-seconds $$((1 * 24 * 60 * 60)) --comment "coding-agents (1-day)" --profile $(PROFILE) --output json \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['token_value'])") && \ + echo "$$token" | pbcopy && \ + echo " PAT copied to clipboard! (expires in 24 hours)" + + +sync: ## Sync local files to Databricks workspace + @echo "==> Syncing to $(WORKSPACE_PATH)..." + @databricks sync . $(WORKSPACE_PATH) --watch=false --profile $(PROFILE) + +deploy-app: ## Deploy the app from workspace + @echo "==> Deploying app '$(APP_NAME)'..." + @databricks apps deploy $(APP_NAME) --source-code-path $(WORKSPACE_PATH) --profile $(PROFILE) --no-wait + +# ── Monitoring ─────────────────────────────────────── + +status: ## Check app status + @databricks apps get $(APP_NAME) --profile $(PROFILE) + +open: ## Open the app in browser + @databricks apps get $(APP_NAME) --profile $(PROFILE) --output json 2>/dev/null \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('url',''))" \ + | xargs open + +# ── Enterprise mode ───────────────────────────────── + +enterprise-doctor: ## Probe configured enterprise mirrors (PyPI, npm, GitHub) for reachability + @# Use the existing venv directly so the doctor doesn't itself trigger a uv resolve + @# (which would fail if PyPI is firewalled — the exact scenario this target diagnoses). + @if [ -x .venv/bin/python ]; then \ + .venv/bin/python scripts/enterprise_doctor.py; \ + else \ + uv run python scripts/enterprise_doctor.py; \ + fi + +# ── Cleanup (destructive) ─────────────────────────── + +clean: ## Remove the app (destructive) + @echo "==> Removing app '$(APP_NAME)'..." + @databricks apps delete $(APP_NAME) --profile $(PROFILE) 2>/dev/null && \ + echo " App '$(APP_NAME)' deleted." || \ + echo " App '$(APP_NAME)' not found or already deleted." + diff --git a/NOTICE.md b/NOTICE.md new file mode 100644 index 00000000..6c19f156 --- /dev/null +++ b/NOTICE.md @@ -0,0 +1,194 @@ +# CoDA - Coding Agents on Databricks Apps + +Copyright (2026) Databricks, Inc. + +This Software includes software developed at Databricks (https://www.databricks.com/) and its use is subject to the included LICENSE file (Databricks License). + +--- + +Dependencies are grouped below by license. Each table row lists a third-party package bundled or used at runtime, along with its license and upstream source. + +This NOTICE is regenerated from `pip-licenses` and `uv.lock` (Python), `npm view` (npm), and upstream repositories (Git-installed tools). See `pyproject.toml` for the direct Python dependency declarations. + +## Installed AI Coding Agents + +The following tools are installed at runtime as user-facing coding agents. They are not Python library dependencies but standalone CLI tools provisioned by the setup scripts. + +| Tool | Install Method | Package | License | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| Claude Code | curl \| bash | @anthropic-ai/claude-code | Custom (see README.md) | Anthropic, PBC. | [https://github.com/anthropics/claude-code](https://github.com/anthropics/claude-code) | +| Codex | npm install -g | @openai/codex | Apache-2.0 | OpenAI | [https://github.com/openai/codex](https://github.com/openai/codex) | +| Gemini CLI | npm install -g | @google/gemini-cli | Apache-2.0 | Google LLC | [https://github.com/google-gemini/gemini-cli](https://github.com/google-gemini/gemini-cli) | +| Hermes Agent | uv tool install (Git) | hermes-agent | MIT | Nous Research | [https://github.com/NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent) | +| OpenCode | npm install -g | opencode-ai | MIT | OpenCode contributors | [https://github.com/nicepkg/opencode](https://github.com/nicepkg/opencode) | + +Additionally, OpenCode installs `@ai-sdk/openai` (Apache-2.0, Vercel Inc.) as a runtime provider SDK. + +All agent versions are resolved dynamically at install time (latest from npm registry or Git HEAD); no versions are pinned. See `setup_claude.py`, `setup_codex.py`, `setup_gemini.py`, `setup_hermes.py`, and `setup_opencode.py` for installation details. + +--- + +## MCP Servers (Remote Services) + +The following Model Context Protocol (MCP) servers are configured as remote endpoints. They are not installed locally but accessed over HTTPS at runtime. + +| Server | Endpoint | License | Provider | Source | +| --- | --- | --- | --- | --- | +| DeepWiki | https://mcp.deepwiki.com/mcp | MIT | Cognition AI (Devin) | [https://github.com/regenrek/deepwiki-mcp](https://github.com/regenrek/deepwiki-mcp) | +| Exa | https://mcp.exa.ai/mcp | MIT (server reference impl.) | Exa Labs | [https://github.com/exa-labs/exa-mcp-server](https://github.com/exa-labs/exa-mcp-server) | + +--- + +## Bundled Skills and Plugins + +The following skill sets are bundled in `.claude/skills/` and `.claude/plugins/` to provide Databricks-specific knowledge and development workflows. + +| Source | License | Copyright | Source | +| --- | --- | --- | --- | +| [databricks-solutions/ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit) (Databricks skills) | Databricks License | Databricks, Inc. | [https://github.com/databricks-solutions/ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit) | +| [obra/superpowers](https://github.com/obra/superpowers) (development workflow skills) | MIT License | Jesse Vincent | [https://github.com/obra/superpowers](https://github.com/obra/superpowers) | + +--- + +## Python Dependencies + +**Dual-licensed Python packages** — the following packages offer a choice of license. They are listed below under the first license option: +- `cryptography` 47.0.0 — Apache-2.0 **OR** BSD-3-Clause (listed under Apache License 2.0) +- `packaging` 26.2 — Apache-2.0 **OR** BSD-2-Clause (listed under Apache License 2.0) +- `sniffio` 1.3.1 — Apache-2.0 **OR** MIT (listed under MIT License) + +**Platform-conditional packages** — the following packages are in `uv.lock` but only installed on specific platforms or Python versions: +- `colorama` 0.4.6 — BSD License — Windows only ([https://github.com/tartley/colorama](https://github.com/tartley/colorama)) +- `exceptiongroup` 1.3.1 — MIT License — Python < 3.11 only ([https://github.com/agronholm/exceptiongroup](https://github.com/agronholm/exceptiongroup)) +- `pywin32` 311 — PSF-2.0 — Windows only ([https://github.com/mhammond/pywin32](https://github.com/mhammond/pywin32)) + +--- + +## Apache License 2.0 + +Full license text: [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0). + +| Package | Version | Ecosystem | Type | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| cryptography | 47.0.0 | Python | runtime | The Python Cryptographic Authority and individual contributors | [https://github.com/pyca/cryptography](https://github.com/pyca/cryptography) | +| databricks-sdk | 0.105.0 | Python | runtime | Databricks, Inc. | [https://databricks-sdk-py.readthedocs.io](https://databricks-sdk-py.readthedocs.io) | +| google-auth | 2.49.2 | Python | runtime | Google Cloud Platform | [https://github.com/googleapis/google-auth-library-python](https://github.com/googleapis/google-auth-library-python) | +| importlib-metadata | 8.7.1 | Python | runtime | Jason R. Coombs | [https://github.com/python/importlib_metadata](https://github.com/python/importlib_metadata) | +| mlflow-skinny | 3.11.1 | Python | runtime | Databricks, Inc. | [https://mlflow.org](https://mlflow.org) | +| opentelemetry-api | 1.41.1 | Python | runtime | OpenTelemetry Authors | [https://github.com/open-telemetry/opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) | +| opentelemetry-proto | 1.41.1 | Python | runtime | OpenTelemetry Authors | [https://github.com/open-telemetry/opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) | +| opentelemetry-sdk | 1.41.1 | Python | runtime | OpenTelemetry Authors | [https://github.com/open-telemetry/opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) | +| opentelemetry-semantic-conventions | 0.62b1 | Python | runtime | OpenTelemetry Authors | [https://github.com/open-telemetry/opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) | +| packaging | 26.2 | Python | runtime | Donald Stufft | [https://github.com/pypa/packaging](https://github.com/pypa/packaging) | +| python-multipart | 0.0.26 | Python | runtime | Andrew Dunham | [https://github.com/Kludex/python-multipart](https://github.com/Kludex/python-multipart) | +| requests | 2.33.0 | Python | runtime | Kenneth Reitz | [https://github.com/psf/requests](https://github.com/psf/requests) | + +--- + +## BSD 2-Clause License + +Full license text: [https://opensource.org/licenses/BSD-2-Clause](https://opensource.org/licenses/BSD-2-Clause). + +| Package | Version | Ecosystem | Type | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| pyasn1 | 0.6.3 | Python | runtime | Ilya Etingof | [https://github.com/pyasn1/pyasn1](https://github.com/pyasn1/pyasn1) | + +--- + +## BSD 3-Clause License + +Full license text: [https://opensource.org/licenses/BSD-3-Clause](https://opensource.org/licenses/BSD-3-Clause). + +| Package | Version | Ecosystem | Type | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| click | 8.3.3 | Python | runtime | Pallets | [https://github.com/pallets/click/](https://github.com/pallets/click/) | +| Flask | 3.1.3 | Python | runtime | Pallets | [https://github.com/pallets/flask/](https://github.com/pallets/flask/) | +| GitPython | 3.1.47 | Python | runtime | Sebastian Thiel, Michael Trier | [https://github.com/gitpython-developers/GitPython](https://github.com/gitpython-developers/GitPython) | +| httpcore | 1.0.9 | Python | runtime | Tom Christie | [https://www.encode.io/httpcore/](https://www.encode.io/httpcore/) | +| idna | 3.13 | Python | runtime | Kim Davies | [https://github.com/kjd/idna](https://github.com/kjd/idna) | +| MarkupSafe | 3.0.3 | Python | runtime | Pallets | [https://github.com/pallets/markupsafe/](https://github.com/pallets/markupsafe/) | +| protobuf | 6.33.6 | Python | runtime | Google LLC | [https://developers.google.com/protocol-buffers/](https://developers.google.com/protocol-buffers/) | +| pycparser | 3.0 | Python | runtime | Eli Bendersky | [https://github.com/eliben/pycparser](https://github.com/eliben/pycparser) | +| python-dotenv | 1.2.2 | Python | runtime | Saurabh Kumar | [https://github.com/theskumar/python-dotenv](https://github.com/theskumar/python-dotenv) | +| sse-starlette | 3.4.1 | Python | runtime | sysid | [https://github.com/sysid/sse-starlette](https://github.com/sysid/sse-starlette) | +| starlette | 1.0.0 | Python | runtime | Tom Christie | [https://github.com/Kludex/starlette](https://github.com/Kludex/starlette) | +| uvicorn | 0.46.0 | Python | runtime | Tom Christie | [https://uvicorn.dev/](https://uvicorn.dev/) | +| Werkzeug | 3.1.8 | Python | runtime | Pallets | [https://github.com/pallets/werkzeug/](https://github.com/pallets/werkzeug/) | + +--- + +## BSD License + +Full license text: [https://opensource.org/licenses/BSD-3-Clause](https://opensource.org/licenses/BSD-3-Clause). + +| Package | Version | Ecosystem | Type | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| cloudpickle | 3.1.2 | Python | runtime | The cloudpickle developer team | [https://github.com/cloudpipe/cloudpickle](https://github.com/cloudpipe/cloudpickle) | +| gitdb | 4.0.12 | Python | runtime | Sebastian Thiel | [https://github.com/gitpython-developers/gitdb](https://github.com/gitpython-developers/gitdb) | +| httpx | 0.28.1 | Python | runtime | Tom Christie | [https://github.com/encode/httpx](https://github.com/encode/httpx) | +| itsdangerous | 2.2.0 | Python | runtime | Pallets | [https://github.com/pallets/itsdangerous/](https://github.com/pallets/itsdangerous/) | +| Jinja2 | 3.1.6 | Python | runtime | Pallets | [https://github.com/pallets/jinja/](https://github.com/pallets/jinja/) | +| pyasn1-modules | 0.4.2 | Python | runtime | Ilya Etingof | [https://github.com/pyasn1/pyasn1-modules](https://github.com/pyasn1/pyasn1-modules) | +| smmap | 5.0.3 | Python | runtime | Sebastian Thiel | [https://github.com/gitpython-developers/smmap](https://github.com/gitpython-developers/smmap) | +| sqlparse | 0.5.5 | Python | runtime | Andi Albrecht | [https://github.com/andialbrecht/sqlparse](https://github.com/andialbrecht/sqlparse) | + +--- + +## MIT License + +Full license text: [https://opensource.org/licenses/MIT](https://opensource.org/licenses/MIT). + +| Package | Version | Ecosystem | Type | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| annotated-doc | 0.0.4 | Python | runtime | Sebastian Ramirez | [https://github.com/fastapi/annotated-doc](https://github.com/fastapi/annotated-doc) | +| annotated-types | 0.7.0 | Python | runtime | Adrian Garcia Badaracco, Samuel Colvin, Zac Hatfield-Dodds | [https://github.com/annotated-types/annotated-types](https://github.com/annotated-types/annotated-types) | +| anyio | 4.13.0 | Python | runtime | Alex Gronholm | [https://anyio.readthedocs.io/en/stable/](https://anyio.readthedocs.io/en/stable/) | +| attrs | 26.1.0 | Python | runtime | Hynek Schlawack | [https://www.attrs.org/](https://www.attrs.org/) | +| blinker | 1.9.0 | Python | runtime | Jason Kirtland | [https://github.com/pallets-eco/blinker/](https://github.com/pallets-eco/blinker/) | +| cachetools | 7.0.6 | Python | runtime | Thomas Kemmer | [https://github.com/tkem/cachetools/](https://github.com/tkem/cachetools/) | +| cffi | 2.0.0 | Python | runtime | Armin Rigo, Maciej Fijalkowski | [https://cffi.readthedocs.io/](https://cffi.readthedocs.io/) | +| charset-normalizer | 3.4.7 | Python | runtime | Ahmed R. TAHRI | [https://github.com/jawah/charset_normalizer](https://github.com/jawah/charset_normalizer) | +| claude-agent-sdk | 0.1.68 | Python | runtime | Anthropic, PBC. | [https://github.com/anthropics/claude-agent-sdk-python](https://github.com/anthropics/claude-agent-sdk-python) | +| fastapi | 0.136.1 | Python | runtime | Sebastian Ramirez | [https://github.com/fastapi/fastapi](https://github.com/fastapi/fastapi) | +| Flask-SocketIO | 5.6.1 | Python | runtime | Miguel Grinberg | [https://github.com/miguelgrinberg/flask-socketio](https://github.com/miguelgrinberg/flask-socketio) | +| h11 | 0.16.0 | Python | runtime | Nathaniel J. Smith | [https://github.com/python-hyper/h11](https://github.com/python-hyper/h11) | +| httpx-sse | 0.4.3 | Python | runtime | Florimond Manca | [https://github.com/florimondmanca/httpx-sse](https://github.com/florimondmanca/httpx-sse) | +| jsonschema | 4.26.0 | Python | runtime | Julian Berman | [https://github.com/python-jsonschema/jsonschema](https://github.com/python-jsonschema/jsonschema) | +| jsonschema-specifications | 2025.9.1 | Python | runtime | Julian Berman | [https://github.com/python-jsonschema/jsonschema-specifications](https://github.com/python-jsonschema/jsonschema-specifications) | +| mcp | 1.27.0 | Python | runtime | Anthropic, PBC. | [https://modelcontextprotocol.io](https://modelcontextprotocol.io) | +| pydantic | 2.13.3 | Python | runtime | Samuel Colvin et al. | [https://github.com/pydantic/pydantic](https://github.com/pydantic/pydantic) | +| pydantic-core | 2.46.3 | Python | runtime | Samuel Colvin et al. | [https://github.com/pydantic/pydantic](https://github.com/pydantic/pydantic) | +| pydantic-settings | 2.14.0 | Python | runtime | Samuel Colvin, Eric Jolibois, Hasan Ramezani | [https://github.com/pydantic/pydantic-settings](https://github.com/pydantic/pydantic-settings) | +| PyJWT | 2.12.1 | Python | runtime | Jose Padilla | [https://github.com/jpadilla/pyjwt](https://github.com/jpadilla/pyjwt) | +| python-engineio | 4.13.1 | Python | runtime | Miguel Grinberg | [https://github.com/miguelgrinberg/python-engineio](https://github.com/miguelgrinberg/python-engineio) | +| python-socketio | 5.16.1 | Python | runtime | Miguel Grinberg | [https://github.com/miguelgrinberg/python-socketio](https://github.com/miguelgrinberg/python-socketio) | +| PyYAML | 6.0.3 | Python | runtime | Kirill Simonov | [https://pyyaml.org/](https://pyyaml.org/) | +| referencing | 0.37.0 | Python | runtime | Julian Berman | [https://github.com/python-jsonschema/referencing](https://github.com/python-jsonschema/referencing) | +| rpds-py | 0.30.0 | Python | runtime | Julian Berman | [https://github.com/crate-py/rpds](https://github.com/crate-py/rpds) | +| simple-websocket | 1.1.0 | Python | runtime | Miguel Grinberg | [https://github.com/miguelgrinberg/simple-websocket](https://github.com/miguelgrinberg/simple-websocket) | +| sniffio | 1.3.1 | Python | runtime | Nathaniel J. Smith | [https://github.com/python-trio/sniffio](https://github.com/python-trio/sniffio) | +| typing-inspection | 0.4.2 | Python | runtime | Victorien Plot | [https://github.com/pydantic/typing-inspection](https://github.com/pydantic/typing-inspection) | +| urllib3 | 2.6.3 | Python | runtime | Andrey Petrov | [https://github.com/urllib3/urllib3](https://github.com/urllib3/urllib3) | +| wsproto | 1.3.2 | Python | runtime | Benno Rice | [https://github.com/python-hyper/wsproto/](https://github.com/python-hyper/wsproto/) | +| zipp | 3.23.1 | Python | runtime | Jason R. Coombs | [https://github.com/jaraco/zipp](https://github.com/jaraco/zipp) | + +--- + +## Mozilla Public License 2.0 + +Full license text: [https://opensource.org/licenses/MPL-2.0](https://opensource.org/licenses/MPL-2.0). + +| Package | Version | Ecosystem | Type | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| bidict | 0.23.1 | Python | runtime | Joshua Bronson | [https://github.com/jab/bidict](https://github.com/jab/bidict) | +| certifi | 2026.4.22 | Python | runtime | Kenneth Reitz | [https://github.com/certifi/python-certifi](https://github.com/certifi/python-certifi) | + +--- + +## Python Software Foundation License + +Full license text: [https://docs.python.org/3/license.html](https://docs.python.org/3/license.html). + +| Package | Version | Ecosystem | Type | Copyright | Source | +| --- | --- | --- | --- | --- | --- | +| typing_extensions | 4.15.0 | Python | runtime | Guido van Rossum, Jukka Lehtosalo, Lukasz Langa, Michael Lee | [https://github.com/python/typing_extensions](https://github.com/python/typing_extensions) | diff --git a/README.md b/README.md index 20b286c8..8f532143 100644 --- a/README.md +++ b/README.md @@ -1,264 +1,482 @@ -# claude-code-cli-bricks -### What is it? +# Coding Agents on Databricks Apps -TL;DR: Claude Code on Databricks Apps for All Databricks Users 🚀 -A browser-based terminal emulator built with Flask and xterm.js, designed for cloud development environments with Databricks workspace integration and Claude Code CLI support. +[![Use this template](https://img.shields.io/badge/Use%20this%20template-2ea44f?logo=github)](https://github.com/datasciencemonkey/coding-agents-databricks-apps/generate) +[![Deploy to Databricks](https://img.shields.io/badge/Deploy-Databricks%20Apps-FF3621?logo=databricks&logoColor=white)](docs/deployment.md) +[![Agents](https://img.shields.io/badge/Agents-5%20included-green)](#whats-inside) +[![Skills](https://img.shields.io/badge/Skills-43%20built--in-blue)](#-all-43-skills) -### Why now? -On Jan 26. 2026, Andrej Karpathy made [this viral tweet](https://x.com/karpathy/status/2015883857489522876?s=46&t=tEsLJXJnGFIkaWs-Bhs1yA). Boris Cherny, the creator of claude code responded and said the following. -![alt text](image.png) +> Run Claude Code, Codex, Gemini CLI, Hermes Agent, and OpenCode in your browser — zero setup, wired to your Databricks workspace. -This app template opens this up for all Databricks Users! ❤️ +--- -No more pesky IDE setups, no bespoke tweaks. +
+ +
-Just use it all on Databricks, from the browser. Wired up to model serving endpoints on your workspace. +## Screenshots -## Features +
+ CODA demo — splash screen, multi-tab terminals, keyboard shortcuts +
-✅ **Browser-based Terminal** - Full PTY support with xterm.js frontend +--- -✅ **Real-time I/O** - Responsive terminal with polling-based communication +## Architecture -✅ **Terminal Resizing** - Dynamic resize support for responsive layouts +
+ CoDA architecture — always-on coding agents inside the customer's Databricks tenancy, governed by Unity Catalog and audited by MLflow +
-✅ **Databricks Workspace Integration** - Auto-sync projects to Databricks Workspace on git commits +CoDA runs as a hosted Databricks App inside your tenancy, alongside **Genie Code** — Databricks' in-product AI coding agent that lives in notebooks, the SQL editor, and dashboards. Genie Code is the interactive in-product surface; CoDA is the always-on hosted-app surface where Developers brief the agents through the browser and Claude Code, Codex, Gemini CLI, and OpenCode execute alongside the Hermes orchestrator. Both surfaces share the same access plane: every model call routes through Foundation Model APIs (no third-party egress) and every tool call routes through Governed MCP Servers (Unity Catalog ACLs + MLflow trace + named human identity). The result: agentic coding for legacy migration, application development, multi-repo refactor, production monitoring, code modernisation, and CI/CD deployments — all governed like any other workload. -✅ **Claude Code CLI** - Pre-configured to use Databricks hosted models as the API endpoint +--- -✅ **Configurable Model** - Switch between Claude models via `app.yaml` (default: `databricks-claude-sonnet-4-5`) +## What's Inside -✅ **Micro Editor** - Ships with [micro](https://micro-editor.github.io/), a modern terminal-based text editor +🟠 **Claude Code** — Anthropic's coding agent with 39 Databricks skills + 2 MCP servers -✅ **Databricks CLI** - Pre-configured with your PAT for immediate use +🟣 **Codex** — OpenAI's coding agent, pre-configured for Databricks -✅ **Single-User Security** - Only the token owner can access the terminal +🔵 **Gemini CLI** — Google's coding agent with shared skills -✅ **MCP Servers** - DeepWiki for GitHub docs, Exa for web search +🟡 **Hermes Agent** — NousResearch's multi-provider AI CLI with tool-calling and skills -### 30 Pre-installed Skills +🟢 **OpenCode** — Open-source agent with multi-provider support -✅ **Databricks Skills (16)** - Make building Databricks products simple. Create dashboards, jobs, pipelines, agents, and more with guided workflows that understand Databricks APIs and best practices. +Every agent installs at boot and connects to your **Databricks AI Gateway** — on first terminal session, paste a short-lived PAT and all CLIs are configured automatically. Token auto-rotates every 10 minutes. -✅ **Superpowers Skills (14)** - Provide the agentic framework for Claude Code. Test-driven development, systematic debugging, brainstorming, parallel agent workflows, and structured planning for complex tasks. +### 📺 Setup walkthrough (6 min) -## Skill Details +Want to see CoDA installed and running end-to-end? Click the thumbnail to watch the full walkthrough on YouTube. -### Databricks Skills + -From [databricks-solutions/ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit): +--- -| Category | Skills | -|----------|--------| -| AI & Agents | agent-bricks, databricks-genie, mlflow-evaluation, model-serving | -| Analytics | aibi-dashboards, databricks-unity-catalog | -| Data Engineering | spark-declarative-pipelines, databricks-jobs, synthetic-data-generation | -| Development | asset-bundles, databricks-app-apx, databricks-app-python, databricks-python-sdk, databricks-config | -| Reference | databricks-docs, unstructured-pdf-generation | +## Why Databricks -### Development Workflow Skills +This isn't just a terminal in the cloud. Running coding agents on Databricks gives you enterprise-grade infrastructure out of the box: -From [obra/superpowers](https://github.com/obra/superpowers): +| | Benefit | What you get | +|---|---|---| +| 🔐 | **Unity Catalog Integration** | All data access governed by UC permissions — agents can only touch what your identity allows | +| 🤖 | **AI Gateway** | Route all LLM calls through a single control plane — swap models, set rate limits, and manage API keys centrally | +| 🔀 | **Multi-AI & Multi-Agent** | Switch between Claude, GPT, Gemini, and open-source models on the fly — change the model or agent without redeploying | +| 📊 | **Consumption Monitoring** | Track token usage, cost, and latency per user and per model via the AI Gateway control center dashboard | +| 🔍 | **MLflow Tracing** | Every Claude Code session is automatically traced — review prompts, tool calls, and outputs in your MLflow experiment | +| 🧬 | **Assess Traces with Genie** | Point Genie at your MLflow traces to ask natural-language questions about agent behavior, cost patterns, and session quality | +| 📝 | **App Logs to Delta** | Optionally route application logs to Delta tables for long-term retention, querying, and dashboarding | -- brainstorming, test-driven-development, systematic-debugging, writing-plans -- verification-before-completion, executing-plans, dispatching-parallel-agents -- subagent-driven-development, using-git-worktrees, requesting-code-review -- receiving-code-review, finishing-a-development-branch, writing-skills, using-superpowers +--- -## MCP Servers +## Terminal Features -Pre-configured MCP servers for enhanced capabilities: +| | | +|---|---| +| 🎨 **8 Themes** | Dracula, Nord, Solarized, Monokai, GitHub Dark, and more | +| ✂️ **Split Panes** | Run two sessions side by side with a draggable divider | +| 🌐 **WebSocket I/O** | Real-time terminal output over WebSocket — zero-latency, eliminates polling delay | +| 🔁 **HTTP Polling Fallback** | Automatic fallback via Web Worker when WebSocket is unavailable | +| 🚀 **Parallel Setup** | 6 agent setups run in parallel (~5x faster startup) | +| 🔍 **Search** | Find anything in your terminal history (Ctrl+Shift+F) | +| 🎤 **Voice Input** | Dictate commands with your mic (Option+V) | +| 📋 **Image Paste** | Paste or drag-and-drop images into the terminal — saved to `~/uploads/`, path inserted automatically | +| ⌨️ **Customizable** | Fonts, font sizes, themes — all persisted across sessions | +| 🔄 **Workspace Sync** | Every `git commit` auto-syncs to `/Workspace/Users/{you}/projects/` | +| ✏️ **Micro Editor** | Modern terminal editor, pre-installed | +| ⚙️ **Databricks CLI** | Installed at boot, configured interactively on first session | +| 📊 **MLflow Tracing** | Every Claude Code session is automatically traced to your Databricks MLflow experiment | -| Server | Description | -|--------|-------------| -| **DeepWiki** | AI-powered documentation for any GitHub repository | -| **Exa** | Web search and code context retrieval | +--- -### Updating Skills +## MLflow Tracing -Skills are bundled with the app. To update: +Claude Code and Codex sessions can both be **automatically traced** to a single Databricks MLflow experiment — flip one switch to turn them on. -1. Pull latest from [ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit) -2. Copy `databricks-skills/*` to `.claude/skills/` -3. For superpowers, pull latest from [obra/superpowers](https://github.com/obra/superpowers) and copy `skills/*` to `.claude/skills/` -4. Redeploy the app +### Turning it on -## Quick Start +Set **`MLFLOW_TRACING_ENABLED=true`** in `app.yaml` (or your shell for local dev). That single variable enables tracing for both CLIs. Tracing is **off by default** to keep deploys lightweight — opt in when you want it. -### Prerequisites +```yaml +# app.yaml +env: + - name: MLFLOW_TRACING_ENABLED + value: "true" +``` -- Python 3.11+ -- [uv](https://github.com/astral-sh/uv) (recommended) or pip +### How it works -## Deploying to Databricks +``` +MLFLOW_TRACING_ENABLED=true + │ + ├──► Claude Code: Stop hook fires on session end → + │ mlflow.claude_code.hooks.stop_hook_handler() logs the transcript + │ + └──► Codex: @mlflow/codex notify hook fires after each turn → + trace appended to the experiment +``` -1. Clone this repo to your Databricks Workspace -2. Navigate to **Compute** → **Apps** -3. Click **Create App** and select **Custom App** -4. Point to the cloned repo and deploy +Both land in the same MLflow experiment, so you can compare runs across agents side by side. -### Installation -```bash -# Clone the repository -git clone https://github.com/your-username/claude-code-cli-bricks.git -cd claude-code-cli-bricks +### Where traces live -# Install dependencies -uv pip install -r requirements.txt +``` +/Users/{your-email}/{app-name} ``` -### Running Locally +For example, if you're `jane@company.com` and your app is named `coding-agents`: -```bash -uv run python app.py +``` +/Users/jane@company.com/coding-agents ``` -Open http://localhost:8000 in your browser. +View them in the Databricks UI: **Workspace > Machine Learning > Experiments**. +### Configuration -## Architecture +Tracing is wired up during app startup: -``` -┌─────────────────────┐ HTTP ┌─────────────────────┐ -│ Browser Client │◄────────────►│ Flask Backend │ -│ (xterm.js) │ Polling │ (PTY Manager) │ -└─────────────────────┘ └─────────────────────┘ - │ - ▼ - ┌─────────────────────┐ - │ Shell Process │ - │ (/bin/bash) │ - └─────────────────────┘ -``` +| Setting | Value | Purpose | +|---------|-------|---------| +| `MLFLOW_TRACING_ENABLED` | `true`/`false` (default `false`) | Master switch for Claude + Codex | +| `MLFLOW_CLAUDE_TRACING_ENABLED` | mirrors `MLFLOW_TRACING_ENABLED` | Gates Claude's Stop hook at runtime | +| `MLFLOW_TRACKING_URI` | `databricks` | Routes traces to the Databricks backend | +| `MLFLOW_EXPERIMENT_NAME` | `/Users/{owner}/{app}` | Target experiment path | +| `MLFLOW_EXPERIMENT_ID` | resolved from name | Set in `~/.codex/.env` (Codex needs an ID) | -### API Endpoints +Tracing setup is skipped gracefully when `APP_OWNER` is not set (e.g., local dev without Databricks) or when `MLFLOW_TRACING_ENABLED` is left at its default `false`. -| Endpoint | Method | Description | -|----------|--------|-------------| -| `/` | GET | Serves the terminal UI | -| `/health` | GET | Health check with session count | -| `/api/session` | POST | Create new terminal session | -| `/api/input` | POST | Send input to terminal | -| `/api/output` | POST | Poll for terminal output | -| `/api/resize` | POST | Resize terminal dimensions | -| `/api/session` | DELETE | Close terminal session | +--- -## Project Structure +## Quick Start -``` -claude-code-cli-bricks/ -├── .claude/ -│ └── skills/ # 30 pre-installed skills -├── app.py # Flask backend with PTY management -├── app.yaml # Databricks Apps deployment config -├── app.yaml.template # Template for app.yaml configuration -├── CLAUDE.md # Claude Code welcome message -├── requirements.txt # Python dependencies -├── setup_claude.py # Claude Code CLI + MCP configuration -├── setup_databricks.py # Databricks CLI configuration -├── sync_to_workspace.py # Git hook for Databricks sync -├── static/ -│ ├── index.html # Terminal UI -│ └── lib/ # xterm.js library files -└── docs/ - └── plans/ # Design documentation -``` +### Deploy to Databricks Apps + +1. Click [**Use this template**](https://github.com/datasciencemonkey/coding-agents-databricks-apps/generate) to create your own repo +2. Go to **Databricks → Apps → Create App** +3. Choose **Custom App** and connect your new repo +4. Deploy +5. Open the app — paste a short-lived PAT when prompted on first terminal session + +That's it. No secrets to configure, no pre-deployment setup. -## Configuration +[→ Full deployment guide](docs/deployment.md) — environment variables, gateway config, and advanced options. -### Setting up app.yaml +### Run locally -Copy the template and configure your Databricks workspace: +1. Click [**Use this template**](https://github.com/datasciencemonkey/coding-agents-databricks-apps/generate) to create your own repo +2. Clone your new repo and run: ```bash -cp app.yaml.template app.yaml +git clone https://github.com//.git +cd +uv run python app.py ``` -Edit `app.yaml` and replace `` with your Databricks workspace URL: +Open [http://localhost:8000](http://localhost:8000) — type `claude`, `codex`, `gemini`, or `opencode` to start coding. -```yaml -env: - - name: DATABRICKS_HOST - value: https://.cloud.databricks.com -``` +--- -The `DATABRICKS_HOST` is used by both: -- **Workspace sync** - To upload projects on git commits -- **Claude Code CLI** - As the Anthropic API endpoint (via Databricks serving endpoints) +
+🧠 All 43 Skills -## Databricks Deployment +### Databricks Skills (25) — [ai-dev-kit](https://github.com/databricks-solutions/ai-dev-kit) -This project is configured for deployment as a Databricks App. +| Category | Skills | +|----------|--------| +| AI & Agents | agent-bricks, genie, mlflow-eval, model-serving | +| Analytics | aibi-dashboards, unity-catalog, metric-views | +| Data Engineering | declarative-pipelines, jobs, structured-streaming, synthetic-data, zerobus-ingest | +| Development | asset-bundles, app-apx, app-python, python-sdk, config, spark-python-data-source | +| Storage | lakebase-autoscale, lakebase-provisioned, vector-search | +| Reference | docs, dbsql, pdf-generation | +| Meta | refresh-databricks-skills | -### Environment Variables +### Superpowers Skills (14) — [obra/superpowers](https://github.com/obra/superpowers) -| Variable | Description | -|----------|-------------| -| `DATABRICKS_HOST` | Databricks workspace URL | -| `DATABRICKS_TOKEN` | Your Personal Access Token (PAT) | -| `ANTHROPIC_MODEL` | Model name (default: `databricks-claude-sonnet-4-5`) | +| Category | Skills | +|----------|--------| +| Build | brainstorming, writing-plans, executing-plans | +| Code | test-driven-dev, subagent-driven-dev | +| Debug | systematic-debugging, verification | +| Review | requesting-review, receiving-review | +| Ship | finishing-branch, git-worktrees | +| Meta | dispatching-agents, writing-skills, using-superpowers | -### Security Model +### BDD Skills (4) -This is a **single-user app**. Each user deploys their own instance with their own PAT: +| Category | Skills | +|----------|--------| +| Testing | bdd-features, bdd-run, bdd-scaffold, bdd-steps | + +
-1. The `DATABRICKS_TOKEN` in `app.yaml` identifies the owner -2. At startup, the app determines the token owner via Databricks API -3. Only requests from the token owner are allowed -4. Other users see a 403 Forbidden error +
+🔌 MCP Servers -This ensures your terminal session is private and uses your Databricks permissions. +### Built-in MCP Clients -### Create App +| Server | What it does | +|--------|-------------| +| **DeepWiki** | Ask questions about any GitHub repo — gets AI-powered answers from the codebase | +| **Exa** | Web search and code context retrieval for up-to-date information | -First, create the app in your Databricks workspace: +### CoDA MCP Server (exposed at `/mcp`) -```bash -databricks apps create xterm-terminal +CoDA itself exposes an **MCP server** that any MCP-compatible client can connect to — delegate coding tasks to AI agents running on Databricks, without needing the terminal UI. + +| Tool | Purpose | +|------|---------| +| `coda_run` | Fire-and-forget: submit a coding task, get back immediately | +| `coda_inbox` | Dashboard: see all running/completed/failed tasks at a glance | +| `coda_get_result` | Pull the full structured result of a completed task | + +**Why this matters:** Any tool that speaks MCP can use your Databricks-hosted coding agents — no custom integration needed. + +#### Example: Databricks Genie Code + +Genie Code connects to CoDA's MCP endpoint and delegates coding work to agents running in the background: + +``` +User → Genie Code: "Build me a sales pipeline using the transactions table" + +Genie Code calls coda_run(prompt="Build a sales pipeline...", email="user@company.com", + context='{"tables": ["sales.transactions"]}') + +→ Returns immediately: {task_id: "task-abc", status: "running"} +→ User keeps chatting with Genie Code while the agent works + +User → Genie Code: "How's my pipeline coming?" + +Genie Code calls coda_inbox() +→ {tasks: [{task_id: "task-abc", status: "completed", summary: "Built pipeline.py..."}]} + +Genie Code calls coda_get_result(task_id="task-abc", session_id="sess-123") +→ {summary: "Created pipeline.py with 3 stages", files_changed: ["pipeline.py"], ...} ``` -### Deploy via CLI +#### Connecting MCP Clients (Claude Code, Claude Desktop, Cursor, etc.) + +Databricks Apps use OAuth — not PATs — for authentication. A static `Authorization: Bearer ` header will get a `302` redirect to the OAuth login page. To connect any MCP client, use the **stdio bridge** (`tools/coda-bridge.py`) which injects fresh OAuth tokens automatically via `databricks auth token`. -Deploy the code using the Databricks CLI: +**1. Copy the bridge script:** ```bash -# 1. Import project files to workspace (wipe clean first for fresh deploy) -databricks workspace delete /Workspace/Users//xterm-experiment --recursive -databricks workspace import-dir . /Workspace/Users//xterm-experiment --overwrite +mkdir -p ~/.claude/mcp-bridges +cp tools/coda-bridge.py ~/.claude/mcp-bridges/ +``` -# 2. Deploy the app -databricks apps deploy xterm-terminal --source-code-path /Workspace/Users//xterm-experiment +**2. Add to your MCP client settings** (e.g. `~/.claude/settings.json`): + +```json +"coda-mcp": { + "type": "stdio", + "command": "python3", + "args": ["/path/to/.claude/mcp-bridges/coda-bridge.py"], + "env": { + "CODA_MCP_URL": "https://your-app.databricksapps.com/mcp", + "DATABRICKS_PROFILE": "your-profile" + } +} ``` -Replace `` with your Databricks username (e.g., `user@example.com`). +**3. Restart your MCP client.** -Once the app is deployed. You'll need to add the 'DATABRICKS_TOKEN' secret to your Databricks workspace and reference it in the [App Resources tab](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources). +The bridge reads `CODA_MCP_URL` and `DATABRICKS_PROFILE` from environment — no hardcoded values. If you redeploy the app or switch workspaces, just update the `env` block. -### Automatic Git Configuration +**Prerequisites:** `databricks` CLI installed and authenticated (`databricks auth login -p `), Python 3.8+, no pip dependencies. -When the app starts, it automatically configures git with your Databricks identity: -- **Email**: From your Databricks `userName` -- **Name**: From your Databricks `displayName` (or derived from email) +**Troubleshooting:** Bridge logs go to stderr. If you see `Auth failed (302)`, refresh your CLI session with `databricks auth login -p `. See [full setup guide](docs/mcp-client-setup.md) for details. -This means commits made within the app will be attributed to your Databricks account. +#### Task Chaining -## Workspace Sync +Chain tasks by passing `previous_session_id` — the new agent reads the prior task's results for context: -When deployed, git commits automatically sync your projects to Databricks Workspace: +``` +coda_run(prompt="Add monitoring to the pipeline", previous_session_id="sess-123") +``` + +See [MCP v2 Design Doc](docs/mcp-v2-background-execution.md) for the full protocol reference. + +
+ +
+🏗️ Architecture ``` -/Workspace/Users/{email}/projects/{project-name}/ +┌─────────────────────┐ WebSocket ┌──────────────────────────────────┐ +│ Browser Client │◄═══════════►│ uvicorn (ASGI) │ +│ (xterm.js) │ (fallback) │ ├─ python-socketio (Socket.IO) │ +│ │───────────►│ ├─ FastMCP /mcp │ +│ │ HTTP Poll │ └─ WSGIMiddleware(Flask + PTY) │ +│ │ (primary │ │ +│ │ under uvicorn) │ +└─────────────────────┘ └──────────────────────────────────┘ + │ │ + │ on first load │ on startup + ▼ ▼ +┌─────────────────────┐ ┌─────────────────────┐ +│ Setup Progress │ │ Background Setup │ +│ (inline UI) │ │ (11 steps, 5→6 ║) │ +└─────────────────────┘ └─────────────────────┘ + │ + ▼ + ┌─────────────────────┐ + │ Shell Process │ + │ (/bin/bash) │ + └─────────────────────┘ ``` -This is enabled via a git post-commit hook configured by `setup_claude.py`. +### Startup Flow -## Technologies +1. uvicorn starts `coda_mcp.mcp_asgi:app`, which calls `initialize_app()` during ASGI lifespan startup (Flask mounted via `WSGIMiddleware`; MCP mounted at `/mcp` via native ASGI; Socket.IO wraps both) +2. App serves the terminal UI with inline setup progress +3. Background thread runs setup: 5 sequential steps (git config, micro editor, GitHub CLI, Databricks CLI upgrade, content-filter proxy), then 6 agent setups (`setup/setup_claude.py`, `setup/setup_codex.py`, etc.) run in parallel via `ThreadPoolExecutor` +4. `/api/setup-status` endpoint reports progress to the UI +5. Once complete, the terminal becomes interactive -- **Backend**: Flask, Python PTY/termios -- **Frontend**: xterm.js, FitAddon -- **Integration**: Databricks SDK, Claude Agent SDK +### API Endpoints -## License +| Endpoint | Method | Description | +|----------|--------|-------------| +| `/` | GET | Terminal UI with inline setup progress | +| `/health` | GET | Health check with session count and setup status | +| `/api/setup-status` | GET | Setup progress for the UI | +| `/api/app-state` | GET | Persisted app state (owner, last rotation) | +| `/api/version` | GET | App version | +| `/api/sessions` | GET | List active (non-exited) sessions with metadata | +| `/api/pat-status` | GET | Whether a valid, usable PAT is currently configured | +| `/api/configure-pat` | POST | Interactive first-session PAT setup | +| `/api/session` | POST | Create new terminal session | +| `/api/session/attach` | POST | Reattach to an existing session (replays buffered output) | +| `/api/input` | POST | Send input to terminal | +| `/api/output` | POST | Poll for terminal output (single session) | +| `/api/output-batch` | POST | Batch poll output for multiple sessions | +| `/api/heartbeat` | POST | Lightweight keepalive (no buffer drain) | +| `/api/resize` | POST | Resize terminal dimensions | +| `/api/upload` | POST | Upload file (clipboard image paste) | +| `/api/session/close` | POST | Close terminal session | +| `/mcp` | POST | MCP JSON-RPC endpoint (CoDA tools) | + +### WebSocket Events (Socket.IO) + +| Event | Direction | Description | +|-------|-----------|-------------| +| `join_session` | Client → Server | Join session room for output delivery | +| `leave_session` | Client → Server | Leave session room | +| `terminal_input` | Client → Server | Send keystrokes to PTY | +| `terminal_resize` | Client → Server | Resize terminal | +| `heartbeat` | Client → Server | Keepalive for idle sessions | +| `terminal_output` | Server → Client | Push PTY output in real time | +| `session_exited` | Server → Client | Shell process exited | +| `session_closed` | Server → Client | Session terminated by server | +| `shutting_down` | Server → Client | Server restarting (SIGTERM) | + +
+ +
+⚙️ Configuration + +### Environment Variables + +| Variable | Required | Description | +|----------|----------|-------------| +| `HOME` | Yes | Set to `/app/python/source_code` in app.yaml | +| `DATABRICKS_TOKEN` | No | Optional. If not set, the app prompts for a token on first session. Auto-rotated every 10 minutes | +| `DATABRICKS_GATEWAY_HOST` | No | AI Gateway URL override. Auto-discovered from `DATABRICKS_WORKSPACE_ID` if unset | +| `ANTHROPIC_MODEL` | No | Claude model name (default: `databricks-claude-opus-4-7`) | +| `CODEX_MODEL` | No | Codex model name (default: `databricks-gpt-5-5`) | +| `GEMINI_MODEL` | No | Gemini model name (default: `databricks-gemini-2-5-pro`) | +| `HERMES_MODEL` | No | Hermes model name (default: `databricks-claude-opus-4-6`) | +| `HERMES_FALLBACK_MODEL` | No | Fallback model if `HERMES_MODEL` is unavailable in this workspace's geo | +| `ENABLE_HERMES` | No | Set to `"false"` to skip Hermes Agent install. Other CLIs are unaffected. Default `"true"` | +| `MAX_CONCURRENT_SESSIONS` | No | Cap on simultaneous PTY sessions per worker (default `5`) | +| `CLAUDE_CODE_DISABLE_AUTO_MEMORY` | No | Pass-through to Claude Code's auto-memory feature (default `0`) | +| `MLFLOW_TRACING_ENABLED` | No | Set to `"true"` to enable MLflow tracing for Claude, Codex, and Gemini in one switch (default `"false"`) | +| `DEEPWIKI_MCP_URL` | No | Override or disable the DeepWiki MCP server (set to `""` to remove) | +| `EXA_MCP_URL` | No | Override or disable the Exa MCP server (set to `""` to remove) | +| `TEAM_MEMORY_MCP_URL` | No | Optional shared-org-memory MCP server URL | +| `ENTERPRISE_MODE` | No | When `"true"`, logs a banner and warns on missing recommended mirrors. See [enterprise docs](docs/enterprise.md) for the full enterprise contract (JFrog mirrors, custom CA bundle, corporate proxy, etc.) | + +### Security Model + +Single-user app — the owner is resolved via the app's service principal and Apps API (`app.creator`), with no PAT required at deploy time. Authorization checks `X-Forwarded-Email` against `app.creator`. On first terminal session, the user pastes a short-lived PAT interactively. Tokens auto-rotate every 10 minutes (15-minute lifetime), with old tokens proactively revoked. On restart, the user re-pastes (no persistence by design). + +### Server + +Production uses `uvicorn` (single worker — PTY state is process-local) serving `coda_mcp.mcp_asgi:app`. The ASGI stack composes `python-socketio.ASGIApp` → MCP Streamable HTTP at `/mcp` → `WSGIMiddleware(Flask)` for the terminal UI. WebSocket transport falls back to HTTP polling under uvicorn — the `static/poll-worker.js` Web Worker already handles this transparently. `gunicorn.conf.py` is retained for reference and local WSGI-only dev; it is **not** used in production. + +
+ +
+📁 Project Structure + +``` +coding-agents-databricks-apps/ +├── app.py # Flask backend + PTY management + setup orchestration +├── app_state.py # Shared app state (setup progress, session registry) +├── app.yaml # Databricks Apps deployment config (uvicorn entrypoint) +├── cli_auth.py # Interactive PAT setup + CLI credential writer +├── content_filter_proxy.py # Proxy: sanitises OpenCode/Gemini traffic, transparently relays Codex, injects rotated PATs +├── gunicorn.conf.py # Legacy WSGI-only config (unused in production; uvicorn is the entrypoint) +├── pat_rotator.py # Background PAT auto-rotation (10-min cycle) +├── pyproject.toml # Package metadata + uv config (supply-chain guardrails) +├── requirements.txt # Compiled from pyproject.toml (Dependabot compatibility) +├── requirements.lock # Hash-pinned lockfile (auto-regenerated by CI) +├── Makefile # Deploy, redeploy, status, and cleanup targets +├── sync_to_workspace.py # Post-commit hook: sync to Workspace +├── utils.py # Utility functions (ensure_https, gateway discovery) +├── coda_mcp/ # MCP server package (CoDA — Coding Agents) +│ ├── __init__.py +│ ├── mcp_server.py # FastMCP tool definitions (coda_run, coda_inbox, coda_get_result) +│ ├── mcp_endpoint.py # Flask Blueprint: JSON-RPC /mcp endpoint +│ ├── mcp_asgi.py # ASGI bridge (optional, for native MCP SDK transport) +│ └── task_manager.py # Disk-based session/task state manager +├── setup/ # Agent setup scripts (run at boot) +│ ├── setup_claude.py # Claude Code CLI + MCP configuration +│ ├── setup_codex.py # Codex CLI configuration +│ ├── setup_gemini.py # Gemini CLI configuration +│ ├── setup_opencode.py # OpenCode configuration +│ ├── setup_hermes.py # Hermes Agent configuration +│ ├── setup_databricks.py # Databricks CLI configuration +│ ├── setup_mlflow.py # MLflow tracing auto-configuration +│ └── setup_proxy.py # Content-filter proxy startup +├── scripts/ # Shell scripts +│ ├── install_micro.sh # Micro editor installer +│ ├── install_gh.sh # GitHub CLI installer (OS/arch-aware) +│ └── install_databricks_cli.sh # Databricks CLI upgrade script +├── static/ +│ ├── index.html # Terminal UI (xterm.js + split panes + WebSocket) +│ ├── favicon.svg # App favicon +│ ├── poll-worker.js # Web Worker for HTTP polling fallback +│ └── lib/ +│ ├── xterm.js # xterm.js terminal emulator +│ └── socket.io.min.js # Vendored Socket.IO client +├── .claude/ +│ └── skills/ # 39 pre-installed skills +├── .github/ +│ └── workflows/ +│ ├── dependency-audit.yml # Weekly CVE audit + lockfile drift check +│ └── update-lockfile.yml # Auto-regenerate requirements.lock on push +├── tools/ +│ └── coda-bridge.py # Stdio-to-HTTP MCP bridge (OAuth token injection) +└── docs/ + ├── deployment.md # Full Databricks Apps deployment guide + ├── mcp-client-setup.md # MCP client setup guide (bridge config) + ├── mcp-v2-background-execution.md # MCP server design doc + ├── prd/ # Product requirement documents + └── plans/ # Design documentation +``` + +
+ +--- + +## Technologies -MIT +Flask · Flask-SocketIO · Socket.IO · uvicorn · MCP (Streamable HTTP) · xterm.js · Python PTY · uv · Databricks SDK · Databricks AI Gateway · MLflow diff --git a/agents/build-feature.md b/agents/build-feature.md new file mode 100644 index 00000000..9a357776 --- /dev/null +++ b/agents/build-feature.md @@ -0,0 +1,66 @@ +--- +name: build-feature +description: End-to-end feature builder. Chains prd-writer → test-generator → implementer → web-devloop-tester in TDD flow. Use when asked to "build", "create", or "implement" a feature from scratch. Orchestrates the full cycle including bug fix loops and visual UI testing. +tools: Read, Write, Edit, Glob, Grep, Bash, Agent, AskUserQuestion, WebSearch, WebFetch +--- + +# Role +You are a tech lead orchestrating a TDD feature build. You coordinate four phases and handle failures. + +# Phase 1: PRD +1. Invoke yourself as a prd-writer: interview the user, write `docs/prd/.md` +2. Do NOT proceed until the user approves the PRD +3. PRD must have status `READY_FOR_IMPLEMENTATION` before moving on + +# Phase 2: Tests (TDD) +1. Read the approved PRD +2. Extract all Acceptance Criteria (AC-*) +3. Scan the codebase for test framework and conventions +4. Write failing tests that define the contract — one or more tests per AC +5. Run the tests to confirm they fail for the right reasons (missing implementation, not broken tests) +6. Update PRD status to `TESTS_WRITTEN` + +# Phase 3: Implementation +1. Read the PRD and all test files +2. Run the test suite to see current failures +3. Create an implementation plan, present it to the user for approval +4. Implement code to make tests pass, working through one group at a time +5. After each group, run tests to verify progress + +# Bug Fix Loop +If tests fail after implementation: + +1. Read the failure output carefully +2. Identify whether the bug is in the **test** or the **implementation** +3. If test is wrong (doesn't match PRD): fix the test +4. If implementation is wrong: fix the code +5. Re-run tests +6. **Max 3 fix loops** — if still failing after 3 rounds, stop and report to the user with: + - Which tests are failing + - The error messages + - Your hypothesis on the root cause + - Ask the user how to proceed + +# Phase 4: Visual Testing (Web Apps Only) +If the feature has a UI component (React, Vue, Streamlit, Dash, etc.): + +1. Spawn a `web-devloop-tester` agent (subagent_type: `fe-specialized-agents:web-devloop-tester`) +2. Tell it to: start the dev server, navigate to the relevant page, take screenshots, check console for errors, and test key interactions from the AC-* list +3. Review the tester's report: + - **All clear** → proceed to Completion + - **Issues found** → create fix tasks for the implementer, then re-test +4. **Max 3 visual fix loops** — if issues persist after 3 rounds, stop and report to the user with screenshots and logs + +Skip this phase for: +- CLI tools, libraries, backend-only APIs +- Projects with no dev server or browser UI + +# Completion +When all tests pass and visual testing is complete (or skipped): +1. Run the full test suite one final time +2. Update PRD status to `COMPLETE` +3. Summarize what was built: + - Files created/modified + - Test coverage (AC-* mapping) + - Visual test results (screenshots, if applicable) + - Any open items or manual testing needed diff --git a/agents/implementer.md b/agents/implementer.md new file mode 100644 index 00000000..2f6d0881 --- /dev/null +++ b/agents/implementer.md @@ -0,0 +1,59 @@ +--- +name: implementer +description: Reads a PRD and makes all tests pass. Implements code to satisfy the test suite written by test-generator. Use after test-generator has written failing tests. Runs tests iteratively until green. +tools: Read, Write, Edit, Glob, Grep, Bash, Agent +--- + +# Role +You are a senior software engineer who makes failing tests pass. You implement exactly what's needed to satisfy the test suite and PRD requirements — nothing more. + +# Startup +1. Read the PRD file specified (or scan `docs/prd/` for files with status `TESTS_WRITTEN`) +2. Read ALL test files listed in the PRD status section +3. Run the test suite to see the current failures +4. Read any files referenced in the PRD's Technical Notes or Dependencies sections +5. Scan the codebase with Glob/Grep to understand existing patterns and architecture + +# Planning Phase +Before writing any code, create a numbered implementation plan: + +1. List every failing test and what it expects +2. Group tests by module/component +3. Identify files to create or modify +4. Note the order of operations (what depends on what) +5. Flag any Open Questions from the PRD that block implementation + +Present the plan and wait for approval before proceeding. + +# Implementation Phase — Red-Green Loop +For each group of related tests: + +1. **Read the tests** — understand exactly what they expect +2. **Write minimal code** to make those tests pass +3. **Run tests** — check if they pass +4. **If tests fail** — read the error, fix the code, run again +5. **Repeat** until that group is green +6. **Commit** — use `git commit -m "message"` directly +7. Move to the next group + +Rules: +- **Read before writing** — always read existing files before modifying +- **Follow existing patterns** — match the codebase's style and conventions +- **Keep it simple** — don't over-engineer; make the tests pass +- **Max 3 fix attempts per test** — if a test won't pass after 3 tries, flag it and move on + +# Final Validation +After all implementation: + +1. Run the FULL test suite +2. If any tests still fail, attempt fixes (max 2 more rounds) +3. If tests still fail after retries, document the failures + +# Handoff +When complete, update the PRD status: + +> **Status: IMPLEMENTED** +> Commits: +> Test results: +> If all green: **Status: COMPLETE** +> If failures remain: **Status: NEEDS_REVIEW** with failure details diff --git a/agents/prd-writer.md b/agents/prd-writer.md new file mode 100644 index 00000000..baf4aa01 --- /dev/null +++ b/agents/prd-writer.md @@ -0,0 +1,81 @@ +--- +name: prd-writer +description: Use when creating a new feature, epic, or project requirement. Interviews the user with clarifying questions, then generates a structured PRD markdown file ready for implementation. Use proactively when asked about new features or "what should we build". +tools: Read, Write, Glob, Grep, AskUserQuestion, WebSearch, WebFetch +--- + +# Role +You are a senior product manager who turns raw ideas into implementation-ready PRDs through Socratic questioning. + +# Discovery Phase +Before writing anything, interview the user with numbered clarifying questions (max 6 per round) covering: + +1. **Problem** — What problem are we solving and who does it affect? +2. **Success metrics** — How will we know this worked? What are the acceptance criteria? +3. **Scope boundaries** — What is explicitly OUT of scope? +4. **Technical constraints** — Any dependencies, existing systems, or limitations? +5. **Priority & timeline** — How urgent is this? What's the desired delivery window? +6. **Edge cases** — What happens when things go wrong? Error states? + +Use AskUserQuestion to present these as structured questions. WAIT for answers before proceeding. Ask follow-up rounds if answers are vague or incomplete. + +# Research Phase +If the feature involves external APIs, libraries, or patterns: +- Use WebSearch to find current best practices +- Use Glob/Grep to scan the existing codebase for related patterns, data models, and conventions +- Reference any existing PRDs in `docs/prd/` to follow established format and naming + +# Output Format +Write the PRD to `docs/prd/.md` using this structure: + +```markdown +# PRD: +**Author:** | **Date:** | **Status:** DRAFT + +## Problem Statement + + +## User Personas & Stories +- As a [user type], I want [action] so that [outcome] +- ... + +## Functional Requirements +1. FR-1: +2. FR-2: ... + +## Non-Functional Requirements +1. NFR-1: +2. NFR-2: ... + +## Acceptance Criteria +1. AC-1: Given [context], when [action], then [result] +2. AC-2: ... + +## Out of Scope +- + +## Dependencies +- + +## Open Questions +- + +## Technical Notes +- +- +``` + +# Iteration +After writing the first draft: +1. Present a summary to the user +2. Ask if any sections need refinement +3. Update the PRD based on feedback +4. Repeat until the user approves + +# Handoff +Once approved, update the status line and append: + +> **Status: READY_FOR_IMPLEMENTATION** +> Next steps (TDD flow): +> 1. test-generator writes failing tests from the Acceptance Criteria +> 2. implementer makes all tests pass diff --git a/agents/test-generator.md b/agents/test-generator.md new file mode 100644 index 00000000..f2f2d21b --- /dev/null +++ b/agents/test-generator.md @@ -0,0 +1,56 @@ +--- +name: test-generator +description: Reads a PRD's acceptance criteria and generates comprehensive tests BEFORE implementation (TDD). Maps each AC-* criterion to one or more test cases. Tests should initially fail — that's expected. Use after prd-writer and BEFORE the implementer. +tools: Read, Write, Edit, Glob, Grep, Bash +--- + +# Role +You are a senior QA engineer who writes tests FIRST (TDD style). You translate acceptance criteria into failing tests that define the contract the implementer must satisfy. + +# Startup +1. Read the PRD file specified by the user (or scan `docs/prd/` for files with status `READY_FOR_IMPLEMENTATION`) +2. Extract all Acceptance Criteria (AC-*) +3. Scan the codebase to understand the test framework, conventions, and existing test patterns +4. If code already exists, read it to understand the interfaces; if not, define the expected interfaces from the PRD + +# Test Strategy +Before writing tests, produce a test matrix: + +| AC | Test Name | Type | Description | +|----|-----------|------|-------------| +| AC-1 | test_... | unit | ... | +| AC-1 | test_... | integration | ... | +| AC-2 | test_... | unit | ... | + +Every AC must have at least one test. Include: +- **Happy path** — the AC scenario works as described +- **Edge cases** — boundary values, empty inputs, max limits +- **Error cases** — what happens when preconditions aren't met + +# Implementation Rules +1. **Match existing test patterns** — use the same framework, fixtures, helpers, and directory structure already in the project +2. **Name tests after ACs** — include the AC number in the test name or docstring (e.g., `test_ac1_user_can_login`) +3. **Keep tests independent** — no test should depend on another test's state +4. **Test behavior, not implementation** — tests should survive refactoring +5. **Define interfaces** — if the code doesn't exist yet, write tests against the interfaces/function signatures described in the PRD. Import from expected module paths. + +# Test Frameworks +Detect and use whatever the project already has: +- **Python**: pytest (use `uv run pytest`) +- **JS/TS**: jest, vitest, or mocha (use `npx`) +- **Other**: follow existing patterns + +# TDD Validation +After writing all tests: +1. Run the test suite — **tests SHOULD fail** (no implementation yet) +2. Confirm tests fail for the RIGHT reasons (import errors or missing functions, not syntax errors in tests) +3. List the expected failure count + +# Handoff +When complete, update the PRD status: + +> **Status: TESTS_WRITTEN** +> Test files: +> Failing tests: (expected — no implementation yet) +> AC coverage: +> Next: Ask the implementer to read `docs/prd/.md` and make all tests pass diff --git a/app.py b/app.py index 07069c25..9a39beb3 100644 --- a/app.py +++ b/app.py @@ -1,3 +1,4 @@ +import asyncio import os import pty import fcntl @@ -9,65 +10,582 @@ import threading import signal import time +import copy import logging +from concurrent.futures import ThreadPoolExecutor, wait from flask import Flask, send_from_directory, request, jsonify, session +from flask_socketio import SocketIO, emit, join_room, leave_room, disconnect +from werkzeug.utils import secure_filename from collections import deque -os.environ.pop("DATABRICKS_CLIENT_ID", None) -os.environ.pop("DATABRICKS_CLIENT_SECRET", None) +import tomllib +import requests + +import app_state +import enterprise_config +from utils import ensure_https, get_gateway_host +from pat_rotator import PATRotator +from telemetry import log_telemetry, set_product_info + +# Sanitize DATABRICKS_TOKEN early — the platform sometimes injects trailing +# newlines / whitespace which causes auth failures. Cleaning it here prevents +# the agent from "fixing" it in the terminal and leaking the raw token. +_raw_token = os.environ.get("DATABRICKS_TOKEN", "") +if _raw_token != _raw_token.strip(): + os.environ["DATABRICKS_TOKEN"] = _raw_token.strip() + +# App version (single source of truth: pyproject.toml) +_pyproject_file = os.path.join(os.path.dirname(__file__), 'pyproject.toml') +try: + with open(_pyproject_file, 'rb') as _f: + APP_VERSION = tomllib.load(_f)['project']['version'] +except Exception: + APP_VERSION = '0.0.0' # Session timeout configuration -SESSION_TIMEOUT_SECONDS = 60 # No poll for 60s = dead session -CLEANUP_INTERVAL_SECONDS = 30 # How often to check for stale sessions +SESSION_TIMEOUT_SECONDS = 86400 # No poll for 24 hours = dead session +CLEANUP_INTERVAL_SECONDS = 900 # Check for stale sessions every 15 min GRACEFUL_SHUTDOWN_WAIT = 3 # Seconds to wait after SIGHUP before SIGKILL +MAX_CONCURRENT_SESSIONS = int(os.environ.get("MAX_CONCURRENT_SESSIONS", "5")) +TRANSCRIPT_CAP_BYTES = 10 * 1024 * 1024 # 10 MB soft cap per transcript # Logging setup logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) +# PAT auto-rotation — initialized after sessions dict is defined (see below) + app = Flask(__name__, static_folder='static', static_url_path='/static') app.secret_key = os.urandom(24) +app.config['MAX_CONTENT_LENGTH'] = 32 * 1024 * 1024 # 32 MB — aligned with Claude Code's 30 MB file limit + +# WebSocket support via Flask-SocketIO (simple-websocket transport, threading mode) +# Used for local dev (python app.py). Under uvicorn/ASGI, the AsyncServer in +# mcp_asgi.py intercepts /socket.io/ before WSGIMiddleware, so these handlers +# are only active in WSGI mode. +socketio = SocketIO(app, async_mode='threading', cors_allowed_origins=[], logger=False, engineio_logger=False) + +# ── ASGI WebSocket support (python-socketio AsyncServer) ───────────── +# Set by mcp_asgi.py at startup. Background threads use _emit_from_thread() +# which routes to the async server (ASGI) or Flask-SocketIO (WSGI) automatically. +_async_sio = None +_event_loop = None + + +def set_async_sio(sio_instance, loop): + """Called by mcp_asgi.py to wire up the ASGI Socket.IO server.""" + global _async_sio, _event_loop + _async_sio = sio_instance + _event_loop = loop + + +def _emit_from_thread(event, data, room=None): + """Thread-safe emit for background threads (PTY reader, cleanup, SIGTERM). -# Store sessions: {session_id: {"master_fd": fd, "pid": pid, "output_buffer": deque}} + Routes to AsyncServer (ASGI mode) or Flask-SocketIO (WSGI mode) automatically. + """ + if _async_sio and _event_loop and _event_loop.is_running(): + try: + asyncio.run_coroutine_threadsafe( + _async_sio.emit(event, data, room=room), + _event_loop, + ) + except Exception: + pass + else: + # WSGI mode (local dev) — use Flask-SocketIO directly + try: + socketio.emit(event, data, room=room) + except Exception: + pass + + +# Store sessions: {session_id: {"master_fd": fd, "pid": pid, "output_buffer": deque, "lock": Lock, ...}} +# sessions_lock guards dict-level ops (add/remove/iterate); each session["lock"] guards per-session state sessions = {} sessions_lock = threading.Lock() +# PAT auto-rotation (short-lived tokens, background refresh) +# Only rotates while active sessions exist — stops when all sessions are reaped +pat_rotator = PATRotator( + session_count_fn=lambda: len(sessions), +) + +# SIGTERM graceful shutdown: notify clients before gunicorn stops the worker +shutting_down = False + +_start_time = time.time() + +def handle_sigterm(signum, frame): + """Notify clients that app is shutting down, then let gunicorn handle the rest.""" + global shutting_down + # Ignore SIGTERMs in the first 10s — likely stale signals from a prior process kill + if time.time() - _start_time < 10: + logger.info("SIGTERM received during startup — ignoring (likely stale signal)") + return + shutting_down = True + logger.info("SIGTERM received — setting shutting_down flag for clients") + # Notify WS clients immediately (HTTP poll clients will see shutting_down on next poll) + _emit_from_thread('shutting_down', {}) + +# NOTE: Do not register SIGTERM handler at module level. +# It is installed in initialize_app() for gunicorn only. +# For local dev (__main__), we keep SIG_DFL so the process just exits. + +# Setup state tracking +setup_lock = threading.Lock() +setup_state = { + "status": "pending", + "started_at": None, + "completed_at": None, + "error": None, + "steps": [ + {"id": "git", "label": "Configuring git identity", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "micro", "label": "Installing micro editor", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "gh", "label": "Installing GitHub CLI", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "dbcli", "label": "Upgrading Databricks CLI", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "proxy", "label": "Starting content-filter proxy", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "claude", "label": "Configuring Claude CLI", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "codex", "label": "Configuring Codex CLI", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "opencode", "label": "Configuring OpenCode CLI", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "gemini", "label": "Configuring Gemini CLI", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "hermes", "label": "Configuring Hermes Agent", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "databricks", "label": "Setting up Databricks CLI", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + {"id": "mlflow", "label": "Enabling MLflow tracing", "status": "pending", "started_at": None, "completed_at": None, "error": None}, + ] +} + + +def _update_step(step_id, **kwargs): + with setup_lock: + for step in setup_state["steps"]: + if step["id"] == step_id: + step.update(kwargs) + break + + +def _get_setup_state_snapshot(): + with setup_lock: + return copy.deepcopy(setup_state) + + # Single-user security: only the token owner can access the terminal app_owner = None -def get_token_owner(): - """Get the owner email from DATABRICKS_TOKEN at startup.""" +def _run_step(step_id, command): + _update_step(step_id, status="running", started_at=time.time()) + try: + env = os.environ.copy() + if not env.get("HOME") or env["HOME"] == "/": + env["HOME"] = "/app/python/source_code" + home = env.get("HOME", "/app/python/source_code") + # Ensure uv and other tools in ~/.local/bin are on PATH + local_bin = os.path.join(home, ".local", "bin") + if local_bin not in env.get("PATH", ""): + env["PATH"] = f"{local_bin}:{env.get('PATH', '')}" + env.pop("DATABRICKS_CLIENT_ID", None) + env.pop("DATABRICKS_CLIENT_SECRET", None) + + # Ensure setup scripts can still import from repo root (e.g. `from utils import ...`) + app_dir = os.path.dirname(os.path.abspath(__file__)) + existing_pp = env.get("PYTHONPATH", "") + env["PYTHONPATH"] = f"{app_dir}:{existing_pp}" if existing_pp else app_dir + + result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=300) + if result.returncode == 0: + _update_step(step_id, status="complete", completed_at=time.time()) + else: + err = result.stderr.strip() or result.stdout.strip() or "Unknown error" + _update_step(step_id, status="error", completed_at=time.time(), error=err[:500]) + except subprocess.TimeoutExpired: + _update_step(step_id, status="error", completed_at=time.time(), error="Timed out after 300s") + except Exception as e: + _update_step(step_id, status="error", completed_at=time.time(), error=str(e)) + + +def _build_terminal_shell_env(base_env: dict) -> dict: + """Build the env dict for a user terminal PTY. + + Starts from ``base_env`` (typically ``os.environ``) and strips the + credentials and CLI-state vars that should never reach a user shell: + + - ``CLAUDECODE`` / ``CLAUDE_CODE_SESSION`` — would mark the terminal as + a nested-Claude session. + - ``DATABRICKS_TOKEN`` / ``DATABRICKS_HOST`` — forces CLIs to read + ``~/.databrickscfg`` per-request so they pick up rotated PATs without + an env-snapshot rewrite. + - ``GEMINI_API_KEY`` — same pattern, read from config file instead. + - ``NPM_TOKEN`` / ``UV_DEFAULT_INDEX`` / ``UV_INDEX_*_PASSWORD`` / + ``UV_INDEX_*_USERNAME`` / ``npm_config_//host/:_authToken`` — + deployer-level credentials from app.yaml that must not be readable + via ``env`` inside the user terminal. The user's npm/uv operations + still work because ``~/.npmrc`` (written by + ``enterprise_config.bootstrap``) holds the registry config — they + just can't see the bearer token in plaintext. (F-01) + """ + shell_env = base_env.copy() + shell_env["TERM"] = "xterm-256color" + + # Always-strip fixed names + for key in ( + "CLAUDECODE", "CLAUDE_CODE_SESSION", + "DATABRICKS_TOKEN", "DATABRICKS_HOST", + "GEMINI_API_KEY", + "NPM_TOKEN", "UV_DEFAULT_INDEX", + ): + shell_env.pop(key, None) + + # Pattern-strip operator-named registry credentials + for key in list(shell_env.keys()): + if ( + key.startswith("npm_config_//") # derived registry-auth tokens + or ( + key.startswith("UV_INDEX_") + and (key.endswith("_PASSWORD") or key.endswith("_USERNAME")) + ) + ): + shell_env.pop(key, None) + + return shell_env + + +def _setup_git_config(): + """Configure git identity and hooks by writing files directly (no subprocess).""" + home = os.environ.get("HOME", "/app/python/source_code") + if not home or home == "/": + home = "/app/python/source_code" + + # Get user identity from Databricks token + user_email = None + display_name = None try: from databricks.sdk import WorkspaceClient - host = os.environ.get("DATABRICKS_HOST") + db_host = ensure_https(os.environ.get("DATABRICKS_HOST", "")) + db_token = os.environ.get("DATABRICKS_TOKEN") + if db_host and db_token: + w = WorkspaceClient(host=db_host, token=db_token, auth_type="pat") + set_product_info(w) + me = w.current_user.me() + user_email = me.user_name + display_name = me.display_name or user_email.split("@")[0] + except Exception as e: + logger.warning(f"Could not get user identity from token: {e}") + + # Write ~/.gitconfig directly (more reliable than subprocess git config) + gitconfig_path = os.path.join(home, ".gitconfig") + hooks_dir = os.path.join(home, ".githooks") + os.makedirs(hooks_dir, exist_ok=True) + + lines = [] + if user_email and display_name: + lines.append("[user]") + lines.append(f"\temail = {user_email}") + lines.append(f"\tname = {display_name}") + lines.append("[core]") + lines.append(f"\thooksPath = {hooks_dir}") + + with open(gitconfig_path, "w") as f: + f.write("\n".join(lines) + "\n") + logger.info(f"Git config written to {gitconfig_path}") + + # Write post-commit hook for workspace sync (works from any CLI: Claude, Gemini, OpenCode, etc.) + # Only syncs repos inside ~/projects/ — skips the app source and any other repos + post_commit = os.path.join(hooks_dir, "post-commit") + with open(post_commit, "w") as f: + f.write('#!/bin/bash\n') + f.write('# Auto-sync to Databricks Workspace on commit (works from any CLI)\n') + f.write('SYNC_LOG="$HOME/.sync.log"\n') + f.write('\n') + f.write('# Resolve git repo root (handles commits from subdirectories)\n') + f.write('REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)"\n') + f.write('if [ -z "$REPO_ROOT" ]; then\n') + f.write(' echo "[post-commit] $(date +%H:%M:%S) SKIP: not inside a git repo" >> "$SYNC_LOG"\n') + f.write(' exit 0\n') + f.write('fi\n') + f.write('\n') + f.write('# Only sync repos inside ~/projects/\n') + f.write('PROJECTS_DIR="$HOME/projects"\n') + f.write('case "$REPO_ROOT" in\n') + f.write(' "$PROJECTS_DIR"/*)\n') + f.write(' ;; # allowed - continue\n') + f.write(' *)\n') + f.write(' echo "[post-commit] $(date +%H:%M:%S) SKIP: $REPO_ROOT is outside $PROJECTS_DIR" >> "$SYNC_LOG"\n') + f.write(' exit 0\n') + f.write(' ;;\n') + f.write('esac\n') + f.write('\n') + f.write('echo "[post-commit] $(date +%H:%M:%S) syncing $REPO_ROOT" >> "$SYNC_LOG"\n') + f.write('\n') + f.write('# Use uv run so sync script gets the correct Python + deps\n') + f.write('APP_DIR="/app/python/source_code"\n') + f.write('SYNC_SCRIPT="$APP_DIR/sync_to_workspace.py"\n') + f.write('\n') + f.write('if [ -f "$SYNC_SCRIPT" ]; then\n') + f.write(' nohup uv run --project "$APP_DIR" python "$SYNC_SCRIPT" "$REPO_ROOT" >> "$SYNC_LOG" 2>&1 & disown\n') + f.write('else\n') + f.write(' echo "[post-commit] $(date +%H:%M:%S) SKIP: sync script not found" >> "$SYNC_LOG"\n') + f.write('fi\n') + os.chmod(post_commit, 0o755) + logger.info(f"Post-commit hook written to {post_commit}") + + # Reinit app source git to remove template origin (Databricks Apps only) + _reinit_app_git() + + +def _reinit_app_git(): + """On Databricks Apps, reinit git to remove template origin remote.""" + app_dir = os.path.dirname(os.path.abspath(__file__)) + if app_dir != "/app/python/source_code": + return # Local dev — leave git intact + + git_dir = os.path.join(app_dir, ".git") + if not os.path.isdir(git_dir): + return # Already clean + + import shutil + shutil.rmtree(git_dir) + subprocess.run(["git", "init"], cwd=app_dir, capture_output=True) + subprocess.run(["git", "add", "."], cwd=app_dir, capture_output=True) + subprocess.run( + ["git", "commit", "-m", "Initial commit from coding-agents template"], + cwd=app_dir, capture_output=True, + ) + logger.info("Reinitialized app source git (template origin removed)") + + +def _configure_all_cli_auth(token): + """Configure auth for ALL coding-agent CLIs after a PAT is provided. + + Called from /api/configure-pat when a user supplies a PAT interactively. + Handles: Claude CLI (inline), Databricks CLI (via pat_rotator), and + Codex/OpenCode/Gemini CLIs (by re-running their setup scripts with token in env). + """ + import json + + from utils import resolve_and_cache_gateway + resolve_and_cache_gateway() + + home = os.environ.get("HOME", "/app/python/source_code") + if not home or home == "/": + home = "/app/python/source_code" + + # 1. Configure Claude CLI (~/.claude/settings.json) + claude_dir = os.path.join(home, ".claude") + os.makedirs(claude_dir, exist_ok=True) + + gateway_host = get_gateway_host() + databricks_host = ensure_https(os.environ.get("DATABRICKS_HOST", "").rstrip("/")) + + if gateway_host: + anthropic_base_url = f"{gateway_host}/anthropic" + else: + anthropic_base_url = f"{databricks_host}/serving-endpoints/anthropic" + + # Read-merge-write to preserve env vars from other setup scripts (e.g. setup_mlflow.py) + settings_path = os.path.join(claude_dir, "settings.json") + try: + with open(settings_path) as f: + settings = json.load(f) + except (FileNotFoundError, json.JSONDecodeError): + settings = {} + + settings.setdefault("env", {}) + settings["env"]["ANTHROPIC_MODEL"] = os.environ.get("ANTHROPIC_MODEL", "databricks-claude-opus-4-7") + settings["env"]["ANTHROPIC_BASE_URL"] = anthropic_base_url + settings["env"]["ANTHROPIC_AUTH_TOKEN"] = token + settings["env"]["ANTHROPIC_DEFAULT_OPUS_MODEL"] = "databricks-claude-opus-4-7" + settings["env"]["ANTHROPIC_DEFAULT_SONNET_MODEL"] = "databricks-claude-sonnet-4-6" + settings["env"]["ANTHROPIC_DEFAULT_HAIKU_MODEL"] = "databricks-claude-haiku-4-5" + settings["env"]["ANTHROPIC_CUSTOM_HEADERS"] = "x-databricks-use-coding-agent-mode: true" + settings["env"]["CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS"] = "1" + + with open(settings_path, "w") as f: + json.dump(settings, f, indent=2) + + logger.info(f"Claude CLI auth configured: {settings_path}") + + # 2. Configure Databricks CLI (~/.databrickscfg) — already called by + # configure_pat() via pat_rotator, but explicit for clarity + pat_rotator._write_databrickscfg(token) + logger.info("Databricks CLI auth configured: ~/.databrickscfg") + + # 3. Re-run Codex, OpenCode, Gemini setup scripts with token in env + # They are idempotent: detect CLI already installed, just write config files + app_dir = os.path.dirname(os.path.abspath(__file__)) + existing_pp = os.environ.get("PYTHONPATH", "") + env = { + **os.environ, + "DATABRICKS_TOKEN": token, + "PYTHONPATH": f"{app_dir}:{existing_pp}" if existing_pp else app_dir, + } + for script in ["setup/setup_codex.py", "setup/setup_opencode.py", "setup/setup_gemini.py", "setup/setup_hermes.py"]: + try: + result = subprocess.run( + ["uv", "run", "python", script], + env=env, capture_output=True, text=True, timeout=60 + ) + if result.returncode == 0: + logger.info(f"CLI config updated: {script}") + else: + logger.warning(f"CLI config failed: {script}: {result.stderr[:200]}") + except Exception as e: + logger.warning(f"CLI config error: {script}: {e}") + + +def run_setup(): + with setup_lock: + setup_state["status"] = "running" + setup_state["started_at"] = time.time() + + # Apply enterprise (proxy/registry) config before any subprocess runs: + # writes ~/.npmrc, pushes derived env vars (npm_config_registry, CURL_CA_BUNDLE, + # etc.) into os.environ so every child process inherits them, and logs a + # banner of the effective config. No-op when no enterprise env vars are set. + enterprise_config.bootstrap() + + # Probe AI Gateway once; result is cached in _GATEWAY_RESOLVED for subprocesses + from utils import resolve_and_cache_gateway + resolve_and_cache_gateway() + + # --- Sequential prerequisites (git identity + editor) --- + # Git config — done directly in Python, not as a subprocess + _update_step("git", status="running", started_at=time.time()) + try: + _setup_git_config() + _update_step("git", status="complete", completed_at=time.time()) + except Exception as e: + _update_step("git", status="error", completed_at=time.time(), error=str(e)) + + _run_step("micro", ["bash", "-c", + "mkdir -p ~/.local/bin && bash scripts/install_micro.sh && mv micro ~/.local/bin/ 2>/dev/null || true"]) + + _run_step("gh", ["bash", "scripts/install_gh.sh"]) + + # --- Upgrade Databricks CLI (runtime image ships an older version) --- + _run_step("dbcli", ["bash", "scripts/install_databricks_cli.sh"]) + + # --- Content-filter proxy (must be running before OpenCode starts) --- + # Sanitizes requests/responses between OpenCode and Databricks + # (see OpenCode #5028, docs/plans/2026-03-11-litellm-empty-content-blocks-design.md) + _run_step("proxy", ["uv", "run", "python", "setup/setup_proxy.py"]) + + # --- Parallel agent setup (all independent of each other) --- + parallel_steps = [ + ("claude", ["uv", "run", "python", "setup/setup_claude.py"]), + ("codex", ["uv", "run", "python", "setup/setup_codex.py"]), + ("opencode", ["uv", "run", "python", "setup/setup_opencode.py"]), + ("gemini", ["uv", "run", "python", "setup/setup_gemini.py"]), + ("hermes", ["uv", "run", "python", "setup/setup_hermes.py"]), + ("databricks", ["uv", "run", "python", "setup/setup_databricks.py"]), + ] + + with ThreadPoolExecutor(max_workers=len(parallel_steps)) as executor: + futures = [ + executor.submit(_run_step, step_id, command) + for step_id, command in parallel_steps + ] + wait(futures) + + # --- MLflow setup runs AFTER claude setup to avoid settings.json race --- + # setup_mlflow.py merges env vars into ~/.claude/settings.json which + # setup_claude.py also writes; running sequentially prevents clobbering. + _run_step("mlflow", ["uv", "run", "python", "setup/setup_mlflow.py"]) + + # Sync latest token into all CLI configs — covers the race where PAT + # rotation happened while a setup script was still installing (the + # rotation's update_cli_tokens() call silently skips missing config files). + current_token = os.environ.get("DATABRICKS_TOKEN", "") + if current_token: + try: + from cli_auth import update_cli_tokens + update_cli_tokens(current_token) + logger.info("Post-setup token sync: all CLI configs updated with current token") + except Exception as e: + logger.warning(f"Post-setup token sync failed: {e}") + + with setup_lock: + any_error = any(s["status"] == "error" for s in setup_state["steps"]) + setup_state["status"] = "error" if any_error else "complete" + setup_state["completed_at"] = time.time() + + +def get_token_owner(): + """Get the owner email. Priority: Apps API (app.creator) > PAT (current_user.me). + + Uses the auto-provisioned SP to call the Apps API — no PAT needed for + owner resolution. Falls back to PAT-based lookup for backward compat. + """ + from databricks.sdk import WorkspaceClient + + # 1. Try Apps API via SP credentials (no PAT needed) + app_name = os.environ.get("DATABRICKS_APP_NAME") + if app_name: + try: + w = WorkspaceClient() # auto-detects SP credentials + set_product_info(w) + app = w.apps.get(name=app_name) + owner = (app.creator or "").lower() + logger.info(f"Owner resolved from app.creator: {owner}") + return owner + except Exception as e: + logger.warning(f"Could not resolve owner via Apps API: {e}") + + # 2. Fallback: PAT-based resolution + try: + host = ensure_https(os.environ.get("DATABRICKS_HOST", "")) token = os.environ.get("DATABRICKS_TOKEN") if not host or not token: return None w = WorkspaceClient(host=host, token=token, auth_type="pat") - return w.current_user.me().user_name + set_product_info(w) + username = w.current_user.me().user_name + return username.lower() if username else username except Exception as e: logger.warning(f"Could not determine token owner: {e}") return None def get_request_user(): - """Extract user email from Databricks Apps request headers.""" - return request.headers.get("X-Forwarded-Email") or \ - request.headers.get("X-Forwarded-User") or \ - request.headers.get("X-Databricks-User-Email") + """Extract user email from Databricks Apps request headers. + + Returns lowercase email to ensure case-insensitive matching against app_owner. + """ + email = ( + request.headers.get("X-Forwarded-Email") + or request.headers.get("X-Forwarded-User") + or request.headers.get("X-Databricks-User-Email") + ) + return email.lower() if email else email + + +def _is_databricks_apps(): + """Detect if we're running on Databricks Apps (not local dev).""" + return os.environ.get("DATABRICKS_APP_PORT") or os.path.isdir("/app/python/source_code") def check_authorization(): - """Check if the current user is authorized to access the app.""" - # If owner not set (local dev or SDK unavailable), allow access + """Check if the current user is authorized to access the app. + + Fails CLOSED on Databricks Apps: if we can't determine the owner, + deny all access rather than allowing unauthenticated terminal access. + Fails open only for local development. + Fixes: https://github.com/datasciencemonkey/coding-agents-databricks-apps/issues/57 + """ + # Fail closed on Databricks Apps if owner couldn't be resolved if not app_owner: - return True, None + if _is_databricks_apps(): + logger.error("SECURITY: app_owner not resolved — denying all access (fail-closed)") + return False, "unknown" + return True, None # Local dev only current_user = get_request_user() # If no user identity in request (local dev), allow access if not current_user: + if _is_databricks_apps(): + logger.warning("No user identity in request on Databricks Apps — denying access") + return False, "unknown" return True, None # Check if current user is the owner @@ -78,25 +596,392 @@ def check_authorization(): return True, None +def _check_ws_authorization(): + """Check authorization for WebSocket connections — mirrors HTTP check_authorization(). + + Fails CLOSED on Databricks Apps: if app_owner is unresolved or no user identity + in headers, deny WebSocket access. Matches the HTTP handler's behavior exactly. + """ + if not app_owner: + if _is_databricks_apps(): + logger.error("SECURITY: app_owner not resolved — denying WebSocket (fail-closed)") + return False + return True # Local dev only + + # Socket.IO passes HTTP headers from the initial handshake via request context + raw_user = ( + request.headers.get("X-Forwarded-Email") + or request.headers.get("X-Forwarded-User") + or request.headers.get("X-Databricks-User-Email") + ) + current_user = raw_user.lower() if raw_user else raw_user + + if not current_user: + if _is_databricks_apps(): + logger.warning("No user identity in WebSocket request on Databricks Apps — denying") + return False + return True # Local dev only + + if current_user != app_owner: + logger.warning(f"WebSocket unauthorized: {current_user} (owner: {app_owner})") + return False + return True + + +def _check_ws_authorization_from_environ(environ): + """Check authorization from WSGI environ dict (for ASGI WebSocket via python-socketio). + + Same logic as _check_ws_authorization() but reads headers from the environ + dict instead of Flask's request context. WSGI environ stores HTTP headers as + HTTP_X_FORWARDED_EMAIL (uppercase, underscores, HTTP_ prefix). + """ + if not app_owner: + if _is_databricks_apps(): + logger.error("SECURITY: app_owner not resolved — denying WebSocket (fail-closed)") + return False + return True # Local dev only + + raw_user = ( + environ.get("HTTP_X_FORWARDED_EMAIL") + or environ.get("HTTP_X_FORWARDED_USER") + or environ.get("HTTP_X_DATABRICKS_USER_EMAIL") + ) + current_user = raw_user.lower() if raw_user else raw_user + + if not current_user: + if _is_databricks_apps(): + logger.warning("No user identity in WebSocket request on Databricks Apps — denying") + return False + return True # Local dev only + + if current_user != app_owner: + logger.warning(f"WebSocket unauthorized: {current_user} (owner: {app_owner})") + return False + return True + + +def register_sio_handlers(sio): + """Register Socket.IO event handlers on an AsyncServer for ASGI mode. + + Called by mcp_asgi.py. The handlers mirror the Flask-SocketIO handlers below + but use python-socketio's async API (explicit sid, enter_room/leave_room, + async def, ConnectionRefusedError for auth denial). + """ + + @sio.on('connect') + async def handle_connect(sid, environ, auth): + # Capture event loop on first connection for _emit_from_thread() + set_async_sio(sio, asyncio.get_running_loop()) + + # Diagnostic: log transport and header presence for debugging proxy behavior + transport = environ.get('QUERY_STRING', '') + has_email = bool(environ.get('HTTP_X_FORWARDED_EMAIL')) + has_user = bool(environ.get('HTTP_X_FORWARDED_USER')) + logger.info(f"WS connect: sid={sid}, qs={transport}, " + f"has_email={has_email}, has_user={has_user}") + + if not _check_ws_authorization_from_environ(environ): + raise ConnectionRefusedError('unauthorized') + logger.info("WebSocket client connected (ASGI)") + + @sio.on('join_session') + async def handle_join_session(sid, data): + session_id = data.get('session_id') + if not session_id: + return {'status': 'error', 'message': 'session_id required'} + sess = _get_session(session_id) + if not sess: + return {'status': 'error', 'message': 'Session not found'} + with sess["lock"]: + sess["last_poll_time"] = time.time() + sess["output_buffer"].clear() + await sio.enter_room(sid, session_id) + logger.info(f"WebSocket client joined session room {session_id}") + return {'status': 'ok'} + + @sio.on('leave_session') + async def handle_leave_session(sid, data): + session_id = data.get('session_id') + if session_id: + await sio.leave_room(sid, session_id) + logger.info(f"WebSocket client left session room {session_id}") + + @sio.on('terminal_input') + async def handle_terminal_input(sid, data): + session_id = data.get('session_id') + input_data = data.get('input', '') + sess = _get_session(session_id) + if not sess: + return + with sess["lock"]: + sess["last_poll_time"] = time.time() + fd = sess["master_fd"] + try: + os.write(fd, input_data.encode()) + except OSError as e: + logger.warning(f"WebSocket input write error for {session_id}: {e}") + + @sio.on('terminal_resize') + async def handle_terminal_resize(sid, data): + session_id = data.get('session_id') + cols = data.get('cols', 80) + rows = data.get('rows', 24) + sess = _get_session(session_id) + if not sess: + return + with sess["lock"]: + sess["last_poll_time"] = time.time() + fd = sess["master_fd"] + try: + winsize = struct.pack("HHHH", rows, cols, 0, 0) + fcntl.ioctl(fd, termios.TIOCSWINSZ, winsize) + except OSError as e: + logger.warning(f"WebSocket resize error for {session_id}: {e}") + + @sio.on('heartbeat') + async def handle_heartbeat(sid, data): + session_ids = data.get('session_ids', []) + now = time.time() + for s_id in session_ids: + sess = _get_session(s_id) + if sess: + with sess["lock"]: + sess["last_poll_time"] = now + + @sio.on('disconnect') + async def handle_disconnect(sid): + logger.info("WebSocket client disconnected (ASGI)") + + +# ── WebSocket Event Handlers (Flask-SocketIO — WSGI/local dev only) ────── + +@socketio.on('connect') +def handle_ws_connect(): + """Authenticate WebSocket connections (AC-3).""" + if not _check_ws_authorization(): + disconnect() + return False + logger.info("WebSocket client connected") + + +@socketio.on('join_session') +def handle_join_session(data): + """Client joins a session room to receive output (AC-4).""" + session_id = data.get('session_id') + if not session_id: + return {'status': 'error', 'message': 'session_id required'} + + session = _get_session(session_id) + if not session: + return {'status': 'error', 'message': 'Session not found'} + + with session["lock"]: + session["last_poll_time"] = time.time() + session["output_buffer"].clear() # Prevent duplicate output on WS↔HTTP switch + + join_room(session_id) + logger.info(f"WebSocket client joined session room {session_id}") + return {'status': 'ok'} + + +@socketio.on('leave_session') +def handle_leave_session(data): + """Client leaves a session room (AC-5).""" + session_id = data.get('session_id') + if session_id: + leave_room(session_id) + logger.info(f"WebSocket client left session room {session_id}") + + +@socketio.on('terminal_input') +def handle_terminal_input(data): + """Receive keystrokes from client, write to PTY (AC-6).""" + session_id = data.get('session_id') + input_data = data.get('input', '') + + session = _get_session(session_id) + if not session: + return + + with session["lock"]: + session["last_poll_time"] = time.time() + fd = session["master_fd"] + + try: + os.write(fd, input_data.encode()) + except OSError as e: + logger.warning(f"WebSocket input write error for {session_id}: {e}") + + +@socketio.on('terminal_resize') +def handle_terminal_resize(data): + """Receive resize events from client (AC-7).""" + session_id = data.get('session_id') + cols = data.get('cols', 80) + rows = data.get('rows', 24) + + session = _get_session(session_id) + if not session: + return + + with session["lock"]: + session["last_poll_time"] = time.time() + fd = session["master_fd"] + + try: + winsize = struct.pack("HHHH", rows, cols, 0, 0) + fcntl.ioctl(fd, termios.TIOCSWINSZ, winsize) + except OSError as e: + logger.warning(f"WebSocket resize error for {session_id}: {e}") + + +@socketio.on('heartbeat') +def handle_ws_heartbeat(data): + """Periodic keepalive from WS client — prevents idle session reaping (AC-17).""" + session_ids = data.get('session_ids', []) + now = time.time() + for sid in session_ids: + session = _get_session(sid) + if session: + with session["lock"]: + session["last_poll_time"] = now + + +@socketio.on('disconnect') +def handle_ws_disconnect(): + """Log WebSocket disconnections. Do NOT auto-close PTY — client may reconnect.""" + logger.info("WebSocket client disconnected") + + +def _get_session(session_id): + """Get a session dict reference under the global lock. Returns None if not found.""" + with sessions_lock: + return sessions.get(session_id) + + +def _tee_transcript_chunk(session, output: bytes, cap: int = TRANSCRIPT_CAP_BYTES) -> None: + """Append PTY output to the transcript file. Single-writer (read_pty_output). + + All file-handle access is under ``session["lock"]`` so we never race the + Timer-driven close path in ``terminate_session``. The ``ValueError`` catch + is belt-and-suspenders for the tiny window where the handle is closed + between the ``is not None`` check and the actual ``write`` call (the lock + prevents this, but be defensive). + """ + with session["lock"]: + fh = session.get("transcript_fh") + written = session.get("transcript_bytes", 0) + if fh is None: + return + remaining = cap - written + if remaining <= 0: + return + chunk = output[:remaining] + try: + fh.write(chunk) + fh.flush() + session["transcript_bytes"] = written + len(chunk) + if len(chunk) < len(output): + fh.write(b"\n[transcript truncated at %d bytes]\n" % cap) + fh.flush() + fh.close() + session["transcript_fh"] = None + except (OSError, ValueError) as exc: + logger.warning("transcript write failed: %s", exc) + try: + fh.close() + except Exception: + pass + session["transcript_fh"] = None + + def read_pty_output(session_id, fd): - """Background thread to read PTY output into buffer.""" + """Background thread to read PTY output into buffer and push via WebSocket.""" + session = _get_session(session_id) + if not session: + return + pid = session["pid"] + session_lock = session["lock"] + while True: with sessions_lock: if session_id not in sessions: break try: - if select.select([fd], [], [], 0.1)[0]: - output = os.read(fd, 4096).decode(errors="replace") - with sessions_lock: - if session_id in sessions: - sessions[session_id]["output_buffer"].append(output) + readable, _, errors = select.select([fd], [], [fd], 0.05) + if readable or errors: + output = os.read(fd, 65536) + if not output: + # EOF — process exited + break + decoded = output.decode(errors="replace") + with session_lock: + # Buffer for HTTP polling fallback (AC-15) + session["output_buffer"].append(decoded) + session["last_poll_time"] = time.time() # Keep session alive during WS output + # Push via WebSocket to the session room (AC-8) + _emit_from_thread('terminal_output', + {'session_id': session_id, 'output': decoded}, + room=session_id) + # Tee to transcript file if enabled for this session + _tee_transcript_chunk(session, output) + else: + # select timed out — check if process is still alive + try: + pid_result, _ = os.waitpid(pid, os.WNOHANG) + if pid_result != 0: + # Process exited + break + except ChildProcessError: + # Process already reaped + break except OSError: break + # Process exited or fd closed — notify WebSocket clients (AC-9) + _emit_from_thread('session_exited', {'session_id': session_id}, room=session_id) + + logger.info(f"Session {session_id} process exited") + + # Clean up immediately — no zombie sessions in the picker + if session: + terminate_session(session_id, session["pid"], session["master_fd"]) + def terminate_session(session_id, pid, master_fd): - """Gracefully terminate a session: SIGHUP -> wait -> SIGKILL -> cleanup.""" + """Gracefully terminate a session: SIGHUP -> wait -> SIGKILL -> cleanup. + + Idempotent. Both the explicit close path (``mcp_close_pty_session``) and the + read-thread exit path (``read_pty_output``) call this for the same session. + We atomically *claim* the session by popping it from ``sessions`` — only the + caller that wins the pop kills the process and closes ``master_fd``. This + guarantees ``os.close()`` runs exactly once: a second close could land on a + since-reused fd (e.g. an asyncio event loop's self-pipe allocated by a later + test) and corrupt unrelated I/O, surfacing as intermittent EBADF. + """ + # Atomically claim the session. If it's already gone, the other teardown + # path handled it — bail out WITHOUT touching the (possibly reused) fd. + with sessions_lock: + sess = sessions.pop(session_id, None) + if sess is None: + return + logger.info(f"Terminating stale session {session_id} (pid={pid})") + + # Notify WebSocket clients that the session is closed + _emit_from_thread('session_closed', {'session_id': session_id}, room=session_id) + + # Close transcript handle (if any) under per-session lock; swap-then-close + # outside the lock to avoid blocking on slow filesystems. + with sess["lock"]: + transcript_fh = sess.get("transcript_fh") + sess["transcript_fh"] = None + if transcript_fh is not None: + try: + transcript_fh.close() + except Exception: + pass + try: os.kill(pid, signal.SIGHUP) time.sleep(GRACEFUL_SHUTDOWN_WAIT) @@ -113,8 +998,74 @@ def terminate_session(session_id, pid, master_fd): except OSError: pass # Process or fd already gone - with sessions_lock: - sessions.pop(session_id, None) + # Clean up the project dir if coda_interactive created one. + # Done here (not in mcp_close_pty_session) so BOTH the graceful close + # path AND the idle reaper (which calls terminate_session directly) hit + # this cleanup. Safe for HTTP-created sessions too — they never planted + # a dir at this path, so os.path.isdir short-circuits. + import shutil + project_dir = os.path.join( + os.path.expanduser("~/.coda/projects"), + session_id, + ) + if os.path.isdir(project_dir): + try: + shutil.rmtree(project_dir) + except OSError as e: + logger.warning("Failed to clean up project dir %s: %s", project_dir, e) + + +def _get_session_process(pid): + """Return the name of the foreground child process for *pid*. + + Uses ``pgrep -P`` to find children (works on both macOS and Linux), + then ``ps -o comm=`` to resolve the process name. + + Returns: + str: process name, or ``"unknown"`` on any error / dead PID. + """ + if not isinstance(pid, int) or pid <= 0: + return "unknown" + + try: + # Step 1 — find child PIDs via pgrep (cross-platform) + child_result = subprocess.run( + ["pgrep", "-P", str(pid)], + capture_output=True, + text=True, + timeout=5, + ) + + if child_result.returncode == 0 and child_result.stdout.strip(): + child_pids = child_result.stdout.strip().splitlines() + last_child_pid = child_pids[-1].strip() + + # Step 2 — resolve child name + name_result = subprocess.run( + ["ps", "-o", "comm=", "-p", last_child_pid], + capture_output=True, + text=True, + timeout=5, + ) + if name_result.returncode == 0 and name_result.stdout.strip(): + name = name_result.stdout.strip().splitlines()[0].strip() + # ps may return the full path; take basename + return os.path.basename(name) + + # Step 3 — no children: fall back to the process itself + self_result = subprocess.run( + ["ps", "-o", "comm=", "-p", str(pid)], + capture_output=True, + text=True, + timeout=5, + ) + if self_result.returncode == 0 and self_result.stdout.strip(): + name = self_result.stdout.strip().splitlines()[0].strip() + return os.path.basename(name) + + return "unknown" + except Exception: + return "unknown" def cleanup_stale_sessions(): @@ -124,12 +1075,18 @@ def cleanup_stale_sessions(): now = time.time() stale_sessions = [] + warning_threshold = SESSION_TIMEOUT_SECONDS * 0.8 - # Find stale sessions with sessions_lock: - for session_id, session in sessions.items(): - if now - session["last_poll_time"] > SESSION_TIMEOUT_SECONDS: + session_snapshot = list(sessions.items()) + + for session_id, session in session_snapshot: + with session["lock"]: + idle = now - session["last_poll_time"] + if idle > SESSION_TIMEOUT_SECONDS: stale_sessions.append((session_id, session["pid"], session["master_fd"])) + elif idle > warning_threshold: + session["timeout_warning"] = True if stale_sessions: logger.info(f"Found {len(stale_sessions)} stale session(s) to clean up") @@ -142,8 +1099,8 @@ def cleanup_stale_sessions(): @app.before_request def authorize_request(): """Check authorization before processing any request.""" - # Skip auth for health check - if request.path == "/health": + # Skip auth for health check, setup status, and Socket.IO (has own auth via connect event) + if request.path in ("/health", "/api/setup-status", "/api/pat-status", "/api/configure-pat", "/api/app-state") or request.path.startswith("/socket.io") or request.path.startswith("/mcp"): return None authorized, user = check_authorization() @@ -156,29 +1113,276 @@ def authorize_request(): return None +@app.after_request +def set_security_headers(response): + # MCP endpoint handles its own CORS/headers — skip security headers + # that might interfere (CSP connect-src, X-Frame-Options, etc.) + if request.path.startswith("/mcp"): + return response + response.headers["X-Content-Type-Options"] = "nosniff" + response.headers["X-Frame-Options"] = "DENY" + response.headers["X-XSS-Protection"] = "1; mode=block" + response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin" + # CSP: restrict scripts to self + inline (needed for embedded + + + +