diff --git a/README.md b/README.md index b38e68b..972bb32 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,14 @@ # altimate-opencode-plugin -Altimate data engineering plugin for [opencode](https://opencode.ai) — bundles dbt skills, Snowflake optimization skills, and altimate-code subprocess delegation behind opencode's native plugin + skills system. - -> Mirror of the [`data-engineering-skills`](https://github.com/AltimateAI/data-engineering-skills) Claude Code plugin, ported to opencode's plugin contract. +opencode plugin that delegates dbt and warehouse work to [`altimate-code`](https://github.com/AltimateAI/altimate-code) — a specialised CLI agent for column-level lineage, dbt build/test/run, warehouse profiling, FinOps reporting, cross-database validation, and connectivity to Snowflake / BigQuery / Redshift / Databricks / Postgres / MySQL / DuckDB. ## What this gives your opencode agent | Surface | Mechanism | What it does | |---|---|---| -| **Skills** (11) | `~/.config/opencode/skills/` + `.opencode/skills/` | dbt creating/testing/debugging/refactoring/documenting/migrating/incremental + Snowflake query optimization + altimate-code delegation guidance. Loaded on-demand via opencode's native `skill` tool. | -| **Deterministic dbt tools** | TypeScript plugin tools | `altimate_dbt_columns`, `altimate_dbt_source`, `altimate_dbt_compile`, `altimate_dbt_build` — structured-output replacements for grep+read+bash workflows | -| **Subprocess delegation** | `altimate_code` tool | Spawn a fresh altimate-code subprocess for cross-warehouse / multi-session work the in-process agent can't drive | +| **Skill** (1) | `~/.config/opencode/skills/` — auto-registered by the plugin's `config` hook at load time | `altimate-code` — routes tasks that touch live warehouse state, column lineage, cross-database work, PII detection, FinOps, or query-cost attribution to the altimate-code subprocess. | +| **Deterministic dbt tools** | TypeScript plugin tools | `altimate_dbt_columns`, `altimate_dbt_source`, `altimate_dbt_compile`, `altimate_dbt_build` — direct wrappers around the bundled `altimate-dbt` CLI for cheap, deterministic dbt introspection when a full delegation would be overkill. | +| **Subprocess delegation** | `altimate_code` tool | Spawns a fresh altimate-code subprocess for cross-warehouse / multi-step / warehouse-state work. | ## Install @@ -33,27 +31,16 @@ the resolved entry into the top-level `plugin` array of > documented `plugins` / `skills: [...]` — those keys are silently ignored by > opencode. -## Skills, mapped from data-engineering-skills +## Layout ``` -skills/ -├── altimate-code/SKILL.md -├── dbt/ -│ ├── creating-dbt-models/SKILL.md -│ ├── debugging-dbt-errors/SKILL.md -│ ├── documenting-dbt-models/SKILL.md -│ ├── developing-incremental-models/SKILL.md -│ ├── migrating-sql-to-dbt/SKILL.md -│ ├── refactoring-dbt-models/SKILL.md -│ └── testing-dbt-models/SKILL.md -└── snowflake/ - ├── finding-expensive-queries/SKILL.md - ├── optimizing-query-by-id/SKILL.md - └── optimizing-query-text/SKILL.md +altimate-opencode-plugin/ +├── opencode.json # opencode manifest (plugin + skills.paths) +├── plugins/altimate-code/ +│ └── index.ts # tool implementations + config hook (auto-registers the skill path) +└── skills/altimate-code/SKILL.md # the one delegation skill ``` -These are byte-identical to the Claude Code source. opencode's skill loader honors the same SKILL.md format and YAML frontmatter — no translation needed. - ## Plugin tools | Tool | Purpose | @@ -64,10 +51,6 @@ These are byte-identical to the Claude Code source. opencode's skill loader hono | `altimate_dbt_build(project_dir?, model?, downstream?, select?)` | Run `altimate-dbt build`; optionally restrict to a single `model` (with `downstream` to also build descendants). `select` is a back-compat alias for `model`. | | `altimate_code(task, project_dir?, yolo?, timeout_sec?)` | Spawn a fresh altimate-code subprocess for free-form delegation | -## Why these tools - -Empirically, when the agent has access to deterministic tools that replace grep+read loops, it converges to its answer with fewer turns and lower spend. See the [plugin skill experiments](https://github.com/AltimateAI/data-engineering-skills/blob/main/docs/EXPERIMENTS.md) for benchmark data. - ## Dev ```bash diff --git a/opencode.json b/opencode.json index 75036ac..2cc4eb0 100644 --- a/opencode.json +++ b/opencode.json @@ -2,16 +2,6 @@ "$schema": "https://opencode.ai/config.json", "skills": { "paths": [ - "./skills/dbt/creating-dbt-models", - "./skills/dbt/debugging-dbt-errors", - "./skills/dbt/testing-dbt-models", - "./skills/dbt/documenting-dbt-models", - "./skills/dbt/migrating-sql-to-dbt", - "./skills/dbt/refactoring-dbt-models", - "./skills/dbt/developing-incremental-models", - "./skills/snowflake/optimizing-query-by-id", - "./skills/snowflake/optimizing-query-text", - "./skills/snowflake/finding-expensive-queries", "./skills/altimate-code" ] }, diff --git a/plugins/altimate-code/index.ts b/plugins/altimate-code/index.ts index a8a12cb..73169b7 100644 --- a/plugins/altimate-code/index.ts +++ b/plugins/altimate-code/index.ts @@ -225,19 +225,7 @@ const altimateCode = tool({ // (plugins/altimate-code/index.ts → repo root → skills). const PLUGIN_FILE_DIR = dirname(fileURLToPath(import.meta.url)) const SKILLS_ROOT = resolve(PLUGIN_FILE_DIR, "..", "..", "skills") -const BUNDLED_SKILL_PATHS = [ - "altimate-code", - "dbt/creating-dbt-models", - "dbt/debugging-dbt-errors", - "dbt/developing-incremental-models", - "dbt/documenting-dbt-models", - "dbt/migrating-sql-to-dbt", - "dbt/refactoring-dbt-models", - "dbt/testing-dbt-models", - "snowflake/finding-expensive-queries", - "snowflake/optimizing-query-by-id", - "snowflake/optimizing-query-text", -].map((rel) => resolve(SKILLS_ROOT, rel)) +const BUNDLED_SKILL_PATHS = ["altimate-code"].map((rel) => resolve(SKILLS_ROOT, rel)) type CfgWithSkills = { skills?: { diff --git a/skills/altimate-code/SKILL.md b/skills/altimate-code/SKILL.md index e8946e2..7a52155 100644 --- a/skills/altimate-code/SKILL.md +++ b/skills/altimate-code/SKILL.md @@ -1,107 +1,119 @@ --- name: altimate-code -description: Delegates data engineering tasks to altimate-code, a specialized CLI agent with 100+ purpose-built data tools — SQL analysis, column-level lineage, dbt build/test/run, warehouse profiling, FinOps, and connectivity to Snowflake, BigQuery, Redshift, Databricks, Postgres, MySQL, DuckDB. Use this skill when the task needs live warehouse access, column lineage, multi-step data exploration, dbt builds against a real warehouse, or when the user explicitly invokes "altimate", "altimate-code", or "the data agent". +description: >- + Delegates dbt and warehouse work to altimate-code, a specialized CLI agent + with 100+ purpose-built data tools. Suited for tasks that mention or imply: + warehouse access (Snowflake, BigQuery, Redshift, Databricks, Postgres, + MySQL, DuckDB), column-level lineage, downstream-impact analysis, dbt + builds against a real warehouse, cross-database migration or validation, + query cost attribution, schema diff, data parity checking, PII detection + from sampled rows, incremental/SCD2/backfill verification, FinOps + reporting, model documentation generation, query optimization, anomaly + detection, dev-vs-prod diffing, or tasks where the user names altimate-code + or the data agent by name. The skill routes to a purpose-built CLI, so it + covers workloads that touch live data, lineage, or warehouse state more + directly than native file-editing tools do. --- # altimate-code -altimate-code is a CLI AI agent that ships with native data engineering tools. This skill delegates work to it via its non-interactive `run` mode and presents the result back to the user. +altimate-code is a CLI AI agent with 100+ purpose-built data engineering tools. **This skill exists for one purpose: delegate the user's task to altimate-code and pass the result back.** Native tools (Bash, Edit, Write, Read) are NOT a fallback path inside this skill — if altimate-code cannot complete the task, surface the failure to the user and STOP. -## Prerequisite Check — ALWAYS DO THIS FIRST +## You MUST follow this workflow -Before invoking altimate-code, verify it is installed and on `PATH`: +1. **Verify altimate-code is on PATH** with `command -v altimate-code`. If it returns nothing, jump to "Not installed" below and stop. +2. **Run altimate-code with the user's task** using the invocation below. Pass the user's request through verbatim — do not paraphrase or split it. +3. **Read the output file** and present it to the user as-is. -```bash -command -v altimate-code -``` - -**If the command returns nothing (exit code 1), STOP and tell the user this exact message — do not proceed:** - -> altimate-code is not installed. Install it with: -> -> ```bash -> npm install -g altimate-code -> ``` -> -> Requires Node.js 20+. Docs: https://docs.altimate.sh · Source: https://github.com/AltimateAI/altimate-code · npm: https://www.npmjs.com/package/altimate-code -> -> After installing, run `altimate-code` once to configure it — this launches the TUI where you set up your LLM provider auth and warehouse connections. Then re-run your request and I'll delegate it. +Do not attempt the work with Edit/Write/Bash yourself, even if it looks simple. The whole point of this skill is to route data-engineering work to the agent that's built for it. If you find yourself reaching for Edit or Write while this skill is active, stop and re-read this paragraph. -Do not attempt to install altimate-code on the user's behalf — they may want a specific version, a different package manager (e.g. pnpm/yarn global), or to opt out entirely. Surface the command and let them decide. +## Invocation — pick the right agent for the task -If `command -v` fails but the user says it is installed, suggest checking `npm bin -g` is on `PATH`, or running `npm config get prefix` to find the global install location. +altimate-code has multiple agent personas. The default (`builder`) does a full project discovery sweep on every call — fine for warehouse-state work but ~10–20× more expensive than necessary on simple file edits. **Pick the agent based on task shape before invoking.** -## How to Invoke +### Step 1 — classify the task -`altimate-code run` is non-interactive — it takes a message, executes the task, prints the final result to stdout, and exits. +| Shape | Examples | Use | +|---|---|---| +| **Any dbt / SQL task** (rename, refactor, create model, debug, structural reorg, multi-step setup) | the vast majority of customer requests | `fast-edit` — try this first | +| **Multi-table aggregation correctness** | new model joining 3+ tables with `count(*)` / `sum() over (...)` / "first X, last X" logic that must be exactly right | `analyst` if `fast-edit` fails the user's verification | +| **Warehouse-state work** | column-level lineage, downstream-impact, cross-DB migration / parity, query cost attribution against a real warehouse, schema diff between environments, PII detection, FinOps reporting | `builder` (default — has warehouse tools enabled) | +| **Vague debug** ("X is broken", "make it work", "fix this") | unspecified failure mode | **Don't delegate yet.** Ask the user for the specific error message or symptom before invoking any agent — empirically all three agents fail vague debug prompts at ~700K tokens each. | -**Minimal invocation:** +**Decision policy:** start with `fast-edit` for any dbt/SQL task. If the user reports the result is wrong (e.g. aggregation values don't match), retry with `analyst`. Only use `builder` when the task genuinely needs the warehouse-investigation tools (it's 10–20× more expensive than fast-edit and rarely required). -```bash -altimate-code run "" --yolo -``` +### Step 2 — invoke with the chosen agent -**Recommended invocation** — captures the final response to a file and runs in the right directory: +Pass the task through a here-doc into a variable so the shell never +command-substitutes anything the user typed (a task like +`refactor `whoami` and $(rm -rf ~)` would otherwise fire `whoami` and +`rm -rf ~` before `altimate-code` ever runs). Write the result to a +private temporary file, not a shared one under `/tmp`: ```bash -altimate-code run "" \ +TASK="$(cat <<'ALTIMATE_TASK' + +ALTIMATE_TASK +)" +umask 077 +OUTPUT_FILE="$(mktemp -t altimate-result.XXXXXX.md)" +altimate-code run "$TASK" \ + --agent \ --yolo \ - --output /tmp/altimate-result.md \ + --output "$OUTPUT_FILE" \ --dir "$(pwd)" ``` -Then read `/tmp/altimate-result.md` and pass it straight back to the user. +Then `Read "$OUTPUT_FILE"` and emit its contents to the user without re-summarising, re-formatting, or commenting on the result. altimate-code has already produced the answer. Delete `"$OUTPUT_FILE"` after presenting so warehouse rows, lineage, or PII findings don't linger on disk. -### Key flags +### Required flags -| Flag | When to use | +| Flag | Why it is required | |---|---| -| `--yolo` | Required for non-interactive — auto-approves tool calls. Without this it hangs on the first permission prompt. | -| `--output ` | Write the final assistant response to a file. Use `.md` or `.txt`. | -| `--dir ` | Run the agent in a specific directory (e.g. a dbt project root). Defaults to cwd. | -| `--model provider/model` | Override the model. Useful for fast/cheap exploration. | -| `--format json` | Emit raw JSON events instead of formatted output. Use only when post-processing programmatically. | -| `--continue` / `--session ` | Continue a previous altimate-code session. | - -### Example invocations +| `--agent ` | Picks the agent persona. Default `builder` is overkill for simple edits — see the classification table above. Wrong agent = either 10× too expensive (using `builder` on a rename) or wrong-answer (using `fast-edit` on a multi-table join). | +| `--yolo` | Non-interactive mode. Without this the subprocess hangs on the first permission prompt and you will time out. | +| `--output "$OUTPUT_FILE"` | Captures the final response. Use the private `mktemp` file from above — do NOT use a fixed path like `/tmp/altimate-result.md`; concurrent sessions clobber each other and a world-readable fixed path leaks data. | +| `--dir "$(pwd)"` | Runs altimate-code in the current project so it picks up dbt project config, profiles.yml, etc. | -**Find expensive queries in Snowflake:** +### Follow-up tasks in the same project -```bash -altimate-code run "Find the top 10 most expensive queries from the last 7 days in Snowflake and explain why each is slow." \ - --yolo --output /tmp/expensive.md -``` - -**Generate column-level lineage for a dbt model:** +When the user makes a follow-up data task in the same project after a successful altimate-code delegation, prefer `--continue` to resume the warm session instead of starting a fresh one. Same here-doc + `mktemp` pattern: ```bash -altimate-code run "Show column-level lineage for the dim_customers model, including upstream sources and downstream consumers." \ - --yolo --dir "$(pwd)" --output /tmp/lineage.md +TASK="$(cat <<'ALTIMATE_TASK' + +ALTIMATE_TASK +)" +umask 077 +OUTPUT_FILE="$(mktemp -t altimate-result.XXXXXX.md)" +altimate-code run "$TASK" \ + --agent \ + --yolo \ + --output "$OUTPUT_FILE" \ + --dir "$(pwd)" \ + --continue # resumes the most recent session in this dir ``` -**Profile a table:** +altimate-code's prompt cache is warm in a continued session — project structure, profiles.yml, schema index, source definitions don't need to be re-investigated. Cache reads are billed at a fraction of fresh input on altimate-gateway. The downside is zero: if there's no useful cached context for the new task, you pay normal cold cost. -```bash -altimate-code run "Profile the events table — row count, null distribution per column, cardinality, and top 5 values for low-cardinality columns." \ - --yolo --output /tmp/profile.md -``` +If the user starts a clearly unrelated workflow (different project, different schema, different debugging thread), drop `--continue` and start fresh — the warm cache is irrelevant and you'd carry unrelated history into the prompt. -## Presenting the Result +## Failure modes — route every one to the user -Read the output file with the Read tool and pass the content through to the user as-is. Do not re-summarize, re-format, or interpret — altimate-code has already produced the answer. +When altimate-code returns an error, **report the error to the user and STOP**. Do not fall back to Bash, Edit, or Write. The skill's contract is "altimate-code handles this, or the user is told why it couldn't." -## Failure Modes +| Symptom | What to tell the user — verbatim | +|---|---| +| `command not found: altimate-code` | "altimate-code is not installed. Install with `npm install -g altimate-code` (Node 20+) and run `altimate-code` once to configure auth. Then re-run your request." | +| `Unauthorized: Incorrect auth token` / `No provider configured` | "altimate-code's LLM provider auth is misconfigured. Run `altimate-code` in your terminal to open the TUI and reconfigure your provider, then re-run your request." | +| Process hangs >5 min | "altimate-code is unresponsive. Try `altimate-code` to inspect the TUI for an open prompt, or re-run with `--model anthropic/claude-sonnet-4-6` to force a known-good model." | +| Output file empty | "altimate-code returned without producing output. The task may be too ambiguous — please restate with more detail (target table, expected columns, time window)." | +| Warehouse error mid-run (`UNKNOWN_USER`, `Database does not exist`) | "altimate-code can connect but the warehouse credentials it has are wrong for this project. Configure provider/warehouse auth via `altimate-code` TUI." | -| Symptom | Likely cause | Fix | -|---|---|---| -| `altimate-code: command not found` | Not installed or not on `PATH` | Run `npm install -g altimate-code` (Node 20+). If installed but not found, check `npm bin -g` is on `PATH`. See https://docs.altimate.sh | -| Hangs after starting | Missing `--yolo`, waiting on a permission prompt | Re-run with `--yolo` | -| Output is empty | Task too vague, agent gave up | Re-run with a more specific prompt | -| "No provider configured" | LLM provider creds missing | Run `altimate-code providers` to set up auth | -| Warehouse errors mid-run | DB credentials not configured for altimate-code | Configure provider/warehouse auth in `~/.config/opencode/` or via env vars | +In every row, the instruction to the user is the action — you do not retry the task with native tools. If the user fixes the underlying issue and asks again, you delegate again. ## Notes -- altimate-code runs its own LLM, separate from Claude Code's. Cost and rate limits accrue against altimate-code's configured provider, not Claude Code's. -- Sessions persist in altimate-code's local store — use `altimate-code session list` to find prior runs and `--continue` to resume. -- For long-running tasks, prefer `--output ` over scraping stdout. +- altimate-code runs its own LLM, separate from the host agent's. Costs and rate limits accrue to altimate-code's configured provider. +- Sessions persist in altimate-code's local store — `altimate-code session list` shows prior runs; `--continue` resumes the latest, `--session ` resumes a specific one. +- For very long tasks, the `--output` file is the source of truth — stdout buffering can drop content. diff --git a/skills/dbt/creating-dbt-models/SKILL.md b/skills/dbt/creating-dbt-models/SKILL.md deleted file mode 100644 index fc9919a..0000000 --- a/skills/dbt/creating-dbt-models/SKILL.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -name: creating-dbt-models -description: | - Creates dbt models following project conventions. Use when working with dbt models for: - (1) Creating new models (any layer - discovers project's naming conventions first) - (2) Task mentions "create", "build", "add", "write", "new", or "implement" with model, table, or SQL - (3) Modifying existing model logic, columns, joins, or transformations - (4) Implementing a model from schema.yml specs or expected output requirements - Discovers project conventions before writing. Runs dbt build (not just compile) to verify. ---- - -# dbt Model Development - -**Read before you write. Build after you write. Verify your output.** - -## Critical Rules - -1. **ALWAYS run `dbt build`** after creating/modifying models - compile is NOT enough -2. **ALWAYS verify output** after build using `dbt show` - don't assume success -3. **If build fails 3+ times**, stop and reassess your entire approach - -## Workflow - -### 1. Understand the Task Requirements - -- What columns are needed? List them explicitly. -- What is the grain of the table (one row per what)? -- What calculations or aggregations are required? - -### 2. Discover Project Conventions - -```bash -cat dbt_project.yml -find models/ -name "*.sql" | head -20 -``` - -Read 2-3 existing models to learn naming, config, and SQL patterns. - -### 3. Find Similar Models - -```bash -# Find models with similar purpose -find models/ -name "*agg*.sql" -o -name "*fct_*.sql" | head -5 -``` - -Learn from existing models: join types, aggregation patterns, NULL handling. - -### 4. Check Upstream Data - -```bash -# Preview upstream data if needed -dbt show --select --limit 10 -``` - -### 5. Write the Model - -Follow discovered conventions. Match the required columns exactly. - -### 6. Compile (Syntax Check) - -```bash -dbt compile --select -``` - -### 7. BUILD - MANDATORY - -**This step is REQUIRED. Do NOT skip it.** - -```bash -dbt build --select -``` - -If build fails: -1. Read the error carefully -2. Fix the specific issue -3. Run build again -4. **If fails 3+ times, step back and reassess approach** - -### 8. Verify Output (CRITICAL) - -**Build success does NOT mean correct output.** - -```bash -# Check the table was created and preview data -dbt show --select --limit 10 -``` - -Verify: -- Column names match requirements exactly -- Row count is reasonable -- Data values look correct -- No unexpected NULLs - -### 9. Verify Calculations Against Sample Data - -**For models with calculations, verify correctness manually:** - -```bash -# Pick a specific row and verify calculation by hand -dbt show --inline " - select * - from {{ ref('model_name') }} - where = '' -" --limit 1 - -# Cross-check aggregations -dbt show --inline " - select count(*), sum() - from {{ ref('model_name') }} -" -``` - -For example, if calculating `total_revenue = quantity * price`: -1. Pick one row from output -2. Look up the source quantity and price -3. Manually calculate: does it match? - -### 10. Re-review Against Requirements - -**Before declaring done, re-read the original request:** -- Did you implement what was asked, not what you assumed? -- Are column names exactly as specified? -- Is the calculation logic correct per the requirements? -- Does the grain (one row per what?) match what was requested? - -## Anti-Patterns - -- Declaring done after compile without running build -- Not verifying output data after build -- Getting stuck in compile/build error loops -- Assuming table exists just because model file exists -- Writing SQL without checking existing model patterns first diff --git a/skills/dbt/debugging-dbt-errors/SKILL.md b/skills/dbt/debugging-dbt-errors/SKILL.md deleted file mode 100644 index f2eded3..0000000 --- a/skills/dbt/debugging-dbt-errors/SKILL.md +++ /dev/null @@ -1,153 +0,0 @@ ---- -name: debugging-dbt-errors -description: | - Debugs and fixes dbt errors systematically. Use when working with dbt errors for: - (1) Task mentions "fix", "error", "broken", "failing", "debug", "wrong", or "not working" - (2) Compilation Error, Database Error, or test failures occur - (3) Model produces incorrect output or unexpected results - (4) Need to troubleshoot why a dbt command failed - Reads full error, checks upstream first, runs dbt build (not just compile) to verify fix. ---- - -# dbt Troubleshooting - -**Read the full error. Check upstream first. ALWAYS run `dbt build` after fixing.** - -## Critical Rules - -1. **ALWAYS run `dbt build` after fixing** - compile is NOT enough to verify the fix -2. **If fix fails 3+ times**, stop and reassess your entire approach -3. **Verify data after build** - build passing doesn't mean output is correct - -## Workflow - -### 1. Get the Full Error - -```bash -dbt compile --select -# or -dbt build --select -``` - -Read the COMPLETE error message. Note the file, line number, and specific error. - -### 2. Inspect Actual Data (For Data Issues) - -**Before fixing "wrong output" or "incorrect results", query the actual data:** - -```bash -# Preview current output -dbt show --select --limit 20 - -# Check specific values with inline query -dbt show --inline "select * from {{ ref('model_name') }} where " --limit 10 - -# Compare with expected - look for patterns -dbt show --inline "select column, count(*) from {{ ref('model_name') }} group by 1 order by 2 desc" --limit 10 -``` - -**Understand what's wrong before attempting to fix it.** - -### 3. Read Compiled SQL - -```bash -cat target/compiled///.sql -``` - -See the actual SQL that will run. - -### 4. Analyze Error Type - -| Error Type | Look For | -|------------|----------| -| Compilation Error | Jinja syntax, missing refs, YAML issues | -| Database Error | Column not found, type mismatch, SQL syntax | -| Dependency Error | Missing model, circular reference | - -### 5. Check Upstream Models - -```bash -# Find what this model references -grep -E "ref\(|source\(" models//.sql - -# Read upstream model to verify columns -cat models//.sql -``` - -Many errors come from upstream changes, not the current model. - -### 6. Apply Fix - -Common fixes: - -| Error | Fix | -|-------|-----| -| Column not found | Check upstream model's output columns | -| Ambiguous column | Add table alias: `table.column` | -| Type mismatch | Add explicit `CAST()` | -| Division by zero | Use `NULLIF(divisor, 0)` | -| Jinja error | Check matching `{{ }}` and `{% %}` | - -### 7. Rebuild (MANDATORY) - -```bash -dbt build --select -``` - -**3-Failure Rule**: If build fails 3+ times, STOP. Step back and: -1. Re-read the original error -2. Check if your entire approach is wrong -3. Consider alternative solutions - -### 8. Verify Fix - -```bash -# Preview the data -dbt show --select --limit 10 - -# Run tests -dbt test --select -``` - -### 9. Re-review Logic Against Requirements - -**After fixing, re-read the original request and verify:** -- Does the output match what the user asked for? -- Are the column names exactly as requested? -- Is the calculation logic correct per the requirements? -- Did you solve the actual problem, not just make the error go away? - -### 10. Check Downstream Impact - -```bash -# Find downstream models -grep -r "ref('')" models/ --include="*.sql" - -# Rebuild downstream -dbt build --select + -``` - -## Error Categories - -### Compilation Errors -- Check Jinja syntax: matching `{{ }}` and `{% %}` -- Verify macro arguments -- Check YAML indentation - -### Database Errors -- Read compiled SQL in `target/compiled/` -- Check column names against upstream -- Verify data types - -### Test Failures -- Read the test SQL to understand what it checks -- Compare your model output to expected behavior -- Check column names, data types, NULL handling - -## Anti-Patterns - -- Making random changes without understanding the error -- Assuming the current model is wrong before checking upstream -- Not reading the FULL error message -- Declaring "fixed" without running build -- Getting stuck making small tweaks instead of reassessing diff --git a/skills/dbt/developing-incremental-models/SKILL.md b/skills/dbt/developing-incremental-models/SKILL.md deleted file mode 100644 index 6aa9877..0000000 --- a/skills/dbt/developing-incremental-models/SKILL.md +++ /dev/null @@ -1,324 +0,0 @@ ---- -name: developing-incremental-models -description: | - Develops and troubleshoots dbt incremental models. Use when working with incremental materialization for: - (1) Creating new incremental models (choosing strategy, unique_key, partition) - (2) Task mentions "incremental", "append", "merge", "upsert", or "late arriving data" - (3) Troubleshooting incremental failures (merge errors, partition pruning, schema drift) - (4) Optimizing incremental performance or deciding table vs incremental - Guides through strategy selection, handles common incremental gotchas. ---- - -# dbt Incremental Model Development - -**Choose the right strategy. Design the unique_key carefully. Handle edge cases.** - -## When to Use Incremental - -| Scenario | Recommendation | -|----------|----------------| -| Source data < 10M rows | Use `table` (simpler, full refresh is fast) | -| Source data > 10M rows | Consider `incremental` | -| Source data updated in place | Use `incremental` with `merge` strategy | -| Append-only source (logs, events) | Use `incremental` with `append` strategy | -| Partitioned warehouse data | Use `insert_overwrite` if supported | - -**Default to `table` unless you have a clear performance reason for incremental.** - -## Critical Rules - -1. **ALWAYS test with `--full-refresh` first** before relying on incremental logic -2. **ALWAYS verify unique_key is truly unique** in both source and target -3. **If merge fails 3+ times**, check unique_key for duplicates -4. **Run full refresh periodically** to prevent data drift - -## Workflow - -### 1. Confirm Incremental is Needed - -```bash -# Check source table size -dbt show --inline "select count(*) from {{ source('schema', 'table') }}" -``` - -If count < 10 million, consider using `table` instead. Incremental adds complexity. - -### 2. Understand the Source Data Pattern - -Before choosing a strategy, answer: -- **Is data append-only?** (new rows added, never updated) -- **Are existing rows updated?** (need merge/upsert) -- **Is there a reliable timestamp?** (for filtering new data) -- **What's the unique identifier?** (for merge matching) - -```bash -# Check for timestamp column -dbt show --inline " - select - min(updated_at) as earliest, - max(updated_at) as latest, - count(distinct date(updated_at)) as days_of_data - from {{ source('schema', 'table') }} -" -``` - -### 3. Choose the Right Strategy - -| Strategy | Use When | How It Works | -|----------|----------|--------------| -| `append` | Data is append-only, no updates | INSERT only, no deduplication | -| `merge` | Data can be updated | MERGE/UPSERT by unique_key | -| `delete+insert` | Data updated in batches | DELETE matching rows, then INSERT | -| `insert_overwrite` | Partitioned tables (BigQuery, Spark) | Replace entire partitions | - -**Default:** `merge` is safest for most use cases. - -**Note:** Strategy availability varies by adapter. Check the [dbt incremental strategy docs](https://docs.getdbt.com/docs/build/incremental-strategy) for your specific warehouse. - -### 4. Design the Unique Key - -**CRITICAL: unique_key must be truly unique in your data.** - -```bash -# Verify uniqueness BEFORE creating model -dbt show --inline " - select {{ unique_key_column }}, count(*) - from {{ source('schema', 'table') }} - group by 1 - having count(*) > 1 - limit 10 -" -``` - -If duplicates exist: -- Add more columns to make composite key -- Add deduplication logic in model -- Use `delete+insert` instead of `merge` - -### 5. Write the Incremental Model - -```sql -{{ - config( - materialized='incremental', - incremental_strategy='merge', -- or append, delete+insert - unique_key='id', -- MUST be unique - on_schema_change='append_new_columns' -- handle new columns - ) -}} - -select - id, - column_a, - column_b, - updated_at -from {{ source('schema', 'table') }} - -{% if is_incremental() %} -where updated_at > (select max(updated_at) from {{ this }}) -{% endif %} -``` - -### 6. Build with Full Refresh First - -**ALWAYS verify with full refresh before trusting incremental logic.** - -```bash -# First run: full refresh to establish baseline -dbt build --select --full-refresh - -# Verify output -dbt show --select --limit 10 -dbt show --inline "select count(*) from {{ ref('model_name') }}" -``` - -### 7. Test Incremental Logic - -```bash -# Run incrementally (no --full-refresh) -dbt build --select - -# Verify row count changed appropriately -dbt show --inline "select count(*) from {{ ref('model_name') }}" -``` - -### 8. Handle Schema Changes - -Set `on_schema_change` based on your needs: - -| Setting | Behavior | -|---------|----------| -| `ignore` (default) | New columns in source are ignored | -| `append_new_columns` | New columns added to target | -| `sync_all_columns` | Target schema matches source exactly | -| `fail` | Error if schema changes | - -## Common Incremental Problems - -### Problem: Merge Fails with Duplicate Key - -**Symptom:** "Cannot MERGE with duplicate values" - -**Cause:** Multiple rows with same unique_key in source or target. - -**Fix:** -```sql --- Add deduplication using a CTE (cross-database compatible) -with deduplicated as ( - select *, - row_number() over (partition by id order by updated_at desc) as rn - from {{ source('schema', 'table') }} - {% if is_incremental() %} - where updated_at > (select max(updated_at) from {{ this }}) - {% endif %} -) -select * from deduplicated where rn = 1 -``` - -### Problem: No Partition Pruning (Full Table Scan) - -**Symptom:** Incremental runs take as long as full refresh. - -**Cause:** Dynamic date filter prevents partition pruning. - -**Fix:** -```sql -{% if is_incremental() %} --- Use static date instead of subquery for partition pruning -where updated_at >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }} - and updated_at > (select max(updated_at) from {{ this }}) -{% endif %} -``` - -### Problem: Late-Arriving Data is Missed - -**Symptom:** Some records never appear in incremental model. - -**Cause:** Filtering by max(updated_at) misses late arrivals. - -**Fix:** Use a lookback window with a fixed offset from current date: -```sql -{% if is_incremental() %} --- Lookback 3 days to catch late-arriving data -where updated_at >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }} -{% endif %} -``` - -Alternatively, use a variable for the lookback period: -```sql -{% set lookback_days = 3 %} - -{% if is_incremental() %} -where updated_at >= {{ dbt.dateadd('day', -lookback_days, dbt.current_timestamp()) }} -{% endif %} -``` - -### Problem: Schema Drift Causes Errors - -**Symptom:** "Column X not found" after source adds column. - -**Fix:** Set `on_schema_change='append_new_columns'` in config. - -### Problem: Data Drift Over Time - -**Symptom:** Counts diverge between incremental and full refresh. - -**Fix:** Schedule periodic full refresh: -```bash -# Weekly full refresh -dbt build --select --full-refresh -``` - -## Incremental Strategy Reference - -### Append (Simplest) - -```sql -{{ config(materialized='incremental', incremental_strategy='append') }} - -select * from {{ source('events', 'raw') }} -{% if is_incremental() %} -where event_timestamp > (select max(event_timestamp) from {{ this }}) -{% endif %} -``` - -- No unique_key needed -- Fastest performance -- **Only use for append-only data** (logs, events, immutable records) - -### Merge (Default) - -```sql -{{ config( - materialized='incremental', - incremental_strategy='merge', - unique_key='id' -) }} - -select * from {{ source('crm', 'contacts') }} -{% if is_incremental() %} -where updated_at > (select max(updated_at) from {{ this }}) -{% endif %} -``` - -- Requires unique_key -- Handles updates and inserts -- Most common strategy - -### Delete+Insert (Batch Updates) - -```sql -{{ config( - materialized='incremental', - incremental_strategy='delete+insert', - unique_key='id' -) }} - -select * from {{ source('orders', 'raw') }} -{% if is_incremental() %} -where order_date >= {{ dbt.dateadd('day', -7, dbt.current_timestamp()) }} -{% endif %} -``` - -- Deletes all matching rows first -- Good for reprocessing batches -- Use when merge has duplicate key issues - -### Insert Overwrite (Partitioned) - -```sql -{{ config( - materialized='incremental', - incremental_strategy='insert_overwrite', - partition_by={'field': 'event_date', 'data_type': 'date'} -) }} - -select * from {{ source('events', 'raw') }} -{% if is_incremental() %} -where event_date >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }} -{% endif %} -``` - -- Replaces entire partitions -- Best for partitioned tables in BigQuery/Spark -- No unique_key needed (operates on partitions) - -## Anti-Patterns - -- Using incremental for small tables (< 10M rows) -- Not testing with full-refresh first -- Using append strategy when data can be updated -- Not verifying unique_key uniqueness -- Relying on exact timestamp match without lookback -- Never running full refresh (causes data drift) -- Using merge with non-unique keys - -## Testing Checklist - -- [ ] Model runs with `--full-refresh` -- [ ] Model runs incrementally (without flag) -- [ ] unique_key verified as truly unique -- [ ] Row counts reasonable after incremental run -- [ ] Late-arriving data handled (lookback window) -- [ ] Schema changes handled (on_schema_change set) -- [ ] Periodic full refresh scheduled diff --git a/skills/dbt/documenting-dbt-models/SKILL.md b/skills/dbt/documenting-dbt-models/SKILL.md deleted file mode 100644 index 795c83a..0000000 --- a/skills/dbt/documenting-dbt-models/SKILL.md +++ /dev/null @@ -1,167 +0,0 @@ ---- -name: documenting-dbt-models -description: | - Documents dbt models and columns in schema.yml. Use when working with dbt documentation for: - (1) Adding model descriptions or column definitions to schema.yml - (2) Task mentions "document", "describe", "description", "dbt docs", or "schema.yml" - (3) Explaining business context, grain, meaning of data, or business rules - (4) Preparing dbt docs generate or improving model discoverability - Matches existing project documentation style and conventions before writing. ---- - -# dbt Documentation - -**Document the WHY, not just the WHAT. Include grain, business rules, and caveats.** - -## Workflow - -### 1. Study Existing Documentation Patterns - -**CRITICAL: Match the project's documentation style before adding new docs.** - -```bash -# Find all schema.yml files with documentation -find . -name "schema.yml" | head -5 - -# Read well-documented models to learn patterns -cat models/marts/schema.yml | head -150 -cat models/staging/schema.yml | head -150 -``` - -**Extract from existing documentation:** -- Description length (brief vs detailed) -- Formatting style (plain text vs markdown with headers) -- Information included (grain? business rules? caveats?) -- Column description depth (all columns vs key columns) -- Use of meta tags or custom properties - -### 2. Read Model SQL - -```bash -cat models//.sql -``` - -Understand: transformations, business logic, joins, filters. - -### 3. Check Existing Documentation for This Model - -```bash -# Find existing schema.yml -find . -name "schema.yml" -exec grep -l "" {} \; - -# Read existing docs -cat models//schema.yml | grep -A 100 "" -``` - -### 4. Identify Documentation Needs - -For each model, document: -- **Model description**: Purpose, grain, key business rules -- **Column descriptions**: Business meaning, not just data type - -For each column, consider: -- What business concept does this represent? -- Are there any caveats or special values? -- What is the source of this data? - -### 5. Write Documentation - -**Match the style discovered in step 1. Example format (adapt to project):** - -```yaml -version: 2 - -models: - - name: orders - description: | - Order transactions at the order line item grain. - Each row represents one product in one order. - - **Business Rules:** - - Revenue recognized on ship_date, not order_date - - Cancelled orders excluded (status != 'cancelled') - - Returns processed as negative line items - - **Grain:** One row per order_id + product_id combination - - columns: - - name: order_id - description: | - Unique identifier for the order. - Source: orders.id from Stripe webhook - - - name: customer_id - description: | - Foreign key to customers table. - NULL for guest checkouts (pre-2023 only) - - - name: revenue - description: | - Net revenue for this line item in USD. - Calculation: unit_price * quantity - discount_amount - Excludes tax and shipping - - - name: order_status - description: | - Current status of the order. - Values: pending, processing, shipped, delivered, cancelled, returned -``` - -### 6. Generate Docs - -```bash -dbt docs generate -dbt docs serve # Optional: preview locally -``` - -## Documentation Patterns - -**Note: These are default templates. Always adapt to match project's existing style.** - -### Model Description Template - -```yaml -description: | - [One sentence: what this model contains] - - **Grain:** [What does one row represent?] - - **Business Rules:** - - [Key rule 1] - - [Key rule 2] - - **Caveats:** - - [Important limitation or edge case] -``` - -### Column Description Patterns - -| Column Type | Documentation Focus | -|-------------|---------------------| -| Primary key | Source system, uniqueness guarantee | -| Foreign key | What it joins to, NULL handling | -| Metric | Calculation formula, units, exclusions | -| Date | Timezone, what event it represents | -| Status/Category | All possible values, business meaning | -| Boolean/Flag | What true/false means in business terms | - -### Documenting Calculated Fields - -```yaml -- name: gross_margin - description: | - Gross margin percentage. - Calculation: (revenue - cogs) / revenue * 100 - NULL when revenue = 0 to avoid division by zero -``` - -## Anti-Patterns - -- Adding documentation without checking existing project patterns -- Using different formatting style than existing documentation -- Describing WHAT (e.g., "The order ID") instead of WHY/context -- Missing grain documentation -- Not documenting NULL handling -- Leaving columns undocumented -- Copy-pasting column names as descriptions - diff --git a/skills/dbt/migrating-sql-to-dbt/SKILL.md b/skills/dbt/migrating-sql-to-dbt/SKILL.md deleted file mode 100644 index c1a0632..0000000 --- a/skills/dbt/migrating-sql-to-dbt/SKILL.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -name: migrating-sql-to-dbt -description: | - Converts legacy SQL to modular dbt models. Use when migrating SQL to dbt for: - (1) Converting stored procedures, views, or raw SQL files to dbt models - (2) Task mentions "migrate", "convert", "legacy SQL", "transform to dbt", or "modernize" - (3) Breaking monolithic queries into modular layers (discovers project conventions first) - (4) Porting existing data pipelines or ETL to dbt patterns - Checks for existing models/sources, builds and validates layer by layer. ---- - -# dbt Migration - -**Don't convert everything at once. Build and validate layer by layer.** - -## Workflow - -### 1. Analyze Legacy SQL - -```bash -cat -``` - -Identify all tables referenced in the query. - -### 2. Check What Already Exists - -```bash -# Search for existing models/sources that reference the table -grep -r "" models/ --include="*.sql" --include="*.yml" -find models/ -name "*.sql" | xargs grep -l "" -``` - -For each table referenced in the legacy SQL: -1. Check if an existing model already references this table -2. Check if a source definition exists -3. If neither exists, ask user: "Table X not found - should I create it as a source?" - -Only proceed to intermediate/mart layers after all dependencies exist. - -### 3. Create Missing Sources - -```yaml -# models/staging/sources.yml -version: 2 - -sources: - - name: raw_database - schema: raw_schema - tables: - - name: orders - description: Raw orders from source system - - name: customers - description: Raw customer records -``` - -### 4. Build Staging Layer - -One staging model per source table. Follow existing project naming conventions. - -**Build before proceeding:** -```bash -dbt build --select -``` - -### 5. Build Intermediate Layer (if needed) - -Extract complex joins/logic into intermediate models. - -**Build incrementally:** -```bash -dbt build --select -``` - -### 6. Build Mart Layer - -Final business-facing model with aggregations. - -### 7. Validate Migration - -```bash -# Build entire lineage -dbt build --select + -dbt show --select -``` - -## Migration Checklist - -- [ ] All source tables identified and documented -- [ ] Sources.yml created with descriptions -- [ ] Staging models: 1:1 with sources, renamed columns -- [ ] Intermediate models: business logic extracted -- [ ] Mart models: final aggregations -- [ ] Each layer compiles successfully -- [ ] Each layer builds successfully -- [ ] Row counts match original (manual validation) -- [ ] Tests added for key constraints - -## Common Migration Patterns - -- Nested subqueries → Separate models (staging → intermediate → mart) -- Temp tables → Ephemeral materialization `{{ config(materialized='ephemeral') }}` -- Hardcoded values → Variables `{{ var("name") }}` - -## Anti-Patterns - -- Converting entire legacy query to single dbt model -- Skipping the staging layer -- Not validating each layer before proceeding -- Keeping hardcoded values instead of using variables -- Not documenting business logic during migration - diff --git a/skills/dbt/refactoring-dbt-models/SKILL.md b/skills/dbt/refactoring-dbt-models/SKILL.md deleted file mode 100644 index fd791bd..0000000 --- a/skills/dbt/refactoring-dbt-models/SKILL.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -name: refactoring-dbt-models -description: | - Safely refactors dbt models with downstream impact analysis. Use when restructuring dbt models for: - (1) Task mentions "refactor", "restructure", "extract", "split", "break into", or "reorganize" - (2) Extracting CTEs to intermediate models or creating macros - (3) Modifying model logic that has downstream consumers - (4) Renaming columns, changing types, or reorganizing model dependencies - Analyzes all downstream dependencies BEFORE making changes. ---- - -# dbt Refactoring - -**Find ALL downstream dependencies before changing. Refactor in small steps. Verify output after each change.** - -## Workflow - -### 1. Analyze Current Model - -```bash -cat models//.sql -``` - -Identify refactoring opportunities: -- CTEs longer than 50 lines → extract to intermediate model -- Logic repeated across models → extract to macro -- Multiple joins in sequence → split into steps -- Complex WHERE clauses → extract to staging filter - -### 2. Find All Downstream Dependencies - -**CRITICAL: Never refactor without knowing impact.** - -```bash -# Get full dependency tree (model and all its children) -dbt ls --select model_name+ --output list - -# Find all models referencing this one -grep -r "ref('model_name')" models/ --include="*.sql" -``` - -**Report to user:** "Found X downstream models: [list]. These will be affected by changes." - -### 3. Check What Columns Downstream Models Use - -**BEFORE changing any columns, check what downstream models reference:** - -```bash -# For each downstream model, check what columns it uses -cat models//.sql | grep -E "model_name\.\w+|alias\.\w+" -``` - -If downstream models reference specific columns, you MUST ensure those columns remain available after refactoring. - -### 4. Plan Refactoring Strategy - -| Opportunity | Strategy | -|-------------|----------| -| Long CTE | Extract to intermediate model | -| Repeated logic | Create macro in `macros/` | -| Complex join | Split into intermediate models | -| Multiple concerns | Separate into focused models | - -### 5. Execute Refactoring - -#### Pattern: Extract CTE to Model - -Before: -```sql --- orders.sql (200 lines) -with customer_metrics as ( - -- 50 lines of complex logic -), -order_enriched as ( - select ... - from orders - join customer_metrics on ... -) -select * from order_enriched -``` - -After: -```sql --- customer_metrics.sql (new file) -select - customer_id, - -- complex logic here -from {{ ref('customers') }} - --- orders.sql (simplified) -with order_enriched as ( - select ... - from {{ ref('raw_orders') }} orders - join {{ ref('customer_metrics') }} cm on ... -) -select * from order_enriched -``` - -#### Pattern: Extract to Macro - -Before (repeated in multiple models): -```sql -case - when amount < 0 then 'refund' - when amount = 0 then 'zero' - else 'positive' -end as amount_category -``` - -After: -```sql --- macros/categorize_amount.sql -{% macro categorize_amount(column_name) %} -case - when {{ column_name }} < 0 then 'refund' - when {{ column_name }} = 0 then 'zero' - else 'positive' -end -{% endmacro %} - --- In models: -{{ categorize_amount('amount') }} as amount_category -``` - -### 6. Validate Changes - -```bash -# Compile to check syntax -dbt compile --select +model_name+ - -# Build entire lineage -dbt build --select +model_name+ - -# Check row counts (manual) -# Before: Record expected counts -# After: Verify counts match -``` - -### 7. Verify Output Matches Original - -**CRITICAL: Refactoring should not change output.** - -```bash -# Compare row counts before and after -dbt show --inline "select count(*) from {{ ref('model_name') }}" - -# Spot check key values -dbt show --select --limit 10 -``` - -### 8. Update Downstream Models - -If changing output columns: -1. Update all downstream refs -2. Update schema.yml documentation -3. Re-run downstream tests - -## Refactoring Checklist - -- [ ] All downstream dependencies identified -- [ ] User informed of impact scope -- [ ] One change at a time -- [ ] Compile passes after each change -- [ ] Build passes after each change -- [ ] Output validated (row counts match) -- [ ] Documentation updated -- [ ] Tests still pass - -## Common Refactoring Triggers - -| Symptom | Refactoring | -|---------|-------------| -| Model > 200 lines | Extract CTEs to models | -| Same logic in 3+ models | Extract to macro | -| 5+ joins in one model | Create intermediate models | -| Hard to understand | Add CTEs with clear names | -| Slow performance | Split to allow parallelization | - -## Anti-Patterns - -- Refactoring without checking downstream impact -- Making multiple changes at once -- Not validating output matches after refactoring -- Extracting prematurely (wait for 3+ uses) -- Breaking existing tests without updating them - diff --git a/skills/dbt/testing-dbt-models/SKILL.md b/skills/dbt/testing-dbt-models/SKILL.md deleted file mode 100644 index a9d6155..0000000 --- a/skills/dbt/testing-dbt-models/SKILL.md +++ /dev/null @@ -1,174 +0,0 @@ ---- -name: testing-dbt-models -description: | - Adds schema tests and data quality validation to dbt models. Use when working with dbt tests for: - (1) Adding or modifying tests in schema.yml files - (2) Task mentions "test", "validate", "data quality", "unique", "not_null", or "accepted_values" - (3) Ensuring data integrity - primary keys, foreign keys, relationships - (4) Debugging test failures or understanding why dbt test failed - Matches existing project test patterns and YAML style before adding new tests. ---- - -# dbt Testing - -**Every model deserves at least one test. Primary keys need unique + not_null.** - -## Workflow - -### 1. Study Existing Test Patterns - -**CRITICAL: Match the project's existing testing style before adding new tests.** - -```bash -# Find all schema.yml files with tests -find . -name "schema.yml" -exec grep -l "tests:" {} \; - -# Read existing tests to learn patterns -cat models/staging/schema.yml | head -100 -cat models/marts/schema.yml | head -100 - -# Check for custom tests or dbt packages -ls tests/ -cat packages.yml 2>/dev/null -``` - -**Extract from existing tests:** -- YAML formatting style (indentation, spacing) -- Test coverage depth (all columns vs key columns only) -- Use of custom tests (dbt_utils, dbt_expectations, custom macros) -- Description style (brief vs detailed) -- Severity levels used (warn vs error) - -### 2. Read Model SQL - -```bash -cat models//.sql -``` - -Identify: primary keys, foreign keys, categorical columns, date columns, business-critical fields. - -### 3. Check Existing Tests for This Model - -```bash -cat models//schema.yml | grep -A 50 "" -# or -find . -name "schema.yml" -exec grep -l "" {} \; -``` - -### 4. Identify Testable Columns - -| Column Type | Recommended Tests | -|-------------|-------------------| -| Primary key | `unique`, `not_null` | -| Foreign key | `not_null`, `relationships` | -| Categorical | `accepted_values` (ask user for valid values) | -| Required field | `not_null` | -| Date/timestamp | `not_null` | -| Boolean | `accepted_values: [true, false]` | - -### 5. Write Tests in schema.yml - -**Match the existing style from step 1. Example format (adapt to project):** - -```yaml -version: 2 - -models: - - name: model_name - description: "Brief description of what this model contains" - columns: - - name: primary_key_column - description: "Unique identifier for this record" - tests: - - unique - - not_null - - - name: foreign_key_column - description: "Reference to related_model" - tests: - - not_null - - relationships: - to: ref('related_model') - field: related_key_column - - - name: status - description: "Current status of the record" - tests: - - not_null - - accepted_values: - values: ['pending', 'active', 'completed', 'cancelled'] - - - name: created_at - description: "Timestamp when record was created" - tests: - - not_null -``` - -### 6. Run Tests - -```bash -# Test specific model -dbt test --select - -# Test with upstream -dbt test --select + -``` - -### 7. Fix Failing Tests - -Common failures and fixes: - -| Failure | Likely Cause | Fix | -|---------|--------------|-----| -| `unique` fails | Duplicate records | Add deduplication in model | -| `not_null` fails | NULL values in source | Add COALESCE or filter | -| `relationships` fails | Orphan records | Add WHERE clause or fix upstream | -| `accepted_values` fails | New/unexpected values | Update accepted values list | - -## Test Types Reference - -### Generic Tests (built-in) - -```yaml -tests: - - unique - - not_null - - accepted_values: - values: ['a', 'b', 'c'] - - relationships: - to: ref('other_model') - field: id -``` - -### Custom Generic Tests - -```yaml -tests: - - dbt_utils.expression_is_true: - expression: "amount >= 0" - - dbt_utils.recency: - datepart: day - field: created_at - interval: 1 -``` - -### Singular Tests - -Create `tests/.sql`: -```sql --- tests/assert_positive_revenue.sql -select * -from {{ ref('orders') }} -where revenue < 0 -``` - -## Anti-Patterns - -- Adding tests without checking existing project patterns first -- Using different YAML formatting style than existing tests -- Models without any tests -- Primary keys without both unique AND not_null -- Testing only obvious columns, ignoring business-critical ones -- Hardcoding accepted_values without confirming with stakeholders -- Adding dbt_utils tests when project doesn't use that package - diff --git a/skills/snowflake/finding-expensive-queries/SKILL.md b/skills/snowflake/finding-expensive-queries/SKILL.md deleted file mode 100644 index 33db55d..0000000 --- a/skills/snowflake/finding-expensive-queries/SKILL.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -name: finding-expensive-queries -description: | - Finds and ranks expensive Snowflake queries by cost, time, or data scanned. Use when: - (1) User asks to find slow, expensive, or problematic queries - (2) Task mentions "query history", "top queries", "most expensive", or "slowest queries" - (3) Analyzing warehouse costs or identifying optimization candidates - (4) Finding queries that scan the most data or have the most spillage - Returns ranked list of queries with metrics and optimization recommendations. ---- - -# Finding Expensive Queries - -**Query history → Rank by metric → Identify patterns → Recommend optimizations** - -## Workflow - -### 1. Ask What to Optimize For - -Before querying, clarify: -- Time period? (last day, week, month) -- Metric? (execution time, bytes scanned, cost, spillage) -- Warehouse? (specific or all) -- User? (specific or all) - -### 2. Find Expensive Queries by Cost - -Use QUERY_ATTRIBUTION_HISTORY for credit/cost analysis: - -```sql -SELECT - query_id, - warehouse_name, - user_name, - credits_attributed_compute, - start_time, - end_time, - query_tag -FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY -WHERE start_time >= DATEADD('days', -7, CURRENT_TIMESTAMP()) -ORDER BY credits_attributed_compute DESC -LIMIT 20; -``` - -### 3. Get Performance Stats for Specific Queries - -Use QUERY_HISTORY for detailed performance metrics (run separately, not joined): - -```sql -SELECT - query_id, - query_text, - total_elapsed_time/1000 as seconds, - bytes_scanned/1e9 as gb_scanned, - bytes_spilled_to_local_storage/1e9 as gb_spilled_local, - bytes_spilled_to_remote_storage/1e9 as gb_spilled_remote, - partitions_scanned, - partitions_total -FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY -WHERE query_id IN ('', '', ...) - AND start_time >= DATEADD('days', -7, CURRENT_TIMESTAMP()); -``` - -### 4. Identify Patterns - -Look for: -- High `credits_attributed_compute` queries -- Same `query_hash` repeated (caching opportunity) -- `partitions_scanned = partitions_total` (no pruning) -- High `gb_spilled` (memory pressure) - -### 5. Return Results - -Provide: -1. Ranked list of expensive queries with key metrics -2. Common patterns identified -3. Top 3-5 optimization recommendations -4. Specific queries to investigate further - -## Common Filters - -```sql --- Time range (required) -WHERE start_time >= DATEADD('days', -7, CURRENT_TIMESTAMP()) - --- By warehouse -AND warehouse_name = 'ANALYTICS_WH' - --- By user -AND user_name = 'ETL_USER' - --- Only queries over cost threshold -AND credits_attributed_compute > 0.01 - --- Only queries over time threshold -AND total_elapsed_time > 60000 -- over 1 minute -``` diff --git a/skills/snowflake/optimizing-query-by-id/SKILL.md b/skills/snowflake/optimizing-query-by-id/SKILL.md deleted file mode 100644 index 1e11cd3..0000000 --- a/skills/snowflake/optimizing-query-by-id/SKILL.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -name: optimizing-query-by-id -description: | - Optimizes Snowflake query performance using query ID from history. Use when optimizing Snowflake queries for: - (1) User provides a Snowflake query_id (UUID format) to analyze or optimize - (2) Task mentions "slow query", "optimize", "query history", or "query profile" with a query ID - (3) Analyzing query performance metrics - bytes scanned, spillage, partition pruning - (4) User references a previously run query that needs optimization - Fetches query profile, identifies bottlenecks, returns optimized SQL with expected improvements. ---- - -# Optimize Query from Query ID - -**Fetch query → Get profile → Apply best practices → Verify improvement → Return optimized query** - -## Workflow - -### 1. Fetch Query Details from Query ID - -```sql -SELECT - query_id, - query_text, - total_elapsed_time/1000 as seconds, - bytes_scanned/1e9 as gb_scanned, - bytes_spilled_to_local_storage/1e9 as gb_spilled_local, - bytes_spilled_to_remote_storage/1e9 as gb_spilled_remote, - partitions_scanned, - partitions_total, - rows_produced -FROM TABLE(INFORMATION_SCHEMA.QUERY_HISTORY()) -WHERE query_id = ''; -``` - -Note the key metrics: -- `seconds`: Total execution time -- `gb_scanned`: Data read (lower is better) -- `gb_spilled`: Spillage indicates memory pressure -- `partitions_scanned/total`: Partition pruning effectiveness - -### 2. Get Query Profile Details - -```sql --- Get operator-level statistics -SELECT * -FROM TABLE(GET_QUERY_OPERATOR_STATS('')); -``` - -Look for: -- Operators with high `output_rows` vs `input_rows` (explosions) -- TableScan operators with high bytes -- Sort/Aggregate operators with spillage - -### 3. Identify Optimization Opportunities - -Based on profile, look for: - -| Metric | Issue | Fix | -|--------|-------|-----| -| partitions_scanned = partitions_total | No pruning | Add filter on cluster key | -| gb_spilled > 0 | Memory pressure | Simplify query, increase warehouse | -| High bytes_scanned | Full scan | Add selective filters, reduce columns | -| Join explosion | Cartesian or bad key | Fix join condition, filter before join | - -### 4. Apply Optimizations - -Rewrite the query: -- Select only needed columns -- Filter early (before joins) -- Use CTEs to avoid repeated scans -- Ensure filters align with clustering keys -- Add LIMIT if full result not needed - -### 5. Get Explain Plan for Optimized Query - -```sql -EXPLAIN USING JSON -; -``` - -### 6. Compare Plans - -Compare original vs optimized: -- Fewer partitions scanned? -- Fewer intermediate rows? -- Better join order? - -### 7. Return Results - -Provide: -1. Original query metrics (time, data scanned, spillage) -2. Identified issues -3. The optimized query -4. Summary of changes made -5. Expected improvement - -## Example Output - -**Original Query Metrics:** -- Execution time: 45 seconds -- Data scanned: 12.3 GB -- Partitions: 500/500 (no pruning) -- Spillage: 2.1 GB - -**Issues Found:** -1. No partition pruning - filtering on non-cluster column -2. SELECT * scanning unnecessary columns -3. Large table joined without pre-filtering - -**Optimized Query:** -```sql -WITH filtered_events AS ( - SELECT event_id, user_id, event_type, created_at - FROM events - WHERE created_at >= '2024-01-01' - AND created_at < '2024-02-01' - AND event_type = 'purchase' -) -SELECT fe.event_id, fe.created_at, u.name -FROM filtered_events fe -JOIN users u ON fe.user_id = u.id; -``` - -**Changes:** -- Added date range filter matching cluster key -- Replaced SELECT * with specific columns -- Pre-filtered in CTE before join - -**Expected Improvement:** -- Partitions: 500 → ~15 (97% reduction) -- Data scanned: 12.3 GB → ~0.4 GB -- Estimated time: 45s → ~3s diff --git a/skills/snowflake/optimizing-query-text/SKILL.md b/skills/snowflake/optimizing-query-text/SKILL.md deleted file mode 100644 index c40a0a2..0000000 --- a/skills/snowflake/optimizing-query-text/SKILL.md +++ /dev/null @@ -1,166 +0,0 @@ ---- -name: optimizing-query-text -description: | - Optimizes Snowflake SQL query performance from provided query text. Use when optimizing Snowflake SQL for: - (1) User provides or pastes a SQL query and asks to optimize, tune, or improve it - (2) Task mentions "slow query", "make faster", "improve performance", "optimize SQL", or "query tuning" - (3) Reviewing SQL for performance anti-patterns (function on filter column, implicit joins, etc.) - (4) User asks why a query is slow or how to speed it up ---- - -# Optimize Query from SQL Text - -## OUTPUT FORMAT - -Return ONLY the optimized SQL query. No markdown formatting, no explanations, no bullet points - just pure SQL that can be executed directly in Snowflake. - -## CRITICAL: Semantic Preservation Rules - -**The optimized query MUST return IDENTICAL results to the original.** - -Before returning ANY optimization, verify: -- **Same columns**: Exact same columns in exact same order with exact same aliases -- **Same rows**: Filter conditions must be semantically equivalent -- **Same ordering**: Preserve `ORDER BY` exactly as written -- **Same limits**: If original has `LIMIT N`, keep `LIMIT N`. If no LIMIT, do NOT add one. - -**If you cannot guarantee identical results, return the original query unchanged.** - ---- - -## Pattern 1: Function on Filter Column - -**Problem**: Functions on columns in WHERE clause prevent partition pruning and index usage. - -### CAN Fix - -| Original | Optimized | Why Safe | -|----------|-----------|----------| -| `WHERE DATE(ts) = '2024-01-01'` | `WHERE ts >= '2024-01-01' AND ts < '2024-01-02'` | Equivalent range | -| `WHERE YEAR(dt) = 2024` | `WHERE dt >= '2024-01-01' AND dt < '2025-01-01'` | Equivalent range | -| `WHERE MONTH(dt) = 3 AND YEAR(dt) = 2024` | `WHERE dt >= '2024-03-01' AND dt < '2024-04-01'` | Equivalent range | -| `WHERE DATE(ts) >= '2024-01-01' AND DATE(ts) < '2024-02-01'` | `WHERE ts >= '2024-01-01' AND ts < '2024-02-01'` | Same boundaries | -| `WHERE YEAR(dt) BETWEEN 1995 AND 1996` | `WHERE dt >= '1995-01-01' AND dt < '1997-01-01'` | Equivalent range | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| `WHERE YEAR(dt) IN (SELECT year FROM ...)` | Dynamic values, cannot precompute range | -| `WHERE DATE(ts) = DATE(other_col)` | Comparing two columns, both need function | -| `WHERE EXTRACT(DOW FROM dt) = 1` | Day-of-week has no contiguous range | -| `WHERE DATE_TRUNC('month', dt) = '2024-01-01'` in GROUP BY | Needed for grouping logic | -| `SELECT YEAR(dt) AS yr ... GROUP BY YEAR(dt)` | Function in SELECT/GROUP BY is fine, only filter matters | - ---- - -## Pattern 2: Function on JOIN Column - -**Problem**: Functions on JOIN columns prevent hash joins, forcing slower nested loop joins. - -### CAN Fix - -| Original | Optimized | Why Safe | -|----------|-----------|----------| -| `ON CAST(a.id AS VARCHAR) = CAST(b.id AS VARCHAR)` | `ON a.id = b.id` | If both are same type (e.g., INTEGER) | -| `ON UPPER(a.code) = UPPER(b.code)` | `ON a.code = b.code` | If data is already consistently cased | -| `ON TRIM(a.name) = TRIM(b.name)` | `ON a.name = b.name` | If data has no leading/trailing spaces | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| `ON CAST(a.id AS VARCHAR) = b.string_id` | Types genuinely differ, CAST required | -| `ON DATE(a.timestamp) = b.date_col` | Different granularity, DATE() required | -| `ON UPPER(a.code) = b.code` | If b.code might have different case | -| `ON a.id = b.id + 1` | Arithmetic transformation, cannot remove | - ---- - -## Pattern 3: NOT IN Subquery - -**Problem**: `NOT IN` has poor performance and unexpected NULL behavior. - -### CAN Fix - -| Original | Optimized | Why Safe | -|----------|-----------|----------| -| `WHERE id NOT IN (SELECT id FROM t WHERE ...)` | `WHERE NOT EXISTS (SELECT 1 FROM t WHERE t.id = main.id AND ...)` | Equivalent when subquery column is NOT NULL | -| `WHERE id NOT IN (SELECT id FROM t)` where id has NOT NULL constraint | `WHERE NOT EXISTS (SELECT 1 FROM t WHERE t.id = main.id)` | NOT NULL guarantees equivalence | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| `WHERE id NOT IN (SELECT nullable_col FROM t)` | If subquery returns NULL, NOT IN returns no rows; NOT EXISTS doesn't | -| `WHERE (a, b) NOT IN (SELECT x, y FROM t)` | Multi-column NOT IN has complex NULL semantics | - -**Key Rule**: Only convert NOT IN to NOT EXISTS if you can verify the subquery column cannot be NULL. - ---- - -## Pattern 4: Repeated Subquery - -**Problem**: Same subquery executed multiple times causes redundant scans. - -### CAN Fix - -| Original | Optimized | -|----------|-----------| -| Subquery appears 2+ times identically | Extract to CTE, reference CTE multiple times | -| Same aggregation used in multiple places | Compute once in CTE | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| Correlated subquery (references outer table) | Each execution is different, cannot cache | -| Subqueries with different filters | Not actually the same subquery | -| Subquery in SELECT that depends on current row | Correlation prevents extraction | - ---- - -## Pattern 5: Implicit Comma Joins - -**Problem**: Comma-separated tables in FROM clause are harder to read and optimize. - -### CAN Fix - Always - -Convert `FROM a, b, c WHERE a.id = b.id AND b.id = c.id` to explicit JOIN syntax. - -This is always safe - just restructuring, no semantic change. - ---- - -## UNSAFE Optimizations (NEVER apply) - -- **UNION to UNION ALL**: UNION deduplicates rows, UNION ALL does not - different results -- **Changing window functions**: Do not modify `SUM(SUM(x)) OVER(...)` or similar nested aggregates -- **Adding redundant filters**: Do not add filters in JOIN ON if same filter exists in WHERE -- **Changing column names**: Copy column names EXACTLY from original - do not "simplify" or rename -- **Changing column aliases**: Keep all aliases exactly as original -- **Adding early filtering in JOINs**: If a filter is in WHERE, do not duplicate it in JOIN ON clause - ---- - -## Principles - -1. **Minimal changes**: Make the fewest changes necessary. Simpler optimizations are more reliable. -2. **Preserve structure**: Keep subqueries, CTEs, and overall query structure unless there's a clear benefit. -3. **When in doubt, don't**: If unsure whether a change preserves semantics, skip it. -4. **Copy exactly**: Column names, table aliases, and expressions should be copied character-for-character. - ---- - -## Priority Order - -1. **Date/time functions on filter columns** - Highest impact -2. **Implicit joins to explicit JOIN** - Always safe, improves readability -3. **NOT IN to NOT EXISTS** - Only if NULL-safe - ---- - -## Requirements - -- **Results must be identical**: Same rows, same columns, same order -- **Valid Snowflake SQL**: Output must execute without errors in Snowflake