From 4a1f5ed7a3ddb237d83a9f744017f4a5cf00ae90 Mon Sep 17 00:00:00 2001 From: Haider Date: Thu, 16 Jul 2026 11:18:23 +0530 Subject: [PATCH 1/3] fix(skills): use folded block scalar for SKILL.md description frontmatter 10 SKILL.md files across dbt/* and snowflake/* used `description: |` (literal block scalar) which preserves embedded newlines in the parsed string. Some downstream consumers (Claude Code plugin browser, strict-parser YAML validators, marketplace review tooling) render or validate that string as-is instead of collapsing the per-numbered-item newlines into paragraph spaces. Switched all 10 to `description: >-` (folded strip block scalar), which folds single newlines into single spaces on parse and strips the trailing newline. Content is otherwise unchanged. Verified via PyYAML that all frontmatters now parse cleanly: - description is a single-line string on the parsed side - no double-space or leading/trailing-space artifacts - text content preserved verbatim Same fix landed independently in `altimate-claude-plugin` (the Anthropic-compliance rewrite). Applied here + in the two other sibling plugins (opencode, codex) in parallel commits so the SKILL.md source stays in sync across all four surfaces. Co-Authored-By: Claude Opus 4.7 (1M context) --- skills/dbt/creating-dbt-models/SKILL.md | 2 +- skills/dbt/debugging-dbt-errors/SKILL.md | 2 +- skills/dbt/developing-incremental-models/SKILL.md | 2 +- skills/dbt/documenting-dbt-models/SKILL.md | 2 +- skills/dbt/migrating-sql-to-dbt/SKILL.md | 2 +- skills/dbt/refactoring-dbt-models/SKILL.md | 2 +- skills/dbt/testing-dbt-models/SKILL.md | 2 +- skills/snowflake/finding-expensive-queries/SKILL.md | 2 +- skills/snowflake/optimizing-query-by-id/SKILL.md | 2 +- skills/snowflake/optimizing-query-text/SKILL.md | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) diff --git a/skills/dbt/creating-dbt-models/SKILL.md b/skills/dbt/creating-dbt-models/SKILL.md index fc9919a..87efc8b 100644 --- a/skills/dbt/creating-dbt-models/SKILL.md +++ b/skills/dbt/creating-dbt-models/SKILL.md @@ -1,6 +1,6 @@ --- name: creating-dbt-models -description: | +description: >- Creates dbt models following project conventions. Use when working with dbt models for: (1) Creating new models (any layer - discovers project's naming conventions first) (2) Task mentions "create", "build", "add", "write", "new", or "implement" with model, table, or SQL diff --git a/skills/dbt/debugging-dbt-errors/SKILL.md b/skills/dbt/debugging-dbt-errors/SKILL.md index f2eded3..0bf3bb9 100644 --- a/skills/dbt/debugging-dbt-errors/SKILL.md +++ b/skills/dbt/debugging-dbt-errors/SKILL.md @@ -1,6 +1,6 @@ --- name: debugging-dbt-errors -description: | +description: >- Debugs and fixes dbt errors systematically. Use when working with dbt errors for: (1) Task mentions "fix", "error", "broken", "failing", "debug", "wrong", or "not working" (2) Compilation Error, Database Error, or test failures occur diff --git a/skills/dbt/developing-incremental-models/SKILL.md b/skills/dbt/developing-incremental-models/SKILL.md index 6aa9877..8d2f684 100644 --- a/skills/dbt/developing-incremental-models/SKILL.md +++ b/skills/dbt/developing-incremental-models/SKILL.md @@ -1,6 +1,6 @@ --- name: developing-incremental-models -description: | +description: >- Develops and troubleshoots dbt incremental models. Use when working with incremental materialization for: (1) Creating new incremental models (choosing strategy, unique_key, partition) (2) Task mentions "incremental", "append", "merge", "upsert", or "late arriving data" diff --git a/skills/dbt/documenting-dbt-models/SKILL.md b/skills/dbt/documenting-dbt-models/SKILL.md index 795c83a..5ff0ed4 100644 --- a/skills/dbt/documenting-dbt-models/SKILL.md +++ b/skills/dbt/documenting-dbt-models/SKILL.md @@ -1,6 +1,6 @@ --- name: documenting-dbt-models -description: | +description: >- Documents dbt models and columns in schema.yml. Use when working with dbt documentation for: (1) Adding model descriptions or column definitions to schema.yml (2) Task mentions "document", "describe", "description", "dbt docs", or "schema.yml" diff --git a/skills/dbt/migrating-sql-to-dbt/SKILL.md b/skills/dbt/migrating-sql-to-dbt/SKILL.md index c1a0632..5bc1cb0 100644 --- a/skills/dbt/migrating-sql-to-dbt/SKILL.md +++ b/skills/dbt/migrating-sql-to-dbt/SKILL.md @@ -1,6 +1,6 @@ --- name: migrating-sql-to-dbt -description: | +description: >- Converts legacy SQL to modular dbt models. Use when migrating SQL to dbt for: (1) Converting stored procedures, views, or raw SQL files to dbt models (2) Task mentions "migrate", "convert", "legacy SQL", "transform to dbt", or "modernize" diff --git a/skills/dbt/refactoring-dbt-models/SKILL.md b/skills/dbt/refactoring-dbt-models/SKILL.md index fd791bd..87bc352 100644 --- a/skills/dbt/refactoring-dbt-models/SKILL.md +++ b/skills/dbt/refactoring-dbt-models/SKILL.md @@ -1,6 +1,6 @@ --- name: refactoring-dbt-models -description: | +description: >- Safely refactors dbt models with downstream impact analysis. Use when restructuring dbt models for: (1) Task mentions "refactor", "restructure", "extract", "split", "break into", or "reorganize" (2) Extracting CTEs to intermediate models or creating macros diff --git a/skills/dbt/testing-dbt-models/SKILL.md b/skills/dbt/testing-dbt-models/SKILL.md index a9d6155..a18d185 100644 --- a/skills/dbt/testing-dbt-models/SKILL.md +++ b/skills/dbt/testing-dbt-models/SKILL.md @@ -1,6 +1,6 @@ --- name: testing-dbt-models -description: | +description: >- Adds schema tests and data quality validation to dbt models. Use when working with dbt tests for: (1) Adding or modifying tests in schema.yml files (2) Task mentions "test", "validate", "data quality", "unique", "not_null", or "accepted_values" diff --git a/skills/snowflake/finding-expensive-queries/SKILL.md b/skills/snowflake/finding-expensive-queries/SKILL.md index 33db55d..05305ad 100644 --- a/skills/snowflake/finding-expensive-queries/SKILL.md +++ b/skills/snowflake/finding-expensive-queries/SKILL.md @@ -1,6 +1,6 @@ --- name: finding-expensive-queries -description: | +description: >- Finds and ranks expensive Snowflake queries by cost, time, or data scanned. Use when: (1) User asks to find slow, expensive, or problematic queries (2) Task mentions "query history", "top queries", "most expensive", or "slowest queries" diff --git a/skills/snowflake/optimizing-query-by-id/SKILL.md b/skills/snowflake/optimizing-query-by-id/SKILL.md index 1e11cd3..abd2017 100644 --- a/skills/snowflake/optimizing-query-by-id/SKILL.md +++ b/skills/snowflake/optimizing-query-by-id/SKILL.md @@ -1,6 +1,6 @@ --- name: optimizing-query-by-id -description: | +description: >- Optimizes Snowflake query performance using query ID from history. Use when optimizing Snowflake queries for: (1) User provides a Snowflake query_id (UUID format) to analyze or optimize (2) Task mentions "slow query", "optimize", "query history", or "query profile" with a query ID diff --git a/skills/snowflake/optimizing-query-text/SKILL.md b/skills/snowflake/optimizing-query-text/SKILL.md index c40a0a2..b4914bf 100644 --- a/skills/snowflake/optimizing-query-text/SKILL.md +++ b/skills/snowflake/optimizing-query-text/SKILL.md @@ -1,6 +1,6 @@ --- name: optimizing-query-text -description: | +description: >- Optimizes Snowflake SQL query performance from provided query text. Use when optimizing Snowflake SQL for: (1) User provides or pastes a SQL query and asks to optimize, tune, or improve it (2) Task mentions "slow query", "make faster", "improve performance", "optimize SQL", or "query tuning" From 15b543f50e71874158884551bc4c18a02c24c16c Mon Sep 17 00:00:00 2001 From: Haider Date: Thu, 16 Jul 2026 11:31:57 +0530 Subject: [PATCH 2/3] refactor: single-skill delegation bundle (altimate-code) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This bundle exposes one skill — `altimate-code` — whose job is to route data-engineering tasks (dbt, warehouse state, lineage, FinOps, PII, cross-database work) to the altimate-code CLI. Layout: - `skills/altimate-code/SKILL.md` — the one skill. - `opencode.json` — `skills.paths` points at that dir. - `plugins/altimate-code/index.ts` — `BUNDLED_SKILL_PATHS` matches; the plugin's `config` hook auto-registers the path so users don't have to hand-edit their global `opencode.jsonc` after `opencode plugin install`. Also keeps the 4 deterministic dbt tools + the `altimate_code` subprocess-spawn tool from the existing plugin surface — those are CLI wrappers around the bundled `altimate-dbt` binary, not skills. Typecheck: clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 37 +- opencode.json | 10 - plugins/altimate-code/index.ts | 14 +- skills/dbt/creating-dbt-models/SKILL.md | 132 ------- skills/dbt/debugging-dbt-errors/SKILL.md | 153 --------- .../developing-incremental-models/SKILL.md | 324 ------------------ skills/dbt/documenting-dbt-models/SKILL.md | 167 --------- skills/dbt/migrating-sql-to-dbt/SKILL.md | 112 ------ skills/dbt/refactoring-dbt-models/SKILL.md | 186 ---------- skills/dbt/testing-dbt-models/SKILL.md | 174 ---------- .../finding-expensive-queries/SKILL.md | 97 ------ .../snowflake/optimizing-query-by-id/SKILL.md | 132 ------- .../snowflake/optimizing-query-text/SKILL.md | 166 --------- 13 files changed, 11 insertions(+), 1693 deletions(-) delete mode 100644 skills/dbt/creating-dbt-models/SKILL.md delete mode 100644 skills/dbt/debugging-dbt-errors/SKILL.md delete mode 100644 skills/dbt/developing-incremental-models/SKILL.md delete mode 100644 skills/dbt/documenting-dbt-models/SKILL.md delete mode 100644 skills/dbt/migrating-sql-to-dbt/SKILL.md delete mode 100644 skills/dbt/refactoring-dbt-models/SKILL.md delete mode 100644 skills/dbt/testing-dbt-models/SKILL.md delete mode 100644 skills/snowflake/finding-expensive-queries/SKILL.md delete mode 100644 skills/snowflake/optimizing-query-by-id/SKILL.md delete mode 100644 skills/snowflake/optimizing-query-text/SKILL.md diff --git a/README.md b/README.md index b38e68b..972bb32 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,14 @@ # altimate-opencode-plugin -Altimate data engineering plugin for [opencode](https://opencode.ai) — bundles dbt skills, Snowflake optimization skills, and altimate-code subprocess delegation behind opencode's native plugin + skills system. - -> Mirror of the [`data-engineering-skills`](https://github.com/AltimateAI/data-engineering-skills) Claude Code plugin, ported to opencode's plugin contract. +opencode plugin that delegates dbt and warehouse work to [`altimate-code`](https://github.com/AltimateAI/altimate-code) — a specialised CLI agent for column-level lineage, dbt build/test/run, warehouse profiling, FinOps reporting, cross-database validation, and connectivity to Snowflake / BigQuery / Redshift / Databricks / Postgres / MySQL / DuckDB. ## What this gives your opencode agent | Surface | Mechanism | What it does | |---|---|---| -| **Skills** (11) | `~/.config/opencode/skills/` + `.opencode/skills/` | dbt creating/testing/debugging/refactoring/documenting/migrating/incremental + Snowflake query optimization + altimate-code delegation guidance. Loaded on-demand via opencode's native `skill` tool. | -| **Deterministic dbt tools** | TypeScript plugin tools | `altimate_dbt_columns`, `altimate_dbt_source`, `altimate_dbt_compile`, `altimate_dbt_build` — structured-output replacements for grep+read+bash workflows | -| **Subprocess delegation** | `altimate_code` tool | Spawn a fresh altimate-code subprocess for cross-warehouse / multi-session work the in-process agent can't drive | +| **Skill** (1) | `~/.config/opencode/skills/` — auto-registered by the plugin's `config` hook at load time | `altimate-code` — routes tasks that touch live warehouse state, column lineage, cross-database work, PII detection, FinOps, or query-cost attribution to the altimate-code subprocess. | +| **Deterministic dbt tools** | TypeScript plugin tools | `altimate_dbt_columns`, `altimate_dbt_source`, `altimate_dbt_compile`, `altimate_dbt_build` — direct wrappers around the bundled `altimate-dbt` CLI for cheap, deterministic dbt introspection when a full delegation would be overkill. | +| **Subprocess delegation** | `altimate_code` tool | Spawns a fresh altimate-code subprocess for cross-warehouse / multi-step / warehouse-state work. | ## Install @@ -33,27 +31,16 @@ the resolved entry into the top-level `plugin` array of > documented `plugins` / `skills: [...]` — those keys are silently ignored by > opencode. -## Skills, mapped from data-engineering-skills +## Layout ``` -skills/ -├── altimate-code/SKILL.md -├── dbt/ -│ ├── creating-dbt-models/SKILL.md -│ ├── debugging-dbt-errors/SKILL.md -│ ├── documenting-dbt-models/SKILL.md -│ ├── developing-incremental-models/SKILL.md -│ ├── migrating-sql-to-dbt/SKILL.md -│ ├── refactoring-dbt-models/SKILL.md -│ └── testing-dbt-models/SKILL.md -└── snowflake/ - ├── finding-expensive-queries/SKILL.md - ├── optimizing-query-by-id/SKILL.md - └── optimizing-query-text/SKILL.md +altimate-opencode-plugin/ +├── opencode.json # opencode manifest (plugin + skills.paths) +├── plugins/altimate-code/ +│ └── index.ts # tool implementations + config hook (auto-registers the skill path) +└── skills/altimate-code/SKILL.md # the one delegation skill ``` -These are byte-identical to the Claude Code source. opencode's skill loader honors the same SKILL.md format and YAML frontmatter — no translation needed. - ## Plugin tools | Tool | Purpose | @@ -64,10 +51,6 @@ These are byte-identical to the Claude Code source. opencode's skill loader hono | `altimate_dbt_build(project_dir?, model?, downstream?, select?)` | Run `altimate-dbt build`; optionally restrict to a single `model` (with `downstream` to also build descendants). `select` is a back-compat alias for `model`. | | `altimate_code(task, project_dir?, yolo?, timeout_sec?)` | Spawn a fresh altimate-code subprocess for free-form delegation | -## Why these tools - -Empirically, when the agent has access to deterministic tools that replace grep+read loops, it converges to its answer with fewer turns and lower spend. See the [plugin skill experiments](https://github.com/AltimateAI/data-engineering-skills/blob/main/docs/EXPERIMENTS.md) for benchmark data. - ## Dev ```bash diff --git a/opencode.json b/opencode.json index 75036ac..2cc4eb0 100644 --- a/opencode.json +++ b/opencode.json @@ -2,16 +2,6 @@ "$schema": "https://opencode.ai/config.json", "skills": { "paths": [ - "./skills/dbt/creating-dbt-models", - "./skills/dbt/debugging-dbt-errors", - "./skills/dbt/testing-dbt-models", - "./skills/dbt/documenting-dbt-models", - "./skills/dbt/migrating-sql-to-dbt", - "./skills/dbt/refactoring-dbt-models", - "./skills/dbt/developing-incremental-models", - "./skills/snowflake/optimizing-query-by-id", - "./skills/snowflake/optimizing-query-text", - "./skills/snowflake/finding-expensive-queries", "./skills/altimate-code" ] }, diff --git a/plugins/altimate-code/index.ts b/plugins/altimate-code/index.ts index a8a12cb..73169b7 100644 --- a/plugins/altimate-code/index.ts +++ b/plugins/altimate-code/index.ts @@ -225,19 +225,7 @@ const altimateCode = tool({ // (plugins/altimate-code/index.ts → repo root → skills). const PLUGIN_FILE_DIR = dirname(fileURLToPath(import.meta.url)) const SKILLS_ROOT = resolve(PLUGIN_FILE_DIR, "..", "..", "skills") -const BUNDLED_SKILL_PATHS = [ - "altimate-code", - "dbt/creating-dbt-models", - "dbt/debugging-dbt-errors", - "dbt/developing-incremental-models", - "dbt/documenting-dbt-models", - "dbt/migrating-sql-to-dbt", - "dbt/refactoring-dbt-models", - "dbt/testing-dbt-models", - "snowflake/finding-expensive-queries", - "snowflake/optimizing-query-by-id", - "snowflake/optimizing-query-text", -].map((rel) => resolve(SKILLS_ROOT, rel)) +const BUNDLED_SKILL_PATHS = ["altimate-code"].map((rel) => resolve(SKILLS_ROOT, rel)) type CfgWithSkills = { skills?: { diff --git a/skills/dbt/creating-dbt-models/SKILL.md b/skills/dbt/creating-dbt-models/SKILL.md deleted file mode 100644 index 87efc8b..0000000 --- a/skills/dbt/creating-dbt-models/SKILL.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -name: creating-dbt-models -description: >- - Creates dbt models following project conventions. Use when working with dbt models for: - (1) Creating new models (any layer - discovers project's naming conventions first) - (2) Task mentions "create", "build", "add", "write", "new", or "implement" with model, table, or SQL - (3) Modifying existing model logic, columns, joins, or transformations - (4) Implementing a model from schema.yml specs or expected output requirements - Discovers project conventions before writing. Runs dbt build (not just compile) to verify. ---- - -# dbt Model Development - -**Read before you write. Build after you write. Verify your output.** - -## Critical Rules - -1. **ALWAYS run `dbt build`** after creating/modifying models - compile is NOT enough -2. **ALWAYS verify output** after build using `dbt show` - don't assume success -3. **If build fails 3+ times**, stop and reassess your entire approach - -## Workflow - -### 1. Understand the Task Requirements - -- What columns are needed? List them explicitly. -- What is the grain of the table (one row per what)? -- What calculations or aggregations are required? - -### 2. Discover Project Conventions - -```bash -cat dbt_project.yml -find models/ -name "*.sql" | head -20 -``` - -Read 2-3 existing models to learn naming, config, and SQL patterns. - -### 3. Find Similar Models - -```bash -# Find models with similar purpose -find models/ -name "*agg*.sql" -o -name "*fct_*.sql" | head -5 -``` - -Learn from existing models: join types, aggregation patterns, NULL handling. - -### 4. Check Upstream Data - -```bash -# Preview upstream data if needed -dbt show --select --limit 10 -``` - -### 5. Write the Model - -Follow discovered conventions. Match the required columns exactly. - -### 6. Compile (Syntax Check) - -```bash -dbt compile --select -``` - -### 7. BUILD - MANDATORY - -**This step is REQUIRED. Do NOT skip it.** - -```bash -dbt build --select -``` - -If build fails: -1. Read the error carefully -2. Fix the specific issue -3. Run build again -4. **If fails 3+ times, step back and reassess approach** - -### 8. Verify Output (CRITICAL) - -**Build success does NOT mean correct output.** - -```bash -# Check the table was created and preview data -dbt show --select --limit 10 -``` - -Verify: -- Column names match requirements exactly -- Row count is reasonable -- Data values look correct -- No unexpected NULLs - -### 9. Verify Calculations Against Sample Data - -**For models with calculations, verify correctness manually:** - -```bash -# Pick a specific row and verify calculation by hand -dbt show --inline " - select * - from {{ ref('model_name') }} - where = '' -" --limit 1 - -# Cross-check aggregations -dbt show --inline " - select count(*), sum() - from {{ ref('model_name') }} -" -``` - -For example, if calculating `total_revenue = quantity * price`: -1. Pick one row from output -2. Look up the source quantity and price -3. Manually calculate: does it match? - -### 10. Re-review Against Requirements - -**Before declaring done, re-read the original request:** -- Did you implement what was asked, not what you assumed? -- Are column names exactly as specified? -- Is the calculation logic correct per the requirements? -- Does the grain (one row per what?) match what was requested? - -## Anti-Patterns - -- Declaring done after compile without running build -- Not verifying output data after build -- Getting stuck in compile/build error loops -- Assuming table exists just because model file exists -- Writing SQL without checking existing model patterns first diff --git a/skills/dbt/debugging-dbt-errors/SKILL.md b/skills/dbt/debugging-dbt-errors/SKILL.md deleted file mode 100644 index 0bf3bb9..0000000 --- a/skills/dbt/debugging-dbt-errors/SKILL.md +++ /dev/null @@ -1,153 +0,0 @@ ---- -name: debugging-dbt-errors -description: >- - Debugs and fixes dbt errors systematically. Use when working with dbt errors for: - (1) Task mentions "fix", "error", "broken", "failing", "debug", "wrong", or "not working" - (2) Compilation Error, Database Error, or test failures occur - (3) Model produces incorrect output or unexpected results - (4) Need to troubleshoot why a dbt command failed - Reads full error, checks upstream first, runs dbt build (not just compile) to verify fix. ---- - -# dbt Troubleshooting - -**Read the full error. Check upstream first. ALWAYS run `dbt build` after fixing.** - -## Critical Rules - -1. **ALWAYS run `dbt build` after fixing** - compile is NOT enough to verify the fix -2. **If fix fails 3+ times**, stop and reassess your entire approach -3. **Verify data after build** - build passing doesn't mean output is correct - -## Workflow - -### 1. Get the Full Error - -```bash -dbt compile --select -# or -dbt build --select -``` - -Read the COMPLETE error message. Note the file, line number, and specific error. - -### 2. Inspect Actual Data (For Data Issues) - -**Before fixing "wrong output" or "incorrect results", query the actual data:** - -```bash -# Preview current output -dbt show --select --limit 20 - -# Check specific values with inline query -dbt show --inline "select * from {{ ref('model_name') }} where " --limit 10 - -# Compare with expected - look for patterns -dbt show --inline "select column, count(*) from {{ ref('model_name') }} group by 1 order by 2 desc" --limit 10 -``` - -**Understand what's wrong before attempting to fix it.** - -### 3. Read Compiled SQL - -```bash -cat target/compiled///.sql -``` - -See the actual SQL that will run. - -### 4. Analyze Error Type - -| Error Type | Look For | -|------------|----------| -| Compilation Error | Jinja syntax, missing refs, YAML issues | -| Database Error | Column not found, type mismatch, SQL syntax | -| Dependency Error | Missing model, circular reference | - -### 5. Check Upstream Models - -```bash -# Find what this model references -grep -E "ref\(|source\(" models//.sql - -# Read upstream model to verify columns -cat models//.sql -``` - -Many errors come from upstream changes, not the current model. - -### 6. Apply Fix - -Common fixes: - -| Error | Fix | -|-------|-----| -| Column not found | Check upstream model's output columns | -| Ambiguous column | Add table alias: `table.column` | -| Type mismatch | Add explicit `CAST()` | -| Division by zero | Use `NULLIF(divisor, 0)` | -| Jinja error | Check matching `{{ }}` and `{% %}` | - -### 7. Rebuild (MANDATORY) - -```bash -dbt build --select -``` - -**3-Failure Rule**: If build fails 3+ times, STOP. Step back and: -1. Re-read the original error -2. Check if your entire approach is wrong -3. Consider alternative solutions - -### 8. Verify Fix - -```bash -# Preview the data -dbt show --select --limit 10 - -# Run tests -dbt test --select -``` - -### 9. Re-review Logic Against Requirements - -**After fixing, re-read the original request and verify:** -- Does the output match what the user asked for? -- Are the column names exactly as requested? -- Is the calculation logic correct per the requirements? -- Did you solve the actual problem, not just make the error go away? - -### 10. Check Downstream Impact - -```bash -# Find downstream models -grep -r "ref('')" models/ --include="*.sql" - -# Rebuild downstream -dbt build --select + -``` - -## Error Categories - -### Compilation Errors -- Check Jinja syntax: matching `{{ }}` and `{% %}` -- Verify macro arguments -- Check YAML indentation - -### Database Errors -- Read compiled SQL in `target/compiled/` -- Check column names against upstream -- Verify data types - -### Test Failures -- Read the test SQL to understand what it checks -- Compare your model output to expected behavior -- Check column names, data types, NULL handling - -## Anti-Patterns - -- Making random changes without understanding the error -- Assuming the current model is wrong before checking upstream -- Not reading the FULL error message -- Declaring "fixed" without running build -- Getting stuck making small tweaks instead of reassessing diff --git a/skills/dbt/developing-incremental-models/SKILL.md b/skills/dbt/developing-incremental-models/SKILL.md deleted file mode 100644 index 8d2f684..0000000 --- a/skills/dbt/developing-incremental-models/SKILL.md +++ /dev/null @@ -1,324 +0,0 @@ ---- -name: developing-incremental-models -description: >- - Develops and troubleshoots dbt incremental models. Use when working with incremental materialization for: - (1) Creating new incremental models (choosing strategy, unique_key, partition) - (2) Task mentions "incremental", "append", "merge", "upsert", or "late arriving data" - (3) Troubleshooting incremental failures (merge errors, partition pruning, schema drift) - (4) Optimizing incremental performance or deciding table vs incremental - Guides through strategy selection, handles common incremental gotchas. ---- - -# dbt Incremental Model Development - -**Choose the right strategy. Design the unique_key carefully. Handle edge cases.** - -## When to Use Incremental - -| Scenario | Recommendation | -|----------|----------------| -| Source data < 10M rows | Use `table` (simpler, full refresh is fast) | -| Source data > 10M rows | Consider `incremental` | -| Source data updated in place | Use `incremental` with `merge` strategy | -| Append-only source (logs, events) | Use `incremental` with `append` strategy | -| Partitioned warehouse data | Use `insert_overwrite` if supported | - -**Default to `table` unless you have a clear performance reason for incremental.** - -## Critical Rules - -1. **ALWAYS test with `--full-refresh` first** before relying on incremental logic -2. **ALWAYS verify unique_key is truly unique** in both source and target -3. **If merge fails 3+ times**, check unique_key for duplicates -4. **Run full refresh periodically** to prevent data drift - -## Workflow - -### 1. Confirm Incremental is Needed - -```bash -# Check source table size -dbt show --inline "select count(*) from {{ source('schema', 'table') }}" -``` - -If count < 10 million, consider using `table` instead. Incremental adds complexity. - -### 2. Understand the Source Data Pattern - -Before choosing a strategy, answer: -- **Is data append-only?** (new rows added, never updated) -- **Are existing rows updated?** (need merge/upsert) -- **Is there a reliable timestamp?** (for filtering new data) -- **What's the unique identifier?** (for merge matching) - -```bash -# Check for timestamp column -dbt show --inline " - select - min(updated_at) as earliest, - max(updated_at) as latest, - count(distinct date(updated_at)) as days_of_data - from {{ source('schema', 'table') }} -" -``` - -### 3. Choose the Right Strategy - -| Strategy | Use When | How It Works | -|----------|----------|--------------| -| `append` | Data is append-only, no updates | INSERT only, no deduplication | -| `merge` | Data can be updated | MERGE/UPSERT by unique_key | -| `delete+insert` | Data updated in batches | DELETE matching rows, then INSERT | -| `insert_overwrite` | Partitioned tables (BigQuery, Spark) | Replace entire partitions | - -**Default:** `merge` is safest for most use cases. - -**Note:** Strategy availability varies by adapter. Check the [dbt incremental strategy docs](https://docs.getdbt.com/docs/build/incremental-strategy) for your specific warehouse. - -### 4. Design the Unique Key - -**CRITICAL: unique_key must be truly unique in your data.** - -```bash -# Verify uniqueness BEFORE creating model -dbt show --inline " - select {{ unique_key_column }}, count(*) - from {{ source('schema', 'table') }} - group by 1 - having count(*) > 1 - limit 10 -" -``` - -If duplicates exist: -- Add more columns to make composite key -- Add deduplication logic in model -- Use `delete+insert` instead of `merge` - -### 5. Write the Incremental Model - -```sql -{{ - config( - materialized='incremental', - incremental_strategy='merge', -- or append, delete+insert - unique_key='id', -- MUST be unique - on_schema_change='append_new_columns' -- handle new columns - ) -}} - -select - id, - column_a, - column_b, - updated_at -from {{ source('schema', 'table') }} - -{% if is_incremental() %} -where updated_at > (select max(updated_at) from {{ this }}) -{% endif %} -``` - -### 6. Build with Full Refresh First - -**ALWAYS verify with full refresh before trusting incremental logic.** - -```bash -# First run: full refresh to establish baseline -dbt build --select --full-refresh - -# Verify output -dbt show --select --limit 10 -dbt show --inline "select count(*) from {{ ref('model_name') }}" -``` - -### 7. Test Incremental Logic - -```bash -# Run incrementally (no --full-refresh) -dbt build --select - -# Verify row count changed appropriately -dbt show --inline "select count(*) from {{ ref('model_name') }}" -``` - -### 8. Handle Schema Changes - -Set `on_schema_change` based on your needs: - -| Setting | Behavior | -|---------|----------| -| `ignore` (default) | New columns in source are ignored | -| `append_new_columns` | New columns added to target | -| `sync_all_columns` | Target schema matches source exactly | -| `fail` | Error if schema changes | - -## Common Incremental Problems - -### Problem: Merge Fails with Duplicate Key - -**Symptom:** "Cannot MERGE with duplicate values" - -**Cause:** Multiple rows with same unique_key in source or target. - -**Fix:** -```sql --- Add deduplication using a CTE (cross-database compatible) -with deduplicated as ( - select *, - row_number() over (partition by id order by updated_at desc) as rn - from {{ source('schema', 'table') }} - {% if is_incremental() %} - where updated_at > (select max(updated_at) from {{ this }}) - {% endif %} -) -select * from deduplicated where rn = 1 -``` - -### Problem: No Partition Pruning (Full Table Scan) - -**Symptom:** Incremental runs take as long as full refresh. - -**Cause:** Dynamic date filter prevents partition pruning. - -**Fix:** -```sql -{% if is_incremental() %} --- Use static date instead of subquery for partition pruning -where updated_at >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }} - and updated_at > (select max(updated_at) from {{ this }}) -{% endif %} -``` - -### Problem: Late-Arriving Data is Missed - -**Symptom:** Some records never appear in incremental model. - -**Cause:** Filtering by max(updated_at) misses late arrivals. - -**Fix:** Use a lookback window with a fixed offset from current date: -```sql -{% if is_incremental() %} --- Lookback 3 days to catch late-arriving data -where updated_at >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }} -{% endif %} -``` - -Alternatively, use a variable for the lookback period: -```sql -{% set lookback_days = 3 %} - -{% if is_incremental() %} -where updated_at >= {{ dbt.dateadd('day', -lookback_days, dbt.current_timestamp()) }} -{% endif %} -``` - -### Problem: Schema Drift Causes Errors - -**Symptom:** "Column X not found" after source adds column. - -**Fix:** Set `on_schema_change='append_new_columns'` in config. - -### Problem: Data Drift Over Time - -**Symptom:** Counts diverge between incremental and full refresh. - -**Fix:** Schedule periodic full refresh: -```bash -# Weekly full refresh -dbt build --select --full-refresh -``` - -## Incremental Strategy Reference - -### Append (Simplest) - -```sql -{{ config(materialized='incremental', incremental_strategy='append') }} - -select * from {{ source('events', 'raw') }} -{% if is_incremental() %} -where event_timestamp > (select max(event_timestamp) from {{ this }}) -{% endif %} -``` - -- No unique_key needed -- Fastest performance -- **Only use for append-only data** (logs, events, immutable records) - -### Merge (Default) - -```sql -{{ config( - materialized='incremental', - incremental_strategy='merge', - unique_key='id' -) }} - -select * from {{ source('crm', 'contacts') }} -{% if is_incremental() %} -where updated_at > (select max(updated_at) from {{ this }}) -{% endif %} -``` - -- Requires unique_key -- Handles updates and inserts -- Most common strategy - -### Delete+Insert (Batch Updates) - -```sql -{{ config( - materialized='incremental', - incremental_strategy='delete+insert', - unique_key='id' -) }} - -select * from {{ source('orders', 'raw') }} -{% if is_incremental() %} -where order_date >= {{ dbt.dateadd('day', -7, dbt.current_timestamp()) }} -{% endif %} -``` - -- Deletes all matching rows first -- Good for reprocessing batches -- Use when merge has duplicate key issues - -### Insert Overwrite (Partitioned) - -```sql -{{ config( - materialized='incremental', - incremental_strategy='insert_overwrite', - partition_by={'field': 'event_date', 'data_type': 'date'} -) }} - -select * from {{ source('events', 'raw') }} -{% if is_incremental() %} -where event_date >= {{ dbt.dateadd('day', -3, dbt.current_timestamp()) }} -{% endif %} -``` - -- Replaces entire partitions -- Best for partitioned tables in BigQuery/Spark -- No unique_key needed (operates on partitions) - -## Anti-Patterns - -- Using incremental for small tables (< 10M rows) -- Not testing with full-refresh first -- Using append strategy when data can be updated -- Not verifying unique_key uniqueness -- Relying on exact timestamp match without lookback -- Never running full refresh (causes data drift) -- Using merge with non-unique keys - -## Testing Checklist - -- [ ] Model runs with `--full-refresh` -- [ ] Model runs incrementally (without flag) -- [ ] unique_key verified as truly unique -- [ ] Row counts reasonable after incremental run -- [ ] Late-arriving data handled (lookback window) -- [ ] Schema changes handled (on_schema_change set) -- [ ] Periodic full refresh scheduled diff --git a/skills/dbt/documenting-dbt-models/SKILL.md b/skills/dbt/documenting-dbt-models/SKILL.md deleted file mode 100644 index 5ff0ed4..0000000 --- a/skills/dbt/documenting-dbt-models/SKILL.md +++ /dev/null @@ -1,167 +0,0 @@ ---- -name: documenting-dbt-models -description: >- - Documents dbt models and columns in schema.yml. Use when working with dbt documentation for: - (1) Adding model descriptions or column definitions to schema.yml - (2) Task mentions "document", "describe", "description", "dbt docs", or "schema.yml" - (3) Explaining business context, grain, meaning of data, or business rules - (4) Preparing dbt docs generate or improving model discoverability - Matches existing project documentation style and conventions before writing. ---- - -# dbt Documentation - -**Document the WHY, not just the WHAT. Include grain, business rules, and caveats.** - -## Workflow - -### 1. Study Existing Documentation Patterns - -**CRITICAL: Match the project's documentation style before adding new docs.** - -```bash -# Find all schema.yml files with documentation -find . -name "schema.yml" | head -5 - -# Read well-documented models to learn patterns -cat models/marts/schema.yml | head -150 -cat models/staging/schema.yml | head -150 -``` - -**Extract from existing documentation:** -- Description length (brief vs detailed) -- Formatting style (plain text vs markdown with headers) -- Information included (grain? business rules? caveats?) -- Column description depth (all columns vs key columns) -- Use of meta tags or custom properties - -### 2. Read Model SQL - -```bash -cat models//.sql -``` - -Understand: transformations, business logic, joins, filters. - -### 3. Check Existing Documentation for This Model - -```bash -# Find existing schema.yml -find . -name "schema.yml" -exec grep -l "" {} \; - -# Read existing docs -cat models//schema.yml | grep -A 100 "" -``` - -### 4. Identify Documentation Needs - -For each model, document: -- **Model description**: Purpose, grain, key business rules -- **Column descriptions**: Business meaning, not just data type - -For each column, consider: -- What business concept does this represent? -- Are there any caveats or special values? -- What is the source of this data? - -### 5. Write Documentation - -**Match the style discovered in step 1. Example format (adapt to project):** - -```yaml -version: 2 - -models: - - name: orders - description: | - Order transactions at the order line item grain. - Each row represents one product in one order. - - **Business Rules:** - - Revenue recognized on ship_date, not order_date - - Cancelled orders excluded (status != 'cancelled') - - Returns processed as negative line items - - **Grain:** One row per order_id + product_id combination - - columns: - - name: order_id - description: | - Unique identifier for the order. - Source: orders.id from Stripe webhook - - - name: customer_id - description: | - Foreign key to customers table. - NULL for guest checkouts (pre-2023 only) - - - name: revenue - description: | - Net revenue for this line item in USD. - Calculation: unit_price * quantity - discount_amount - Excludes tax and shipping - - - name: order_status - description: | - Current status of the order. - Values: pending, processing, shipped, delivered, cancelled, returned -``` - -### 6. Generate Docs - -```bash -dbt docs generate -dbt docs serve # Optional: preview locally -``` - -## Documentation Patterns - -**Note: These are default templates. Always adapt to match project's existing style.** - -### Model Description Template - -```yaml -description: | - [One sentence: what this model contains] - - **Grain:** [What does one row represent?] - - **Business Rules:** - - [Key rule 1] - - [Key rule 2] - - **Caveats:** - - [Important limitation or edge case] -``` - -### Column Description Patterns - -| Column Type | Documentation Focus | -|-------------|---------------------| -| Primary key | Source system, uniqueness guarantee | -| Foreign key | What it joins to, NULL handling | -| Metric | Calculation formula, units, exclusions | -| Date | Timezone, what event it represents | -| Status/Category | All possible values, business meaning | -| Boolean/Flag | What true/false means in business terms | - -### Documenting Calculated Fields - -```yaml -- name: gross_margin - description: | - Gross margin percentage. - Calculation: (revenue - cogs) / revenue * 100 - NULL when revenue = 0 to avoid division by zero -``` - -## Anti-Patterns - -- Adding documentation without checking existing project patterns -- Using different formatting style than existing documentation -- Describing WHAT (e.g., "The order ID") instead of WHY/context -- Missing grain documentation -- Not documenting NULL handling -- Leaving columns undocumented -- Copy-pasting column names as descriptions - diff --git a/skills/dbt/migrating-sql-to-dbt/SKILL.md b/skills/dbt/migrating-sql-to-dbt/SKILL.md deleted file mode 100644 index 5bc1cb0..0000000 --- a/skills/dbt/migrating-sql-to-dbt/SKILL.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -name: migrating-sql-to-dbt -description: >- - Converts legacy SQL to modular dbt models. Use when migrating SQL to dbt for: - (1) Converting stored procedures, views, or raw SQL files to dbt models - (2) Task mentions "migrate", "convert", "legacy SQL", "transform to dbt", or "modernize" - (3) Breaking monolithic queries into modular layers (discovers project conventions first) - (4) Porting existing data pipelines or ETL to dbt patterns - Checks for existing models/sources, builds and validates layer by layer. ---- - -# dbt Migration - -**Don't convert everything at once. Build and validate layer by layer.** - -## Workflow - -### 1. Analyze Legacy SQL - -```bash -cat -``` - -Identify all tables referenced in the query. - -### 2. Check What Already Exists - -```bash -# Search for existing models/sources that reference the table -grep -r "" models/ --include="*.sql" --include="*.yml" -find models/ -name "*.sql" | xargs grep -l "" -``` - -For each table referenced in the legacy SQL: -1. Check if an existing model already references this table -2. Check if a source definition exists -3. If neither exists, ask user: "Table X not found - should I create it as a source?" - -Only proceed to intermediate/mart layers after all dependencies exist. - -### 3. Create Missing Sources - -```yaml -# models/staging/sources.yml -version: 2 - -sources: - - name: raw_database - schema: raw_schema - tables: - - name: orders - description: Raw orders from source system - - name: customers - description: Raw customer records -``` - -### 4. Build Staging Layer - -One staging model per source table. Follow existing project naming conventions. - -**Build before proceeding:** -```bash -dbt build --select -``` - -### 5. Build Intermediate Layer (if needed) - -Extract complex joins/logic into intermediate models. - -**Build incrementally:** -```bash -dbt build --select -``` - -### 6. Build Mart Layer - -Final business-facing model with aggregations. - -### 7. Validate Migration - -```bash -# Build entire lineage -dbt build --select + -dbt show --select -``` - -## Migration Checklist - -- [ ] All source tables identified and documented -- [ ] Sources.yml created with descriptions -- [ ] Staging models: 1:1 with sources, renamed columns -- [ ] Intermediate models: business logic extracted -- [ ] Mart models: final aggregations -- [ ] Each layer compiles successfully -- [ ] Each layer builds successfully -- [ ] Row counts match original (manual validation) -- [ ] Tests added for key constraints - -## Common Migration Patterns - -- Nested subqueries → Separate models (staging → intermediate → mart) -- Temp tables → Ephemeral materialization `{{ config(materialized='ephemeral') }}` -- Hardcoded values → Variables `{{ var("name") }}` - -## Anti-Patterns - -- Converting entire legacy query to single dbt model -- Skipping the staging layer -- Not validating each layer before proceeding -- Keeping hardcoded values instead of using variables -- Not documenting business logic during migration - diff --git a/skills/dbt/refactoring-dbt-models/SKILL.md b/skills/dbt/refactoring-dbt-models/SKILL.md deleted file mode 100644 index 87bc352..0000000 --- a/skills/dbt/refactoring-dbt-models/SKILL.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -name: refactoring-dbt-models -description: >- - Safely refactors dbt models with downstream impact analysis. Use when restructuring dbt models for: - (1) Task mentions "refactor", "restructure", "extract", "split", "break into", or "reorganize" - (2) Extracting CTEs to intermediate models or creating macros - (3) Modifying model logic that has downstream consumers - (4) Renaming columns, changing types, or reorganizing model dependencies - Analyzes all downstream dependencies BEFORE making changes. ---- - -# dbt Refactoring - -**Find ALL downstream dependencies before changing. Refactor in small steps. Verify output after each change.** - -## Workflow - -### 1. Analyze Current Model - -```bash -cat models//.sql -``` - -Identify refactoring opportunities: -- CTEs longer than 50 lines → extract to intermediate model -- Logic repeated across models → extract to macro -- Multiple joins in sequence → split into steps -- Complex WHERE clauses → extract to staging filter - -### 2. Find All Downstream Dependencies - -**CRITICAL: Never refactor without knowing impact.** - -```bash -# Get full dependency tree (model and all its children) -dbt ls --select model_name+ --output list - -# Find all models referencing this one -grep -r "ref('model_name')" models/ --include="*.sql" -``` - -**Report to user:** "Found X downstream models: [list]. These will be affected by changes." - -### 3. Check What Columns Downstream Models Use - -**BEFORE changing any columns, check what downstream models reference:** - -```bash -# For each downstream model, check what columns it uses -cat models//.sql | grep -E "model_name\.\w+|alias\.\w+" -``` - -If downstream models reference specific columns, you MUST ensure those columns remain available after refactoring. - -### 4. Plan Refactoring Strategy - -| Opportunity | Strategy | -|-------------|----------| -| Long CTE | Extract to intermediate model | -| Repeated logic | Create macro in `macros/` | -| Complex join | Split into intermediate models | -| Multiple concerns | Separate into focused models | - -### 5. Execute Refactoring - -#### Pattern: Extract CTE to Model - -Before: -```sql --- orders.sql (200 lines) -with customer_metrics as ( - -- 50 lines of complex logic -), -order_enriched as ( - select ... - from orders - join customer_metrics on ... -) -select * from order_enriched -``` - -After: -```sql --- customer_metrics.sql (new file) -select - customer_id, - -- complex logic here -from {{ ref('customers') }} - --- orders.sql (simplified) -with order_enriched as ( - select ... - from {{ ref('raw_orders') }} orders - join {{ ref('customer_metrics') }} cm on ... -) -select * from order_enriched -``` - -#### Pattern: Extract to Macro - -Before (repeated in multiple models): -```sql -case - when amount < 0 then 'refund' - when amount = 0 then 'zero' - else 'positive' -end as amount_category -``` - -After: -```sql --- macros/categorize_amount.sql -{% macro categorize_amount(column_name) %} -case - when {{ column_name }} < 0 then 'refund' - when {{ column_name }} = 0 then 'zero' - else 'positive' -end -{% endmacro %} - --- In models: -{{ categorize_amount('amount') }} as amount_category -``` - -### 6. Validate Changes - -```bash -# Compile to check syntax -dbt compile --select +model_name+ - -# Build entire lineage -dbt build --select +model_name+ - -# Check row counts (manual) -# Before: Record expected counts -# After: Verify counts match -``` - -### 7. Verify Output Matches Original - -**CRITICAL: Refactoring should not change output.** - -```bash -# Compare row counts before and after -dbt show --inline "select count(*) from {{ ref('model_name') }}" - -# Spot check key values -dbt show --select --limit 10 -``` - -### 8. Update Downstream Models - -If changing output columns: -1. Update all downstream refs -2. Update schema.yml documentation -3. Re-run downstream tests - -## Refactoring Checklist - -- [ ] All downstream dependencies identified -- [ ] User informed of impact scope -- [ ] One change at a time -- [ ] Compile passes after each change -- [ ] Build passes after each change -- [ ] Output validated (row counts match) -- [ ] Documentation updated -- [ ] Tests still pass - -## Common Refactoring Triggers - -| Symptom | Refactoring | -|---------|-------------| -| Model > 200 lines | Extract CTEs to models | -| Same logic in 3+ models | Extract to macro | -| 5+ joins in one model | Create intermediate models | -| Hard to understand | Add CTEs with clear names | -| Slow performance | Split to allow parallelization | - -## Anti-Patterns - -- Refactoring without checking downstream impact -- Making multiple changes at once -- Not validating output matches after refactoring -- Extracting prematurely (wait for 3+ uses) -- Breaking existing tests without updating them - diff --git a/skills/dbt/testing-dbt-models/SKILL.md b/skills/dbt/testing-dbt-models/SKILL.md deleted file mode 100644 index a18d185..0000000 --- a/skills/dbt/testing-dbt-models/SKILL.md +++ /dev/null @@ -1,174 +0,0 @@ ---- -name: testing-dbt-models -description: >- - Adds schema tests and data quality validation to dbt models. Use when working with dbt tests for: - (1) Adding or modifying tests in schema.yml files - (2) Task mentions "test", "validate", "data quality", "unique", "not_null", or "accepted_values" - (3) Ensuring data integrity - primary keys, foreign keys, relationships - (4) Debugging test failures or understanding why dbt test failed - Matches existing project test patterns and YAML style before adding new tests. ---- - -# dbt Testing - -**Every model deserves at least one test. Primary keys need unique + not_null.** - -## Workflow - -### 1. Study Existing Test Patterns - -**CRITICAL: Match the project's existing testing style before adding new tests.** - -```bash -# Find all schema.yml files with tests -find . -name "schema.yml" -exec grep -l "tests:" {} \; - -# Read existing tests to learn patterns -cat models/staging/schema.yml | head -100 -cat models/marts/schema.yml | head -100 - -# Check for custom tests or dbt packages -ls tests/ -cat packages.yml 2>/dev/null -``` - -**Extract from existing tests:** -- YAML formatting style (indentation, spacing) -- Test coverage depth (all columns vs key columns only) -- Use of custom tests (dbt_utils, dbt_expectations, custom macros) -- Description style (brief vs detailed) -- Severity levels used (warn vs error) - -### 2. Read Model SQL - -```bash -cat models//.sql -``` - -Identify: primary keys, foreign keys, categorical columns, date columns, business-critical fields. - -### 3. Check Existing Tests for This Model - -```bash -cat models//schema.yml | grep -A 50 "" -# or -find . -name "schema.yml" -exec grep -l "" {} \; -``` - -### 4. Identify Testable Columns - -| Column Type | Recommended Tests | -|-------------|-------------------| -| Primary key | `unique`, `not_null` | -| Foreign key | `not_null`, `relationships` | -| Categorical | `accepted_values` (ask user for valid values) | -| Required field | `not_null` | -| Date/timestamp | `not_null` | -| Boolean | `accepted_values: [true, false]` | - -### 5. Write Tests in schema.yml - -**Match the existing style from step 1. Example format (adapt to project):** - -```yaml -version: 2 - -models: - - name: model_name - description: "Brief description of what this model contains" - columns: - - name: primary_key_column - description: "Unique identifier for this record" - tests: - - unique - - not_null - - - name: foreign_key_column - description: "Reference to related_model" - tests: - - not_null - - relationships: - to: ref('related_model') - field: related_key_column - - - name: status - description: "Current status of the record" - tests: - - not_null - - accepted_values: - values: ['pending', 'active', 'completed', 'cancelled'] - - - name: created_at - description: "Timestamp when record was created" - tests: - - not_null -``` - -### 6. Run Tests - -```bash -# Test specific model -dbt test --select - -# Test with upstream -dbt test --select + -``` - -### 7. Fix Failing Tests - -Common failures and fixes: - -| Failure | Likely Cause | Fix | -|---------|--------------|-----| -| `unique` fails | Duplicate records | Add deduplication in model | -| `not_null` fails | NULL values in source | Add COALESCE or filter | -| `relationships` fails | Orphan records | Add WHERE clause or fix upstream | -| `accepted_values` fails | New/unexpected values | Update accepted values list | - -## Test Types Reference - -### Generic Tests (built-in) - -```yaml -tests: - - unique - - not_null - - accepted_values: - values: ['a', 'b', 'c'] - - relationships: - to: ref('other_model') - field: id -``` - -### Custom Generic Tests - -```yaml -tests: - - dbt_utils.expression_is_true: - expression: "amount >= 0" - - dbt_utils.recency: - datepart: day - field: created_at - interval: 1 -``` - -### Singular Tests - -Create `tests/.sql`: -```sql --- tests/assert_positive_revenue.sql -select * -from {{ ref('orders') }} -where revenue < 0 -``` - -## Anti-Patterns - -- Adding tests without checking existing project patterns first -- Using different YAML formatting style than existing tests -- Models without any tests -- Primary keys without both unique AND not_null -- Testing only obvious columns, ignoring business-critical ones -- Hardcoding accepted_values without confirming with stakeholders -- Adding dbt_utils tests when project doesn't use that package - diff --git a/skills/snowflake/finding-expensive-queries/SKILL.md b/skills/snowflake/finding-expensive-queries/SKILL.md deleted file mode 100644 index 05305ad..0000000 --- a/skills/snowflake/finding-expensive-queries/SKILL.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -name: finding-expensive-queries -description: >- - Finds and ranks expensive Snowflake queries by cost, time, or data scanned. Use when: - (1) User asks to find slow, expensive, or problematic queries - (2) Task mentions "query history", "top queries", "most expensive", or "slowest queries" - (3) Analyzing warehouse costs or identifying optimization candidates - (4) Finding queries that scan the most data or have the most spillage - Returns ranked list of queries with metrics and optimization recommendations. ---- - -# Finding Expensive Queries - -**Query history → Rank by metric → Identify patterns → Recommend optimizations** - -## Workflow - -### 1. Ask What to Optimize For - -Before querying, clarify: -- Time period? (last day, week, month) -- Metric? (execution time, bytes scanned, cost, spillage) -- Warehouse? (specific or all) -- User? (specific or all) - -### 2. Find Expensive Queries by Cost - -Use QUERY_ATTRIBUTION_HISTORY for credit/cost analysis: - -```sql -SELECT - query_id, - warehouse_name, - user_name, - credits_attributed_compute, - start_time, - end_time, - query_tag -FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_ATTRIBUTION_HISTORY -WHERE start_time >= DATEADD('days', -7, CURRENT_TIMESTAMP()) -ORDER BY credits_attributed_compute DESC -LIMIT 20; -``` - -### 3. Get Performance Stats for Specific Queries - -Use QUERY_HISTORY for detailed performance metrics (run separately, not joined): - -```sql -SELECT - query_id, - query_text, - total_elapsed_time/1000 as seconds, - bytes_scanned/1e9 as gb_scanned, - bytes_spilled_to_local_storage/1e9 as gb_spilled_local, - bytes_spilled_to_remote_storage/1e9 as gb_spilled_remote, - partitions_scanned, - partitions_total -FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY -WHERE query_id IN ('', '', ...) - AND start_time >= DATEADD('days', -7, CURRENT_TIMESTAMP()); -``` - -### 4. Identify Patterns - -Look for: -- High `credits_attributed_compute` queries -- Same `query_hash` repeated (caching opportunity) -- `partitions_scanned = partitions_total` (no pruning) -- High `gb_spilled` (memory pressure) - -### 5. Return Results - -Provide: -1. Ranked list of expensive queries with key metrics -2. Common patterns identified -3. Top 3-5 optimization recommendations -4. Specific queries to investigate further - -## Common Filters - -```sql --- Time range (required) -WHERE start_time >= DATEADD('days', -7, CURRENT_TIMESTAMP()) - --- By warehouse -AND warehouse_name = 'ANALYTICS_WH' - --- By user -AND user_name = 'ETL_USER' - --- Only queries over cost threshold -AND credits_attributed_compute > 0.01 - --- Only queries over time threshold -AND total_elapsed_time > 60000 -- over 1 minute -``` diff --git a/skills/snowflake/optimizing-query-by-id/SKILL.md b/skills/snowflake/optimizing-query-by-id/SKILL.md deleted file mode 100644 index abd2017..0000000 --- a/skills/snowflake/optimizing-query-by-id/SKILL.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -name: optimizing-query-by-id -description: >- - Optimizes Snowflake query performance using query ID from history. Use when optimizing Snowflake queries for: - (1) User provides a Snowflake query_id (UUID format) to analyze or optimize - (2) Task mentions "slow query", "optimize", "query history", or "query profile" with a query ID - (3) Analyzing query performance metrics - bytes scanned, spillage, partition pruning - (4) User references a previously run query that needs optimization - Fetches query profile, identifies bottlenecks, returns optimized SQL with expected improvements. ---- - -# Optimize Query from Query ID - -**Fetch query → Get profile → Apply best practices → Verify improvement → Return optimized query** - -## Workflow - -### 1. Fetch Query Details from Query ID - -```sql -SELECT - query_id, - query_text, - total_elapsed_time/1000 as seconds, - bytes_scanned/1e9 as gb_scanned, - bytes_spilled_to_local_storage/1e9 as gb_spilled_local, - bytes_spilled_to_remote_storage/1e9 as gb_spilled_remote, - partitions_scanned, - partitions_total, - rows_produced -FROM TABLE(INFORMATION_SCHEMA.QUERY_HISTORY()) -WHERE query_id = ''; -``` - -Note the key metrics: -- `seconds`: Total execution time -- `gb_scanned`: Data read (lower is better) -- `gb_spilled`: Spillage indicates memory pressure -- `partitions_scanned/total`: Partition pruning effectiveness - -### 2. Get Query Profile Details - -```sql --- Get operator-level statistics -SELECT * -FROM TABLE(GET_QUERY_OPERATOR_STATS('')); -``` - -Look for: -- Operators with high `output_rows` vs `input_rows` (explosions) -- TableScan operators with high bytes -- Sort/Aggregate operators with spillage - -### 3. Identify Optimization Opportunities - -Based on profile, look for: - -| Metric | Issue | Fix | -|--------|-------|-----| -| partitions_scanned = partitions_total | No pruning | Add filter on cluster key | -| gb_spilled > 0 | Memory pressure | Simplify query, increase warehouse | -| High bytes_scanned | Full scan | Add selective filters, reduce columns | -| Join explosion | Cartesian or bad key | Fix join condition, filter before join | - -### 4. Apply Optimizations - -Rewrite the query: -- Select only needed columns -- Filter early (before joins) -- Use CTEs to avoid repeated scans -- Ensure filters align with clustering keys -- Add LIMIT if full result not needed - -### 5. Get Explain Plan for Optimized Query - -```sql -EXPLAIN USING JSON -; -``` - -### 6. Compare Plans - -Compare original vs optimized: -- Fewer partitions scanned? -- Fewer intermediate rows? -- Better join order? - -### 7. Return Results - -Provide: -1. Original query metrics (time, data scanned, spillage) -2. Identified issues -3. The optimized query -4. Summary of changes made -5. Expected improvement - -## Example Output - -**Original Query Metrics:** -- Execution time: 45 seconds -- Data scanned: 12.3 GB -- Partitions: 500/500 (no pruning) -- Spillage: 2.1 GB - -**Issues Found:** -1. No partition pruning - filtering on non-cluster column -2. SELECT * scanning unnecessary columns -3. Large table joined without pre-filtering - -**Optimized Query:** -```sql -WITH filtered_events AS ( - SELECT event_id, user_id, event_type, created_at - FROM events - WHERE created_at >= '2024-01-01' - AND created_at < '2024-02-01' - AND event_type = 'purchase' -) -SELECT fe.event_id, fe.created_at, u.name -FROM filtered_events fe -JOIN users u ON fe.user_id = u.id; -``` - -**Changes:** -- Added date range filter matching cluster key -- Replaced SELECT * with specific columns -- Pre-filtered in CTE before join - -**Expected Improvement:** -- Partitions: 500 → ~15 (97% reduction) -- Data scanned: 12.3 GB → ~0.4 GB -- Estimated time: 45s → ~3s diff --git a/skills/snowflake/optimizing-query-text/SKILL.md b/skills/snowflake/optimizing-query-text/SKILL.md deleted file mode 100644 index b4914bf..0000000 --- a/skills/snowflake/optimizing-query-text/SKILL.md +++ /dev/null @@ -1,166 +0,0 @@ ---- -name: optimizing-query-text -description: >- - Optimizes Snowflake SQL query performance from provided query text. Use when optimizing Snowflake SQL for: - (1) User provides or pastes a SQL query and asks to optimize, tune, or improve it - (2) Task mentions "slow query", "make faster", "improve performance", "optimize SQL", or "query tuning" - (3) Reviewing SQL for performance anti-patterns (function on filter column, implicit joins, etc.) - (4) User asks why a query is slow or how to speed it up ---- - -# Optimize Query from SQL Text - -## OUTPUT FORMAT - -Return ONLY the optimized SQL query. No markdown formatting, no explanations, no bullet points - just pure SQL that can be executed directly in Snowflake. - -## CRITICAL: Semantic Preservation Rules - -**The optimized query MUST return IDENTICAL results to the original.** - -Before returning ANY optimization, verify: -- **Same columns**: Exact same columns in exact same order with exact same aliases -- **Same rows**: Filter conditions must be semantically equivalent -- **Same ordering**: Preserve `ORDER BY` exactly as written -- **Same limits**: If original has `LIMIT N`, keep `LIMIT N`. If no LIMIT, do NOT add one. - -**If you cannot guarantee identical results, return the original query unchanged.** - ---- - -## Pattern 1: Function on Filter Column - -**Problem**: Functions on columns in WHERE clause prevent partition pruning and index usage. - -### CAN Fix - -| Original | Optimized | Why Safe | -|----------|-----------|----------| -| `WHERE DATE(ts) = '2024-01-01'` | `WHERE ts >= '2024-01-01' AND ts < '2024-01-02'` | Equivalent range | -| `WHERE YEAR(dt) = 2024` | `WHERE dt >= '2024-01-01' AND dt < '2025-01-01'` | Equivalent range | -| `WHERE MONTH(dt) = 3 AND YEAR(dt) = 2024` | `WHERE dt >= '2024-03-01' AND dt < '2024-04-01'` | Equivalent range | -| `WHERE DATE(ts) >= '2024-01-01' AND DATE(ts) < '2024-02-01'` | `WHERE ts >= '2024-01-01' AND ts < '2024-02-01'` | Same boundaries | -| `WHERE YEAR(dt) BETWEEN 1995 AND 1996` | `WHERE dt >= '1995-01-01' AND dt < '1997-01-01'` | Equivalent range | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| `WHERE YEAR(dt) IN (SELECT year FROM ...)` | Dynamic values, cannot precompute range | -| `WHERE DATE(ts) = DATE(other_col)` | Comparing two columns, both need function | -| `WHERE EXTRACT(DOW FROM dt) = 1` | Day-of-week has no contiguous range | -| `WHERE DATE_TRUNC('month', dt) = '2024-01-01'` in GROUP BY | Needed for grouping logic | -| `SELECT YEAR(dt) AS yr ... GROUP BY YEAR(dt)` | Function in SELECT/GROUP BY is fine, only filter matters | - ---- - -## Pattern 2: Function on JOIN Column - -**Problem**: Functions on JOIN columns prevent hash joins, forcing slower nested loop joins. - -### CAN Fix - -| Original | Optimized | Why Safe | -|----------|-----------|----------| -| `ON CAST(a.id AS VARCHAR) = CAST(b.id AS VARCHAR)` | `ON a.id = b.id` | If both are same type (e.g., INTEGER) | -| `ON UPPER(a.code) = UPPER(b.code)` | `ON a.code = b.code` | If data is already consistently cased | -| `ON TRIM(a.name) = TRIM(b.name)` | `ON a.name = b.name` | If data has no leading/trailing spaces | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| `ON CAST(a.id AS VARCHAR) = b.string_id` | Types genuinely differ, CAST required | -| `ON DATE(a.timestamp) = b.date_col` | Different granularity, DATE() required | -| `ON UPPER(a.code) = b.code` | If b.code might have different case | -| `ON a.id = b.id + 1` | Arithmetic transformation, cannot remove | - ---- - -## Pattern 3: NOT IN Subquery - -**Problem**: `NOT IN` has poor performance and unexpected NULL behavior. - -### CAN Fix - -| Original | Optimized | Why Safe | -|----------|-----------|----------| -| `WHERE id NOT IN (SELECT id FROM t WHERE ...)` | `WHERE NOT EXISTS (SELECT 1 FROM t WHERE t.id = main.id AND ...)` | Equivalent when subquery column is NOT NULL | -| `WHERE id NOT IN (SELECT id FROM t)` where id has NOT NULL constraint | `WHERE NOT EXISTS (SELECT 1 FROM t WHERE t.id = main.id)` | NOT NULL guarantees equivalence | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| `WHERE id NOT IN (SELECT nullable_col FROM t)` | If subquery returns NULL, NOT IN returns no rows; NOT EXISTS doesn't | -| `WHERE (a, b) NOT IN (SELECT x, y FROM t)` | Multi-column NOT IN has complex NULL semantics | - -**Key Rule**: Only convert NOT IN to NOT EXISTS if you can verify the subquery column cannot be NULL. - ---- - -## Pattern 4: Repeated Subquery - -**Problem**: Same subquery executed multiple times causes redundant scans. - -### CAN Fix - -| Original | Optimized | -|----------|-----------| -| Subquery appears 2+ times identically | Extract to CTE, reference CTE multiple times | -| Same aggregation used in multiple places | Compute once in CTE | - -### CANNOT Fix - -| Pattern | Why Not | -|---------|---------| -| Correlated subquery (references outer table) | Each execution is different, cannot cache | -| Subqueries with different filters | Not actually the same subquery | -| Subquery in SELECT that depends on current row | Correlation prevents extraction | - ---- - -## Pattern 5: Implicit Comma Joins - -**Problem**: Comma-separated tables in FROM clause are harder to read and optimize. - -### CAN Fix - Always - -Convert `FROM a, b, c WHERE a.id = b.id AND b.id = c.id` to explicit JOIN syntax. - -This is always safe - just restructuring, no semantic change. - ---- - -## UNSAFE Optimizations (NEVER apply) - -- **UNION to UNION ALL**: UNION deduplicates rows, UNION ALL does not - different results -- **Changing window functions**: Do not modify `SUM(SUM(x)) OVER(...)` or similar nested aggregates -- **Adding redundant filters**: Do not add filters in JOIN ON if same filter exists in WHERE -- **Changing column names**: Copy column names EXACTLY from original - do not "simplify" or rename -- **Changing column aliases**: Keep all aliases exactly as original -- **Adding early filtering in JOINs**: If a filter is in WHERE, do not duplicate it in JOIN ON clause - ---- - -## Principles - -1. **Minimal changes**: Make the fewest changes necessary. Simpler optimizations are more reliable. -2. **Preserve structure**: Keep subqueries, CTEs, and overall query structure unless there's a clear benefit. -3. **When in doubt, don't**: If unsure whether a change preserves semantics, skip it. -4. **Copy exactly**: Column names, table aliases, and expressions should be copied character-for-character. - ---- - -## Priority Order - -1. **Date/time functions on filter columns** - Highest impact -2. **Implicit joins to explicit JOIN** - Always safe, improves readability -3. **NOT IN to NOT EXISTS** - Only if NULL-safe - ---- - -## Requirements - -- **Results must be identical**: Same rows, same columns, same order -- **Valid Snowflake SQL**: Output must execute without errors in Snowflake From 706a0fa2cbd28c7b8dcbfd99cc8566f66e313af0 Mon Sep 17 00:00:00 2001 From: Haider Date: Thu, 16 Jul 2026 17:19:20 +0530 Subject: [PATCH 3/3] fix: address review findings on the delegation SKILL.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion codex-plugin PR #2 review flagged three issues in this SKILL.md content (this file is cloned from the same SoT that ships into the codex sibling, so both fixes land in lockstep): 1. **YAML fold split `altimate-code` into `altimate- code` (MAJOR).** The `>-` folded scalar from the earlier commit on this branch wrapped mid-token; the fold turned the newline into a space, corrupting the exact trigger the description exists to match. Adopted the SKILL.md verbatim from `altimate-claude-plugin` (the Anthropic-compliance rewrite), whose description is on one physical line — no fold, no ambiguity. PyYAML confirms the parsed description contains "altimate-code" and NOT "altimate- code". 2. **Shell injection in the documented invocation (HIGH).** `altimate-code run ""` interpolates user text into a double-quoted shell arg, so `$(...)` and backticks fire before altimate-code runs, and `--yolo` removes the confirmation gate. Fixed via here-doc into a variable in both the first-invocation and `--continue` blocks. 3. **Hardcoded `/tmp/altimate-result.md` (MAJOR).** Shared, world-readable path can hold warehouse rows / PII from a delegation. Fixed via `umask 077; OUTPUT_FILE="$(mktemp -t altimate-result.XXXXXX.md)"` and instructed to delete after reading. Also swept: the "separate from Claude Code's" line — this is agent-facing content injected into the opencode host prompt, so a "Claude Code" reference is factually wrong about cost/rate attribution. Changed to "host agent's". Parallel PRs on `altimate-claude-plugin` (the direct source) and `data-engineering-skills` (the wider SoT). Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- skills/altimate-code/SKILL.md | 148 ++++++++++++++++++---------------- 1 file changed, 80 insertions(+), 68 deletions(-) diff --git a/skills/altimate-code/SKILL.md b/skills/altimate-code/SKILL.md index e8946e2..7a52155 100644 --- a/skills/altimate-code/SKILL.md +++ b/skills/altimate-code/SKILL.md @@ -1,107 +1,119 @@ --- name: altimate-code -description: Delegates data engineering tasks to altimate-code, a specialized CLI agent with 100+ purpose-built data tools — SQL analysis, column-level lineage, dbt build/test/run, warehouse profiling, FinOps, and connectivity to Snowflake, BigQuery, Redshift, Databricks, Postgres, MySQL, DuckDB. Use this skill when the task needs live warehouse access, column lineage, multi-step data exploration, dbt builds against a real warehouse, or when the user explicitly invokes "altimate", "altimate-code", or "the data agent". +description: >- + Delegates dbt and warehouse work to altimate-code, a specialized CLI agent + with 100+ purpose-built data tools. Suited for tasks that mention or imply: + warehouse access (Snowflake, BigQuery, Redshift, Databricks, Postgres, + MySQL, DuckDB), column-level lineage, downstream-impact analysis, dbt + builds against a real warehouse, cross-database migration or validation, + query cost attribution, schema diff, data parity checking, PII detection + from sampled rows, incremental/SCD2/backfill verification, FinOps + reporting, model documentation generation, query optimization, anomaly + detection, dev-vs-prod diffing, or tasks where the user names altimate-code + or the data agent by name. The skill routes to a purpose-built CLI, so it + covers workloads that touch live data, lineage, or warehouse state more + directly than native file-editing tools do. --- # altimate-code -altimate-code is a CLI AI agent that ships with native data engineering tools. This skill delegates work to it via its non-interactive `run` mode and presents the result back to the user. +altimate-code is a CLI AI agent with 100+ purpose-built data engineering tools. **This skill exists for one purpose: delegate the user's task to altimate-code and pass the result back.** Native tools (Bash, Edit, Write, Read) are NOT a fallback path inside this skill — if altimate-code cannot complete the task, surface the failure to the user and STOP. -## Prerequisite Check — ALWAYS DO THIS FIRST +## You MUST follow this workflow -Before invoking altimate-code, verify it is installed and on `PATH`: +1. **Verify altimate-code is on PATH** with `command -v altimate-code`. If it returns nothing, jump to "Not installed" below and stop. +2. **Run altimate-code with the user's task** using the invocation below. Pass the user's request through verbatim — do not paraphrase or split it. +3. **Read the output file** and present it to the user as-is. -```bash -command -v altimate-code -``` - -**If the command returns nothing (exit code 1), STOP and tell the user this exact message — do not proceed:** - -> altimate-code is not installed. Install it with: -> -> ```bash -> npm install -g altimate-code -> ``` -> -> Requires Node.js 20+. Docs: https://docs.altimate.sh · Source: https://github.com/AltimateAI/altimate-code · npm: https://www.npmjs.com/package/altimate-code -> -> After installing, run `altimate-code` once to configure it — this launches the TUI where you set up your LLM provider auth and warehouse connections. Then re-run your request and I'll delegate it. +Do not attempt the work with Edit/Write/Bash yourself, even if it looks simple. The whole point of this skill is to route data-engineering work to the agent that's built for it. If you find yourself reaching for Edit or Write while this skill is active, stop and re-read this paragraph. -Do not attempt to install altimate-code on the user's behalf — they may want a specific version, a different package manager (e.g. pnpm/yarn global), or to opt out entirely. Surface the command and let them decide. +## Invocation — pick the right agent for the task -If `command -v` fails but the user says it is installed, suggest checking `npm bin -g` is on `PATH`, or running `npm config get prefix` to find the global install location. +altimate-code has multiple agent personas. The default (`builder`) does a full project discovery sweep on every call — fine for warehouse-state work but ~10–20× more expensive than necessary on simple file edits. **Pick the agent based on task shape before invoking.** -## How to Invoke +### Step 1 — classify the task -`altimate-code run` is non-interactive — it takes a message, executes the task, prints the final result to stdout, and exits. +| Shape | Examples | Use | +|---|---|---| +| **Any dbt / SQL task** (rename, refactor, create model, debug, structural reorg, multi-step setup) | the vast majority of customer requests | `fast-edit` — try this first | +| **Multi-table aggregation correctness** | new model joining 3+ tables with `count(*)` / `sum() over (...)` / "first X, last X" logic that must be exactly right | `analyst` if `fast-edit` fails the user's verification | +| **Warehouse-state work** | column-level lineage, downstream-impact, cross-DB migration / parity, query cost attribution against a real warehouse, schema diff between environments, PII detection, FinOps reporting | `builder` (default — has warehouse tools enabled) | +| **Vague debug** ("X is broken", "make it work", "fix this") | unspecified failure mode | **Don't delegate yet.** Ask the user for the specific error message or symptom before invoking any agent — empirically all three agents fail vague debug prompts at ~700K tokens each. | -**Minimal invocation:** +**Decision policy:** start with `fast-edit` for any dbt/SQL task. If the user reports the result is wrong (e.g. aggregation values don't match), retry with `analyst`. Only use `builder` when the task genuinely needs the warehouse-investigation tools (it's 10–20× more expensive than fast-edit and rarely required). -```bash -altimate-code run "" --yolo -``` +### Step 2 — invoke with the chosen agent -**Recommended invocation** — captures the final response to a file and runs in the right directory: +Pass the task through a here-doc into a variable so the shell never +command-substitutes anything the user typed (a task like +`refactor `whoami` and $(rm -rf ~)` would otherwise fire `whoami` and +`rm -rf ~` before `altimate-code` ever runs). Write the result to a +private temporary file, not a shared one under `/tmp`: ```bash -altimate-code run "" \ +TASK="$(cat <<'ALTIMATE_TASK' + +ALTIMATE_TASK +)" +umask 077 +OUTPUT_FILE="$(mktemp -t altimate-result.XXXXXX.md)" +altimate-code run "$TASK" \ + --agent \ --yolo \ - --output /tmp/altimate-result.md \ + --output "$OUTPUT_FILE" \ --dir "$(pwd)" ``` -Then read `/tmp/altimate-result.md` and pass it straight back to the user. +Then `Read "$OUTPUT_FILE"` and emit its contents to the user without re-summarising, re-formatting, or commenting on the result. altimate-code has already produced the answer. Delete `"$OUTPUT_FILE"` after presenting so warehouse rows, lineage, or PII findings don't linger on disk. -### Key flags +### Required flags -| Flag | When to use | +| Flag | Why it is required | |---|---| -| `--yolo` | Required for non-interactive — auto-approves tool calls. Without this it hangs on the first permission prompt. | -| `--output ` | Write the final assistant response to a file. Use `.md` or `.txt`. | -| `--dir ` | Run the agent in a specific directory (e.g. a dbt project root). Defaults to cwd. | -| `--model provider/model` | Override the model. Useful for fast/cheap exploration. | -| `--format json` | Emit raw JSON events instead of formatted output. Use only when post-processing programmatically. | -| `--continue` / `--session ` | Continue a previous altimate-code session. | - -### Example invocations +| `--agent ` | Picks the agent persona. Default `builder` is overkill for simple edits — see the classification table above. Wrong agent = either 10× too expensive (using `builder` on a rename) or wrong-answer (using `fast-edit` on a multi-table join). | +| `--yolo` | Non-interactive mode. Without this the subprocess hangs on the first permission prompt and you will time out. | +| `--output "$OUTPUT_FILE"` | Captures the final response. Use the private `mktemp` file from above — do NOT use a fixed path like `/tmp/altimate-result.md`; concurrent sessions clobber each other and a world-readable fixed path leaks data. | +| `--dir "$(pwd)"` | Runs altimate-code in the current project so it picks up dbt project config, profiles.yml, etc. | -**Find expensive queries in Snowflake:** +### Follow-up tasks in the same project -```bash -altimate-code run "Find the top 10 most expensive queries from the last 7 days in Snowflake and explain why each is slow." \ - --yolo --output /tmp/expensive.md -``` - -**Generate column-level lineage for a dbt model:** +When the user makes a follow-up data task in the same project after a successful altimate-code delegation, prefer `--continue` to resume the warm session instead of starting a fresh one. Same here-doc + `mktemp` pattern: ```bash -altimate-code run "Show column-level lineage for the dim_customers model, including upstream sources and downstream consumers." \ - --yolo --dir "$(pwd)" --output /tmp/lineage.md +TASK="$(cat <<'ALTIMATE_TASK' + +ALTIMATE_TASK +)" +umask 077 +OUTPUT_FILE="$(mktemp -t altimate-result.XXXXXX.md)" +altimate-code run "$TASK" \ + --agent \ + --yolo \ + --output "$OUTPUT_FILE" \ + --dir "$(pwd)" \ + --continue # resumes the most recent session in this dir ``` -**Profile a table:** +altimate-code's prompt cache is warm in a continued session — project structure, profiles.yml, schema index, source definitions don't need to be re-investigated. Cache reads are billed at a fraction of fresh input on altimate-gateway. The downside is zero: if there's no useful cached context for the new task, you pay normal cold cost. -```bash -altimate-code run "Profile the events table — row count, null distribution per column, cardinality, and top 5 values for low-cardinality columns." \ - --yolo --output /tmp/profile.md -``` +If the user starts a clearly unrelated workflow (different project, different schema, different debugging thread), drop `--continue` and start fresh — the warm cache is irrelevant and you'd carry unrelated history into the prompt. -## Presenting the Result +## Failure modes — route every one to the user -Read the output file with the Read tool and pass the content through to the user as-is. Do not re-summarize, re-format, or interpret — altimate-code has already produced the answer. +When altimate-code returns an error, **report the error to the user and STOP**. Do not fall back to Bash, Edit, or Write. The skill's contract is "altimate-code handles this, or the user is told why it couldn't." -## Failure Modes +| Symptom | What to tell the user — verbatim | +|---|---| +| `command not found: altimate-code` | "altimate-code is not installed. Install with `npm install -g altimate-code` (Node 20+) and run `altimate-code` once to configure auth. Then re-run your request." | +| `Unauthorized: Incorrect auth token` / `No provider configured` | "altimate-code's LLM provider auth is misconfigured. Run `altimate-code` in your terminal to open the TUI and reconfigure your provider, then re-run your request." | +| Process hangs >5 min | "altimate-code is unresponsive. Try `altimate-code` to inspect the TUI for an open prompt, or re-run with `--model anthropic/claude-sonnet-4-6` to force a known-good model." | +| Output file empty | "altimate-code returned without producing output. The task may be too ambiguous — please restate with more detail (target table, expected columns, time window)." | +| Warehouse error mid-run (`UNKNOWN_USER`, `Database does not exist`) | "altimate-code can connect but the warehouse credentials it has are wrong for this project. Configure provider/warehouse auth via `altimate-code` TUI." | -| Symptom | Likely cause | Fix | -|---|---|---| -| `altimate-code: command not found` | Not installed or not on `PATH` | Run `npm install -g altimate-code` (Node 20+). If installed but not found, check `npm bin -g` is on `PATH`. See https://docs.altimate.sh | -| Hangs after starting | Missing `--yolo`, waiting on a permission prompt | Re-run with `--yolo` | -| Output is empty | Task too vague, agent gave up | Re-run with a more specific prompt | -| "No provider configured" | LLM provider creds missing | Run `altimate-code providers` to set up auth | -| Warehouse errors mid-run | DB credentials not configured for altimate-code | Configure provider/warehouse auth in `~/.config/opencode/` or via env vars | +In every row, the instruction to the user is the action — you do not retry the task with native tools. If the user fixes the underlying issue and asks again, you delegate again. ## Notes -- altimate-code runs its own LLM, separate from Claude Code's. Cost and rate limits accrue against altimate-code's configured provider, not Claude Code's. -- Sessions persist in altimate-code's local store — use `altimate-code session list` to find prior runs and `--continue` to resume. -- For long-running tasks, prefer `--output ` over scraping stdout. +- altimate-code runs its own LLM, separate from the host agent's. Costs and rate limits accrue to altimate-code's configured provider. +- Sessions persist in altimate-code's local store — `altimate-code session list` shows prior runs; `--continue` resumes the latest, `--session ` resumes a specific one. +- For very long tasks, the `--output` file is the source of truth — stdout buffering can drop content.