docs: update CLAUDE.md, skills, and add PostgreSQL grammar reference files

.claude/skills/fix_bug/SKILL.md

@@ -78,6 +78,7 @@ End-to-end workflow for fixing bugs reported as GitHub issues. Uses TDD: reprodu
    - `old.sql` - The starting schema state (current database)
    - `new.sql` - The desired schema state (user's SQL files)
    - Leave `diff.sql` empty or with expected content — it will be generated
+   - Note: `plan.json`, `plan.sql`, `plan.txt` will also be generated alongside `diff.sql`
 
 3. **Run the diff test to confirm it fails** (red):
    ```bash
@@ -88,6 +89,7 @@ End-to-end workflow for fixing bugs reported as GitHub issues. Uses TDD: reprodu
    ```bash
    PGSCHEMA_TEST_FILTER="<category>/issue_<NUMBER>_<short_description>" go test -v ./cmd -run TestPlanAndApply --generate
    ```
+   This overwrites `diff.sql`, `plan.json`, `plan.sql`, and `plan.txt` with actual output.
 
 ### Phase 3: Implement the Fix (Green)
 

.claude/skills/postgres_syntax/SKILL.md

@@ -19,14 +19,18 @@ Invoke this skill when:
 
 ## Source Code Locations
 
-**Main parser directory**: https://github.com/postgres/postgres/blob/master/src/backend/parser/
+**Local copies** (preferred - read these directly):
+- `internal/gram.y` - **Main grammar file** - Yacc/Bison grammar defining PostgreSQL SQL syntax
+- `internal/scan.l` - Lexical scanner (Flex/Lex) - tokenization rules
+
+**Upstream reference**: https://github.com/postgres/postgres/blob/master/src/backend/parser/
 
 **Key files to reference**:
 
 ### Grammar and Lexer
-- `gram.y` - **Main grammar file** - Yacc/Bison grammar defining PostgreSQL SQL syntax
-- `scan.l` - Lexical scanner (Flex/Lex) - tokenization rules
-- `keywords.c` - Reserved and non-reserved keywords
+- `internal/gram.y` (local) - **Main grammar file** - Yacc/Bison grammar defining PostgreSQL SQL syntax
+- `internal/scan.l` (local) - Lexical scanner (Flex/Lex) - tokenization rules
+- `keywords.c` (upstream) - Reserved and non-reserved keywords
 
 ### Parser Implementation
 - `parse_clause.c` - Parsing of clauses (WHERE, GROUP BY, ORDER BY, etc.)
@@ -64,12 +68,12 @@ Determine what kind of SQL you're working with:
 
 ### 2. Locate the Grammar Rule in gram.y
 
-Search gram.y for the statement's production rule:
+Search the local gram.y for the statement's production rule:
 
 **Example - Finding CREATE TRIGGER syntax**:
 ```bash
-# In the postgres repository
-grep -n "CreateTrigStmt:" src/backend/parser/gram.y
+# Search the local copy
+grep -n "CreateTrigStmt:" internal/gram.y
 ```
 
 **What to look for**:
@@ -493,25 +497,34 @@ After consulting gram.y and implementing in pgschema:
 
 ## Quick Reference
 
-**Finding syntax in gram.y**:
+**Finding syntax in gram.y** (use local copy):
 ```bash
 # Search for statement type
-grep -n "CreateTrigStmt:" src/backend/parser/gram.y
+grep -n "CreateTrigStmt:" internal/gram.y
 
 # Find keyword definitions
-grep -n "^TRIGGER" src/backend/parser/gram.y
+grep -n "^TRIGGER" internal/gram.y
 
 # Understand an option
-grep -A 10 "TriggerWhen:" src/backend/parser/gram.y
+grep -A 10 "TriggerWhen:" internal/gram.y
+```
+
+**Finding lexer rules in scan.l** (use local copy):
+```bash
+# Search for token patterns
+grep -n "identifier" internal/scan.l
+
+# Find keyword handling
+grep -n "ScanKeywordLookup" internal/scan.l
 ```
 
 **Understanding precedence**:
 ```bash
 # Look at top of gram.y
-head -100 src/backend/parser/gram.y | grep -A 50 "%left\|%right\|%nonassoc"
+head -100 internal/gram.y | grep -A 50 "%left\|%right\|%nonassoc"
 ```
 
-**Find utility command handling**:
+**Find utility command handling** (upstream):
 ```bash
 grep -n "transformCreateTrigStmt" src/backend/parser/parse_utilcmd.c
 ```

.claude/skills/run_tests/SKILL.md

@@ -33,7 +33,7 @@ go test -v ./internal/diff -run TestDiffFromFiles
 **What it tests**:
 - Compares `old.sql` vs `new.sql` from `testdata/diff/`
 - Generates migration DDL
-- Validates against `expected.sql`
+- Validates against `diff.sql`
 - Pure logic testing - no database required
 
 **Speed**: Very fast (~1-2 seconds)
@@ -97,22 +97,22 @@ PGSCHEMA_TEST_FILTER="create_trigger/add_trigger_when_distinct" go test -v ./cmd
 ```
 
 **Test filter paths** (from `testdata/diff/`):
-- `comment/` - Comment operations (10 test cases)
-- `create_domain/` - Domain types (3 test cases)
-- `create_function/` - Functions (5 test cases)
+- `comment/` - Comment operations (11 test cases)
+- `create_domain/` - Domain types (5 test cases)
+- `create_function/` - Functions (8 test cases)
 - `create_index/` - Indexes (2 test cases)
 - `create_materialized_view/` - Materialized views (3 test cases)
 - `create_policy/` - RLS policies (10 test cases)
 - `create_procedure/` - Procedures (3 test cases)
 - `create_sequence/` - Sequences (3 test cases)
-- `create_table/` - Tables (30 test cases)
+- `create_table/` - Tables (37 test cases)
 - `create_trigger/` - Triggers (7 test cases)
 - `create_type/` - Custom types (3 test cases)
 - `create_view/` - Views (4 test cases)
-- `default_privilege/` - Default privileges (8 test cases)
-- `privilege/` - Privileges/permissions (10 test cases)
-- `dependency/` - Dependencies (8 test cases)
-- `online/` - Online migrations (12 test cases)
+- `default_privilege/` - Default privileges (9 test cases)
+- `privilege/` - Privileges/permissions (13 test cases)
+- `dependency/` - Dependencies (13 test cases)
+- `online/` - Online migrations (14 test cases)
 - `migrate/` - Complex migrations (5 test cases)
 
 ### Workflow 2: Regenerate Expected Output
@@ -132,7 +132,7 @@ PGSCHEMA_TEST_FILTER="create_trigger/add_trigger" go test -v ./cmd -run TestPlan
 
 **What `--generate` does**:
 - Runs the test normally
-- Overwrites `expected.sql` with actual generated output
+- Overwrites `diff.sql`, `plan.json`, `plan.sql`, and `plan.txt` with actual output
 - Use when you've intentionally changed how DDL is generated
 - **Warning**: Only use when you're sure the new output is correct!
 
@@ -143,10 +143,10 @@ PGSCHEMA_TEST_FILTER="create_trigger/add_trigger" go test -v ./cmd -run TestPlan
 - Changed normalization logic
 
 **Verification steps after `--generate`**:
-1. Review the diff in git: `git diff testdata/diff/path/to/test/expected.sql`
+1. Review the diff in git: `git diff testdata/diff/path/to/test/`
 2. Ensure changes are intentional and correct
 3. Run test again without `--generate` to verify it passes
-4. Commit the updated expected.sql
+4. Commit the updated `diff.sql` and plan files
 
 ### Workflow 3: Test Across PostgreSQL Versions
 
@@ -219,7 +219,7 @@ cat testdata/diff/create_trigger/add_trigger/old.sql
 cat testdata/diff/create_trigger/add_trigger/new.sql
 
 # View expected migration
-cat testdata/diff/create_trigger/add_trigger/expected.sql
+cat testdata/diff/create_trigger/add_trigger/diff.sql
 ```
 
 3. **Run with debugger** (optional):
@@ -254,15 +254,18 @@ Located in `testdata/diff/<category>/<test_name>/`:
 testdata/diff/create_trigger/add_trigger/
 ├── old.sql       # Starting schema state
 ├── new.sql       # Desired schema state
-└── expected.sql  # Expected migration DDL
+├── diff.sql      # Expected migration DDL
+├── plan.json     # Expected plan in JSON format
+├── plan.sql      # Expected plan as SQL statements
+└── plan.txt      # Expected plan as human-readable text
 ```
 
 **Test process**:
 1. Apply `old.sql` to embedded PostgreSQL and inspect into IR
 2. Apply `new.sql` to embedded PostgreSQL and inspect into IR
 3. Diff the two IRs
 4. Generate migration DDL
-5. Compare with `expected.sql`
+5. Compare with `diff.sql`
 
 ### Integration Test Structure
 
@@ -324,16 +327,17 @@ CREATE TRIGGER my_trigger
 EOF
 ```
 
-### Step 4: Generate expected.sql
+### Step 4: Generate diff.sql and plan files
 
-**Option A: Use --generate flag**:
+**Option A: Use --generate flag** (recommended):
 ```bash
 PGSCHEMA_TEST_FILTER="create_trigger/add_trigger_new_feature" go test -v ./cmd -run TestPlanAndApply --generate
 ```
+This generates `diff.sql`, `plan.json`, `plan.sql`, and `plan.txt`.
 
-**Option B: Manually create**:
+**Option B: Manually create diff.sql**:
 ```bash
-cat > testdata/diff/create_trigger/add_trigger_new_feature/expected.sql << 'EOF'
+cat > testdata/diff/create_trigger/add_trigger_new_feature/diff.sql << 'EOF'
 CREATE TRIGGER my_trigger
     BEFORE INSERT ON test_table
     FOR EACH ROW
@@ -446,11 +450,11 @@ PGSCHEMA_TEST_FILTER="create_trigger/" go test -v ./cmd -run TestPlanAndApply -t
     CREATE TRIGGER my_trigger AFTER INSERT ON test_table
 ```
 
-**What this means**: The generated migration DDL doesn't match expected.sql
+**What this means**: The generated migration DDL doesn't match diff.sql
 
 **How to fix**:
 1. Check if the actual output is correct
-2. If correct: Update expected.sql (or use `--generate`)
+2. If correct: Update diff.sql (or use `--generate`)
 3. If incorrect: Fix the diff logic in `internal/diff/trigger.go`
 
 ### Integration Test Failure
@@ -519,7 +523,7 @@ git status
 # Create test case
 mkdir -p testdata/diff/create_feature/test_name
 
-# Add old.sql, new.sql, expected.sql
+# Add old.sql, new.sql, then use --generate for diff.sql
 
 # Test
 PGSCHEMA_TEST_FILTER="create_feature/test_name" go test -v ./internal/diff -run TestDiffFromFiles
@@ -542,7 +546,7 @@ PGSCHEMA_POSTGRES_VERSION=17 PGSCHEMA_TEST_FILTER="create_feature/" go test -v .
 ```bash
 # Add test case that reproduces bug
 mkdir -p testdata/diff/category/bug_reproduction
-# Add old.sql, new.sql, expected.sql
+# Add old.sql, new.sql, then use --generate for diff.sql
 
 # Verify it fails
 PGSCHEMA_TEST_FILTER="category/bug_reproduction" go test -v ./internal/diff -run TestDiffFromFiles
@@ -595,7 +599,7 @@ Before committing changes:
 - [ ] Ran diff tests for affected areas
 - [ ] Ran integration tests for affected areas
 - [ ] Tests pass on at least one PostgreSQL version
-- [ ] If intentionally changed DDL, updated expected.sql files
+- [ ] If intentionally changed DDL, updated diff.sql and plan files
 - [ ] New features have test coverage
 - [ ] Bug fixes have regression tests
 - [ ] No unintended test file modifications

CLAUDE.md

@@ -11,11 +11,12 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - **Plan**: Compare desired state with current database and generate migration plan
 - **Apply**: Execute the migration with safety features like concurrent change detection, transaction-adaptive execution, and lock timeout control
 
-The tool is written in Go 1.24+ (toolchain go1.24.7) and uses:
+The tool is written in Go 1.24.0 (toolchain go1.24.7) and uses:
 
 - Cobra for CLI commands
 - embedded-postgres v1.33.0 for plan command (temporary instances) and testing (no Docker required)
 - pgx/v5 v5.7.5 for database connections
+- BurntSushi/toml for TOML parsing (ignore config)
 - Supports PostgreSQL versions 14-18
 
 Key differentiators:
@@ -33,6 +34,8 @@ This project includes specialized skills for development workflows. Use these sk
 - **PostgreSQL Syntax Reference** - Understand PostgreSQL's parser and grammar (gram.y) for SQL syntax and DDL structure
 - **Validate with Database** - Connect to live PostgreSQL to validate assumptions, compare pg_dump vs pgschema, and query system catalogs
 - **Run Tests** - Execute automated Go tests to validate diff logic, plan generation, and dump functionality
+- **Fix Bug** - Fix a bug from a GitHub issue using TDD (analyze, create reproducing test, implement fix, verify)
+- **Refactor Pass** - Perform a refactor pass focused on simplicity after recent changes
 
 Skills are located in `.claude/skills/` and provide detailed workflows for common development tasks.
 
@@ -87,8 +90,9 @@ PGAPPNAME=pgschema
 - `dump/` - Schema extraction from live database
 - `plan/` - Migration planning by comparing schemas
 - `apply/` - Migration execution with safety checks
-- `util/` - Shared utilities (connection, env, ignore file processing)
+- `util/` - Shared utilities (connection, env, ignore file loading, SQL logging)
 - `root.go` - Main CLI setup with Cobra
+- `migrate_integration_test.go`, `schema_integration_test.go`, `include_integration_test.go`, `ignore_integration_test.go` - Integration test suites
 
 **Core Packages**:
 
@@ -104,18 +108,23 @@ PGAPPNAME=pgschema
 **Internal Packages** (`internal/`):
 
 - `diff/` - Schema comparison and migration DDL generation
-- `plan/` - Migration plan structures and execution
+- `plan/` - Migration plan structures, execution, and plan rewriting (`rewrite.go`)
 - `dump/` - Schema dump formatting and output
-- `fingerprint/` - Schema fingerprinting for change detection
+- `fingerprint/` - Schema fingerprinting and comparison for change detection
 - `include/` - Include file processing for modular schemas
 - `postgres/` - Database provider implementations (embedded and external)
   - `desired_state.go` - DesiredStateProvider interface
   - `embedded.go` - Embedded PostgreSQL implementation
   - `external.go` - External database implementation
 - `color/` - Terminal output colorization
-- `logger/` - Structured logging
+- `logger/` - Structured logging using slog
 - `version/` - Version information
 
+**PostgreSQL Reference Sources** (`internal/`):
+
+- `gram.y` - Local copy of PostgreSQL's Yacc/Bison grammar for SQL syntax reference
+- `scan.l` - Local copy of PostgreSQL's Flex lexer for tokenization reference
+
 **Test Utilities** (`testutil/`):
 
 - `postgres.go` - Shared test utilities for setting up embedded PostgreSQL
@@ -220,6 +229,7 @@ The tool supports comprehensive PostgreSQL schema objects (see `ir/ir.go` for co
 
 - PostgreSQL's pg_dump serves as reference for system catalog queries (see **pg_dump Reference** skill)
 - PostgreSQL's gram.y defines canonical SQL syntax (see **PostgreSQL Syntax Reference** skill)
+- Local copies of PostgreSQL parser sources are available at `internal/gram.y` and `internal/scan.l` for quick reference
 
 ## Key Files Reference
 
@@ -235,34 +245,51 @@ The tool supports comprehensive PostgreSQL schema objects (see `ir/ir.go` for co
 - `ir/normalize.go` - Schema normalization (version-specific differences, type mappings)
 - `ir/quote.go` - Identifier quoting utilities
 - `ir/ignore.go` - IgnoreConfig for filtering database objects with glob patterns
-- `ir/queries/` - sqlc-generated code for type-safe SQL queries (`queries.sql.go`, `models.sql.go`, `dml.sql.go`)
-- Note: `ir/parser.go` removed - now using embedded-postgres for desired state
+- `ir/queries/` - sqlc-generated code for type-safe SQL queries (`queries.sql`, `queries.sql.go`, `models.sql.go`, `dml.sql.go`, `sqlc.yaml`)
 
 **Diff Package** (`internal/diff/`):
 
-- `diff.go` - Main diff logic, topological sorting
+- `diff.go` - Main diff logic, schema comparison
 - `table.go`, `column.go`, `constraint.go`, `index.go`, `trigger.go`, `view.go`, `function.go`, `procedure.go`, `sequence.go`, `type.go`, `policy.go` - Object-specific diff operations
-- `privilege.go`, `default_privilege.go` - Permission management (GRANT/REVOKE)
+- `privilege.go`, `column_privilege.go`, `default_privilege.go` - Permission management (GRANT/REVOKE, column-level privileges)
 - `collector.go` - SQL collection with context
 - `header.go` - Dump header generation
 - `topological.go` - Dependency sorting for migration ordering
 
 **Testing**:
 
 - `cmd/migrate_integration_test.go` - Main integration test suite (TestPlanAndApply)
-- `testdata/diff/` - 100+ test cases covering all schema object types
+- `cmd/schema_integration_test.go` - Schema-level integration tests
+- `cmd/include_integration_test.go` - Include file processing tests
+- `cmd/ignore_integration_test.go` - Ignore configuration tests
+- `testdata/diff/` - 150+ diff test cases covering all schema object types
+- `testdata/dump/` - 18 dump test suites (schema examples and issue-specific regressions)
+- `testdata/include/` - Include file tests (domains, functions, materialized views, procedures, sequences, tables, types, views)
 - See **Run Tests** skill for complete testing workflows
 
 ## Test Data Structure
 
-Tests are organized in `testdata/diff/` by object type:
+### Diff Tests (`testdata/diff/`)
 
-- `comment/` (10 tests), `create_domain/` (3), `create_function/` (5), `create_index/` (2)
+Tests are organized by object type (150+ test cases):
+
+- `comment/` (11), `create_domain/` (5), `create_function/` (8), `create_index/` (2)
 - `create_materialized_view/` (3), `create_policy/` (10), `create_procedure/` (3), `create_sequence/` (3)
-- `create_table/` (30 tests), `create_trigger/` (7), `create_type/` (3), `create_view/` (4)
-- `default_privilege/` (8), `privilege/` (10)
-- `dependency/` (8), `online/` (12), `migrate/` (5)
+- `create_table/` (37), `create_trigger/` (7), `create_type/` (3), `create_view/` (4)
+- `default_privilege/` (9), `privilege/` (13)
+- `dependency/` (13), `online/` (14), `migrate/` (5)
+
+Each test case contains: `old.sql` (starting state), `new.sql` (desired state), `diff.sql` (expected migration DDL), plus `plan.json`, `plan.sql`, `plan.txt` (plan output formats)
+
+### Dump Tests (`testdata/dump/`)
+
+18 test suites with `manifest.json`, `raw.sql`, `pgdump.sql`, `pgschema.sql`:
+
+- Schema examples: `bytebase/`, `employee/`, `sakila/`, `tenant/`
+- Issue regressions: `issue_125_*`, `issue_133_*`, `issue_183_*`, `issue_191_*`, `issue_252_*`, `issue_275_*`, `issue_307_*`, `issue_318_*`, `issue_320_*`, `issue_78_*`, `issue_80_*`, `issue_82_*`, `issue_83_*`
+
+### Include Tests (`testdata/include/`)
 
-Each test case contains: `old.sql` (starting state), `new.sql` (desired state), `expected.sql` (expected migration DDL)
+8 categories: `domains/`, `functions/`, `materialized_views/`, `procedures/`, `sequences/`, `tables/`, `types/`, `views/`
 
 For detailed test workflows, filtering, and regeneration, see the **Run Tests** skill.