Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions pgcopydb-helpers/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,14 @@ trigger_to_skip

---

#### `filters-lib.sh`

Shared Bash library that parses `~/filters.ini` and turns the active filter into SQL scope fragments. **Sourced, not executed** — it defines functions and has no standalone behavior. `preflight-check.sh` and `verify-migration.sh` both source it (via `source "$SCRIPT_DIR/filters-lib.sh"`), so they interpret the filter identically: preflight scopes its per-schema permission checks to what *will* migrate, and verify scopes its source-vs-target comparison to what *was* migrated.

**Key functions:** `parse_filters_ini <path>` (populates `FILTER_*` arrays from `[exclude-schema]`, `[exclude-table]`, `[include-only-schema]`, `[include-only-table]`, `[exclude-extension]`), `filter_scope_mode` (precedence: `include-only-table` > `include-only-schema` > `exclude-schema` > `all`), `schema_clause <col>` / `table_clause <schema-col> <rel-col>` / `extension_clause <col>` (emit `AND ... IN/NOT IN (...)` fragments for any catalog column), `filter_scope_describe` (header summary), and `filter_conflicts` (pgcopydb-disallowed section combinations).

---

#### Source-Specific: Supabase RLS

Supabase source databases typically have Row-Level Security on application tables. Without `BYPASSRLS`, the migration user reads zero rows from RLS-protected tables and the migration silently produces an incomplete target. Detect and fix on the source:
Expand Down
3 changes: 3 additions & 0 deletions pgcopydb-helpers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,8 @@ trigger_to_skip

**Important:** No comments are allowed inside sections — pgcopydb parses `#` lines as object names. Place all comments before the first section.

**Shared parser:** `preflight-check.sh` and `verify-migration.sh` both interpret `~/filters.ini` through `filters-lib.sh`, a sourced helper library (not a standalone script). Keeping the parsing and scope logic in one place ensures both scripts honor the filter identically — preflight scopes its permission checks to what will be migrated, and verify scopes its comparison to what was migrated.

### Common Exclusions by Source

**Amazon RDS:**
Expand Down Expand Up @@ -411,6 +413,7 @@ sqlite3 ~/migration_*/schema/filter.db "SELECT COUNT(*) FROM s_depend;"
| `preflight-check.sh` | Prepare | Validate migration prerequisites (connectivity, WAL level, permissions, slots, extension compatibility) |
| `fix-replica-identity.sh` | Prepare | Set REPLICA IDENTITY FULL on tables without primary keys |
| `filters.ini` | Prepare | pgcopydb filter configuration |
| `filters-lib.sh` | Prepare | Shared `filters.ini` parser sourced by `preflight-check.sh` and `verify-migration.sh` (not run directly) |
| `run-migration.sh` | Migrate | Start a pgcopydb clone --follow migration |
| `start-migration-screen.sh` | Migrate | Run the migration in a detached screen session. |
| `check-migration-status.sh` | Monitor | Migration progress dashboard |
Expand Down
128 changes: 128 additions & 0 deletions pgcopydb-helpers/filters-lib.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# =============================================================================
# filters-lib.sh — shared filters.ini parser & SQL-scope helpers
# =============================================================================
# Sourced by verify-migration.sh and preflight-check.sh so both interpret
# ~/filters.ini identically. Not executable on its own — `source` it:
#
# SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# source "$SCRIPT_DIR/filters-lib.sh"
#
# The migration only copies the subset of objects allowed by ~/filters.ini.
# Without this awareness, intentionally-excluded objects show up as "missing in target"
# — false-positive noise. These helpers turn the active filter into
# SQL WHERE fragments so every catalog query can be scoped to exactly
# what the migration migrates.
#
# Supported sections: exclude-schema, exclude-table, include-only-schema,
# include-only-table, exclude-extension. (exclude-event-trigger is recognised by
# pgcopydb but not relevant to these scripts' checks, so it is ignored here.)
# =============================================================================

# Scope-relevant filter state. Initialised here so sourcing is safe under
# `set -u` before parse_filters_ini runs (or when no filters.ini is loaded).
FILTER_EXCLUDE_SCHEMAS=(); FILTER_EXCLUDE_TABLES=()
FILTER_INCLUDE_ONLY_TABLES=(); FILTER_INCLUDE_ONLY_SCHEMAS=()
FILTER_EXCLUDE_EXTENSIONS=()

# Parse scope-relevant sections from filters.ini into the global arrays above.
parse_filters_ini() {
local ini_file="$1"
FILTER_EXCLUDE_SCHEMAS=(); FILTER_EXCLUDE_TABLES=()
FILTER_INCLUDE_ONLY_TABLES=(); FILTER_INCLUDE_ONLY_SCHEMAS=()
FILTER_EXCLUDE_EXTENSIONS=()
local section="" line
while IFS= read -r line; do
line="${line#"${line%%[![:space:]]*}"}" # ltrim
line="${line%"${line##*[![:space:]]}"}" # rtrim
[[ -z "$line" || "$line" == \#* ]] && continue
if [[ "$line" =~ ^\[(.+)\]$ ]]; then section="${BASH_REMATCH[1]}"; continue; fi
case "$section" in
exclude-schema) FILTER_EXCLUDE_SCHEMAS+=("$line") ;;
exclude-table) FILTER_EXCLUDE_TABLES+=("$line") ;;
include-only-table) FILTER_INCLUDE_ONLY_TABLES+=("$line") ;;
include-only-schema) FILTER_INCLUDE_ONLY_SCHEMAS+=("$line") ;;
exclude-extension) FILTER_EXCLUDE_EXTENSIONS+=("$line") ;;
esac
done < "$ini_file"
}

# Returns the effective filter mode: include-table | include-schema | exclude-schema | all
filter_scope_mode() {
[ ${#FILTER_INCLUDE_ONLY_TABLES[@]} -gt 0 ] && { echo "include-table"; return; }
[ ${#FILTER_INCLUDE_ONLY_SCHEMAS[@]} -gt 0 ] && { echo "include-schema"; return; }
[ ${#FILTER_EXCLUDE_SCHEMAS[@]} -gt 0 ] && { echo "exclude-schema"; return; }
echo "all"
}

# Formats values as a SQL-safe single-quoted comma list: 'a','b','c'
_sql_list() {
local result="" v sq="'"
for v in "$@"; do
v="${v//$sq/$sq$sq}"
result="${result:+$result,}'${v}'"
done
echo "$result"
}

# Build SQL-quoted list of unique schema names extracted from "schema.table" entries
_it_schema_sql_list() {
local result="" sq="'"
while IFS= read -r s; do
s="${s//$sq/$sq$sq}"
result="${result:+$result,}'${s}'"
done < <(printf '%s\n' "$@" | cut -d. -f1 | sort -u)
echo "$result"
}

# schema_clause <schema-column-expr> — "AND <col> IN/NOT IN (...)" or "" for all mode.
# Applied to every object type so excluded/included schemas scope the whole comparison.
schema_clause() {
local col="$1" mode list
mode=$(filter_scope_mode)
case "$mode" in
include-table) list=$(_it_schema_sql_list "${FILTER_INCLUDE_ONLY_TABLES[@]}"); echo "AND ${col} IN (${list})" ;;
include-schema) list=$(_sql_list "${FILTER_INCLUDE_ONLY_SCHEMAS[@]}"); echo "AND ${col} IN (${list})" ;;
exclude-schema) list=$(_sql_list "${FILTER_EXCLUDE_SCHEMAS[@]}"); echo "AND ${col} NOT IN (${list})" ;;
all) echo "" ;;
esac
}

# table_clause <schema-col> <rel-col> — restricts table-keyed checks to the
# in-scope table set. " AND (<schema>.<rel>) IN/NOT IN (...)" or "".
table_clause() {
local scol="$1" rcol="$2" mode
mode=$(filter_scope_mode)
if [ "$mode" = "include-table" ]; then
echo " AND (${scol} || '.' || ${rcol}) IN ($(_sql_list "${FILTER_INCLUDE_ONLY_TABLES[@]}"))"
elif [ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ]; then
echo " AND (${scol} || '.' || ${rcol}) NOT IN ($(_sql_list "${FILTER_EXCLUDE_TABLES[@]}"))"
else
echo ""
fi
}

# extension_clause <extname-col> — excludes [exclude-extension] entries, or "".
extension_clause() {
local col="$1"
[ ${#FILTER_EXCLUDE_EXTENSIONS[@]} -eq 0 ] && { echo ""; return; }
echo "AND ${col} NOT IN ($(_sql_list "${FILTER_EXCLUDE_EXTENSIONS[@]}"))"
}

# Human-readable one-line summary of the active scope
filter_scope_describe() {
case "$(filter_scope_mode)" in
include-table) echo "include-only-table (${#FILTER_INCLUDE_ONLY_TABLES[@]} table(s))" ;;
include-schema) echo "include-only-schema (${#FILTER_INCLUDE_ONLY_SCHEMAS[@]} schema(s))" ;;
exclude-schema) echo "exclude-schema (${#FILTER_EXCLUDE_SCHEMAS[@]} schema(s))$([ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ] && echo " + exclude-table (${#FILTER_EXCLUDE_TABLES[@]})")" ;;
all) [ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ] && { echo "exclude-table (${#FILTER_EXCLUDE_TABLES[@]} table(s))"; return; }; echo "none" ;;
esac
}

# Lists disallowed section combinations present in filters.ini (pgcopydb rejects these), or ""
filter_conflicts() {
local c=()
[ ${#FILTER_INCLUDE_ONLY_TABLES[@]} -gt 0 ] && [ ${#FILTER_EXCLUDE_SCHEMAS[@]} -gt 0 ] && c+=("include-only-table + exclude-schema")
[ ${#FILTER_INCLUDE_ONLY_TABLES[@]} -gt 0 ] && [ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ] && c+=("include-only-table + exclude-table")
[ ${#FILTER_INCLUDE_ONLY_SCHEMAS[@]} -gt 0 ] && [ ${#FILTER_EXCLUDE_SCHEMAS[@]} -gt 0 ] && c+=("include-only-schema + exclude-schema")
local IFS="; "; echo "${c[*]:-}"
}
101 changes: 9 additions & 92 deletions pgcopydb-helpers/preflight-check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -68,95 +68,11 @@ tgt_query() {
}

# ── Filter scope helpers ─────────────────────────────────────────────

# Parse scope-relevant sections from filters.ini into global arrays
parse_filters_ini() {
local ini_file="$1"
FILTER_EXCLUDE_SCHEMAS=(); FILTER_EXCLUDE_TABLES=()
FILTER_INCLUDE_ONLY_TABLES=(); FILTER_INCLUDE_ONLY_SCHEMAS=()
local section="" line
while IFS= read -r line; do
line="${line#"${line%%[![:space:]]*}"}" # ltrim
line="${line%"${line##*[![:space:]]}"}" # rtrim
[[ -z "$line" || "$line" == \#* ]] && continue
if [[ "$line" =~ ^\[(.+)\]$ ]]; then section="${BASH_REMATCH[1]}"; continue; fi
case "$section" in
exclude-schema) FILTER_EXCLUDE_SCHEMAS+=("$line") ;;
exclude-table) FILTER_EXCLUDE_TABLES+=("$line") ;;
include-only-table) FILTER_INCLUDE_ONLY_TABLES+=("$line") ;;
include-only-schema) FILTER_INCLUDE_ONLY_SCHEMAS+=("$line") ;;
esac
done < "$ini_file"
}

# Returns the effective filter mode: include-table | include-schema | exclude-schema | all
filter_scope_mode() {
[ ${#FILTER_INCLUDE_ONLY_TABLES[@]} -gt 0 ] && { echo "include-table"; return; }
[ ${#FILTER_INCLUDE_ONLY_SCHEMAS[@]} -gt 0 ] && { echo "include-schema"; return; }
[ ${#FILTER_EXCLUDE_SCHEMAS[@]} -gt 0 ] && { echo "exclude-schema"; return; }
echo "all"
}

# Formats values as a SQL-safe single-quoted comma list: 'a','b','c'
_sql_list() {
local result="" v sq="'"
for v in "$@"; do
v="${v//$sq/$sq$sq}"
result="${result:+$result,}'${v}'"
done
echo "$result"
}

# Build SQL-quoted list of unique schema names extracted from "schema.table" entries
_it_schema_sql_list() {
local result="" sq="'"
while IFS= read -r s; do
s="${s//$sq/$sq$sq}"
result="${result:+$result,}'${s}'"
done < <(printf '%s\n' "$@" | cut -d. -f1 | sort -u)
echo "$result"
}

# Returns "AND n.nspname IN/NOT IN (...)" SQL fragment, or "" for all mode
build_schema_where() {
local mode list
mode=$(filter_scope_mode)
case "$mode" in
include-table) list=$(_it_schema_sql_list "${FILTER_INCLUDE_ONLY_TABLES[@]}") ;;
include-schema) list=$(_sql_list "${FILTER_INCLUDE_ONLY_SCHEMAS[@]}") ;;
exclude-schema) list=$(_sql_list "${FILTER_EXCLUDE_SCHEMAS[@]}") ;;
all) echo ""; return ;;
esac
local op="IN"; [ "$mode" = "exclude-schema" ] && op="NOT IN"
echo "AND n.nspname ${op} (${list})"
}

# Returns SQL fragment to append to table-count sub-queries (starts with " AND"), or ""
build_table_filter() {
local mode
mode=$(filter_scope_mode)
if [ "$mode" = "include-table" ]; then
echo " AND (n.nspname || '.' || c.relname) IN ($(_sql_list "${FILTER_INCLUDE_ONLY_TABLES[@]}"))"
elif [ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ]; then
echo " AND (n.nspname || '.' || c.relname) NOT IN ($(_sql_list "${FILTER_EXCLUDE_TABLES[@]}"))"
else
echo ""
fi
}

# Lists disallowed section combinations present in filters.ini (pgcopydb rejects these), or ""
filter_conflicts() {
local c=()
[ ${#FILTER_INCLUDE_ONLY_TABLES[@]} -gt 0 ] && [ ${#FILTER_EXCLUDE_SCHEMAS[@]} -gt 0 ] && c+=("include-only-table + exclude-schema")
[ ${#FILTER_INCLUDE_ONLY_TABLES[@]} -gt 0 ] && [ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ] && c+=("include-only-table + exclude-table")
[ ${#FILTER_INCLUDE_ONLY_SCHEMAS[@]} -gt 0 ] && [ ${#FILTER_EXCLUDE_SCHEMAS[@]} -gt 0 ] && c+=("include-only-schema + exclude-schema")
local IFS="; "; echo "${c[*]:-}"
}
# Shared with verify-migration.sh so both interpret filters.ini identically.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/filters-lib.sh"

# ── Pre-parse filters.ini (scope needed for source permission checks) ─
FILTER_EXCLUDE_SCHEMAS=(); FILTER_EXCLUDE_TABLES=()
FILTER_INCLUDE_ONLY_TABLES=(); FILTER_INCLUDE_ONLY_SCHEMAS=()

[ -f ~/filters.ini ] && parse_filters_ini ~/filters.ini

# ══════════════════════════════════════════════════════════════════
Expand Down Expand Up @@ -283,8 +199,8 @@ else
fi

# Per-schema: USAGE + SELECT on tables/mat views/sequences, scoped to what filters.ini will migrate
_schema_where=$(build_schema_where)
_table_filter=$(build_table_filter)
_schema_where=$(schema_clause "n.nspname")
_table_filter=$(table_clause "n.nspname" "c.relname")
[ "$(filter_scope_mode)" = "include-table" ] && _check_seqs=false || _check_seqs=true

if [ "$_check_seqs" = "true" ]; then
Expand Down Expand Up @@ -359,10 +275,11 @@ if [ -n "$TGT_VER" ]; then
pass "Existing pgcopydb schema" "none"
fi

# 14. Extension compatibility
# 14. Extension compatibility — exclusions come from the shared filters.ini
# parser ([exclude-extension] → FILTER_EXCLUDE_EXTENSIONS), as newline list.
FILTER_EXCLUDED_EXTS=""
if [ -f ~/filters.ini ]; then
FILTER_EXCLUDED_EXTS=$(awk '/^\[exclude-extension\]/{found=1; next} /^\[/{found=0} found && /[^[:space:]]/{gsub(/^[[:space:]]+|[[:space:]]+$/, ""); print}' ~/filters.ini)
if [ ${#FILTER_EXCLUDE_EXTENSIONS[@]} -gt 0 ]; then
FILTER_EXCLUDED_EXTS=$(printf '%s\n' "${FILTER_EXCLUDE_EXTENSIONS[@]}")
fi

SRC_EXTS=$(src_query "SELECT extname FROM pg_extension ORDER BY extname;")
Expand Down
106 changes: 3 additions & 103 deletions pgcopydb-helpers/verify-migration.sh
Original file line number Diff line number Diff line change
Expand Up @@ -96,109 +96,9 @@ line_count() { printf '%s' "${1:-}" | grep -c . || true; }
abs() { local v=$1; echo "${v#-}"; }

# ── filters.ini scope helpers ─────────────────────────────────────────────────
# The migration only copies the subset of objects allowed by ~/filters.ini.
# Without this awareness, intentionally-excluded objects (e.g. Supabase's auth,
# storage, realtime schemas) show up as "missing in target" — false-positive
# noise. These helpers mirror preflight-check.sh's interpretation of filters.ini.

# Parse scope-relevant sections from filters.ini into global arrays
parse_filters_ini() {
local ini_file="$1"
FILTER_EXCLUDE_SCHEMAS=(); FILTER_EXCLUDE_TABLES=()
FILTER_INCLUDE_ONLY_TABLES=(); FILTER_INCLUDE_ONLY_SCHEMAS=()
FILTER_EXCLUDE_EXTENSIONS=()
local section="" line
while IFS= read -r line; do
line="${line#"${line%%[![:space:]]*}"}" # ltrim
line="${line%"${line##*[![:space:]]}"}" # rtrim
[[ -z "$line" || "$line" == \#* ]] && continue
if [[ "$line" =~ ^\[(.+)\]$ ]]; then section="${BASH_REMATCH[1]}"; continue; fi
case "$section" in
exclude-schema) FILTER_EXCLUDE_SCHEMAS+=("$line") ;;
exclude-table) FILTER_EXCLUDE_TABLES+=("$line") ;;
include-only-table) FILTER_INCLUDE_ONLY_TABLES+=("$line") ;;
include-only-schema) FILTER_INCLUDE_ONLY_SCHEMAS+=("$line") ;;
exclude-extension) FILTER_EXCLUDE_EXTENSIONS+=("$line") ;;
esac
done < "$ini_file"
}

# Returns the effective filter mode: include-table | include-schema | exclude-schema | all
filter_scope_mode() {
[ ${#FILTER_INCLUDE_ONLY_TABLES[@]} -gt 0 ] && { echo "include-table"; return; }
[ ${#FILTER_INCLUDE_ONLY_SCHEMAS[@]} -gt 0 ] && { echo "include-schema"; return; }
[ ${#FILTER_EXCLUDE_SCHEMAS[@]} -gt 0 ] && { echo "exclude-schema"; return; }
echo "all"
}

# Formats values as a SQL-safe single-quoted comma list: 'a','b','c'
_sql_list() {
local result="" v sq="'"
for v in "$@"; do
v="${v//$sq/$sq$sq}"
result="${result:+$result,}'${v}'"
done
echo "$result"
}

# Build SQL-quoted list of unique schema names extracted from "schema.table" entries
_it_schema_sql_list() {
local result="" sq="'"
while IFS= read -r s; do
s="${s//$sq/$sq$sq}"
result="${result:+$result,}'${s}'"
done < <(printf '%s\n' "$@" | cut -d. -f1 | sort -u)
echo "$result"
}

# schema_clause <schema-column-expr> — "AND <col> IN/NOT IN (...)" or "" for all mode.
# Applied to every object type so excluded/included schemas scope the whole comparison.
schema_clause() {
local col="$1" mode list
mode=$(filter_scope_mode)
case "$mode" in
include-table) list=$(_it_schema_sql_list "${FILTER_INCLUDE_ONLY_TABLES[@]}"); echo "AND ${col} IN (${list})" ;;
include-schema) list=$(_sql_list "${FILTER_INCLUDE_ONLY_SCHEMAS[@]}"); echo "AND ${col} IN (${list})" ;;
exclude-schema) list=$(_sql_list "${FILTER_EXCLUDE_SCHEMAS[@]}"); echo "AND ${col} NOT IN (${list})" ;;
all) echo "" ;;
esac
}

# table_clause <schema-col> <rel-col> — restricts table-keyed checks to the
# in-scope table set. " AND (<schema>.<rel>) IN/NOT IN (...)" or "".
table_clause() {
local scol="$1" rcol="$2" mode
mode=$(filter_scope_mode)
if [ "$mode" = "include-table" ]; then
echo " AND (${scol} || '.' || ${rcol}) IN ($(_sql_list "${FILTER_INCLUDE_ONLY_TABLES[@]}"))"
elif [ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ]; then
echo " AND (${scol} || '.' || ${rcol}) NOT IN ($(_sql_list "${FILTER_EXCLUDE_TABLES[@]}"))"
else
echo ""
fi
}

# extension_clause <extname-col> — excludes [exclude-extension] entries, or "".
extension_clause() {
local col="$1"
[ ${#FILTER_EXCLUDE_EXTENSIONS[@]} -eq 0 ] && { echo ""; return; }
echo "AND ${col} NOT IN ($(_sql_list "${FILTER_EXCLUDE_EXTENSIONS[@]}"))"
}

# Human-readable one-line summary of the active scope
filter_scope_describe() {
case "$(filter_scope_mode)" in
include-table) echo "include-only-table (${#FILTER_INCLUDE_ONLY_TABLES[@]} table(s))" ;;
include-schema) echo "include-only-schema (${#FILTER_INCLUDE_ONLY_SCHEMAS[@]} schema(s))" ;;
exclude-schema) echo "exclude-schema (${#FILTER_EXCLUDE_SCHEMAS[@]} schema(s))$([ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ] && echo " + exclude-table (${#FILTER_EXCLUDE_TABLES[@]})")" ;;
all) [ ${#FILTER_EXCLUDE_TABLES[@]} -gt 0 ] && { echo "exclude-table (${#FILTER_EXCLUDE_TABLES[@]} table(s))"; return; }; echo "none" ;;
esac
}

# Arrays must exist before any helper references them (set -u)
FILTER_EXCLUDE_SCHEMAS=(); FILTER_EXCLUDE_TABLES=()
FILTER_INCLUDE_ONLY_TABLES=(); FILTER_INCLUDE_ONLY_SCHEMAS=()
FILTER_EXCLUDE_EXTENSIONS=()
# Shared with preflight-check.sh so both interpret filters.ini identically.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/filters-lib.sh"

# ── Argument parsing ──────────────────────────────────────────────────────────
while [[ $# -gt 0 ]]; do
Expand Down