feat: Index Migrator (Pre-release)

[nkanu17]

feat: Index Migrator (Pre-release)

Zero-downtime, crash-safe index migration for RedisVL. Plan, apply, and rollback schema changes, including vector quantization, field renames, prefix changes, and algorithm swaps, through a single CLI or programmatic API.

Summary

This PR adds a complete index migration system to RedisVL, enabling users to evolve their index schemas without data loss. The migrator handles the full lifecycle: plan → review → apply → validate → rollback.

Key Capabilities

Category	Operations
Index-only	Change algorithm (FLAT ↔ HNSW ↔ SVS-VAMANA), distance metric, HNSW params (M, EF_CONSTRUCTION), make fields sortable
Schema + Data	Add/remove fields, rename fields, rename index, change key prefix, change field options (separator, stemming)
Vector Quantization	float32 → float16, bfloat16, int8, uint8 with automatic re-encoding and crash-safe backup

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    CLI: rvl migrate                          │
│  plan │ wizard │ apply │ rollback │ estimate │ validate     │
├───────┴────────┴───────┴──────────┴──────────┴──────────────┤
│                  MigrationPlanner                           │
│  Schema diffing, change classification, plan generation     │
├─────────────────────────────────────────────────────────────┤
│            MigrationExecutor (sync + async)                 │
│  enumerate → field-rename → dump → drop → key-rename →     │
│  quantize → create → index → validate                      │
├─────────────────────────────────────────────────────────────┤
│  VectorBackup    │  Quantize Pipeline  │  Reliability Layer │
│  Crash-safe dump │  Batched R/W/Convert│  Validation checks │
└──────────────────┴─────────────────────┴────────────────────┘

Usage

CLI: Interactive Wizard

# List available indexes
rvl migrate list --url redis://localhost:6379

# Interactive wizard: walks through changes step by step
rvl migrate wizard --url redis://localhost:6379

# Generate a plan from a target schema YAML
rvl migrate plan --index my_index --schema target_schema.yaml --url redis://localhost:6379

# Apply with crash-safe backup and multi-worker quantization
rvl migrate apply --plan migration_plan.yaml \
  --backup-dir /tmp/migration_backup \
  --workers 4 --batch-size 1000 \
  --url redis://localhost:6379

# Estimate disk space (dry-run, no mutations)
rvl migrate estimate --plan migration_plan.yaml --url redis://localhost:6379

# Rollback if needed
rvl migrate rollback --backup-dir /tmp/migration_backup \
  --index my_index --url redis://localhost:6379

# Validate post-migration
rvl migrate validate --plan migration_plan.yaml --url redis://localhost:6379

CLI: Batch Migration (Multiple Indexes)

rvl migrate batch-plan --config batch_config.yaml --url redis://localhost:6379
rvl migrate batch-apply --plan batch_plan.yaml --url redis://localhost:6379
rvl migrate batch-status --plan batch_plan.yaml

Programmatic API

from redisvl.migration import MigrationPlanner, MigrationExecutor

# Plan
planner = MigrationPlanner()
plan = planner.create_plan(
    index_name="my_index",
    target_schema=target_schema,
    redis_url="redis://localhost:6379",
)

# Apply
executor = MigrationExecutor()
report = executor.apply(
    plan,
    redis_url="redis://localhost:6379",
    backup_dir="/tmp/migration_backup",
    num_workers=4,
    batch_size=1000,
)

print(f"Result: {report.result}")  # "succeeded"
print(f"Duration: {report.timings.total_duration}s")

Async API

from redisvl.migration import AsyncMigrationExecutor

executor = AsyncMigrationExecutor()
report = await executor.apply(
    plan,
    redis_url="redis://localhost:6379",
    backup_dir="/tmp/migration_backup",
    num_workers=4,
)

Crash Safety & Resume

Migrations are crash-safe by default when --backup-dir is provided:

1. Before drop: Original vectors are dumped to a binary backup file on disk
2. On crash: Re-running the same command detects the backup and resumes from the last completed batch
3. Rollback: rvl migrate rollback restores original vectors from the backup at any time

The backup file tracks phase (dump → ready → active → completed) and batch progress, so resume skips already-completed work.

Performance

• Pipelined reads/writes: Batch HGET/HSET operations (configurable --batch-size)
• Multi-worker quantization: --workers N parallelizes vector re-encoding via ThreadPoolExecutor (sync) or asyncio.gather (async)
• Redis Cluster support: Batched DUMP/RESTORE/DEL for cross-slot key renames (100 keys/pipeline)
• Disk space estimation: rvl migrate estimate calculates RDB + AOF impact before any mutations

What's Blocked

Change	Why	Workaround
Change vector dimensions	Requires re-embedding	Re-embed with new model, reload data
Change storage type (hash ↔ JSON)	Different data format	Export, transform, reload
Add a new vector field	Requires vectors for all docs	Add vectors first, then migrate

New Files

redisvl/migration/ (core module)

File	Description
models.py	Data models: MigrationPlan, MigrationReport, MigrationTimings, etc.
planner.py	Schema diffing, change classification, plan generation
executor.py	Sync migration executor, full apply lifecycle
async_executor.py	Async migration executor
async_planner.py	Async planner
validation.py / async_validation.py	Pre/post-migration validation
backup.py	VectorBackup: crash-safe binary backup format
quantize.py	Pipelined vector quantization + multi-worker orchestration
reliability.py	Dtype conversion safety checks, width analysis
wizard.py	Interactive migration wizard
batch_planner.py / batch_executor.py	Multi-index batch migration
utils.py	Shared utilities (disk estimation, key enumeration, etc.)

redisvl/cli/migrate.py

Full CLI with 12 subcommands: list, wizard, plan, apply, estimate, rollback, validate, helper, batch-plan, batch-apply, batch-resume, batch-status

Tests

Suite	Count	Description
test_migration_planner.py	67	Schema diffing, change classification
test_migration_wizard.py	76	Interactive wizard, adversarial inputs
test_vector_backup.py	32	Backup create/load/resume/rollback/cleanup
test_pipeline_quantize.py	12	Pipelined read/write/convert
test_executor_backup_quantize.py	7	Executor backup integration
test_multi_worker_quantize.py	22	Multi-worker, resume, deprecation
test_async_migration_executor.py	13	Async executor
test_async_migration_planner.py	11	Async planner
test_batch_migration.py	44	Batch planner/executor
Integration tests	6 files	Full end-to-end with live Redis

Total: 178+ unit tests passing, all pre-commit checks clean.

Review Notes

• This is a pre-release: API surface is stable but may evolve based on feedback
• 6 rounds of automated code review (nkode-review) have been applied, addressing correctness, security, performance, and backward compatibility findings
• The branch includes removal of the MCP module (previously merged separately). Those deletions are unrelated to the migrator
• Documentation is in docs/user_guide/how_to_guides/migrate-indexes.md

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a pre-release RedisVL index migration system (sync + async) with planning, execution, validation, batch workflows, and extensive documentation/tests to enable crash-safe, document-preserving schema evolution.

Changes:

• Introduces migration core modules (planner/executor/validator/backup/quantize/reliability) plus batch migration support.
• Adds async equivalents and a new rvl migrate CLI entry point + docs.
• Adds comprehensive unit/integration tests and benchmark/e2e helper scripts.

Reviewed changes

Copilot reviewed 45 out of 53 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/unit/test_async_migration_planner.py	Adds async planner unit coverage mirroring sync planner tests.
tests/unit/test_async_migration_executor.py	Adds async executor + disk space estimator unit tests and dtype-detection tests.
tests/integration/test_migration_v1.py	End-to-end integration test for sync plan/apply/validate flow.
tests/integration/test_migration_routes.py	Integration coverage for supported migration “routes” (algo/metric/dtype/params).
tests/integration/test_field_modifier_ordering_integration.py	Adds integration tests for new/related field modifiers (INDEXEMPTY/UNF/NOINDEX).
tests/integration/test_batch_migration_integration.py	Adds integration tests for batch plan/apply/resume/progress callback.
tests/integration/test_async_migration_v1.py	End-to-end integration test for async plan/apply/validate flow.
tests/benchmarks/visualize_results.py	Adds benchmark visualization script for retrieval/memory/latency charts.
scripts/verify_data_correctness.py	Adds manual script to verify float32→float16 migration correctness.
scripts/test_migration_e2e.py	Adds large-scale e2e migration benchmark script.
scripts/test_crash_resume_e2e.py	Adds crash/resume robustness test script for quantization checkpointing.
redisvl/redis/connection.py	Enhances vector attribute parsing to include HNSW params (m, ef_construction).
redisvl/migration/validation.py	Adds sync migration validation (schema/doc counts/key samples/functional checks).
redisvl/migration/utils.py	Adds YAML helpers, schema canonicalization, readiness polling, disk estimation utilities.
redisvl/migration/reliability.py	Adds crash-safety utilities: dtype detection, checkpointing, BGSAVE helpers, undo buffer.
redisvl/migration/quantize.py	Adds pipelined (and multi-worker) quantization for vector dtype conversions.
redisvl/migration/models.py	Adds migration/batch models and disk space estimate models/helpers.
redisvl/migration/batch_planner.py	Adds batch planner for applying a shared patch across many indexes.
redisvl/migration/batch_executor.py	Adds batch executor with checkpointing/resume and reporting.
redisvl/migration/backup.py	Adds crash-safe on-disk backup format for vector dumps and resume.
redisvl/migration/async_validation.py	Adds async validator parity with sync validation checks.
redisvl/migration/async_planner.py	Adds async planner wrapper over sync diff/classification logic.
redisvl/migration/init.py	Exposes new migration/batch APIs at package boundary.
redisvl/cli/utils.py	Fixes redis URL scheme building and refactors CLI option helpers.
redisvl/cli/main.py	Wires in new migrate CLI command group.
docs/user_guide/index.md	Adds migration to user guide landing page highlights.
docs/user_guide/how_to_guides/index.md	Adds “Migrate an Index” how-to link and toctree entry.
docs/user_guide/cli.ipynb	Updates CLI notebook with rvl migrate commands and reorganizes connection section.
docs/concepts/search-and-indexing.md	Updates concept docs to point to new migration workflow and docs.
docs/concepts/index.md	Adds “Index Migrations” concept card and toctree entry.
docs/concepts/index-migrations.md	New concept doc describing migration modes, supported changes, and sync/async behavior.
docs/concepts/field-attributes.md	Expands vector datatype docs + migration support notes for modifiers.
docs/api/cli.rst	Adds a full CLI reference including rvl migrate command group.
CLAUDE.md	Adds protected directory note (local_docs/).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Using EXISTS with multiple keys can fail on Redis Cluster when keys span multiple hash slots (multi-key commands are restricted). To make validation work reliably on clustered deployments, check key existence per-key (or via a pipeline of single-key EXISTS calls) and then compare the total count.

Suggested change

	existing_count = target_index.client.exists(*keys_to_check)
	existing_count = sum(target_index.client.exists(key) for key in keys_to_check)

[Copilot]

Same Redis Cluster concern as the sync validator: await client.exists(*keys_to_check) is a multi-key operation and can error if keys live in different hash slots. Prefer checking keys individually (or pipelined single-key EXISTS) to keep async validation cluster-safe.

Suggested change

	existing_count = await client.exists(*keys_to_check)
	existing_count = 0
	for key in keys_to_check:
	existing_count += await client.exists(key)

[Copilot]

Prefix-change validation assumes plan.source.keyspace.prefixes[0] is the only (or correct) source prefix. If the index has multiple prefixes, translating sampled keys using only the first prefix can produce incorrect keys_to_check and false validation failures. Consider translating each sampled key by matching against all configured prefixes (or storing the actual matched prefix per sampled key in the snapshot).

[Copilot]

Async validator has the same multi-prefix assumption as the sync validator, which can mis-translate key samples when more than one prefix is configured. Matching the sampled key against all prefixes (instead of using only prefixes[0]) would prevent incorrect existence checks.

Suggested change

	old_prefix = plan.source.keyspace.prefixes[0]
	new_prefix = plan.rename_operations.change_prefix
	keys_to_check = []
	for k in key_sample:
	if k.startswith(old_prefix):
	keys_to_check.append(new_prefix + k[len(old_prefix) :])
	else:
	keys_to_check.append(k)
	old_prefixes = plan.source.keyspace.prefixes
	new_prefix = plan.rename_operations.change_prefix
	keys_to_check = []
	for k in key_sample:
	translated_key = k
	for old_prefix in old_prefixes:
	if k.startswith(old_prefix):
	translated_key = new_prefix + k[len(old_prefix) :]
	break
	keys_to_check.append(translated_key)

[Copilot]

The docstring states the returned list has num_workers slices and 'some may be empty if keys < workers', but the implementation returns fewer slices (never empty slices). Either update the docstring to reflect the current behavior, or adjust the function to always return exactly num_workers slices (padding with empty lists) if that invariant is important for worker bookkeeping and stable shard naming.

Suggested change

	return []
	n = len(keys)
	chunk_size = math.ceil(n / num_workers)
	return [keys[i : i + chunk_size] for i in range(0, n, chunk_size)]
	return [[] for _ in range(num_workers)]
	n = len(keys)
	chunk_size = math.ceil(n / num_workers)
	slices = [keys[i : i + chunk_size] for i in range(0, n, chunk_size)]
	slices.extend([[] for _ in range(num_workers - len(slices))])
	return slices

[Copilot]

In the 'completed' phase, the sum(...) computation is dead code because docs_quantized is immediately overwritten with total. Removing the unused computation reduces confusion and avoids implying that a more complex accounting is needed here.

Suggested change

	docs_quantized = sum(
	1 for _ in range(0, total, batch_size) for _ in keys[:batch_size]
	)

[Copilot]

This test doesn't exercise the documented behavior ('requires redis_url or redis_client')—it only asserts the executor can be instantiated. To make it meaningful, assert the actual failure mode (e.g., await executor.apply(plan) raises/returns a failed report) by mocking the connection path (AsyncSearchIndex.from_existing) or by passing neither redis_url nor redis_client and asserting the expected error/report content.

Suggested change

	# The executor should raise an error internally when trying to connect
	# but let's verify it doesn't crash before it tries to apply
	# For a proper test, we'd need to mock AsyncSearchIndex.from_existing
	# For now, we just verify the executor is created
	assert executor is not None
	with pytest.raises(ValueError, match="redis_url or redis_client"):
	await executor.apply(plan)

[jit-ci]

🛡️ Jit Security Scan Results

✅ No security findings were detected in this PR

---

^{Security scan by Jit}