From 69c01911c92fc721caf4dff02b177b0d6a7ae996 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Thu, 2 Apr 2026 11:55:54 -0400 Subject: [PATCH 1/8] docs: add migration documentation, concept guides, and benchmarks - docs/concepts/index-migrations.md: migration concepts and architecture - docs/user_guide/how_to_guides/migrate-indexes.md: step-by-step migration guide - docs/api/cli.rst: CLI reference for rvl migrate commands - tests/benchmarks/: migration benchmark scripts and visualization - Updated field-attributes, search-and-indexing, and user guide indexes --- docs/api/cli.rst | 614 ++++++++ docs/concepts/field-attributes.md | 91 +- docs/concepts/index-migrations.md | 255 ++++ docs/concepts/index.md | 8 + docs/concepts/search-and-indexing.md | 11 +- docs/user_guide/13_sql_query_exercises.ipynb | 1239 +++++++++++++++++ docs/user_guide/cli.ipynb | 13 +- docs/user_guide/how_to_guides/index.md | 3 + .../how_to_guides/migrate-indexes.md | 933 +++++++++++++ docs/user_guide/index.md | 4 +- .../index_migrator_real_benchmark.py | 647 +++++++++ tests/benchmarks/migration_benchmark.py | 642 +++++++++ tests/benchmarks/retrieval_benchmark.py | 680 +++++++++ tests/benchmarks/visualize_results.py | 529 +++++++ 14 files changed, 5661 insertions(+), 8 deletions(-) create mode 100644 docs/api/cli.rst create mode 100644 docs/concepts/index-migrations.md create mode 100644 docs/user_guide/13_sql_query_exercises.ipynb create mode 100644 docs/user_guide/how_to_guides/migrate-indexes.md create mode 100644 tests/benchmarks/index_migrator_real_benchmark.py create mode 100644 tests/benchmarks/migration_benchmark.py create mode 100644 tests/benchmarks/retrieval_benchmark.py create mode 100644 tests/benchmarks/visualize_results.py diff --git a/docs/api/cli.rst b/docs/api/cli.rst new file mode 100644 index 000000000..4f651a38c --- /dev/null +++ b/docs/api/cli.rst @@ -0,0 +1,614 @@ +********************** +Command Line Interface +********************** + +RedisVL provides a command line interface (CLI) called ``rvl`` for managing vector search indices. The CLI enables you to create, inspect, and delete indices directly from your terminal without writing Python code. + +Installation +============ + +The ``rvl`` command is included when you install RedisVL. + +.. code-block:: bash + + pip install redisvl + +Verify the installation by running: + +.. code-block:: bash + + rvl version + +Connection Configuration +======================== + +The CLI connects to Redis using the following resolution order: + +1. The ``REDIS_URL`` environment variable, if set +2. Explicit connection flags (``--host``, ``--port``, ``--url``) +3. Default values (``localhost:6379``) + +**Connection Flags** + +All commands that interact with Redis accept these optional flags: + +.. list-table:: + :widths: 20 15 50 15 + :header-rows: 1 + + * - Flag + - Type + - Description + - Default + * - ``-u``, ``--url`` + - string + - Full Redis URL (e.g., ``redis://localhost:6379``) + - None + * - ``--host`` + - string + - Redis server hostname + - ``localhost`` + * - ``-p``, ``--port`` + - integer + - Redis server port + - ``6379`` + * - ``--user`` + - string + - Redis username for authentication + - ``default`` + * - ``-a``, ``--password`` + - string + - Redis password for authentication + - Empty + * - ``--ssl`` + - flag + - Enable SSL/TLS encryption + - Disabled + +**Examples** + +Connect using environment variable: + +.. code-block:: bash + + export REDIS_URL="redis://localhost:6379" + rvl index listall + +Connect with explicit host and port: + +.. code-block:: bash + + rvl index listall --host myredis.example.com --port 6380 + +Connect with authentication and SSL: + +.. code-block:: bash + + rvl index listall --user admin --password secret --ssl + +Getting Help +============ + +All commands support the ``-h`` and ``--help`` flags to display usage information. + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Flag + - Description + * - ``-h``, ``--help`` + - Display usage information for the command + +**Examples** + +.. code-block:: bash + + # Display top-level help + rvl --help + + # Display help for a command group + rvl index --help + + # Display help for a specific subcommand + rvl index create --help + +Running ``rvl`` without any arguments also displays the top-level help message. + +.. tip:: + + For a hands-on tutorial with practical examples, see the :doc:`/user_guide/cli`. + +Commands +======== + +rvl version +----------- + +Display the installed RedisVL version. + +**Syntax** + +.. code-block:: bash + + rvl version [OPTIONS] + +**Options** + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Option + - Description + * - ``-s``, ``--short`` + - Print only the version number without additional formatting + +**Examples** + +.. code-block:: bash + + # Full version output + rvl version + + # Version number only + rvl version --short + +rvl index +--------- + +Manage vector search indices. This command group provides subcommands for creating, inspecting, listing, and removing indices. + +**Syntax** + +.. code-block:: bash + + rvl index [OPTIONS] + +**Subcommands** + +.. list-table:: + :widths: 15 85 + :header-rows: 1 + + * - Subcommand + - Description + * - ``create`` + - Create a new index from a YAML schema file + * - ``info`` + - Display detailed information about an index + * - ``listall`` + - List all existing indices in the Redis instance + * - ``delete`` + - Remove an index while preserving the underlying data + * - ``destroy`` + - Remove an index and delete all associated data + +rvl index create +^^^^^^^^^^^^^^^^ + +Create a new vector search index from a YAML schema definition. + +**Syntax** + +.. code-block:: bash + + rvl index create -s [CONNECTION_OPTIONS] + +**Required Options** + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Option + - Description + * - ``-s``, ``--schema`` + - Path to the YAML schema file defining the index structure + +**Example** + +.. code-block:: bash + + rvl index create -s schema.yaml + +**Schema File Format** + +The schema file must be valid YAML with the following structure: + +.. code-block:: yaml + + version: '0.1.0' + + index: + name: my_index + prefix: doc + storage_type: hash + + fields: + - name: content + type: text + - name: embedding + type: vector + attrs: + dims: 768 + algorithm: hnsw + distance_metric: cosine + +rvl index info +^^^^^^^^^^^^^^ + +Display detailed information about an existing index, including field definitions and index options. + +**Syntax** + +.. code-block:: bash + + rvl index info (-i | -s ) [OPTIONS] + +**Options** + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Option + - Description + * - ``-i``, ``--index`` + - Name of the index to inspect + * - ``-s``, ``--schema`` + - Path to the schema file (alternative to specifying index name) + +**Example** + +.. code-block:: bash + + rvl index info -i my_index + +**Output** + +The command displays two tables: + +1. **Index Information** containing the index name, storage type, key prefixes, index options, and indexing status +2. **Index Fields** listing each field with its name, attribute, type, and any additional field options + +rvl index listall +^^^^^^^^^^^^^^^^^ + +List all vector search indices in the connected Redis instance. + +**Syntax** + +.. code-block:: bash + + rvl index listall [CONNECTION_OPTIONS] + +**Example** + +.. code-block:: bash + + rvl index listall + +**Output** + +Returns a numbered list of all index names: + +.. code-block:: text + + Indices: + 1. products_index + 2. documents_index + 3. embeddings_index + +rvl index delete +^^^^^^^^^^^^^^^^ + +Remove an index from Redis while preserving the underlying data. Use this when you want to rebuild an index with a different schema without losing your data. + +**Syntax** + +.. code-block:: bash + + rvl index delete (-i | -s ) [CONNECTION_OPTIONS] + +**Options** + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Option + - Description + * - ``-i``, ``--index`` + - Name of the index to delete + * - ``-s``, ``--schema`` + - Path to the schema file (alternative to specifying index name) + +**Example** + +.. code-block:: bash + + rvl index delete -i my_index + +rvl index destroy +^^^^^^^^^^^^^^^^^ + +Remove an index and permanently delete all associated data from Redis. This operation cannot be undone. + +**Syntax** + +.. code-block:: bash + + rvl index destroy (-i | -s ) [CONNECTION_OPTIONS] + +**Options** + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Option + - Description + * - ``-i``, ``--index`` + - Name of the index to destroy + * - ``-s``, ``--schema`` + - Path to the schema file (alternative to specifying index name) + +**Example** + +.. code-block:: bash + + rvl index destroy -i my_index + +.. warning:: + + This command permanently deletes both the index and all documents stored with the index prefix. Ensure you have backups before running this command. + +rvl stats +--------- + +Display statistics about an existing index, including document counts, memory usage, and indexing performance metrics. + +**Syntax** + +.. code-block:: bash + + rvl stats (-i | -s ) [OPTIONS] + +**Options** + +.. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Option + - Description + * - ``-i``, ``--index`` + - Name of the index to query + * - ``-s``, ``--schema`` + - Path to the schema file (alternative to specifying index name) + +**Example** + +.. code-block:: bash + + rvl stats -i my_index + +**Statistics Reference** + +The command returns the following metrics: + +.. list-table:: + :widths: 35 65 + :header-rows: 1 + + * - Metric + - Description + * - ``num_docs`` + - Total number of indexed documents + * - ``num_terms`` + - Number of distinct terms in text fields + * - ``max_doc_id`` + - Highest internal document ID + * - ``num_records`` + - Total number of index records + * - ``percent_indexed`` + - Percentage of documents fully indexed + * - ``hash_indexing_failures`` + - Number of documents that failed to index + * - ``number_of_uses`` + - Number of times the index has been queried + * - ``bytes_per_record_avg`` + - Average bytes per index record + * - ``doc_table_size_mb`` + - Document table size in megabytes + * - ``inverted_sz_mb`` + - Inverted index size in megabytes + * - ``key_table_size_mb`` + - Key table size in megabytes + * - ``offset_bits_per_record_avg`` + - Average offset bits per record + * - ``offset_vectors_sz_mb`` + - Offset vectors size in megabytes + * - ``offsets_per_term_avg`` + - Average offsets per term + * - ``records_per_doc_avg`` + - Average records per document + * - ``sortable_values_size_mb`` + - Sortable values size in megabytes + * - ``total_indexing_time`` + - Total time spent indexing in milliseconds + * - ``total_inverted_index_blocks`` + - Number of inverted index blocks + * - ``vector_index_sz_mb`` + - Vector index size in megabytes + +rvl migrate +----------- + +Manage document-preserving index migrations. This command group provides subcommands for planning, executing, and validating schema migrations that preserve existing data. + +**Syntax** + +.. code-block:: bash + + rvl migrate [OPTIONS] + +**Subcommands** + +.. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - Subcommand + - Description + * - ``helper`` + - Show migration guidance and supported capabilities + * - ``list`` + - List all available indexes + * - ``plan`` + - Generate a migration plan from a schema patch or target schema + * - ``wizard`` + - Interactively build a migration plan and schema patch + * - ``apply`` + - Execute a reviewed drop/recreate migration plan + * - ``estimate`` + - Estimate disk space required for a migration (dry-run) + * - ``validate`` + - Validate a completed migration against the live index + * - ``batch-plan`` + - Generate a batch migration plan for multiple indexes + * - ``batch-apply`` + - Execute a batch migration plan with checkpointing + * - ``batch-resume`` + - Resume an interrupted batch migration + * - ``batch-status`` + - Show status of an in-progress or completed batch migration + +rvl migrate plan +^^^^^^^^^^^^^^^^ + +Generate a migration plan for a document-preserving drop/recreate migration. + +**Syntax** + +.. code-block:: bash + + rvl migrate plan --index (--patch | --target-schema ) [OPTIONS] + +**Required Options** + +.. list-table:: + :widths: 30 70 + :header-rows: 1 + + * - Option + - Description + * - ``--index``, ``-i`` + - Name of the source index to migrate + * - ``--patch`` + - Path to a YAML schema patch file (mutually exclusive with ``--target-schema``) + * - ``--target-schema`` + - Path to a full target schema YAML file (mutually exclusive with ``--patch``) + +**Optional Options** + +.. list-table:: + :widths: 30 70 + :header-rows: 1 + + * - Option + - Description + * - ``--output``, ``-o`` + - Output path for the migration plan YAML (default: ``migration_plan.yaml``) + +**Example** + +.. code-block:: bash + + rvl migrate plan -i my_index --patch changes.yaml -o plan.yaml + +rvl migrate apply +^^^^^^^^^^^^^^^^^ + +Execute a reviewed drop/recreate migration plan. Use ``--async`` for large migrations involving vector quantization. + +**Syntax** + +.. code-block:: bash + + rvl migrate apply --plan [OPTIONS] + +**Required Options** + +.. list-table:: + :widths: 30 70 + :header-rows: 1 + + * - Option + - Description + * - ``--plan`` + - Path to the migration plan YAML file + +**Optional Options** + +.. list-table:: + :widths: 30 70 + :header-rows: 1 + + * - Option + - Description + * - ``--async`` + - Run migration asynchronously (recommended for large quantization jobs) + * - ``--query-check`` + - Path to a YAML file with post-migration query checks + * - ``--resume`` + - Path to a checkpoint file for crash-safe recovery + +**Example** + +.. code-block:: bash + + rvl migrate apply --plan plan.yaml + rvl migrate apply --plan plan.yaml --async --resume checkpoint.yaml + +rvl migrate wizard +^^^^^^^^^^^^^^^^^^ + +Interactively build a schema patch and migration plan through a guided wizard. + +**Syntax** + +.. code-block:: bash + + rvl migrate wizard [--index ] [OPTIONS] + +**Example** + +.. code-block:: bash + + rvl migrate wizard -i my_index -o plan.yaml + +Exit Codes +========== + +The CLI returns the following exit codes: + +.. list-table:: + :widths: 15 85 + :header-rows: 1 + + * - Code + - Description + * - ``0`` + - Command completed successfully + * - ``1`` + - Command failed due to missing required arguments or invalid input + +Related Resources +================= + +- :doc:`/user_guide/cli` for a tutorial-style walkthrough +- :doc:`schema` for YAML schema format details +- :doc:`searchindex` for the Python ``SearchIndex`` API + diff --git a/docs/concepts/field-attributes.md b/docs/concepts/field-attributes.md index c7764a4a7..96060d2fb 100644 --- a/docs/concepts/field-attributes.md +++ b/docs/concepts/field-attributes.md @@ -267,7 +267,7 @@ Key vector attributes: - `dims`: Vector dimensionality (required) - `algorithm`: `flat`, `hnsw`, or `svs-vamana` - `distance_metric`: `COSINE`, `L2`, or `IP` -- `datatype`: `float16`, `float32`, `float64`, or `bfloat16` +- `datatype`: Vector precision (see table below) - `index_missing`: Allow searching for documents without vectors ```yaml @@ -281,6 +281,48 @@ Key vector attributes: index_missing: true # Handle documents without embeddings ``` +### Vector Datatypes + +The `datatype` attribute controls how vector components are stored. Smaller datatypes reduce memory usage but may affect precision. + +| Datatype | Bits | Memory (768 dims) | Use Case | +|----------|------|-------------------|----------| +| `float32` | 32 | 3 KB | Default. Best precision for most applications. | +| `float16` | 16 | 1.5 KB | Good balance of memory and precision. Recommended for large-scale deployments. | +| `bfloat16` | 16 | 1.5 KB | Better dynamic range than float16. Useful when embeddings have large value ranges. | +| `float64` | 64 | 6 KB | Maximum precision. Rarely needed. | +| `int8` | 8 | 768 B | Integer quantization. Significant memory savings with some precision loss. | +| `uint8` | 8 | 768 B | Unsigned integer quantization. For embeddings with non-negative values. | + +**Algorithm Compatibility:** + +| Datatype | FLAT | HNSW | SVS-VAMANA | +|----------|------|------|------------| +| `float32` | Yes | Yes | Yes | +| `float16` | Yes | Yes | Yes | +| `bfloat16` | Yes | Yes | No | +| `float64` | Yes | Yes | No | +| `int8` | Yes | Yes | No | +| `uint8` | Yes | Yes | No | + +**Choosing a Datatype:** + +- **Start with `float32`** unless you have memory constraints +- **Use `float16`** for production systems with millions of vectors (50% memory savings, minimal precision loss) +- **Use `int8`/`uint8`** only after benchmarking recall on your specific dataset +- **SVS-VAMANA users**: Must use `float16` or `float32` + +**Quantization with the Migrator:** + +You can change vector datatypes on existing indexes using the migration wizard: + +```bash +rvl migrate wizard --index my_index --url redis://localhost:6379 +# Select "Update field" > choose vector field > change datatype +``` + +The migrator automatically re-encodes stored vectors to the new precision. See {doc}`/user_guide/how_to_guides/migrate-indexes` for details. + ## Redis-Specific Subtleties ### Modifier Ordering @@ -304,6 +346,53 @@ Not all attributes work with all field types: | `unf` | ✓ | ✗ | ✓ | ✗ | ✗ | | `withsuffixtrie` | ✓ | ✓ | ✗ | ✗ | ✗ | +### Migration Support + +The migration wizard (`rvl migrate wizard`) supports updating field attributes on existing indexes. The table below shows which attributes can be updated via the wizard vs requiring manual schema patch editing. + +**Wizard Prompts:** + +| Attribute | Text | Tag | Numeric | Geo | Vector | +|-----------|------|-----|---------|-----|--------| +| `sortable` | Wizard | Wizard | Wizard | Wizard | N/A | +| `index_missing` | Wizard | Wizard | Wizard | Wizard | N/A | +| `index_empty` | Wizard | Wizard | N/A | N/A | N/A | +| `no_index` | Wizard | Wizard | Wizard | Wizard | N/A | +| `unf` | Wizard* | N/A | Wizard* | N/A | N/A | +| `separator` | N/A | Wizard | N/A | N/A | N/A | +| `case_sensitive` | N/A | Wizard | N/A | N/A | N/A | +| `no_stem` | Wizard | N/A | N/A | N/A | N/A | +| `weight` | Wizard | N/A | N/A | N/A | N/A | +| `algorithm` | N/A | N/A | N/A | N/A | Wizard | +| `datatype` | N/A | N/A | N/A | N/A | Wizard | +| `distance_metric` | N/A | N/A | N/A | N/A | Wizard | +| `m`, `ef_construction` | N/A | N/A | N/A | N/A | Wizard | + +*\* `unf` is only prompted when `sortable` is enabled.* + +**Manual Schema Patch Required:** + +| Attribute | Notes | +|-----------|-------| +| `phonetic_matcher` | Enable phonetic search | +| `withsuffixtrie` | Suffix/contains search optimization | + +**Example manual patch** for adding `index_missing` to a field: + +```yaml +# schema_patch.yaml +version: 1 +changes: + update_fields: + - name: category + attrs: + index_missing: true +``` + +```bash +rvl migrate plan --index my_index --schema-patch schema_patch.yaml +``` + ### JSON Path for Nested Fields When using JSON storage, use the `path` attribute to index nested fields: diff --git a/docs/concepts/index-migrations.md b/docs/concepts/index-migrations.md new file mode 100644 index 000000000..065522d98 --- /dev/null +++ b/docs/concepts/index-migrations.md @@ -0,0 +1,255 @@ +--- +myst: + html_meta: + "description lang=en": | + Learn how RedisVL index migrations work and which schema changes are supported. +--- + +# Index Migrations + +Redis Search indexes are immutable. To change an index schema, you must drop the existing index and create a new one. RedisVL provides a migration workflow that automates this process while preserving your data. + +This page explains how migrations work and which changes are supported. For step by step instructions, see the [migration guide](../user_guide/how_to_guides/migrate-indexes.md). + +## Supported and blocked changes + +The migrator classifies schema changes into two categories: + +| Change | Status | +|--------|--------| +| Add or remove a field | Supported | +| Rename a field | Supported | +| Change field options (sortable, separator) | Supported | +| Change key prefix | Supported | +| Rename the index | Supported | +| Change vector algorithm (FLAT, HNSW, SVS-VAMANA) | Supported | +| Change distance metric (COSINE, L2, IP) | Supported | +| Tune algorithm parameters (M, EF_CONSTRUCTION) | Supported | +| Quantize vectors (float32 to float16/bfloat16/int8/uint8) | Supported | +| Change vector dimensions | Blocked | +| Change storage type (hash to JSON) | Blocked | +| Add a new vector field | Blocked | + +**Note:** INT8 and UINT8 vector datatypes require Redis 8.0+. SVS-VAMANA algorithm requires Redis 8.2+ and Intel AVX-512 hardware. + +**Supported** changes can be applied automatically using `rvl migrate`. The migrator handles the index rebuild and any necessary data transformations. + +**Blocked** changes require manual intervention because they involve incompatible data formats or missing data. The migrator will reject these changes and explain why. + +## How the migrator works + +The migrator uses a plan first workflow: + +1. **Plan**: Capture the current schema, classify your changes, and generate a migration plan +2. **Review**: Inspect the plan before making any changes +3. **Apply**: Drop the index, transform data if needed, and recreate with the new schema +4. **Validate**: Verify the result matches expectations + +This separation ensures you always know what will happen before any changes are made. + +## Migration mode: drop_recreate + +The `drop_recreate` mode rebuilds the index in place while preserving your documents. + +The process: + +1. Drop only the index structure (documents remain in Redis) +2. For datatype changes, re-encode vectors to the target precision +3. Recreate the index with the new schema +4. Wait for Redis to re-index the existing documents +5. Validate the result + +**Tradeoff**: The index is unavailable during the rebuild. Review the migration plan carefully before applying. + +## Index only vs document dependent changes + +Schema changes fall into two categories based on whether they require modifying stored data. + +**Index only changes** affect how Redis Search indexes data, not the data itself: + +- Algorithm changes: The stored vector bytes are identical. Only the index structure differs. +- Distance metric changes: Same vectors, different similarity calculation. +- Adding or removing fields: The documents already contain the data. The index just starts or stops indexing it. + +These changes complete quickly because they only require rebuilding the index. + +**Document dependent changes** require modifying the stored data: + +- Datatype changes (float32 to float16): Stored vector bytes must be re-encoded. +- Field renames: Stored field names must be updated in every document. +- Dimension changes: Vectors must be re-embedded with a different model. + +The migrator handles datatype changes automatically. Other document dependent changes are blocked because they require application level logic or external services. + +## Vector quantization + +Changing vector precision from float32 to float16 reduces memory usage at the cost of slight precision loss. The migrator handles this automatically by: + +1. Reading all vectors from Redis +2. Converting to the target precision +3. Writing updated vectors back +4. Recreating the index with the new schema + +Typical reductions: + +| Metric | Value | +|--------|-------| +| Index size reduction | ~50% | +| Memory reduction | ~35% | + +Quantization time is proportional to document count. Plan for downtime accordingly. + +## Why some changes are blocked + +### Vector dimension changes + +Vector dimensions are determined by your embedding model. A 384 dimensional vector from one model is mathematically incompatible with a 768 dimensional index expecting vectors from a different model. There is no way to resize an embedding. + +**Resolution**: Re-embed your documents using the new model and load them into a new index. + +### Storage type changes + +Hash and JSON have different data layouts. Hash stores flat key value pairs. JSON stores nested structures. Converting between them requires understanding your schema and restructuring each document. + +**Resolution**: Export your data, transform it to the new format, and reload into a new index. + +### Adding a vector field + +Adding a vector field means all existing documents need vectors for that field. The migrator cannot generate these vectors because it does not know which embedding model to use or what content to embed. + +**Resolution**: Add vectors to your documents using your application, then run the migration. + +## Downtime considerations + +With `drop_recreate`, your index is unavailable between the drop and when re-indexing completes. + +**CRITICAL**: Downtime requires both reads AND writes to be paused: + +| Requirement | Reason | +|-------------|--------| +| **Pause reads** | Index is unavailable during migration | +| **Pause writes** | Redis updates indexes synchronously. Writes during migration may conflict with vector re-encoding or be missed | + +Plan for: + +- Search unavailability during the migration window +- Partial results while indexing is in progress +- Resource usage from the re-indexing process +- Quantization time if changing vector datatypes + +The duration depends on document count, field count, and vector dimensions. For large indexes, consider running migrations during low traffic periods. + +## Sync vs async execution + +The migrator provides both synchronous and asynchronous execution modes. + +### What becomes async and what stays sync + +The migration workflow has distinct phases. Here is what each mode affects: + +| Phase | Sync mode | Async mode | Notes | +|-------|-----------|------------|-------| +| **Plan generation** | `MigrationPlanner.create_plan()` | `AsyncMigrationPlanner.create_plan()` | Reads index metadata from Redis | +| **Schema snapshot** | Sync Redis calls | Async Redis calls | Single `FT.INFO` command | +| **Enumeration** | FT.AGGREGATE (or SCAN fallback) | FT.AGGREGATE (or SCAN fallback) | Before drop, only if quantization needed | +| **Drop index** | `index.delete()` | `await index.delete()` | Single `FT.DROPINDEX` command | +| **Quantization** | Sequential HGET + HSET | Sequential HGET + batched HSET | Uses pre-enumerated keys | +| **Create index** | `index.create()` | `await index.create()` | Single `FT.CREATE` command | +| **Readiness polling** | `time.sleep()` loop | `asyncio.sleep()` loop | Polls `FT.INFO` until indexed | +| **Validation** | Sync Redis calls | Async Redis calls | Schema and doc count checks | +| **CLI interaction** | Always sync | Always sync | User prompts, file I/O | +| **YAML read/write** | Always sync | Always sync | Local filesystem only | + +### When to use sync (default) + +Sync execution is simpler and sufficient for most migrations: + +- Small to medium indexes (under 100K documents) +- Index-only changes (algorithm, distance metric, field options) +- Interactive CLI usage where blocking is acceptable + +For migrations without quantization, the Redis operations are fast single commands. Sync mode adds no meaningful overhead. + +### When to use async + +Async execution (`--async` flag) provides benefits in specific scenarios: + +**Large quantization jobs (1M+ vectors)** + +Converting float32 to float16 requires reading every vector, converting it, and writing it back. The async executor: + +- Enumerates documents using `FT.AGGREGATE WITHCURSOR` for index-specific enumeration (falls back to `SCAN` only if indexing failures exist) +- Pipelines `HSET` operations in batches (100-1000 operations per pipeline is optimal for Redis) +- Yields to the event loop between batches so other tasks can proceed + +**Large keyspaces (40M+ keys)** + +When your Redis instance has many keys and the index has indexing failures (requiring SCAN fallback), async mode yields between batches. + +**Async application integration** + +If your application uses asyncio, you can integrate migration directly: + +```python +import asyncio +from redisvl.migration import AsyncMigrationPlanner, AsyncMigrationExecutor + +async def migrate(): + planner = AsyncMigrationPlanner() + plan = await planner.create_plan("myindex", redis_url="redis://localhost:6379") + + executor = AsyncMigrationExecutor() + report = await executor.apply(plan, redis_url="redis://localhost:6379") + +asyncio.run(migrate()) +``` + +### Why async helps with quantization + +The migrator uses an optimized enumeration strategy: + +1. **Index-based enumeration**: Uses `FT.AGGREGATE WITHCURSOR` to enumerate only indexed documents (not the entire keyspace) +2. **Fallback for safety**: If the index has indexing failures (`hash_indexing_failures > 0`), falls back to `SCAN` to ensure completeness +3. **Enumerate before drop**: Captures the document list while the index still exists, then drops and quantizes + +This optimization provides 10-1000x speedup for sparse indexes (where only a small fraction of prefix-matching keys are indexed). + +**Sync quantization:** +``` +enumerate keys (FT.AGGREGATE or SCAN) -> store list +for each batch of 500 keys: + for each key: + HGET field (blocks) + convert array + pipeline.HSET(field, new_bytes) + pipeline.execute() (blocks) +``` + +**Async quantization:** +``` +enumerate keys (FT.AGGREGATE or SCAN) -> store list +for each batch of 500 keys: + for each key: + await HGET field (yields) + convert array + pipeline.HSET(field, new_bytes) + await pipeline.execute() (yields) +``` + +Each `await` is a yield point where other coroutines can run. For millions of vectors, this prevents your application from freezing. + +### What async does NOT improve + +Async execution does not reduce: + +- **Total migration time**: Same work, different scheduling +- **Redis server load**: Same commands execute on the server +- **Downtime window**: Index remains unavailable during rebuild +- **Network round trips**: Same number of Redis calls + +The benefit is application responsiveness, not faster migration. + +## Learn more + +- [Migration guide](../user_guide/how_to_guides/migrate-indexes.md): Step by step instructions +- [Search and indexing](search-and-indexing.md): How Redis Search indexes work diff --git a/docs/concepts/index.md b/docs/concepts/index.md index 0e522b1a2..02f4d8b01 100644 --- a/docs/concepts/index.md +++ b/docs/concepts/index.md @@ -26,6 +26,13 @@ How RedisVL components connect: schemas, indexes, queries, and extensions. Schemas, fields, documents, storage types, and query patterns. ::: +:::{grid-item-card} 🔄 Index Migrations +:link: index-migrations +:link-type: doc + +How RedisVL handles migration planning, rebuilds, and future shadow migration. +::: + :::{grid-item-card} 🏷️ Field Attributes :link: field-attributes :link-type: doc @@ -62,6 +69,7 @@ Pre-built patterns: caching, message history, and semantic routing. architecture search-and-indexing +index-migrations field-attributes queries utilities diff --git a/docs/concepts/search-and-indexing.md b/docs/concepts/search-and-indexing.md index b4fe69569..5312d7dfb 100644 --- a/docs/concepts/search-and-indexing.md +++ b/docs/concepts/search-and-indexing.md @@ -106,9 +106,14 @@ To change a schema, you create a new index with the updated configuration, reind Planning your schema carefully upfront reduces the need for migrations, but the capability exists when requirements evolve. ---- +RedisVL now includes a dedicated migration workflow for this lifecycle: + +- `drop_recreate` for document-preserving rebuilds, including vector quantization (`float32` → `float16`) -**Related concepts:** {doc}`field-attributes` explains how to configure field options like `sortable` and `index_missing`. {doc}`queries` covers the different query types available. +That means schema evolution is no longer only a manual operational pattern. It is also a product surface in RedisVL with a planner, CLI, and validation artifacts. + +--- -**Learn more:** {doc}`/user_guide/01_getting_started` walks through building your first index. {doc}`/user_guide/05_hash_vs_json` compares storage options in depth. {doc}`/user_guide/02_complex_filtering` covers query composition. +**Related concepts:** {doc}`field-attributes` explains how to configure field options like `sortable` and `index_missing`. {doc}`queries` covers the different query types available. {doc}`index-migrations` explains migration modes, supported changes, and architecture. +**Learn more:** {doc}`/user_guide/01_getting_started` walks through building your first index. {doc}`/user_guide/05_hash_vs_json` compares storage options in depth. {doc}`/user_guide/02_complex_filtering` covers query composition. {doc}`/user_guide/how_to_guides/migrate-indexes` shows how to use the migration CLI in practice. diff --git a/docs/user_guide/13_sql_query_exercises.ipynb b/docs/user_guide/13_sql_query_exercises.ipynb new file mode 100644 index 000000000..c09af709f --- /dev/null +++ b/docs/user_guide/13_sql_query_exercises.ipynb @@ -0,0 +1,1239 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# SQL-to-Redis Query Translation: Hands-On Exercises\n", + "\n", + "This notebook provides hands-on exercises for learning the new **SQLQuery** feature in RedisVL, which allows you to write familiar SQL syntax that automatically translates to Redis Search commands.\n", + "\n", + "## What You'll Learn\n", + "\n", + "1. How to use the `SQLQuery` class to write SQL-like queries\n", + "2. Three equivalent approaches for the same queries:\n", + " - **RedisVL Python API** - Using native query classes (`FilterQuery`, `VectorQuery`, etc.)\n", + " - **RedisVL SQL** - Using the new `SQLQuery` class with SQL syntax\n", + " - **Raw Redis FT.SEARCH** - The equivalent Redis Search command\n", + "3. Various query types: filtering, numeric ranges, text search, aggregations, and vector similarity\n", + "\n", + "## Prerequisites\n", + "\n", + "- Redis Stack running locally (or Redis Cloud)\n", + "- RedisVL with SQL support: `pip install redisvl[sql-redis]`\n", + "\n", + "## Documentation References\n", + "\n", + "- [RedisVL Documentation](https://docs.redisvl.com)\n", + "- [Redis Search Query Syntax](https://redis.io/docs/latest/develop/ai/search-and-query/query/)\n", + "- [Redis Aggregations](https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/aggregations/)\n", + "- [sql-redis Package](https://pypi.org/project/sql-redis/)\n", + "\n", + "---" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup: Create Sample Dataset and Index\n", + "\n", + "We'll create a realistic e-commerce products dataset with multiple field types to demonstrate various query capabilities." + ] + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T22:52:37.820727Z", + "start_time": "2026-02-05T22:52:37.431065Z" + } + }, + "source": [ + "import numpy as np\n", + "from redis import Redis\n", + "from redisvl.index import SearchIndex\n", + "from redisvl.query import FilterQuery, VectorQuery, CountQuery, SQLQuery\n", + "from redisvl.query.filter import Tag, Num, Text\n", + "\n", + "# Redis connection\n", + "REDIS_URL = \"redis://localhost:6379\"\n", + "client = Redis.from_url(REDIS_URL)\n", + "\n", + "# Define schema with multiple field types\n", + "schema = {\n", + " \"index\": {\n", + " \"name\": \"products_exercise\",\n", + " \"prefix\": \"product_exercise\",\n", + " \"storage_type\": \"hash\",\n", + " },\n", + " \"fields\": [\n", + " {\"name\": \"name\", \"type\": \"text\", \"attrs\": {\"sortable\": True}},\n", + " {\"name\": \"description\", \"type\": \"text\"},\n", + " {\"name\": \"category\", \"type\": \"tag\", \"attrs\": {\"sortable\": True}},\n", + " {\"name\": \"brand\", \"type\": \"tag\"},\n", + " {\"name\": \"price\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", + " {\"name\": \"stock\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", + " {\"name\": \"rating\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", + " {\n", + " \"name\": \"embedding\",\n", + " \"type\": \"vector\",\n", + " \"attrs\": {\n", + " \"dims\": 4,\n", + " \"distance_metric\": \"cosine\",\n", + " \"algorithm\": \"flat\",\n", + " \"datatype\": \"float32\",\n", + " },\n", + " },\n", + " ],\n", + "}\n", + "\n", + "# Create the index\n", + "index = SearchIndex.from_dict(schema, redis_client=client)\n", + "index.create(overwrite=True, drop=True)\n", + "print(f\"Created index: {index.name}\")" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Created index: products_exercise\n" + ] + } + ], + "execution_count": 1 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T22:52:57.150152Z", + "start_time": "2026-02-05T22:52:57.105851Z" + } + }, + "source": [ + "# Sample product data with embeddings\n", + "products = [\n", + " {\"name\": \"Gaming Laptop Pro\", \"description\": \"High-performance laptop for gaming\", \"category\": \"electronics\", \"brand\": \"TechBrand\", \"price\": 1299, \"stock\": 15, \"rating\": 4.7, \"embedding\": np.array([0.9, 0.1, 0.2, 0.3], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Budget Laptop Basic\", \"description\": \"Affordable laptop for everyday tasks\", \"category\": \"electronics\", \"brand\": \"ValueTech\", \"price\": 499, \"stock\": 50, \"rating\": 4.0, \"embedding\": np.array([0.8, 0.2, 0.3, 0.4], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Wireless Mouse\", \"description\": \"Ergonomic wireless mouse\", \"category\": \"electronics\", \"brand\": \"TechBrand\", \"price\": 35, \"stock\": 200, \"rating\": 4.3, \"embedding\": np.array([0.7, 0.3, 0.4, 0.5], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Python Programming Guide\", \"description\": \"Comprehensive Python programming guide\", \"category\": \"books\", \"brand\": \"TechBooks\", \"price\": 45, \"stock\": 100, \"rating\": 4.8, \"embedding\": np.array([0.2, 0.8, 0.1, 0.3], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Redis in Action\", \"description\": \"Learn Redis with practical examples\", \"category\": \"books\", \"brand\": \"TechBooks\", \"price\": 55, \"stock\": 75, \"rating\": 4.6, \"embedding\": np.array([0.3, 0.7, 0.2, 0.4], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Data Science Handbook\", \"description\": \"Essential data science handbook\", \"category\": \"books\", \"brand\": \"DataPress\", \"price\": 65, \"stock\": 40, \"rating\": 4.5, \"embedding\": np.array([0.25, 0.75, 0.15, 0.35], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Mechanical Keyboard\", \"description\": \"Premium mechanical keyboard with RGB\", \"category\": \"electronics\", \"brand\": \"KeyMaster\", \"price\": 149, \"stock\": 80, \"rating\": 4.6, \"embedding\": np.array([0.6, 0.4, 0.5, 0.6], dtype=np.float32).tobytes()},\n", + " {\"name\": \"USB-C Hub\", \"description\": \"Multi-port USB-C hub\", \"category\": \"electronics\", \"brand\": \"TechBrand\", \"price\": 49, \"stock\": 150, \"rating\": 4.2, \"embedding\": np.array([0.65, 0.35, 0.45, 0.55], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Desk Lamp LED\", \"description\": \"Adjustable LED desk lamp\", \"category\": \"accessories\", \"brand\": \"LightCo\", \"price\": 39, \"stock\": 120, \"rating\": 4.1, \"embedding\": np.array([0.4, 0.5, 0.6, 0.7], dtype=np.float32).tobytes()},\n", + " {\"name\": \"Monitor Stand\", \"description\": \"Ergonomic monitor stand\", \"category\": \"accessories\", \"brand\": \"DeskPro\", \"price\": 79, \"stock\": 60, \"rating\": 4.4, \"embedding\": np.array([0.45, 0.55, 0.65, 0.75], dtype=np.float32).tobytes()},\n", + "]\n", + "\n", + "# Load data into Redis\n", + "keys = index.load(products)\n", + "print(f\"Loaded {len(keys)} products into Redis\")" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded 10 products into Redis\n" + ] + } + ], + "execution_count": 2 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Exercise 1: Simple Tag Filtering\n", + "\n", + "**Goal:** Find all products in the \"electronics\" category.\n", + "\n", + "### Do It Yourself\n", + "\n", + "**Documentation:**\n", + "- [RedisVL FilterQuery](https://docs.redisvl.com/en/latest/api/query.html#filterquery)\n", + "- [Redis Tag Queries](https://redis.io/docs/latest/develop/ai/search-and-query/query/exact-match/)" + ] + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T22:56:41.131491Z", + "start_time": "2026-02-05T22:56:41.113689Z" + } + }, + "source": [ + "# YOUR CODE HERE - Method 1: RedisVL Python API\n", + "# Hint: Use Tag(\"category\") == \"electronics\" with FilterQuery\n", + "q= FilterQuery(\n", + " filter_expression=Tag(\"category\") == \"electronics\",\n", + " return_fields=[\"name\", \"category\", \"price\"],\n", + " num_results=10\n", + ")\n", + "\n", + "q = index.query(q)\n", + "q" + ], + "outputs": [ + { + "data": { + "text/plain": [ + "[{'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GK',\n", + " 'name': 'Wireless Mouse',\n", + " 'category': 'electronics',\n", + " 'price': '35'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GQ',\n", + " 'name': 'Mechanical Keyboard',\n", + " 'category': 'electronics',\n", + " 'price': '149'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GH',\n", + " 'name': 'Gaming Laptop Pro',\n", + " 'category': 'electronics',\n", + " 'price': '1299'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GJ',\n", + " 'name': 'Budget Laptop Basic',\n", + " 'category': 'electronics',\n", + " 'price': '499'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", + " 'name': 'USB-C Hub',\n", + " 'category': 'electronics',\n", + " 'price': '49'}]" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 5 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T22:58:38.451826Z", + "start_time": "2026-02-05T22:58:38.269871Z" + } + }, + "source": [ + "# YOUR CODE HERE - Method 2: SQLQuery\n", + "# Hint: SELECT ... FROM products_exercise WHERE category = 'electronics'\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT name, category, price\n", + " FROM {index.name}\n", + " WHERE category = 'electronics'\n", + "\"\"\")\n", + "index.query(sql_query)" + ], + "outputs": [ + { + "data": { + "text/plain": [ + "[{'name': 'Wireless Mouse', 'category': 'electronics', 'price': '35'},\n", + " {'name': 'Mechanical Keyboard', 'category': 'electronics', 'price': '149'},\n", + " {'name': 'Gaming Laptop Pro', 'category': 'electronics', 'price': '1299'},\n", + " {'name': 'Budget Laptop Basic', 'category': 'electronics', 'price': '499'},\n", + " {'name': 'USB-C Hub', 'category': 'electronics', 'price': '49'}]" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 7 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T22:59:22.518910Z", + "start_time": "2026-02-05T22:59:22.495076Z" + } + }, + "source": [ + "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", + "# Hint: client.execute_command(\"FT.SEARCH\", index_name, \"@category:{electronics}\", ...)\n", + "client.execute_command(\"FT.Search\", index.name, \"@category:{electronics}\")" + ], + "outputs": [ + { + "data": { + "text/plain": [ + "[5,\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GK',\n", + " [b'name',\n", + " b'Wireless Mouse',\n", + " b'description',\n", + " b'Ergonomic wireless mouse',\n", + " b'category',\n", + " b'electronics',\n", + " b'brand',\n", + " b'TechBrand',\n", + " b'price',\n", + " b'35',\n", + " b'stock',\n", + " b'200',\n", + " b'rating',\n", + " b'4.3',\n", + " b'embedding',\n", + " b'333?\\x9a\\x99\\x99>\\xcd\\xcc\\xcc>\\x00\\x00\\x00?'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GQ',\n", + " [b'name',\n", + " b'Mechanical Keyboard',\n", + " b'description',\n", + " b'Premium mechanical keyboard with RGB',\n", + " b'category',\n", + " b'electronics',\n", + " b'brand',\n", + " b'KeyMaster',\n", + " b'price',\n", + " b'149',\n", + " b'stock',\n", + " b'80',\n", + " b'rating',\n", + " b'4.6',\n", + " b'embedding',\n", + " b'\\x9a\\x99\\x19?\\xcd\\xcc\\xcc>\\x00\\x00\\x00?\\x9a\\x99\\x19?'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GH',\n", + " [b'name',\n", + " b'Gaming Laptop Pro',\n", + " b'description',\n", + " b'High-performance laptop for gaming',\n", + " b'category',\n", + " b'electronics',\n", + " b'brand',\n", + " b'TechBrand',\n", + " b'price',\n", + " b'1299',\n", + " b'stock',\n", + " b'15',\n", + " b'rating',\n", + " b'4.7',\n", + " b'embedding',\n", + " b'fff?\\xcd\\xcc\\xcc=\\xcd\\xccL>\\x9a\\x99\\x99>'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GJ',\n", + " [b'name',\n", + " b'Budget Laptop Basic',\n", + " b'description',\n", + " b'Affordable laptop for everyday tasks',\n", + " b'category',\n", + " b'electronics',\n", + " b'brand',\n", + " b'ValueTech',\n", + " b'price',\n", + " b'499',\n", + " b'stock',\n", + " b'50',\n", + " b'rating',\n", + " b'4.0',\n", + " b'embedding',\n", + " b'\\xcd\\xccL?\\xcd\\xccL>\\x9a\\x99\\x99>\\xcd\\xcc\\xcc>'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", + " [b'name',\n", + " b'USB-C Hub',\n", + " b'description',\n", + " b'Multi-port USB-C hub',\n", + " b'category',\n", + " b'electronics',\n", + " b'brand',\n", + " b'TechBrand',\n", + " b'price',\n", + " b'49',\n", + " b'stock',\n", + " b'150',\n", + " b'rating',\n", + " b'4.2',\n", + " b'embedding',\n", + " b'ff&?33\\xb3>ff\\xe6>\\xcd\\xcc\\x0c?']]" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 9 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Solution: Exercise 1" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 1: RedisVL Python API\n", + "filter_expr = Tag(\"category\") == \"electronics\"\n", + "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"category\", \"price\"], num_results=10)\n", + "results_api = index.query(query)\n", + "print(\"=== Method 1: RedisVL Python API ===\")\n", + "for r in results_api:\n", + " print(f\" {r['name']} - ${r['price']}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 2: RedisVL SQL\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT name, category, price\n", + " FROM {index.name}\n", + " WHERE category = 'electronics'\n", + "\"\"\")\n", + "results_sql = index.query(sql_query)\n", + "print(\"=== Method 2: RedisVL SQL ===\")\n", + "for r in results_sql:\n", + " print(f\" {r['name']} - ${r['price']}\")\n", + "\n", + "# Show the translated Redis command\n", + "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", + "print(f\"\\nTranslated Redis command: {redis_cmd}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 3: Raw Redis FT.SEARCH\n", + "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"@category:{electronics}\", \"RETURN\", \"3\", \"name\", \"category\", \"price\", \"LIMIT\", \"0\", \"10\")\n", + "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", + "total = raw_results[0]\n", + "print(f\"Total matches: {total}\")\n", + "for i in range(1, len(raw_results), 2):\n", + " if i + 1 < len(raw_results):\n", + " fields = raw_results[i + 1]\n", + " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", + " print(f\" {field_dict.get('name', 'N/A')} - ${field_dict.get('price', 'N/A')}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Exercise 2: Numeric Range Queries\n", + "\n", + "**Goal:** Find all products with price between $40 and $100.\n", + "\n", + "### Do It Yourself\n", + "\n", + "**Documentation:**\n", + "- [RedisVL Numeric Filters](https://docs.redisvl.com/en/latest/api/query.html#redisvl.query.filter.Num)\n", + "- [Redis Numeric Range Queries](https://redis.io/docs/latest/develop/ai/search-and-query/query/range/)" + ] + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:01:53.873244Z", + "start_time": "2026-02-05T23:01:53.847120Z" + } + }, + "source": [ + "# YOUR CODE HERE - Method 1: RedisVL Python API\n", + "# Hint: Use Num(\"price\").between(40, 100) with FilterQuery\n", + "q = FilterQuery(\n", + " filter_expression = Num(\"price\").between(40,100),\n", + " return_fields=[\"name\", \"price\"],\n", + " num_results=10\n", + ")\n", + "index.query(q)" + ], + "outputs": [ + { + "data": { + "text/plain": [ + "[{'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GM',\n", + " 'name': 'Python Programming Guide',\n", + " 'price': '45'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GN',\n", + " 'name': 'Redis in Action',\n", + " 'price': '55'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GP',\n", + " 'name': 'Data Science Handbook',\n", + " 'price': '65'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", + " 'name': 'USB-C Hub',\n", + " 'price': '49'},\n", + " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GT',\n", + " 'name': 'Monitor Stand',\n", + " 'price': '79'}]" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 14 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:02:52.791662Z", + "start_time": "2026-02-05T23:02:52.770651Z" + } + }, + "source": [ + "# YOUR CODE HERE - Method 2: SQLQuery\n", + "# Hint: SELECT ... WHERE price BETWEEN 40 AND 100\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT name, price from {index.name} where price between 40 and 100\n", + "\"\"\")\n", + "\n", + "index.query(sql_query)\n" + ], + "outputs": [ + { + "data": { + "text/plain": [ + "[{'name': 'Python Programming Guide', 'price': '45'},\n", + " {'name': 'Redis in Action', 'price': '55'},\n", + " {'name': 'Data Science Handbook', 'price': '65'},\n", + " {'name': 'USB-C Hub', 'price': '49'},\n", + " {'name': 'Monitor Stand', 'price': '79'}]" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 17 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:03:50.055289Z", + "start_time": "2026-02-05T23:03:50.029265Z" + } + }, + "source": [ + "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", + "# Hint: @price:[40 100]\n", + "client.execute_command(\"FT.SEARCH\", index.name, \"@price:[40 100]\", \"RETURN\", \"2\", \"name\", \"price\", \"LIMIT\", \"0\", \"10\")" + ], + "outputs": [ + { + "data": { + "text/plain": [ + "[5,\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GM',\n", + " [b'name', b'Python Programming Guide', b'price', b'45'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GN',\n", + " [b'name', b'Redis in Action', b'price', b'55'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GP',\n", + " [b'name', b'Data Science Handbook', b'price', b'65'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", + " [b'name', b'USB-C Hub', b'price', b'49'],\n", + " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GT',\n", + " [b'name', b'Monitor Stand', b'price', b'79']]" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "execution_count": 19 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Solution: Exercise 2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 1: RedisVL Python API\n", + "filter_expr = Num(\"price\").between(40, 100)\n", + "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"price\"], num_results=10)\n", + "results_api = index.query(query)\n", + "print(\"=== Method 1: RedisVL Python API ===\")\n", + "for r in results_api:\n", + " print(f\" {r['name']} - ${r['price']}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 2: RedisVL SQL\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT name, price\n", + " FROM {index.name}\n", + " WHERE price BETWEEN 40 AND 100\n", + "\"\"\")\n", + "results_sql = index.query(sql_query)\n", + "print(\"=== Method 2: RedisVL SQL ===\")\n", + "for r in results_sql:\n", + " print(f\" {r['name']} - ${r['price']}\")\n", + "\n", + "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", + "print(f\"\\nTranslated Redis command: {redis_cmd}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 3: Raw Redis FT.SEARCH\n", + "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"@price:[40 100]\", \"RETURN\", \"2\", \"name\", \"price\", \"LIMIT\", \"0\", \"10\")\n", + "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", + "for i in range(1, len(raw_results), 2):\n", + " if i + 1 < len(raw_results):\n", + " fields = raw_results[i + 1]\n", + " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", + " print(f\" {field_dict.get('name', 'N/A')} - ${field_dict.get('price', 'N/A')}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Exercise 3: Combined Filters (AND/OR)\n", + "\n", + "**Goal:** Find electronics products under $100.\n", + "\n", + "### Do It Yourself\n", + "\n", + "**Documentation:**\n", + "- [RedisVL Filter Expressions](https://docs.redisvl.com/en/latest/api/query.html#filter-expressions)\n", + "- [Redis Combined Queries](https://redis.io/docs/latest/develop/ai/search-and-query/query/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 1: RedisVL Python API\n", + "# Hint: Combine filters with & operator: (Tag(\"category\") == \"electronics\") & (Num(\"price\") < 100)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 2: SQLQuery\n", + "# Hint: WHERE category = 'electronics' AND price < 100\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", + "# Hint: (@category:{electronics} @price:[-inf 100])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Solution: Exercise 3" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 1: RedisVL Python API\n", + "filter_expr = (Tag(\"category\") == \"electronics\") & (Num(\"price\") < 100)\n", + "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"category\", \"price\"], num_results=10)\n", + "results_api = index.query(query)\n", + "print(\"=== Method 1: RedisVL Python API ===\")\n", + "for r in results_api:\n", + " print(f\" {r['name']} ({r['category']}) - ${r['price']}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 2: RedisVL SQL\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT name, category, price\n", + " FROM {index.name}\n", + " WHERE category = 'electronics' AND price < 100\n", + "\"\"\")\n", + "results_sql = index.query(sql_query)\n", + "print(\"=== Method 2: RedisVL SQL ===\")\n", + "for r in results_sql:\n", + " print(f\" {r['name']} ({r['category']}) - ${r['price']}\")\n", + "\n", + "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", + "print(f\"\\nTranslated Redis command: {redis_cmd}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 3: Raw Redis FT.SEARCH\n", + "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"(@category:{electronics} @price:[-inf (100])\", \"RETURN\", \"3\", \"name\", \"category\", \"price\", \"LIMIT\", \"0\", \"10\")\n", + "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", + "for i in range(1, len(raw_results), 2):\n", + " if i + 1 < len(raw_results):\n", + " fields = raw_results[i + 1]\n", + " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", + " print(f\" {field_dict.get('name', 'N/A')} ({field_dict.get('category', 'N/A')}) - ${field_dict.get('price', 'N/A')}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Exercise 4: Text Search\n", + "\n", + "**Goal:** Find products with \"laptop\" in the name.\n", + "\n", + "### Do It Yourself\n", + "\n", + "**Documentation:**\n", + "- [RedisVL Text Filters](https://docs.redisvl.com/en/latest/api/query.html#redisvl.query.filter.Text)\n", + "- [Redis Full-Text Search](https://redis.io/docs/latest/develop/ai/search-and-query/query/full-text/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 1: RedisVL Python API\n", + "# Hint: Use Text(\"name\") % \"laptop\" with FilterQuery\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 2: SQLQuery\n", + "# Hint: WHERE name = 'laptop'\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", + "# Hint: @name:laptop\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Solution: Exercise 4" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 1: RedisVL Python API\n", + "filter_expr = Text(\"name\") % \"laptop\"\n", + "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"description\", \"price\"], num_results=10)\n", + "results_api = index.query(query)\n", + "print(\"=== Method 1: RedisVL Python API ===\")\n", + "for r in results_api:\n", + " print(f\" {r['name']} - ${r['price']}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 2: RedisVL SQL\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT name, description, price\n", + " FROM {index.name}\n", + " WHERE name = 'laptop'\n", + "\"\"\")\n", + "results_sql = index.query(sql_query)\n", + "print(\"=== Method 2: RedisVL SQL ===\")\n", + "for r in results_sql:\n", + " print(f\" {r['name']} - ${r['price']}\")\n", + "\n", + "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", + "print(f\"\\nTranslated Redis command: {redis_cmd}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 3: Raw Redis FT.SEARCH\n", + "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"@name:laptop\", \"RETURN\", \"3\", \"name\", \"description\", \"price\", \"LIMIT\", \"0\", \"10\")\n", + "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", + "for i in range(1, len(raw_results), 2):\n", + " if i + 1 < len(raw_results):\n", + " fields = raw_results[i + 1]\n", + " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", + " print(f\" {field_dict.get('name', 'N/A')} - ${field_dict.get('price', 'N/A')}\")\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Exercise 5: Vector Similarity Search\n", + "\n", + "**Goal:** Find products most similar to a query vector (simulating a semantic search).\n", + "\n", + "### Do It Yourself\n", + "\n", + "**Documentation:**\n", + "- [RedisVL VectorQuery](https://docs.redisvl.com/en/latest/api/query.html#vectorquery)\n", + "- [Redis Vector Search](https://redis.io/docs/latest/develop/ai/search-and-query/vectors/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 1: RedisVL Python API\n", + "# Hint: Use VectorQuery with a query vector\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 2: SQLQuery\n", + "# Hint: SELECT ... ORDER BY cosine_distance(embedding, ) LIMIT k\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", + "# Hint: FT.SEARCH with KNN and BLOB parameter\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Solution: Exercise 5" + ] + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:05:53.684001Z", + "start_time": "2026-02-05T23:05:53.656720Z" + } + }, + "source": [ + "# Query vector (similar to electronics products)\n", + "query_vector = np.array([0.85, 0.15, 0.25, 0.35], dtype=np.float32)\n", + "\n", + "# Method 1: RedisVL Python API\n", + "vector_query = VectorQuery(\n", + " vector=query_vector,\n", + " vector_field_name=\"embedding\",\n", + " return_fields=[\"name\", \"category\", \"price\"],\n", + " num_results=3\n", + ")\n", + "results_api = index.query(vector_query)\n", + "print(\"=== Method 1: RedisVL Python API ===\")\n", + "for r in results_api:\n", + " print(f\" {r['name']} ({r['category']}) - distance: {r.get('vector_distance', 'N/A')}\")" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=== Method 1: RedisVL Python API ===\n", + " Gaming Laptop Pro (electronics) - distance: 0.00526285171509\n", + " Budget Laptop Basic (electronics) - distance: 0.00537633895874\n", + " Wireless Mouse (electronics) - distance: 0.0464093089104\n" + ] + } + ], + "execution_count": 21 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:07:23.911432Z", + "start_time": "2026-02-05T23:07:23.889830Z" + } + }, + "source": [ + "# Method 2: RedisVL SQL\n", + "# Note: sql-redis uses cosine_distance() function for vector search\n", + "vector_bytes = query_vector.tobytes()\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT name, category, price\n", + " FROM {index.name}\n", + " ORDER BY cosine_distance(embedding, :vector)\n", + " LIMIT 3\n", + "\"\"\",\n", + "params={\"vector\": vector_bytes}\n", + " )\n", + "\n", + "\n", + "results_sql = index.query(sql_query)\n", + "print(\"=== Method 2: RedisVL SQL ===\")\n", + "for r in results_sql:\n", + " print(f\" {r['name']} ({r['category']})\")\n", + "\n", + "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", + "print(f\"\\nTranslated Redis command: {redis_cmd[:100]}...\")\n", + "\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=== Method 2: RedisVL SQL ===\n", + " Wireless Mouse (electronics)\n", + " Monitor Stand (accessories)\n", + " Redis in Action (books)\n", + "\n", + "Translated Redis command: FT.SEARCH products_exercise \"*\" RETURN 3 name category price LIMIT 0 3...\n" + ] + } + ], + "execution_count": 26 + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Method 3: Raw Redis FT.SEARCH with KNN\n", + "import struct\n", + "vector_blob = struct.pack(f'{len(query_vector)}f', *query_vector)\n", + "raw_results = client.execute_command(\n", + " \"FT.SEARCH\", index.name,\n", + " \"*=>[KNN 3 @embedding $vec AS vector_distance]\",\n", + " \"PARAMS\", \"2\", \"vec\", vector_blob,\n", + " \"RETURN\", \"4\", \"name\", \"category\", \"price\", \"vector_distance\",\n", + " \"SORTBY\", \"vector_distance\", \"ASC\",\n", + " \"DIALECT\", \"2\"\n", + ")\n", + "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", + "for i in range(1, len(raw_results), 2):\n", + " if i + 1 < len(raw_results):\n", + " fields = raw_results[i + 1]\n", + " field_dict = {fields[j].decode() if isinstance(fields[j], bytes) else fields[j]:\n", + " fields[j+1].decode() if isinstance(fields[j+1], bytes) else fields[j+1]\n", + " for j in range(0, len(fields), 2)}\n", + " print(f\" {field_dict.get('name', 'N/A')} ({field_dict.get('category', 'N/A')}) - distance: {field_dict.get('vector_distance', 'N/A')}\")\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Exercise 6: Aggregations (COUNT, GROUP BY, AVG)\n", + "\n", + "**Goal:** Count products by category and calculate average prices.\n", + "\n", + "### Do It Yourself\n", + "\n", + "**Documentation:**\n", + "- [RedisVL CountQuery](https://docs.redisvl.com/en/latest/api/query.html#countquery)\n", + "- [Redis Aggregations](https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/aggregations/)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 1: RedisVL Python API\n", + "# Hint: Use CountQuery for counting\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 2: SQLQuery\n", + "# Hint: SELECT category, COUNT(*), AVG(price) FROM ... GROUP BY category\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# YOUR CODE HERE - Method 3: Raw FT.AGGREGATE\n", + "# Hint: FT.AGGREGATE with GROUPBY and REDUCE\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Solution: Exercise 6" + ] + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:08:26.684434Z", + "start_time": "2026-02-05T23:08:26.651689Z" + } + }, + "source": [ + "# Method 1: RedisVL Python API - Count total products\n", + "count_query = CountQuery(filter_expression=Tag(\"category\") == \"electronics\")\n", + "count_result = index.query(count_query)\n", + "print(\"=== Method 1: RedisVL Python API ===\")\n", + "print(f\" Electronics products count: {count_result}\")\n", + "\n", + "# Count for each category\n", + "for cat in [\"electronics\", \"books\", \"accessories\"]:\n", + " count_query = CountQuery(filter_expression=Tag(\"category\") == cat)\n", + " count = index.query(count_query)\n", + " print(f\" {cat}: {count} products\")" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=== Method 1: RedisVL Python API ===\n", + " Electronics products count: 5\n", + " electronics: 5 products\n", + " books: 3 products\n", + " accessories: 2 products\n" + ] + } + ], + "execution_count": 27 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:08:26.968246Z", + "start_time": "2026-02-05T23:08:26.948261Z" + } + }, + "source": [ + "# Method 2: RedisVL SQL - Group by with aggregations\n", + "sql_query = SQLQuery(f\"\"\"\n", + " SELECT category, COUNT(*) as count, AVG(price) as avg_price\n", + " FROM {index.name}\n", + " GROUP BY category\n", + "\"\"\")\n", + "results_sql = index.query(sql_query)\n", + "print(\"=== Method 2: RedisVL SQL ===\")\n", + "for r in results_sql:\n", + " print(f\" {r['category']}: {r['count']} products, avg price: ${float(r['avg_price']):.2f}\")\n", + "\n", + "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", + "print(f\"\\nTranslated Redis command: {redis_cmd}\")" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=== Method 2: RedisVL SQL ===\n", + " books: 3 products, avg price: $55.00\n", + " accessories: 2 products, avg price: $59.00\n", + " electronics: 5 products, avg price: $406.20\n", + "\n", + "Translated Redis command: FT.AGGREGATE products_exercise \"*\" LOAD 2 category price GROUPBY 1 @category REDUCE COUNT 0 AS count REDUCE AVG 1 @price AS avg_price\n" + ] + } + ], + "execution_count": 28 + }, + { + "cell_type": "code", + "metadata": { + "ExecuteTime": { + "end_time": "2026-02-05T23:09:08.232955Z", + "start_time": "2026-02-05T23:09:08.206373Z" + } + }, + "source": [ + "# Method 3: Raw Redis FT.AGGREGATE\n", + "raw_results = client.execute_command(\n", + " \"FT.AGGREGATE\", index.name, \"*\",\n", + " \"GROUPBY\", \"1\", \"@category\",\n", + " \"REDUCE\", \"COUNT\", \"0\", \"AS\", \"count\",\n", + " \"REDUCE\", \"AVG\", \"1\", \"@price\", \"AS\", \"avg_price\"\n", + ")\n", + "print(\"=== Method 3: Raw FT.AGGREGATE ===\")\n", + "for i in range(1, len(raw_results)):\n", + " row = raw_results[i]\n", + " row_dict = {row[j].decode() if isinstance(row[j], bytes) else row[j]:\n", + " row[j+1].decode() if isinstance(row[j+1], bytes) else row[j+1]\n", + " for j in range(0, len(row), 2)}\n", + " cat = row_dict.get('category', 'N/A')\n", + " count = row_dict.get('count', 'N/A')\n", + " avg_price = float(row_dict.get('avg_price', 0))\n", + " print(f\" {cat}: {count} products, avg price: ${avg_price:.2f}\")\n", + "\n" + ], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=== Method 3: Raw FT.AGGREGATE ===\n", + " books: 3 products, avg price: $55.00\n", + " accessories: 2 products, avg price: $59.00\n", + " electronics: 5 products, avg price: $406.20\n" + ] + } + ], + "execution_count": 29 + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Comparison Summary\n", + "\n", + "| Approach | Pros | Cons | Best For |\n", + "|----------|------|------|----------|\n", + "| **RedisVL Python API** | Type-safe, IDE autocomplete, Pythonic | Learning curve for filter expressions | Production applications, complex queries |\n", + "| **RedisVL SQL** | Familiar SQL syntax, easy migration | Limited to SQL capabilities | SQL developers, quick prototyping |\n", + "| **Raw FT.SEARCH** | Full control, all Redis features | Verbose, error-prone | Advanced use cases, debugging |\n", + "\n", + "### Key Takeaways\n", + "\n", + "1. **SQLQuery** is great for developers familiar with SQL who want to quickly query Redis\n", + "2. **RedisVL Python API** provides the best developer experience with type safety\n", + "3. **Raw FT.SEARCH** gives you full control but requires deep Redis knowledge\n", + "4. All three methods can achieve the same results - choose based on your team's expertise\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Cleanup" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Delete the index and clean up\n", + "index.delete(drop=True)\n", + "print(f\"Deleted index: {index.name}\")\n", + "print(\"Cleanup complete!\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "## Next Steps\n", + "\n", + "- Explore more complex queries in the [RedisVL documentation](https://docs.redisvl.com)\n", + "- Try the [sql-redis package](https://pypi.org/project/sql-redis/) for more SQL features\n", + "- Check out other RedisVL features like SemanticCache and SemanticRouter" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/docs/user_guide/cli.ipynb b/docs/user_guide/cli.ipynb index ba9d645a3..02bc68b13 100644 --- a/docs/user_guide/cli.ipynb +++ b/docs/user_guide/cli.ipynb @@ -6,7 +6,7 @@ "source": [ "# The RedisVL CLI\n", "\n", - "RedisVL is a Python library with a dedicated CLI to help load and create vector search indices within Redis.\n", + "RedisVL is a Python library with a dedicated CLI to help load, inspect, migrate, and create vector search indices within Redis.\n", "\n", "This notebook will walk through how to use the Redis Vector Library CLI (``rvl``).\n", "\n", @@ -50,7 +50,16 @@ "| `rvl index` | `delete --index` or `-i ` | remove the specified index, leaving the data still in Redis|\n", "| `rvl index` | `destroy --index` or `-i `| remove the specified index, as well as the associated data|\n", "| `rvl stats` | `--index` or `-i ` | display the index statistics, including number of docs, average bytes per record, indexing time, etc|\n", - "| `rvl stats` | `--schema` or `-s ` | display the index statistics of a schema defined in . The index must have already been created within Redis|" + "| `rvl stats` | `--schema` or `-s ` | display the index statistics of a schema defined in . The index must have already been created within Redis|\n", + "| `rvl migrate` | `helper` or `list` | show migration guidance and list indexes available for migration|\n", + "| `rvl migrate` | `wizard` | interactively build a migration plan and schema patch|\n", + "| `rvl migrate` | `plan` | generate `migration_plan.yaml` from a patch or target schema|\n", + "| `rvl migrate` | `apply` | execute a reviewed `drop_recreate` migration|\n", + "| `rvl migrate` | `validate` | validate a completed migration and emit report artifacts|\n", + "| `rvl migrate` | `batch-plan` | create a batch migration plan for multiple indexes|\n", + "| `rvl migrate` | `batch-apply` | execute a batch migration|\n", + "| `rvl migrate` | `batch-resume` | resume an interrupted batch migration|\n", + "| `rvl migrate` | `batch-status` | check batch migration progress|" ] }, { diff --git a/docs/user_guide/how_to_guides/index.md b/docs/user_guide/how_to_guides/index.md index c03d705da..f6511d54c 100644 --- a/docs/user_guide/how_to_guides/index.md +++ b/docs/user_guide/how_to_guides/index.md @@ -34,6 +34,7 @@ How-to guides are **task-oriented** recipes that help you accomplish specific go :::{grid-item-card} 💾 Storage - [Choose a Storage Type](../05_hash_vs_json.ipynb) -- Hash vs JSON formats and nested data +- [Migrate an Index](migrate-indexes.md) -- use the migrator helper, wizard, plan, apply, and validate workflow ::: :::{grid-item-card} 💻 CLI Operations @@ -59,6 +60,7 @@ How-to guides are **task-oriented** recipes that help you accomplish specific go | Optimize index performance | [Optimize Indexes with SVS-VAMANA](../09_svs_vamana.ipynb) | | Decide on storage format | [Choose a Storage Type](../05_hash_vs_json.ipynb) | | Manage indices from terminal | [Manage Indices with the CLI](../cli.ipynb) | +| Plan and run a supported index migration | [Migrate an Index](migrate-indexes.md) | ```{toctree} :hidden: @@ -74,4 +76,5 @@ Optimize Indexes with SVS-VAMANA <../09_svs_vamana> Cache Embeddings <../10_embeddings_cache> Use Advanced Query Types <../11_advanced_queries> Write SQL Queries for Redis <../12_sql_to_redis_queries> +Migrate an Index ``` diff --git a/docs/user_guide/how_to_guides/migrate-indexes.md b/docs/user_guide/how_to_guides/migrate-indexes.md new file mode 100644 index 000000000..505c70dff --- /dev/null +++ b/docs/user_guide/how_to_guides/migrate-indexes.md @@ -0,0 +1,933 @@ +--- +myst: + html_meta: + "description lang=en": | + How to migrate a RedisVL index schema without losing data. +--- + +# Migrate an Index + +This guide shows how to safely change your index schema using the RedisVL migrator. + +## Quick Start + +Add a field to your index in 4 commands: + +```bash +# 1. See what indexes exist +rvl migrate list --url redis://localhost:6379 + +# 2. Use the wizard to build a migration plan +rvl migrate wizard --index myindex --url redis://localhost:6379 + +# 3. Apply the migration +rvl migrate apply --plan migration_plan.yaml --url redis://localhost:6379 + +# 4. Verify the result +rvl migrate validate --plan migration_plan.yaml --url redis://localhost:6379 +``` + +## Prerequisites + +- Redis with the Search module (Redis Stack, Redis Cloud, or Redis Enterprise) +- An existing index to migrate +- `redisvl` installed (`pip install redisvl`) + +```bash +# Local development with Redis 8.0+ (recommended for full feature support) +docker run -d --name redis -p 6379:6379 redis:8.0 +``` + +**Note:** Redis 8.0+ is required for INT8/UINT8 vector datatypes. SVS-VAMANA algorithm requires Redis 8.2+ and Intel AVX-512 hardware. + +## Step 1: Discover Available Indexes + +```bash +rvl migrate list --url redis://localhost:6379 +``` + +**Example output:** +``` +Available indexes: + 1. products_idx + 2. users_idx + 3. orders_idx +``` + +## Step 2: Build Your Schema Change + +Choose one of these approaches: + +### Option A: Use the Wizard (Recommended) + +The wizard guides you through building a migration interactively. Run: + +```bash +rvl migrate wizard --index myindex --url redis://localhost:6379 +``` + +**Example wizard session (adding a field):** + +```text +Building a migration plan for index 'myindex' +Current schema: +- Index name: myindex +- Storage type: hash + - title (text) + - embedding (vector) + +Choose an action: +1. Add field (text, tag, numeric, geo) +2. Update field (sortable, weight, separator) +3. Remove field +4. Preview patch (show pending changes as YAML) +5. Finish +Enter a number: 1 + +Field name: category +Field type options: text, tag, numeric, geo +Field type: tag + Sortable: enables sorting and aggregation on this field +Sortable [y/n]: n + Separator: character that splits multiple values (default: comma) +Separator [leave blank to keep existing/default]: | + +Choose an action: +1. Add field (text, tag, numeric, geo) +2. Update field (sortable, weight, separator) +3. Remove field +4. Preview patch (show pending changes as YAML) +5. Finish +Enter a number: 5 + +Migration plan written to /path/to/migration_plan.yaml +Mode: drop_recreate +Supported: True +Warnings: +- Index downtime is required +``` + +**Example wizard session (quantizing vectors):** + +```text +Choose an action: +1. Add field (text, tag, numeric, geo) +2. Update field (sortable, weight, separator) +3. Remove field +4. Preview patch (show pending changes as YAML) +5. Finish +Enter a number: 2 + +Updatable fields: +1. title (text) +2. embedding (vector) +Select a field to update by number or name: 2 + +Current vector config for 'embedding': + algorithm: HNSW + datatype: float32 + distance_metric: cosine + dims: 384 (cannot be changed) + m: 16 + ef_construction: 200 + +Leave blank to keep current value. + Algorithm: vector search method (FLAT=brute force, HNSW=graph, SVS-VAMANA=compressed graph) +Algorithm [current: HNSW]: + Datatype: float16, float32, bfloat16, float64, int8, uint8 + (float16 reduces memory ~50%, int8/uint8 reduce ~75%) +Datatype [current: float32]: float16 + Distance metric: how similarity is measured (cosine, l2, ip) +Distance metric [current: cosine]: + M: number of connections per node (higher=better recall, more memory) +M [current: 16]: + EF_CONSTRUCTION: build-time search depth (higher=better recall, slower build) +EF_CONSTRUCTION [current: 200]: + +Choose an action: +... +5. Finish +Enter a number: 5 + +Migration plan written to /path/to/migration_plan.yaml +Mode: drop_recreate +Supported: True +``` + +### Option B: Write a Schema Patch (YAML) + +Create `schema_patch.yaml` manually: + +```yaml +version: 1 +changes: + add_fields: + - name: category + type: tag + path: $.category + attrs: + separator: "|" + remove_fields: + - legacy_field + update_fields: + - name: title + attrs: + sortable: true + - name: embedding + attrs: + datatype: float16 # quantize vectors + algorithm: HNSW + distance_metric: cosine +``` + +Then generate the plan: + +```bash +rvl migrate plan \ + --index myindex \ + --schema-patch schema_patch.yaml \ + --url redis://localhost:6379 \ + --plan-out migration_plan.yaml +``` + +### Option C: Provide a Target Schema + +If you have the complete target schema, use it directly: + +```bash +rvl migrate plan \ + --index myindex \ + --target-schema target_schema.yaml \ + --url redis://localhost:6379 \ + --plan-out migration_plan.yaml +``` + +## Step 3: Review the Migration Plan + +Before applying, review `migration_plan.yaml`: + +```yaml +# migration_plan.yaml (example) +version: 1 +mode: drop_recreate + +source: + schema_snapshot: + index: + name: myindex + prefix: "doc:" + storage_type: json + fields: + - name: title + type: text + - name: embedding + type: vector + attrs: + dims: 384 + algorithm: hnsw + datatype: float32 + stats_snapshot: + num_docs: 10000 + keyspace: + prefixes: ["doc:"] + key_sample: ["doc:1", "doc:2", "doc:3"] + +requested_changes: + add_fields: + - name: category + type: tag + +diff_classification: + supported: true + mode: drop_recreate + warnings: + - "Index will be unavailable during migration" + blocked_reasons: [] + +rename_operations: + rename_index: null + change_prefix: null + rename_fields: [] + +merged_target_schema: + index: + name: myindex + prefix: "doc:" + storage_type: json + fields: + - name: title + type: text + - name: category + type: tag + - name: embedding + type: vector + attrs: + dims: 384 + algorithm: hnsw + datatype: float32 + +warnings: [] +``` + +**Key fields to check:** +- `diff_classification.supported` - Must be `true` to proceed +- `diff_classification.blocked_reasons` - Must be empty +- `merged_target_schema` - The final schema after migration + +## Understanding Downtime Requirements + +**CRITICAL**: During a `drop_recreate` migration, your application must: + +| Requirement | Description | +|-------------|-------------| +| **Pause reads** | Index is unavailable during migration | +| **Pause writes** | Writes during migration may be missed or cause conflicts | + +### Why Both Reads AND Writes Must Be Paused + +- **Reads**: The index definition is dropped and recreated. Any queries during this window will fail. +- **Writes**: Redis updates indexes synchronously on every write. If your app writes documents while the index is dropped, those writes are not indexed. Additionally, if you're quantizing vectors (float32 → float16), concurrent writes may conflict with the migration's re-encoding process. + +### What "Downtime" Means + +| Downtime Type | Reads | Writes | Safe? | +|---------------|-------|--------|-------| +| Full quiesce (recommended) | Stopped | Stopped | **YES** | +| Read-only pause | Stopped | Continuing | **NO** | +| Active | Active | Active | **NO** | + +### Recovery from Interrupted Migration + +| Interruption Point | Documents | Index | Recovery | +|--------------------|-----------|-------|----------| +| After drop, before quantize | Unchanged | **None** | Re-run apply (or `--resume` if checkpoint exists) | +| During quantization | Partially quantized | **None** | Re-run with `--resume` to continue from checkpoint | +| After quantization, before create | Quantized | **None** | Re-run apply (will recreate index) | +| After create | Correct | Rebuilding | Wait for index ready | + +The underlying documents are **never deleted** by `drop_recreate` mode. For large quantization jobs, use `--resume` to enable checkpoint-based recovery. See [Crash-safe resume for quantization](#crash-safe-resume-for-quantization) below. + +## Step 4: Apply the Migration + +The `apply` command executes the migration. The index will be temporarily unavailable during the drop-recreate process. + +```bash +rvl migrate apply \ + --plan migration_plan.yaml \ + --url redis://localhost:6379 \ + --report-out migration_report.yaml \ + --benchmark-out benchmark_report.yaml +``` + +### What `apply` does + +The migration executor follows this sequence: + +**STEP 1: Enumerate keys** (before any modifications) +- Discovers all document keys belonging to the source index +- Uses `FT.AGGREGATE WITHCURSOR` for efficient enumeration +- Falls back to `SCAN` if the index has indexing failures +- Keys are stored in memory for quantization or rename operations + +**STEP 2: Drop source index** +- Issues `FT.DROPINDEX` to remove the index structure +- **The underlying documents remain in Redis** - only the index metadata is deleted +- After this point, the index is unavailable until step 6 completes + +**STEP 3: Quantize vectors** (if changing vector datatype) +- For each document in the enumerated key list: + - Reads the document (including the old vector) + - Converts the vector to the new datatype (e.g., float32 → float16) + - Writes back the converted vector to the same document +- Processes documents in batches of 500 using Redis pipelines +- Skipped for JSON storage (vectors are re-indexed automatically on recreate) +- **Checkpoint support**: For large datasets, use `--resume` to enable crash-safe recovery + +**STEP 4: Key renames** (if changing key prefix) +- If the migration changes the key prefix, renames each key from old prefix to new prefix +- Skipped if no prefix change + +**STEP 5: Create target index** +- Issues `FT.CREATE` with the merged target schema +- Redis begins background indexing of existing documents + +**STEP 6: Wait for re-indexing** +- Polls `FT.INFO` until indexing completes +- The index becomes available for queries when this completes + +**Summary**: The migration preserves all documents, drops only the index structure, performs any document-level transformations (quantization, renames), then recreates the index with the new schema. + +### Async execution for large migrations + +For large migrations (especially those involving vector quantization), use the `--async` flag: + +```bash +rvl migrate apply \ + --plan migration_plan.yaml \ + --async \ + --url redis://localhost:6379 +``` + +**What becomes async:** + +- Document enumeration during quantization (uses `FT.AGGREGATE WITHCURSOR` for index-specific enumeration, falling back to SCAN only if indexing failures exist) +- Vector read/write operations (sequential async HGET, batched HSET via pipeline) +- Index readiness polling (uses `asyncio.sleep()` instead of blocking) +- Validation checks + +**What stays sync:** + +- CLI prompts and user interaction +- YAML file reading/writing +- Progress display + +**When to use async:** + +- Quantizing millions of vectors (float32 to float16) +- Integrating into an async application + +For most migrations (index-only changes, small datasets), sync mode is sufficient and simpler. + +See {doc}`/concepts/index-migrations` for detailed async vs sync guidance. + +### Crash-safe resume for quantization + +When migrating large datasets with vector quantization (e.g. float32 to float16), the re-encoding step can take minutes or hours. If the process is interrupted (crash, network drop, OOM kill), you don't want to start over. The `--resume` flag enables checkpoint-based recovery. + +#### How it works + +1. **Pre-flight estimate** -- before any mutations, `apply` prints a disk space estimate showing RDB snapshot cost, AOF growth (if enabled), and post-migration memory savings. +2. **BGSAVE safety snapshot** -- the migrator triggers a Redis `BGSAVE` and waits for it to complete before modifying any data. This gives you a point-in-time snapshot to fall back on. +3. **Checkpoint file** -- when `--resume` is provided, the migrator writes a YAML checkpoint after every batch of 500 documents. The checkpoint records how many keys have been processed and the last batch of keys written. +4. **Batch undo buffer** -- if a single batch fails mid-write, original vector values are rolled back via pipeline before the error propagates. Only the current batch is held in memory. +5. **Idempotent skip** -- on resume, vectors that were already converted are detected by byte-width inspection and skipped automatically. + +#### Step-by-step: using crash-safe resume + +**1. Estimate disk space (dry-run, no mutations):** + +```bash +rvl migrate estimate --plan migration_plan.yaml +``` + +Example output: + +```text +Pre-migration disk space estimate: + Index: products_idx (1,000,000 documents) + Vector field 'embedding': 768 dims, float32 -> float16 + + RDB snapshot (BGSAVE): ~2.87 GB + AOF growth: not estimated (pass aof_enabled=True if AOF is on) + Total new disk required: ~2.87 GB + + Post-migration memory savings: ~1.43 GB (50% reduction) +``` + +If AOF is enabled: + +```bash +rvl migrate estimate --plan migration_plan.yaml --aof-enabled +``` + +**2. Apply with checkpoint enabled:** + +```bash +rvl migrate apply \ + --plan migration_plan.yaml \ + --resume quantize_checkpoint.yaml \ + --url redis://localhost:6379 \ + --report-out migration_report.yaml +``` + +The `--resume` flag takes a path to a checkpoint file. If the file does not exist, a new checkpoint is created. If it already exists (from a previous interrupted run), the migrator resumes from where it left off. + +**3. If the process crashes or is interrupted:** + +The checkpoint file (`quantize_checkpoint.yaml`) will contain the progress: + +```yaml +index_name: products_idx +total_keys: 1000000 +completed_keys: 450000 +completed_batches: 900 +last_batch_keys: + - 'products:449501' + - 'products:449502' + # ... +status: in_progress +checkpoint_path: quantize_checkpoint.yaml +``` + +**4. Resume the migration:** + +Re-run the exact same command: + +```bash +rvl migrate apply \ + --plan migration_plan.yaml \ + --resume quantize_checkpoint.yaml \ + --url redis://localhost:6379 \ + --report-out migration_report.yaml +``` + +The migrator will: +- Detect the existing checkpoint and skip already-processed keys +- Re-enumerate documents via SCAN (the index was already dropped before the crash) +- Continue quantizing from where it left off +- Print progress like `[4/6] Quantize vectors: 450,000/1,000,000 docs` + +**5. On successful completion:** + +The checkpoint status is set to `completed`. You can safely delete the checkpoint file. + +#### What gets rolled back on batch failure + +If a batch of 500 documents fails mid-write (e.g. Redis returns an error), the migrator: +1. Restores original vector bytes for all documents in that batch using the undo buffer +2. Saves the checkpoint (so progress up to the last successful batch is preserved) +3. Raises the error + +This means you never end up with partially-written vectors in a single batch. + +#### Limitations + +- **Same-width conversions** (float16 to bfloat16, or int8 to uint8) are **not supported** with `--resume`. These conversions cannot be detected by byte-width inspection, so idempotent skip is impossible. The migrator will refuse to proceed and suggest running without `--resume`. +- **JSON storage** does not need vector re-encoding (Redis re-indexes JSON vectors on `FT.CREATE`). The checkpoint is still created for consistency but no batched writes occur. +- The checkpoint file must match the migration plan. If you change the plan, delete the old checkpoint and start fresh. + +#### Python API with checkpoints + +```python +from redisvl.migration import MigrationExecutor + +executor = MigrationExecutor() +report = executor.apply( + plan, + redis_url="redis://localhost:6379", + checkpoint_path="quantize_checkpoint.yaml", +) +``` + +For async: + +```python +from redisvl.migration import AsyncMigrationExecutor + +executor = AsyncMigrationExecutor() +report = await executor.apply( + plan, + redis_url="redis://localhost:6379", + checkpoint_path="quantize_checkpoint.yaml", +) +``` + +## Step 5: Validate the Result + +Validation happens automatically during `apply`, but you can run it separately: + +```bash +rvl migrate validate \ + --plan migration_plan.yaml \ + --url redis://localhost:6379 \ + --report-out migration_report.yaml +``` + +**Validation checks:** +- Live schema matches `merged_target_schema` +- Document count matches the source snapshot +- Sampled keys still exist +- No increase in indexing failures + +## What's Supported + +| Change | Supported | Notes | +|--------|-----------|-------| +| Add text/tag/numeric/geo field | ✅ | | +| Remove a field | ✅ | | +| Rename a field | ✅ | Renames field in all documents | +| Change key prefix | ✅ | Renames keys via RENAME command | +| Rename the index | ✅ | Index-only | +| Make a field sortable | ✅ | | +| Change field options (separator, stemming) | ✅ | | +| Change vector algorithm (FLAT ↔ HNSW ↔ SVS-VAMANA) | ✅ | Index-only | +| Change distance metric (COSINE ↔ L2 ↔ IP) | ✅ | Index-only | +| Tune HNSW parameters (M, EF_CONSTRUCTION) | ✅ | Index-only | +| Quantize vectors (float32 → float16/bfloat16/int8/uint8) | ✅ | Auto re-encode | + +## What's Blocked + +| Change | Why | Workaround | +|--------|-----|------------| +| Change vector dimensions | Requires re-embedding | Re-embed with new model, reload data | +| Change storage type (hash ↔ JSON) | Different data format | Export, transform, reload | +| Add a new vector field | Requires vectors for all docs | Add vectors first, then migrate | + +## CLI Reference + +### Single-Index Commands + +| Command | Description | +|---------|-------------| +| `rvl migrate list` | List all indexes | +| `rvl migrate wizard` | Build a migration interactively | +| `rvl migrate plan` | Generate a migration plan | +| `rvl migrate apply` | Execute a migration | +| `rvl migrate estimate` | Estimate disk space for a migration (dry-run) | +| `rvl migrate validate` | Verify a migration result | + +### Batch Commands + +| Command | Description | +|---------|-------------| +| `rvl migrate batch-plan` | Create a batch migration plan | +| `rvl migrate batch-apply` | Execute a batch migration | +| `rvl migrate batch-resume` | Resume an interrupted batch | +| `rvl migrate batch-status` | Check batch progress | + +**Common flags:** +- `--url` : Redis connection URL +- `--index` : Index name to migrate +- `--plan` / `--plan-out` : Path to migration plan +- `--async` : Use async executor for large migrations (apply only) +- `--resume` : Path to checkpoint file for crash-safe quantization resume (apply only) +- `--report-out` : Path for validation report +- `--benchmark-out` : Path for performance metrics + +**Batch-specific flags:** +- `--pattern` : Glob pattern to match index names (e.g., `*_idx`) +- `--indexes` : Explicit list of index names +- `--indexes-file` : File containing index names (one per line) +- `--schema-patch` : Path to shared schema patch YAML +- `--state` : Path to checkpoint state file +- `--failure-policy` : `fail_fast` or `continue_on_error` +- `--accept-data-loss` : Required for quantization (lossy changes) +- `--retry-failed` : Retry previously failed indexes on resume + +## Troubleshooting + +### Migration blocked: "unsupported change" + +The planner detected a change that requires data transformation. Check `diff_classification.blocked_reasons` in the plan for details. + +### Apply failed: "source schema mismatch" + +The live index schema changed since the plan was generated. Re-run `rvl migrate plan` to create a fresh plan. + +### Apply failed: "timeout waiting for index ready" + +The index is taking longer to rebuild than expected. This can happen with large datasets. Check Redis logs and consider increasing the timeout or running during lower traffic periods. + +### Validation failed: "document count mismatch" + +Documents were added or removed between plan and apply. This is expected if your application is actively writing. Re-run `plan` and `apply` during a quieter period when the document count is stable, or verify the mismatch is due only to normal application traffic. + +### How to recover from a failed migration + +If `apply` fails mid-migration: + +1. **Check if the index exists:** `rvl index info --index myindex` +2. **If the index exists but is wrong:** Re-run `apply` with the same plan +3. **If the index was dropped:** Recreate it from the plan's `merged_target_schema` + +The underlying documents are never deleted by `drop_recreate`. + +## Python API + +For programmatic migrations, use the migration classes directly: + +### Sync API + +```python +from redisvl.migration import MigrationPlanner, MigrationExecutor + +planner = MigrationPlanner() +plan = planner.create_plan( + "myindex", + redis_url="redis://localhost:6379", + schema_patch_path="schema_patch.yaml", +) + +executor = MigrationExecutor() +report = executor.apply(plan, redis_url="redis://localhost:6379") +print(f"Migration result: {report.result}") +``` + +### Async API + +```python +import asyncio +from redisvl.migration import AsyncMigrationPlanner, AsyncMigrationExecutor + +async def migrate(): + planner = AsyncMigrationPlanner() + plan = await planner.create_plan( + "myindex", + redis_url="redis://localhost:6379", + schema_patch_path="schema_patch.yaml", + ) + + executor = AsyncMigrationExecutor() + report = await executor.apply(plan, redis_url="redis://localhost:6379") + print(f"Migration result: {report.result}") + +asyncio.run(migrate()) +``` + +## Batch Migration + +When you need to apply the same schema change to multiple indexes, use batch migration. This is common for: + +- Quantizing all indexes from float32 → float16 +- Standardizing vector algorithms across indexes +- Coordinated migrations during maintenance windows + +### Quick Start: Batch Migration + +```bash +# 1. Create a shared patch (applies to any index with an 'embedding' field) +cat > quantize_patch.yaml << 'EOF' +version: 1 +changes: + update_fields: + - name: embedding + attrs: + datatype: float16 +EOF + +# 2. Create a batch plan for all indexes matching a pattern +rvl migrate batch-plan \ + --pattern "*_idx" \ + --schema-patch quantize_patch.yaml \ + --plan-out batch_plan.yaml \ + --url redis://localhost:6379 + +# 3. Apply the batch plan +rvl migrate batch-apply \ + --plan batch_plan.yaml \ + --accept-data-loss \ + --url redis://localhost:6379 + +# 4. Check status +rvl migrate batch-status --state batch_state.yaml +``` + +### Batch Plan Options + +**Select indexes by pattern:** +```bash +rvl migrate batch-plan \ + --pattern "*_idx" \ + --schema-patch quantize_patch.yaml \ + --plan-out batch_plan.yaml \ + --url redis://localhost:6379 +``` + +**Select indexes by explicit list:** +```bash +rvl migrate batch-plan \ + --indexes "products_idx,users_idx,orders_idx" \ + --schema-patch quantize_patch.yaml \ + --plan-out batch_plan.yaml \ + --url redis://localhost:6379 +``` + +**Select indexes from a file (for 100+ indexes):** +```bash +# Create index list file +echo -e "products_idx\nusers_idx\norders_idx" > indexes.txt + +rvl migrate batch-plan \ + --indexes-file indexes.txt \ + --schema-patch quantize_patch.yaml \ + --plan-out batch_plan.yaml \ + --url redis://localhost:6379 +``` + +### Batch Plan Review + +The generated `batch_plan.yaml` shows which indexes will be migrated: + +```yaml +version: 1 +batch_id: "batch_20260320_100000" +mode: drop_recreate +failure_policy: fail_fast +requires_quantization: true + +shared_patch: + version: 1 + changes: + update_fields: + - name: embedding + attrs: + datatype: float16 + +indexes: + - name: products_idx + applicable: true + skip_reason: null + - name: users_idx + applicable: true + skip_reason: null + - name: legacy_idx + applicable: false + skip_reason: "Field 'embedding' not found" + +created_at: "2026-03-20T10:00:00Z" +``` + +**Key fields:** +- `applicable: true` means the patch applies to this index +- `skip_reason` explains why an index will be skipped + +### Applying a Batch Plan + +```bash +# Apply with fail-fast (default: stop on first error) +rvl migrate batch-apply \ + --plan batch_plan.yaml \ + --accept-data-loss \ + --url redis://localhost:6379 + +# Apply with continue-on-error (set at batch-plan time) +# Note: failure_policy is set during batch-plan, not batch-apply +rvl migrate batch-plan \ + --pattern "*_idx" \ + --schema-patch quantize_patch.yaml \ + --failure-policy continue_on_error \ + --plan-out batch_plan.yaml \ + --url redis://localhost:6379 + +rvl migrate batch-apply \ + --plan batch_plan.yaml \ + --accept-data-loss \ + --url redis://localhost:6379 +``` + +**Flags for batch-apply:** +- `--accept-data-loss` : Required when quantizing vectors (float32 → float16 is lossy) +- `--state` : Path to checkpoint file (default: `batch_state.yaml`) +- `--report-dir` : Directory for per-index reports (default: `./reports/`) + +**Note:** `--failure-policy` is set during `batch-plan`, not `batch-apply`. The policy is stored in the batch plan file. + +### Resume After Failure + +Batch migration automatically checkpoints progress. If interrupted: + +```bash +# Resume from where it left off +rvl migrate batch-resume \ + --state batch_state.yaml \ + --url redis://localhost:6379 + +# Retry previously failed indexes +rvl migrate batch-resume \ + --state batch_state.yaml \ + --retry-failed \ + --url redis://localhost:6379 +``` + +### Checking Batch Status + +```bash +rvl migrate batch-status --state batch_state.yaml +``` + +**Example output:** +``` +Batch Migration Status +====================== +Batch ID: batch_20260320_100000 +Started: 2026-03-20T10:00:00Z +Updated: 2026-03-20T10:25:00Z + +Completed: 2 + - products_idx: succeeded (10:02:30) + - users_idx: failed - Redis connection timeout (10:05:45) + +In Progress: inventory_idx +Remaining: 1 (analytics_idx) +``` + +### Batch Report + +After completion, a `batch_report.yaml` is generated: + +```yaml +version: 1 +batch_id: "batch_20260320_100000" +status: completed # or partial_failure, failed +summary: + total_indexes: 3 + successful: 3 + failed: 0 + skipped: 0 + total_duration_seconds: 127.5 +indexes: + - name: products_idx + status: succeeded + duration_seconds: 45.2 + docs_migrated: 15000 + report_path: ./reports/products_idx_report.yaml + - name: users_idx + status: succeeded + duration_seconds: 38.1 + docs_migrated: 8500 + - name: orders_idx + status: succeeded + duration_seconds: 44.2 + docs_migrated: 22000 +completed_at: "2026-03-20T10:02:07Z" +``` + +### Python API for Batch Migration + +```python +from redisvl.migration import BatchMigrationPlanner, BatchMigrationExecutor + +# Create batch plan +planner = BatchMigrationPlanner() +batch_plan = planner.create_batch_plan( + redis_url="redis://localhost:6379", + pattern="*_idx", + schema_patch_path="quantize_patch.yaml", +) + +# Review applicability +for idx in batch_plan.indexes: + if idx.applicable: + print(f"Will migrate: {idx.name}") + else: + print(f"Skipping {idx.name}: {idx.skip_reason}") + +# Execute batch +executor = BatchMigrationExecutor() +report = executor.apply( + batch_plan, + redis_url="redis://localhost:6379", + state_path="batch_state.yaml", + report_dir="./reports/", + progress_callback=lambda name, pos, total, status: print(f"[{pos}/{total}] {name}: {status}"), +) + +print(f"Batch status: {report.status}") +print(f"Successful: {report.summary.successful}/{report.summary.total_indexes}") +``` + +### Batch Migration Tips + +1. **Test on a single index first**: Run a single-index migration to verify the patch works before applying to a batch. + +2. **Use `continue_on_error` for large batches**: This ensures one failure doesn't block all remaining indexes. + +3. **Schedule during low-traffic periods**: Each index has downtime during migration. + +4. **Review skipped indexes**: The `skip_reason` often indicates schema differences that need attention. + +5. **Keep checkpoint files**: The `batch_state.yaml` is essential for resume. Don't delete it until the batch completes successfully. + +## Learn more + +- {doc}`/concepts/index-migrations`: How migrations work and which changes are supported diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md index 5d2cf6dfd..d85177e73 100644 --- a/docs/user_guide/index.md +++ b/docs/user_guide/index.md @@ -39,7 +39,7 @@ Schema → Index → Load → Query **Solve specific problems.** Task-oriented recipes for LLM extensions, querying, embeddings, optimization, and storage. +++ -LLM Caching • Filtering • Vectorizers • Reranking +LLM Caching • Filtering • Vectorizers • Reranking • Migrations ::: :::{grid-item-card} 💻 CLI Reference @@ -49,7 +49,7 @@ LLM Caching • Filtering • Vectorizers • Reranking **Command-line tools.** Manage indices, inspect stats, and work with schemas using the `rvl` CLI. +++ -rvl index • rvl stats • Schema YAML +rvl index • rvl stats • rvl migrate • Schema YAML ::: :::{grid-item-card} 💡 Use Cases diff --git a/tests/benchmarks/index_migrator_real_benchmark.py b/tests/benchmarks/index_migrator_real_benchmark.py new file mode 100644 index 000000000..ad5e0b0de --- /dev/null +++ b/tests/benchmarks/index_migrator_real_benchmark.py @@ -0,0 +1,647 @@ +from __future__ import annotations + +import argparse +import csv +import json +import statistics +import tempfile +import time +from pathlib import Path +from typing import Any, Dict, Iterable, List, Sequence + +import numpy as np +import yaml +from datasets import load_dataset +from redis import Redis +from sentence_transformers import SentenceTransformer + +from redisvl.index import SearchIndex +from redisvl.migration import MigrationPlanner +from redisvl.query import VectorQuery +from redisvl.redis.utils import array_to_buffer + +AG_NEWS_LABELS = { + 0: "world", + 1: "sports", + 2: "business", + 3: "sci_tech", +} + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Run a real local benchmark for migrating from HNSW/FP32 to FLAT/FP16 " + "with a real internet dataset and sentence-transformers embeddings." + ) + ) + parser.add_argument( + "--redis-url", + default="redis://localhost:6379", + help="Redis URL for the local benchmark target.", + ) + parser.add_argument( + "--sizes", + nargs="+", + type=int, + default=[1000, 10000, 100000], + help="Dataset sizes to benchmark.", + ) + parser.add_argument( + "--query-count", + type=int, + default=25, + help="Number of held-out query documents to benchmark search latency.", + ) + parser.add_argument( + "--top-k", + type=int, + default=10, + help="Number of nearest neighbors to fetch for overlap checks.", + ) + parser.add_argument( + "--embedding-batch-size", + type=int, + default=256, + help="Batch size for sentence-transformers encoding.", + ) + parser.add_argument( + "--load-batch-size", + type=int, + default=500, + help="Batch size for SearchIndex.load calls.", + ) + parser.add_argument( + "--model", + default="sentence-transformers/all-MiniLM-L6-v2", + help="Sentence-transformers model name.", + ) + parser.add_argument( + "--dataset-csv", + default="", + help=( + "Optional path to a local AG News CSV file with label,title,description columns. " + "If provided, the benchmark skips Hugging Face dataset downloads." + ), + ) + parser.add_argument( + "--output", + default="index_migrator_benchmark_results.json", + help="Where to write the benchmark report.", + ) + return parser.parse_args() + + +def build_schema( + *, + index_name: str, + prefix: str, + dims: int, + algorithm: str, + datatype: str, +) -> Dict[str, Any]: + return { + "index": { + "name": index_name, + "prefix": prefix, + "storage_type": "hash", + }, + "fields": [ + {"name": "doc_id", "type": "tag"}, + {"name": "label", "type": "tag"}, + {"name": "text", "type": "text"}, + { + "name": "embedding", + "type": "vector", + "attrs": { + "dims": dims, + "distance_metric": "cosine", + "algorithm": algorithm, + "datatype": datatype, + }, + }, + ], + } + + +def load_ag_news_records(num_docs: int, query_count: int) -> List[Dict[str, Any]]: + dataset = load_dataset("ag_news", split=f"train[:{num_docs + query_count}]") + records: List[Dict[str, Any]] = [] + for idx, row in enumerate(dataset): + records.append( + { + "doc_id": f"ag-news-{idx}", + "text": row["text"], + "label": AG_NEWS_LABELS[int(row["label"])], + } + ) + return records + + +def load_ag_news_records_from_csv( + csv_path: str, + *, + required_docs: int, +) -> List[Dict[str, Any]]: + records: List[Dict[str, Any]] = [] + with open(csv_path, "r", newline="", encoding="utf-8") as f: + reader = csv.reader(f) + for idx, row in enumerate(reader): + if len(row) < 3: + continue + # Skip header row if present (label column should be a digit) + if idx == 0 and not row[0].strip().isdigit(): + continue + if len(records) >= required_docs: + break + label, title, description = row + text = f"{title}. {description}".strip() + records.append( + { + "doc_id": f"ag-news-{len(records)}", + "text": text, + "label": AG_NEWS_LABELS[int(label) - 1], + } + ) + + if len(records) < required_docs: + raise ValueError( + f"Expected at least {required_docs} records in {csv_path}, found {len(records)}" + ) + return records + + +def encode_texts( + model_name: str, + texts: Sequence[str], + batch_size: int, +) -> tuple[np.ndarray, float]: + try: + encoder = SentenceTransformer(model_name, local_files_only=True) + except OSError: + # Model not cached locally yet; download it + print(f"Model '{model_name}' not found locally, downloading...") + encoder = SentenceTransformer(model_name) + start = time.perf_counter() + embeddings = encoder.encode( + list(texts), + batch_size=batch_size, + show_progress_bar=True, + convert_to_numpy=True, + normalize_embeddings=True, + ) + duration = time.perf_counter() - start + return np.asarray(embeddings, dtype=np.float32), duration + + +def iter_documents( + records: Sequence[Dict[str, Any]], + embeddings: np.ndarray, + *, + dtype: str, +) -> Iterable[Dict[str, Any]]: + for record, embedding in zip(records, embeddings): + yield { + "doc_id": record["doc_id"], + "label": record["label"], + "text": record["text"], + "embedding": array_to_buffer(embedding, dtype), + } + + +def wait_for_index_ready( + index: SearchIndex, + *, + timeout_seconds: int = 1800, + poll_interval_seconds: float = 0.5, +) -> Dict[str, Any]: + deadline = time.perf_counter() + timeout_seconds + latest_info = index.info() + while time.perf_counter() < deadline: + latest_info = index.info() + percent_indexed = float(latest_info.get("percent_indexed", 1) or 1) + indexing = latest_info.get("indexing", 0) + if percent_indexed >= 1.0 and not indexing: + return latest_info + time.sleep(poll_interval_seconds) + raise TimeoutError( + f"Index {index.schema.index.name} did not finish indexing within {timeout_seconds} seconds" + ) + + +def get_memory_snapshot(client: Redis) -> Dict[str, Any]: + info = client.info("memory") + used_memory_bytes = int(info.get("used_memory", 0)) + return { + "used_memory_bytes": used_memory_bytes, + "used_memory_mb": round(used_memory_bytes / (1024 * 1024), 3), + "used_memory_human": info.get("used_memory_human"), + } + + +def summarize_index_info(index_info: Dict[str, Any]) -> Dict[str, Any]: + return { + "num_docs": int(index_info.get("num_docs", 0) or 0), + "percent_indexed": float(index_info.get("percent_indexed", 0) or 0), + "hash_indexing_failures": int(index_info.get("hash_indexing_failures", 0) or 0), + "vector_index_sz_mb": float(index_info.get("vector_index_sz_mb", 0) or 0), + "total_indexing_time": float(index_info.get("total_indexing_time", 0) or 0), + } + + +def percentile(values: Sequence[float], pct: float) -> float: + if not values: + return 0.0 + return round(float(np.percentile(np.asarray(values), pct)), 3) + + +def run_query_benchmark( + index: SearchIndex, + query_embeddings: np.ndarray, + *, + dtype: str, + top_k: int, +) -> Dict[str, Any]: + latencies_ms: List[float] = [] + result_sets: List[List[str]] = [] + + for query_embedding in query_embeddings: + query = VectorQuery( + vector=query_embedding.tolist(), + vector_field_name="embedding", + return_fields=["doc_id", "label"], + num_results=top_k, + dtype=dtype, + ) + start = time.perf_counter() + results = index.query(query) + latencies_ms.append((time.perf_counter() - start) * 1000) + result_sets.append( + [result.get("doc_id") or result.get("id") for result in results if result] + ) + + return { + "count": len(latencies_ms), + "p50_ms": percentile(latencies_ms, 50), + "p95_ms": percentile(latencies_ms, 95), + "p99_ms": percentile(latencies_ms, 99), + "mean_ms": round(statistics.mean(latencies_ms), 3), + "result_sets": result_sets, + } + + +def compute_overlap( + source_result_sets: Sequence[Sequence[str]], + target_result_sets: Sequence[Sequence[str]], + *, + top_k: int, +) -> Dict[str, Any]: + overlap_ratios: List[float] = [] + for source_results, target_results in zip(source_result_sets, target_result_sets): + source_set = set(source_results[:top_k]) + target_set = set(target_results[:top_k]) + overlap_ratios.append(len(source_set.intersection(target_set)) / max(top_k, 1)) + return { + "mean_overlap_at_k": round(statistics.mean(overlap_ratios), 4), + "min_overlap_at_k": round(min(overlap_ratios), 4), + "max_overlap_at_k": round(max(overlap_ratios), 4), + } + + +def run_quantization_migration( + planner: MigrationPlanner, + client: Redis, + source_index_name: str, + source_schema: Dict[str, Any], + dims: int, +) -> Dict[str, Any]: + """Run full HNSW/FP32 -> FLAT/FP16 migration with quantization.""" + from redisvl.migration import MigrationExecutor + + target_schema = build_schema( + index_name=source_schema["index"]["name"], + prefix=source_schema["index"]["prefix"], + dims=dims, + algorithm="flat", # Change algorithm + datatype="float16", # Change datatype (quantization) + ) + + with tempfile.TemporaryDirectory() as tmpdir: + target_schema_path = Path(tmpdir) / "target_schema.yaml" + plan_path = Path(tmpdir) / "migration_plan.yaml" + with open(target_schema_path, "w") as f: + yaml.safe_dump(target_schema, f, sort_keys=False) + + plan_start = time.perf_counter() + plan = planner.create_plan( + source_index_name, + redis_client=client, + target_schema_path=str(target_schema_path), + ) + planner.write_plan(plan, str(plan_path)) + plan_duration = time.perf_counter() - plan_start + + if not plan.diff_classification.supported: + raise AssertionError( + f"Expected planner to ALLOW quantization migration, " + f"but it blocked with: {plan.diff_classification.blocked_reasons}" + ) + + # Check datatype changes detected + datatype_changes = MigrationPlanner.get_vector_datatype_changes( + plan.source.schema_snapshot, plan.merged_target_schema + ) + + # Execute migration + executor = MigrationExecutor() + migrate_start = time.perf_counter() + report = executor.apply(plan, redis_client=client) + migrate_duration = time.perf_counter() - migrate_start + + if report.result != "succeeded": + raise AssertionError(f"Migration failed: {report.validation.errors}") + + return { + "test": "quantization_migration", + "plan_duration_seconds": round(plan_duration, 3), + "migration_duration_seconds": round(migrate_duration, 3), + "quantize_duration_seconds": report.timings.quantize_duration_seconds, + "supported": plan.diff_classification.supported, + "datatype_changes": datatype_changes, + "result": report.result, + } + + +def assert_planner_allows_algorithm_change( + planner: MigrationPlanner, + client: Redis, + source_index_name: str, + source_schema: Dict[str, Any], + dims: int, +) -> Dict[str, Any]: + """Test that algorithm-only changes (HNSW -> FLAT) are allowed.""" + target_schema = build_schema( + index_name=source_schema["index"]["name"], + prefix=source_schema["index"]["prefix"], + dims=dims, + algorithm="flat", # Different algorithm - should be allowed + datatype="float32", # Same datatype + ) + + with tempfile.TemporaryDirectory() as tmpdir: + target_schema_path = Path(tmpdir) / "target_schema.yaml" + plan_path = Path(tmpdir) / "migration_plan.yaml" + with open(target_schema_path, "w") as f: + yaml.safe_dump(target_schema, f, sort_keys=False) + + start = time.perf_counter() + plan = planner.create_plan( + source_index_name, + redis_client=client, + target_schema_path=str(target_schema_path), + ) + planner.write_plan(plan, str(plan_path)) + duration = time.perf_counter() - start + + if not plan.diff_classification.supported: + raise AssertionError( + f"Expected planner to ALLOW algorithm change (HNSW -> FLAT), " + f"but it blocked with: {plan.diff_classification.blocked_reasons}" + ) + + return { + "test": "algorithm_change_allowed", + "plan_duration_seconds": round(duration, 3), + "supported": plan.diff_classification.supported, + "blocked_reasons": plan.diff_classification.blocked_reasons, + } + + +def benchmark_scale( + *, + client: Redis, + all_records: Sequence[Dict[str, Any]], + all_embeddings: np.ndarray, + size: int, + query_count: int, + top_k: int, + load_batch_size: int, +) -> Dict[str, Any]: + records = list(all_records[:size]) + query_records = list(all_records[size : size + query_count]) + doc_embeddings = all_embeddings[:size] + query_embeddings = all_embeddings[size : size + query_count] + dims = int(all_embeddings.shape[1]) + + client.flushdb() + + baseline_memory = get_memory_snapshot(client) + planner = MigrationPlanner(key_sample_limit=5) + source_schema = build_schema( + index_name=f"benchmark_source_{size}", + prefix=f"benchmark:source:{size}", + dims=dims, + algorithm="hnsw", + datatype="float32", + ) + + source_index = SearchIndex.from_dict(source_schema, redis_client=client) + migrated_index = None # Will be set after migration + + try: + source_index.create(overwrite=True) + source_load_start = time.perf_counter() + source_index.load( + iter_documents(records, doc_embeddings, dtype="float32"), + id_field="doc_id", + batch_size=load_batch_size, + ) + source_info = wait_for_index_ready(source_index) + source_setup_duration = time.perf_counter() - source_load_start + source_memory = get_memory_snapshot(client) + + # Query source index before migration + source_query_metrics = run_query_benchmark( + source_index, + query_embeddings, + dtype="float32", + top_k=top_k, + ) + + # Run full quantization migration: HNSW/FP32 -> FLAT/FP16 + quantization_result = run_quantization_migration( + planner=planner, + client=client, + source_index_name=source_schema["index"]["name"], + source_schema=source_schema, + dims=dims, + ) + + # Get migrated index info and memory + migrated_index = SearchIndex.from_existing( + source_schema["index"]["name"], redis_client=client + ) + target_info = wait_for_index_ready(migrated_index) + overlap_memory = get_memory_snapshot(client) + + # Query migrated index + target_query_metrics = run_query_benchmark( + migrated_index, + query_embeddings.astype(np.float16), + dtype="float16", + top_k=top_k, + ) + + overlap_metrics = compute_overlap( + source_query_metrics["result_sets"], + target_query_metrics["result_sets"], + top_k=top_k, + ) + + post_cutover_memory = get_memory_snapshot(client) + + return { + "size": size, + "query_count": len(query_records), + "vector_dims": dims, + "source": { + "algorithm": "hnsw", + "datatype": "float32", + "setup_duration_seconds": round(source_setup_duration, 3), + "index_info": summarize_index_info(source_info), + "query_metrics": { + k: v for k, v in source_query_metrics.items() if k != "result_sets" + }, + }, + "migration": { + "quantization": quantization_result, + }, + "target": { + "algorithm": "flat", + "datatype": "float16", + "migration_duration_seconds": quantization_result[ + "migration_duration_seconds" + ], + "quantize_duration_seconds": quantization_result[ + "quantize_duration_seconds" + ], + "index_info": summarize_index_info(target_info), + "query_metrics": { + k: v for k, v in target_query_metrics.items() if k != "result_sets" + }, + }, + "memory": { + "baseline": baseline_memory, + "after_source": source_memory, + "during_overlap": overlap_memory, + "after_cutover": post_cutover_memory, + "overlap_increase_mb": round( + overlap_memory["used_memory_mb"] - source_memory["used_memory_mb"], + 3, + ), + "net_change_after_cutover_mb": round( + post_cutover_memory["used_memory_mb"] + - source_memory["used_memory_mb"], + 3, + ), + }, + "correctness": { + "source_num_docs": int(source_info.get("num_docs", 0) or 0), + "target_num_docs": int(target_info.get("num_docs", 0) or 0), + "doc_count_match": int(source_info.get("num_docs", 0) or 0) + == int(target_info.get("num_docs", 0) or 0), + "migration_succeeded": quantization_result["result"] == "succeeded", + **overlap_metrics, + }, + } + finally: + for idx in (source_index, migrated_index): + try: + if idx is not None: + idx.delete(drop=True) + except Exception: + pass + + +def main() -> None: + args = parse_args() + sizes = sorted(args.sizes) + max_size = max(sizes) + required_docs = max_size + args.query_count + + if args.dataset_csv: + print( + f"Loading AG News CSV from {args.dataset_csv} with {required_docs} records" + ) + records = load_ag_news_records_from_csv( + args.dataset_csv, + required_docs=required_docs, + ) + else: + print(f"Loading AG News dataset with {required_docs} records") + records = load_ag_news_records( + required_docs - args.query_count, + args.query_count, + ) + print(f"Encoding {len(records)} texts with {args.model}") + embeddings, embedding_duration = encode_texts( + args.model, + [record["text"] for record in records], + args.embedding_batch_size, + ) + + client = Redis.from_url(args.redis_url, decode_responses=False) + client.ping() + + report = { + "dataset": "ag_news", + "model": args.model, + "sizes": sizes, + "query_count": args.query_count, + "top_k": args.top_k, + "embedding_duration_seconds": round(embedding_duration, 3), + "results": [], + } + + for size in sizes: + print(f"\nRunning benchmark for {size} documents") + result = benchmark_scale( + client=client, + all_records=records, + all_embeddings=embeddings, + size=size, + query_count=args.query_count, + top_k=args.top_k, + load_batch_size=args.load_batch_size, + ) + report["results"].append(result) + print( + json.dumps( + { + "size": size, + "source_setup_duration_seconds": result["source"][ + "setup_duration_seconds" + ], + "migration_duration_seconds": result["target"][ + "migration_duration_seconds" + ], + "quantize_duration_seconds": result["target"][ + "quantize_duration_seconds" + ], + "migration_succeeded": result["correctness"]["migration_succeeded"], + "mean_overlap_at_k": result["correctness"]["mean_overlap_at_k"], + "memory_change_mb": result["memory"]["net_change_after_cutover_mb"], + }, + indent=2, + ) + ) + + output_path = Path(args.output).resolve() + with open(output_path, "w") as f: + json.dump(report, f, indent=2) + + print(f"\nBenchmark report written to {output_path}") + + +if __name__ == "__main__": + main() diff --git a/tests/benchmarks/migration_benchmark.py b/tests/benchmarks/migration_benchmark.py new file mode 100644 index 000000000..d2ef0a085 --- /dev/null +++ b/tests/benchmarks/migration_benchmark.py @@ -0,0 +1,642 @@ +"""Migration Benchmark: Measure end-to-end migration time at scale. + +Populates a realistic 16-field index (matching the KM production schema) +at 1K, 10K, 100K, and 1M vectors, then migrates: + - Sub-1M: HNSW FP32 -> FLAT FP16 + - 1M: HNSW FP32 -> HNSW FP16 + +Collects full MigrationTimings from MigrationExecutor.apply(). + +Usage: + python tests/benchmarks/migration_benchmark.py \\ + --redis-url redis://localhost:6379 \\ + --sizes 1000 10000 100000 \\ + --trials 3 \\ + --output tests/benchmarks/results_migration.json +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import random +import time +from pathlib import Path +from typing import Any, Dict, List, Optional + +import numpy as np +from redis import Redis + +from redisvl.index import SearchIndex +from redisvl.migration import ( + AsyncMigrationExecutor, + AsyncMigrationPlanner, + MigrationExecutor, + MigrationPlanner, +) +from redisvl.migration.models import FieldUpdate, SchemaPatch, SchemaPatchChanges +from redisvl.migration.utils import wait_for_index_ready +from redisvl.redis.utils import array_to_buffer + +# --------------------------------------------------------------------------- +# Constants +# --------------------------------------------------------------------------- + +VECTOR_DIMS = 3072 +INDEX_PREFIX = "KM:benchmark:" +HNSW_M = 16 +HNSW_EF_CONSTRUCTION = 200 +BATCH_SIZE = 500 + +# Vocabularies for synthetic data +TAG_VOCABS = { + "doc_base_id": [f"base_{i}" for i in range(50)], + "file_id": [f"file_{i:06d}" for i in range(200)], + "created_by": ["alice", "bob", "carol", "dave", "eve"], + "CUSIP": [f"{random.randint(100000000, 999999999)}" for _ in range(100)], +} + +TEXT_WORDS = [ + "financial", + "report", + "quarterly", + "analysis", + "revenue", + "growth", + "market", + "portfolio", + "investment", + "dividend", + "equity", + "bond", + "asset", + "liability", + "balance", + "income", + "statement", + "forecast", + "risk", + "compliance", +] + + +# --------------------------------------------------------------------------- +# Schema helpers +# --------------------------------------------------------------------------- + + +def make_source_schema(index_name: str) -> Dict[str, Any]: + """Build the 16-field HNSW FP32 source schema dict.""" + return { + "index": { + "name": index_name, + "prefix": INDEX_PREFIX, + "storage_type": "hash", + }, + "fields": [ + {"name": "doc_base_id", "type": "tag", "attrs": {"separator": ","}}, + {"name": "file_id", "type": "tag", "attrs": {"separator": ","}}, + {"name": "page_text", "type": "text", "attrs": {"weight": 1}}, + {"name": "chunk_number", "type": "numeric"}, + {"name": "start_page", "type": "numeric"}, + {"name": "end_page", "type": "numeric"}, + {"name": "created_by", "type": "tag", "attrs": {"separator": ","}}, + {"name": "file_name", "type": "text", "attrs": {"weight": 1}}, + {"name": "created_time", "type": "numeric"}, + {"name": "last_updated_by", "type": "text", "attrs": {"weight": 1}}, + {"name": "last_updated_time", "type": "numeric"}, + { + "name": "embedding", + "type": "vector", + "attrs": { + "algorithm": "hnsw", + "datatype": "float32", + "dims": VECTOR_DIMS, + "distance_metric": "COSINE", + "m": HNSW_M, + "ef_construction": HNSW_EF_CONSTRUCTION, + }, + }, + { + "name": "CUSIP", + "type": "tag", + "attrs": {"separator": ",", "index_missing": True}, + }, + { + "name": "description", + "type": "text", + "attrs": {"weight": 1, "index_missing": True}, + }, + { + "name": "name", + "type": "text", + "attrs": {"weight": 1, "index_missing": True}, + }, + {"name": "price", "type": "numeric", "attrs": {"index_missing": True}}, + ], + } + + +def make_migration_patch(target_algo: str) -> SchemaPatch: + """Build a SchemaPatch to change embedding from FP32 to FP16 (and optionally HNSW to FLAT).""" + attrs = {"datatype": "float16"} + if target_algo == "FLAT": + attrs["algorithm"] = "flat" + return SchemaPatch( + version=1, + changes=SchemaPatchChanges( + update_fields=[ + FieldUpdate(name="embedding", attrs=attrs), + ] + ), + ) + + +# --------------------------------------------------------------------------- +# Data generation +# --------------------------------------------------------------------------- + + +def generate_random_text(min_words: int = 10, max_words: int = 50) -> str: + """Generate a random sentence from the vocabulary.""" + n = random.randint(min_words, max_words) + return " ".join(random.choice(TEXT_WORDS) for _ in range(n)) + + +def generate_document(doc_id: int, vector: np.ndarray) -> Dict[str, Any]: + """Generate a single document with all 16 fields.""" + doc: Dict[str, Any] = { + "doc_base_id": random.choice(TAG_VOCABS["doc_base_id"]), + "file_id": random.choice(TAG_VOCABS["file_id"]), + "page_text": generate_random_text(), + "chunk_number": random.randint(0, 100), + "start_page": random.randint(1, 500), + "end_page": random.randint(1, 500), + "created_by": random.choice(TAG_VOCABS["created_by"]), + "file_name": f"document_{doc_id}.pdf", + "created_time": int(time.time()) - random.randint(0, 86400 * 365), + "last_updated_by": random.choice(TAG_VOCABS["created_by"]), + "last_updated_time": int(time.time()) - random.randint(0, 86400 * 30), + "embedding": array_to_buffer(vector, dtype="float32"), + } + # INDEXMISSING fields: populate ~80% of docs + if random.random() < 0.8: + doc["CUSIP"] = random.choice(TAG_VOCABS["CUSIP"]) + if random.random() < 0.8: + doc["description"] = generate_random_text(5, 20) + if random.random() < 0.8: + doc["name"] = f"Entity {doc_id}" + if random.random() < 0.8: + doc["price"] = round(random.uniform(1.0, 10000.0), 2) + return doc + + +# --------------------------------------------------------------------------- +# Population +# --------------------------------------------------------------------------- + + +def populate_index( + redis_url: str, + index_name: str, + num_docs: int, +) -> float: + """Create the source index and populate it with synthetic data. + + Returns the time taken in seconds. + """ + schema_dict = make_source_schema(index_name) + index = SearchIndex.from_dict(schema_dict, redis_url=redis_url) + + # Drop existing index if any + try: + index.delete(drop=True) + except Exception: + pass + + # Clean up any leftover keys from previous runs + client = Redis.from_url(redis_url) + cursor = 0 + while True: + cursor, keys = client.scan(cursor, match=f"{INDEX_PREFIX}*", count=5000) + if keys: + client.delete(*keys) + if cursor == 0: + break + client.close() + + index.create(overwrite=True) + + print(f" Populating {num_docs:,} documents...") + start = time.perf_counter() + + # Generate vectors in batches to manage memory + rng = np.random.default_rng(seed=42) + client = Redis.from_url(redis_url) + + for batch_start in range(0, num_docs, BATCH_SIZE): + batch_end = min(batch_start + BATCH_SIZE, num_docs) + batch_count = batch_end - batch_start + + # Generate batch of random unit-normalized vectors + vectors = rng.standard_normal((batch_count, VECTOR_DIMS)).astype(np.float32) + norms = np.linalg.norm(vectors, axis=1, keepdims=True) + vectors = vectors / norms + + pipe = client.pipeline(transaction=False) + for i in range(batch_count): + doc_id = batch_start + i + key = f"{INDEX_PREFIX}{doc_id}" + doc = generate_document(doc_id, vectors[i]) + pipe.hset(key, mapping=doc) + + pipe.execute() + + if (batch_end % 10000 == 0) or batch_end == num_docs: + elapsed = time.perf_counter() - start + rate = batch_end / elapsed if elapsed > 0 else 0 + print(f" {batch_end:,}/{num_docs:,} docs ({rate:,.0f} docs/sec)") + + populate_duration = time.perf_counter() - start + client.close() + + # Wait for indexing to complete + print(" Waiting for index to be ready...") + idx = SearchIndex.from_existing(index_name, redis_url=redis_url) + _, indexing_wait = wait_for_index_ready(idx) + print( + f" Index ready (waited {indexing_wait:.1f}s after {populate_duration:.1f}s populate)" + ) + + return populate_duration + indexing_wait + + +# --------------------------------------------------------------------------- +# Migration execution +# --------------------------------------------------------------------------- + + +def run_migration( + redis_url: str, + index_name: str, + target_algo: str, +) -> Dict[str, Any]: + """Run a single migration and return the full report as a dict. + + Returns a dict with 'report' (model_dump) and 'enumerate_method' + indicating whether FT.AGGREGATE or SCAN was used for key discovery. + """ + import logging + + patch = make_migration_patch(target_algo) + planner = MigrationPlanner() + plan = planner.create_plan_from_patch( + index_name, + schema_patch=patch, + redis_url=redis_url, + ) + + if not plan.diff_classification.supported: + raise RuntimeError( + f"Migration not supported: {plan.diff_classification.blocked_reasons}" + ) + + executor = MigrationExecutor() + + # Capture enumerate method by intercepting executor logger warnings + enumerate_method = "FT.AGGREGATE" # default (happy path) + _orig_logger = logging.getLogger("redisvl.migration.executor") + _orig_level = _orig_logger.level + + class _EnumMethodHandler(logging.Handler): + def emit(self, record): + nonlocal enumerate_method + msg = record.getMessage() + if "Using SCAN" in msg or "Falling back to SCAN" in msg: + enumerate_method = "SCAN" + + handler = _EnumMethodHandler() + _orig_logger.addHandler(handler) + _orig_logger.setLevel(logging.WARNING) + + def progress(step: str, detail: Optional[str] = None) -> None: + if detail: + print(f" [{step}] {detail}") + + try: + report = executor.apply( + plan, + redis_url=redis_url, + progress_callback=progress, + ) + finally: + _orig_logger.removeHandler(handler) + _orig_logger.setLevel(_orig_level) + + return {"report": report.model_dump(), "enumerate_method": enumerate_method} + + +async def async_run_migration( + redis_url: str, + index_name: str, + target_algo: str, +) -> Dict[str, Any]: + """Run a single migration using AsyncMigrationExecutor. + + Returns a dict with 'report' (model_dump) and 'enumerate_method' + indicating whether FT.AGGREGATE or SCAN was used for key discovery. + """ + import logging + + patch = make_migration_patch(target_algo) + planner = AsyncMigrationPlanner() + plan = await planner.create_plan_from_patch( + index_name, + schema_patch=patch, + redis_url=redis_url, + ) + + if not plan.diff_classification.supported: + raise RuntimeError( + f"Migration not supported: {plan.diff_classification.blocked_reasons}" + ) + + executor = AsyncMigrationExecutor() + + # Capture enumerate method by intercepting executor logger warnings + enumerate_method = "FT.AGGREGATE" # default (happy path) + _orig_logger = logging.getLogger("redisvl.migration.async_executor") + _orig_level = _orig_logger.level + + class _EnumMethodHandler(logging.Handler): + def emit(self, record): + nonlocal enumerate_method + msg = record.getMessage() + if "Using SCAN" in msg or "Falling back to SCAN" in msg: + enumerate_method = "SCAN" + + handler = _EnumMethodHandler() + _orig_logger.addHandler(handler) + _orig_logger.setLevel(logging.WARNING) + + def progress(step: str, detail: Optional[str] = None) -> None: + if detail: + print(f" [{step}] {detail}") + + try: + report = await executor.apply( + plan, + redis_url=redis_url, + progress_callback=progress, + ) + finally: + _orig_logger.removeHandler(handler) + _orig_logger.setLevel(_orig_level) + + return {"report": report.model_dump(), "enumerate_method": enumerate_method} + + +# --------------------------------------------------------------------------- +# Benchmark driver +# --------------------------------------------------------------------------- + + +def run_benchmark( + redis_url: str, + sizes: List[int], + trials: int, + output_path: Optional[str], + use_async: bool = False, +) -> Dict[str, Any]: + """Run the full migration benchmark across all sizes and trials.""" + executor_label = "async" if use_async else "sync" + results: Dict[str, Any] = { + "benchmark": "migration_timing", + "executor": executor_label, + "schema_field_count": 16, + "vector_dims": VECTOR_DIMS, + "trials_per_size": trials, + "results": [], + } + + for size in sizes: + target_algo = "HNSW" if size >= 1_000_000 else "FLAT" + index_name = f"bench_migration_{size}" + print(f"\n{'='*60}") + print( + f"Size: {size:,} | Migration: HNSW FP32 -> {target_algo} FP16 | Executor: {executor_label}" + ) + print(f"{'='*60}") + + size_result = { + "size": size, + "source_algo": "HNSW", + "source_dtype": "FLOAT32", + "target_algo": target_algo, + "target_dtype": "FLOAT16", + "trials": [], + } + + for trial_num in range(1, trials + 1): + print(f"\n Trial {trial_num}/{trials}") + + # Step 1: Populate + populate_time = populate_index(redis_url, index_name, size) + + # Capture source memory + client = Redis.from_url(redis_url) + try: + info_raw = client.execute_command("FT.INFO", index_name) + # Parse the flat list into a dict + info_dict = {} + for i in range(0, len(info_raw), 2): + key = info_raw[i] + if isinstance(key, bytes): + key = key.decode() + info_dict[key] = info_raw[i + 1] + source_mem_mb = float(info_dict.get("vector_index_sz_mb", 0)) + source_total_mb = float(info_dict.get("total_index_memory_sz_mb", 0)) + source_num_docs = int(info_dict.get("num_docs", 0)) + except Exception as e: + print(f" Warning: could not read source FT.INFO: {e}") + source_mem_mb = 0.0 + source_total_mb = 0.0 + source_num_docs = 0 + finally: + client.close() + + print( + f" Source: {source_num_docs:,} docs, " + f"vector_idx={source_mem_mb:.1f}MB, " + f"total_idx={source_total_mb:.1f}MB" + ) + + # Step 2: Migrate + print(f" Running migration ({executor_label})...") + if use_async: + migration_result = asyncio.run( + async_run_migration(redis_url, index_name, target_algo) + ) + else: + migration_result = run_migration(redis_url, index_name, target_algo) + report_dict = migration_result["report"] + enumerate_method = migration_result["enumerate_method"] + + # Capture target memory + target_index_name = report_dict.get("target_index", index_name) + client = Redis.from_url(redis_url) + try: + info_raw = client.execute_command("FT.INFO", target_index_name) + info_dict = {} + for i in range(0, len(info_raw), 2): + key = info_raw[i] + if isinstance(key, bytes): + key = key.decode() + info_dict[key] = info_raw[i + 1] + target_mem_mb = float(info_dict.get("vector_index_sz_mb", 0)) + target_total_mb = float(info_dict.get("total_index_memory_sz_mb", 0)) + except Exception as e: + print(f" Warning: could not read target FT.INFO: {e}") + target_mem_mb = 0.0 + target_total_mb = 0.0 + finally: + client.close() + + timings = report_dict.get("timings", {}) + migrate_s = timings.get("total_migration_duration_seconds", 0) or 0 + total_s = round(populate_time + migrate_s, 3) + + # Vector memory savings (the real savings from FP32 -> FP16) + vec_savings_pct = ( + round((1 - target_mem_mb / source_mem_mb) * 100, 1) + if source_mem_mb > 0 + else 0 + ) + + trial_result = { + "trial": trial_num, + "load_time_seconds": round(populate_time, 3), + "migrate_time_seconds": round(migrate_s, 3), + "total_time_seconds": total_s, + "enumerate_method": enumerate_method, + "timings": timings, + "benchmark_summary": report_dict.get("benchmark_summary", {}), + "source_vector_index_mb": round(source_mem_mb, 3), + "source_total_index_mb": round(source_total_mb, 3), + "target_vector_index_mb": round(target_mem_mb, 3), + "target_total_index_mb": round(target_total_mb, 3), + "vector_memory_savings_pct": vec_savings_pct, + "validation_passed": report_dict.get("result") == "succeeded", + "num_docs": source_num_docs, + } + + # Print isolated timings + _enum_s = timings.get("drop_duration_seconds", 0) or 0 # noqa: F841 + quant_s = timings.get("quantize_duration_seconds") or 0 + index_s = timings.get("initial_indexing_duration_seconds") or 0 + down_s = timings.get("downtime_duration_seconds") or 0 + print( + f""" Results + load = {populate_time:.1f}s + migrate = {migrate_s:.1f}s (enumerate + drop + quantize + create + reindex + validate) + total = {total_s:.1f}s + enumerate = {enumerate_method} + quantize = {quant_s:.1f}s + reindex = {index_s:.1f}s + downtime = {down_s:.1f}s + vec memory = {source_mem_mb:.1f}MB -> {target_mem_mb:.1f}MB ({vec_savings_pct:.1f}% saved) + passed = {trial_result['validation_passed']}""" + ) + + size_result["trials"].append(trial_result) + + # Clean up for next trial (drop index + keys) + client = Redis.from_url(redis_url) + try: + try: + client.execute_command("FT.DROPINDEX", target_index_name) + except Exception: + pass + # Delete document keys + cursor = 0 + while True: + cursor, keys = client.scan( + cursor, match=f"{INDEX_PREFIX}*", count=5000 + ) + if keys: + client.delete(*keys) + if cursor == 0: + break + finally: + client.close() + + results["results"].append(size_result) + + # Save results + if output_path: + output = Path(output_path) + output.parent.mkdir(parents=True, exist_ok=True) + with open(output, "w") as f: + json.dump(results, f, indent=2, default=str) + print(f"\nResults saved to {output}") + + return results + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + + +def main(): + parser = argparse.ArgumentParser(description="Migration timing benchmark") + parser.add_argument( + "--redis-url", default="redis://localhost:6379", help="Redis connection URL" + ) + parser.add_argument( + "--sizes", + nargs="+", + type=int, + default=[1000, 10000, 100000], + help="Corpus sizes to benchmark", + ) + parser.add_argument( + "--trials", type=int, default=3, help="Number of trials per size" + ) + parser.add_argument( + "--output", + default="tests/benchmarks/results_migration.json", + help="Output JSON file", + ) + parser.add_argument( + "--async", + dest="use_async", + action="store_true", + default=False, + help="Use AsyncMigrationExecutor instead of sync MigrationExecutor", + ) + args = parser.parse_args() + + executor_label = "AsyncMigrationExecutor" if args.use_async else "MigrationExecutor" + print( + f"""Migration Benchmark + Redis: {args.redis_url} + Sizes: {args.sizes} + Trials: {args.trials} + Vector dims: {VECTOR_DIMS} + Fields: 16 + Executor: {executor_label}""" + ) + + run_benchmark( + redis_url=args.redis_url, + sizes=args.sizes, + trials=args.trials, + output_path=args.output, + use_async=args.use_async, + ) + + +if __name__ == "__main__": + main() diff --git a/tests/benchmarks/retrieval_benchmark.py b/tests/benchmarks/retrieval_benchmark.py new file mode 100644 index 000000000..6406dd9ad --- /dev/null +++ b/tests/benchmarks/retrieval_benchmark.py @@ -0,0 +1,680 @@ +"""Retrieval Benchmark: FP32 vs FP16 x HNSW vs FLAT + +Replicates the methodology from the Redis SVS-VAMANA study using +pre-embedded datasets from HuggingFace (no embedding step required). + +Comparison matrix (4 configurations): + - HNSW / FLOAT32 (approximate, full precision) + - HNSW / FLOAT16 (approximate, quantized) + - FLAT / FLOAT32 (exact, full precision -- ground truth) + - FLAT / FLOAT16 (exact, quantized) + +Datasets: + - dbpedia: 1536-dim OpenAI embeddings (KShivendu/dbpedia-entities-openai-1M) + - cohere: 768-dim Cohere embeddings (Cohere/wikipedia-22-12-en-embeddings) + +Metrics: + - Overlap@K (precision vs FLAT/FP32 ground truth) + - Query latency: p50, p95, p99, mean + - QPS (queries per second) + - Memory footprint per configuration + - Index build / load time + +Usage: + python tests/benchmarks/retrieval_benchmark.py \\ + --redis-url redis://localhost:6379 \\ + --dataset dbpedia \\ + --sizes 1000 10000 \\ + --top-k 10 \\ + --query-count 100 \\ + --output retrieval_benchmark_results.json +""" + +from __future__ import annotations + +import argparse +import json +import statistics +import time +from pathlib import Path +from typing import Any, Dict, Iterable, List, Sequence, Tuple + +import numpy as np +from redis import Redis + +from redisvl.index import SearchIndex +from redisvl.query import VectorQuery +from redisvl.redis.utils import array_to_buffer + +# --------------------------------------------------------------------------- +# Dataset registry +# --------------------------------------------------------------------------- + +DATASETS = { + "dbpedia": { + "hf_name": "KShivendu/dbpedia-entities-openai-1M", + "embedding_column": "openai", + "dims": 1536, + "distance_metric": "cosine", + "description": "DBpedia entities, OpenAI text-embedding-ada-002, 1536d", + }, + "cohere": { + "hf_name": "Cohere/wikipedia-22-12-en-embeddings", + "embedding_column": "emb", + "dims": 768, + "distance_metric": "cosine", + "description": "Wikipedia EN, Cohere multilingual encoder, 768d", + }, + "random768": { + "hf_name": None, + "embedding_column": None, + "dims": 768, + "distance_metric": "cosine", + "description": "Synthetic random unit vectors, 768d (Cohere-scale proxy)", + }, +} + +# Index configurations to benchmark +INDEX_CONFIGS = [ + {"algorithm": "flat", "datatype": "float32", "label": "FLAT_FP32"}, + {"algorithm": "flat", "datatype": "float16", "label": "FLAT_FP16"}, + {"algorithm": "hnsw", "datatype": "float32", "label": "HNSW_FP32"}, + {"algorithm": "hnsw", "datatype": "float16", "label": "HNSW_FP16"}, +] + +# HNSW parameters matching SVS-VAMANA study +HNSW_M = 16 +HNSW_EF_CONSTRUCTION = 200 +HNSW_EF_RUNTIME = 10 + +# Recall K values to compute recall curves +RECALL_K_VALUES = [1, 5, 10, 20, 50, 100] + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Retrieval benchmark: FP32 vs FP16 x HNSW vs FLAT." + ) + parser.add_argument("--redis-url", default="redis://localhost:6379") + parser.add_argument( + "--dataset", + choices=list(DATASETS.keys()), + default="dbpedia", + ) + parser.add_argument( + "--sizes", + nargs="+", + type=int, + default=[1000, 10000], + ) + parser.add_argument("--query-count", type=int, default=100) + parser.add_argument("--top-k", type=int, default=10) + parser.add_argument("--ef-runtime", type=int, default=10) + parser.add_argument("--load-batch-size", type=int, default=500) + parser.add_argument( + "--recall-k-max", + type=int, + default=100, + help="Max K for recall curve (queries will fetch this many results).", + ) + parser.add_argument( + "--output", + default="retrieval_benchmark_results.json", + ) + return parser.parse_args() + + +# --------------------------------------------------------------------------- +# Dataset loading +# --------------------------------------------------------------------------- + + +def load_dataset_vectors( + dataset_key: str, + num_vectors: int, +) -> Tuple[np.ndarray, int]: + """Load pre-embedded vectors from HuggingFace or generate synthetic.""" + ds_info = DATASETS[dataset_key] + dims = ds_info["dims"] + + if ds_info["hf_name"] is None: + # Synthetic random unit vectors + print(f"Generating {num_vectors} random unit vectors ({dims}d) ...") + rng = np.random.default_rng(42) + vectors = rng.standard_normal((num_vectors, dims)).astype(np.float32) + norms = np.linalg.norm(vectors, axis=1, keepdims=True) + vectors = vectors / norms + print(f" Generated shape: {vectors.shape}") + return vectors, dims + + # Local import to avoid requiring datasets for synthetic mode + from datasets import load_dataset + + hf_name = ds_info["hf_name"] + emb_col = ds_info["embedding_column"] + + print(f"Loading {num_vectors} vectors from {hf_name} ...") + ds = load_dataset(hf_name, split=f"train[:{num_vectors}]") + vectors = np.array(ds[emb_col], dtype=np.float32) + print(f" Loaded shape: {vectors.shape}") + return vectors, dims + + +# --------------------------------------------------------------------------- +# Schema helpers +# --------------------------------------------------------------------------- + + +def build_schema( + *, + index_name: str, + prefix: str, + dims: int, + algorithm: str, + datatype: str, + distance_metric: str, + ef_runtime: int = HNSW_EF_RUNTIME, +) -> Dict[str, Any]: + """Build an index schema dict for a given config.""" + vector_attrs: Dict[str, Any] = { + "dims": dims, + "distance_metric": distance_metric, + "algorithm": algorithm, + "datatype": datatype, + } + if algorithm == "hnsw": + vector_attrs["m"] = HNSW_M + vector_attrs["ef_construction"] = HNSW_EF_CONSTRUCTION + vector_attrs["ef_runtime"] = ef_runtime + + return { + "index": { + "name": index_name, + "prefix": prefix, + "storage_type": "hash", + }, + "fields": [ + {"name": "doc_id", "type": "tag"}, + { + "name": "embedding", + "type": "vector", + "attrs": vector_attrs, + }, + ], + } + + +# --------------------------------------------------------------------------- +# Data loading into Redis +# --------------------------------------------------------------------------- + + +def iter_documents( + vectors: np.ndarray, + *, + dtype: str, +) -> Iterable[Dict[str, Any]]: + """Yield documents ready for SearchIndex.load().""" + for i, vec in enumerate(vectors): + yield { + "doc_id": f"doc-{i}", + "embedding": array_to_buffer(vec, dtype), + } + + +def wait_for_index_ready( + index: SearchIndex, + *, + timeout_seconds: int = 3600, + poll_interval: float = 0.5, +) -> Dict[str, Any]: + """Block until the index reports 100% indexed.""" + deadline = time.perf_counter() + timeout_seconds + info = index.info() + while time.perf_counter() < deadline: + info = index.info() + pct = float(info.get("percent_indexed", 1) or 1) + indexing = info.get("indexing", 0) + if pct >= 1.0 and not indexing: + return info + time.sleep(poll_interval) + raise TimeoutError( + f"Index {index.schema.index.name} not ready within {timeout_seconds}s" + ) + + +# --------------------------------------------------------------------------- +# Memory helpers +# --------------------------------------------------------------------------- + + +def get_memory_mb(client: Redis) -> float: + info = client.info("memory") + return round(int(info.get("used_memory", 0)) / (1024 * 1024), 3) + + +# --------------------------------------------------------------------------- +# Query execution & overlap +# --------------------------------------------------------------------------- + + +def percentile(values: Sequence[float], pct: float) -> float: + if not values: + return 0.0 + return round(float(np.percentile(np.asarray(values), pct)), 6) + + +def run_queries( + index: SearchIndex, + query_vectors: np.ndarray, + *, + dtype: str, + top_k: int, +) -> Dict[str, Any]: + """Run query vectors; return latency stats and result doc-id lists.""" + latencies_ms: List[float] = [] + result_sets: List[List[str]] = [] + + for qvec in query_vectors: + q = VectorQuery( + vector=qvec.tolist(), + vector_field_name="embedding", + return_fields=["doc_id"], + num_results=top_k, + dtype=dtype, + ) + t0 = time.perf_counter() + results = index.query(q) + latencies_ms.append((time.perf_counter() - t0) * 1000) + result_sets.append([r.get("doc_id") or r.get("id", "") for r in results if r]) + + total_s = sum(latencies_ms) / 1000 + qps = len(latencies_ms) / total_s if total_s > 0 else 0 + + return { + "count": len(latencies_ms), + "p50_ms": percentile(latencies_ms, 50), + "p95_ms": percentile(latencies_ms, 95), + "p99_ms": percentile(latencies_ms, 99), + "mean_ms": round(statistics.mean(latencies_ms), 3), + "qps": round(qps, 2), + "result_sets": result_sets, + } + + +def compute_overlap( + ground_truth: List[List[str]], + candidate: List[List[str]], + *, + top_k: int, +) -> Dict[str, Any]: + """Compute Overlap@K (precision) of candidate vs ground truth.""" + ratios: List[float] = [] + for gt, cand in zip(ground_truth, candidate): + gt_set = set(gt[:top_k]) + cand_set = set(cand[:top_k]) + ratios.append(len(gt_set & cand_set) / max(top_k, 1)) + return { + "mean_overlap_at_k": round(statistics.mean(ratios), 4), + "min_overlap_at_k": round(min(ratios), 4), + "max_overlap_at_k": round(max(ratios), 4), + "std_overlap_at_k": ( + round(statistics.stdev(ratios), 4) if len(ratios) > 1 else 0.0 + ), + } + + +def compute_recall( + ground_truth: List[List[str]], + candidate: List[List[str]], + *, + k_values: Sequence[int], + ground_truth_depth: int, +) -> Dict[str, Any]: + """Compute Recall@K at multiple K values. + + For each K, recall is defined as: + |candidate_top_K intersection ground_truth_top_GT_DEPTH| / GT_DEPTH + + The ground truth set is FIXED at ground_truth_depth (e.g., top-100 from + FLAT FP32). As K increases from 1 to ground_truth_depth, recall should + climb from low to 1.0 (for exact search) or near-1.0 (for approximate). + + This is the standard recall metric from ANN benchmarks -- it answers + "what fraction of the true nearest neighbors did we find?" + """ + recall_at_k: Dict[str, float] = {} + recall_detail: Dict[str, Dict[str, float]] = {} + for k in k_values: + ratios: List[float] = [] + for gt, cand in zip(ground_truth, candidate): + gt_set = set(gt[:ground_truth_depth]) + cand_set = set(cand[:k]) + denom = min(ground_truth_depth, len(gt_set)) + if denom == 0: + # Empty ground truth means nothing to recall; use 0.0 + ratios.append(0.0) + else: + ratios.append(len(gt_set & cand_set) / denom) + mean_recall = round(statistics.mean(ratios), 4) + recall_at_k[f"recall@{k}"] = mean_recall + recall_detail[f"recall@{k}"] = { + "mean": mean_recall, + "min": round(min(ratios), 4), + "max": round(max(ratios), 4), + "std": round(statistics.stdev(ratios), 4) if len(ratios) > 1 else 0.0, + } + return { + "recall_at_k": recall_at_k, + "recall_detail": recall_detail, + "ground_truth_depth": ground_truth_depth, + } + + +# --------------------------------------------------------------------------- +# Single-config benchmark +# --------------------------------------------------------------------------- + + +def benchmark_single_config( + *, + client: Redis, + doc_vectors: np.ndarray, + query_vectors: np.ndarray, + config: Dict[str, str], + dims: int, + distance_metric: str, + size: int, + top_k: int, + ef_runtime: int, + load_batch_size: int, +) -> Dict[str, Any]: + """Build one index config, load data, query, and return metrics.""" + label = config["label"] + algo = config["algorithm"] + dtype = config["datatype"] + + index_name = f"bench_{label}_{size}" + prefix = f"bench:{label}:{size}" + + schema = build_schema( + index_name=index_name, + prefix=prefix, + dims=dims, + algorithm=algo, + datatype=dtype, + distance_metric=distance_metric, + ef_runtime=ef_runtime, + ) + + idx = SearchIndex.from_dict(schema, redis_client=client) + try: + idx.create(overwrite=True) + + # Load data + load_start = time.perf_counter() + idx.load( + iter_documents(doc_vectors, dtype=dtype), + id_field="doc_id", + batch_size=load_batch_size, + ) + info = wait_for_index_ready(idx) + load_duration = time.perf_counter() - load_start + + memory_mb = get_memory_mb(client) + + # Query + query_metrics = run_queries( + idx, + query_vectors, + dtype=dtype, + top_k=top_k, + ) + + return { + "label": label, + "algorithm": algo, + "datatype": dtype, + "load_duration_seconds": round(load_duration, 3), + "num_docs": int(info.get("num_docs", 0) or 0), + "vector_index_sz_mb": float(info.get("vector_index_sz_mb", 0) or 0), + "memory_mb": memory_mb, + "latency": { + "queried_top_k": top_k, + **{k: v for k, v in query_metrics.items() if k != "result_sets"}, + }, + "result_sets": query_metrics["result_sets"], + } + finally: + try: + idx.delete(drop=True) + except Exception: + pass + + +# --------------------------------------------------------------------------- +# Scale-level benchmark (runs all 4 configs for one size) +# --------------------------------------------------------------------------- + + +def benchmark_scale( + *, + client: Redis, + all_vectors: np.ndarray, + size: int, + query_count: int, + dims: int, + distance_metric: str, + top_k: int, + ef_runtime: int, + load_batch_size: int, + recall_k_max: int = 100, +) -> Dict[str, Any]: + """Run all 4 index configs for a given dataset size.""" + doc_vectors = all_vectors[:size] + query_vectors = all_vectors[size : size + query_count].copy() + + # Use the larger of top_k and recall_k_max for querying + # so we have enough results for recall curve computation + effective_top_k = max(top_k, recall_k_max) + + baseline_memory = get_memory_mb(client) + + config_results: Dict[str, Any] = {} + ground_truth_results: List[List[str]] = [] + + # Run FLAT_FP32 first to establish ground truth + gt_config = INDEX_CONFIGS[0] # FLAT_FP32 + assert gt_config["label"] == "FLAT_FP32" + + for config in INDEX_CONFIGS: + label = config["label"] + print(f" [{label}] Building and querying ...") + + result = benchmark_single_config( + client=client, + doc_vectors=doc_vectors, + query_vectors=query_vectors, + config=config, + dims=dims, + distance_metric=distance_metric, + size=size, + top_k=effective_top_k, + ef_runtime=ef_runtime, + load_batch_size=load_batch_size, + ) + + if label == "FLAT_FP32": + ground_truth_results = result["result_sets"] + + config_results[label] = result + + # Compute overlap vs ground truth for every config (at original top_k) + overlap_results: Dict[str, Any] = {} + for label, result in config_results.items(): + overlap = compute_overlap( + ground_truth_results, + result["result_sets"], + top_k=top_k, + ) + overlap_results[label] = overlap + + # Compute recall at multiple K values. + # Ground truth depth is fixed at top_k (e.g., 10). We measure what + # fraction of those top_k true results appear in candidate top-K as + # K varies from 1 up to effective_top_k. + valid_k_values = [k for k in RECALL_K_VALUES if k <= effective_top_k] + recall_results: Dict[str, Any] = {} + for label, result in config_results.items(): + recall = compute_recall( + ground_truth_results, + result["result_sets"], + k_values=valid_k_values, + ground_truth_depth=top_k, + ) + recall_results[label] = recall + + # Strip raw result_sets from output (too large for JSON) + for label in config_results: + del config_results[label]["result_sets"] + + return { + "size": size, + "query_count": query_count, + "dims": dims, + "distance_metric": distance_metric, + "top_k": top_k, + "recall_k_max": recall_k_max, + "ef_runtime": ef_runtime, + "hnsw_m": HNSW_M, + "hnsw_ef_construction": HNSW_EF_CONSTRUCTION, + "baseline_memory_mb": baseline_memory, + "configs": config_results, + "overlap_vs_ground_truth": overlap_results, + "recall_vs_ground_truth": recall_results, + } + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + + +def main() -> None: + args = parse_args() + sizes = sorted(args.sizes) + max_needed = max(sizes) + args.query_count + ds_info = DATASETS[args.dataset] + + print( + f"""Retrieval Benchmark + Dataset: {args.dataset} ({ds_info['description']}) + Dims: {ds_info['dims']} + Sizes: {sizes} + Query count: {args.query_count} + Top-K: {args.top_k} + Recall K max: {args.recall_k_max} + EF runtime: {args.ef_runtime} + HNSW M: {HNSW_M} + EF construct: {HNSW_EF_CONSTRUCTION} + Redis URL: {args.redis_url} + Configs: {[c['label'] for c in INDEX_CONFIGS]}""" + ) + + # Load vectors once + all_vectors, dims = load_dataset_vectors(args.dataset, max_needed) + if all_vectors.shape[0] < max_needed: + raise ValueError( + f"Dataset has {all_vectors.shape[0]} vectors but need {max_needed} " + f"(max_size={max(sizes)} + query_count={args.query_count})" + ) + + client = Redis.from_url(args.redis_url, decode_responses=False) + client.ping() + print("Connected to Redis") + + report = { + "benchmark": "retrieval_fp32_vs_fp16", + "dataset": args.dataset, + "dataset_description": ds_info["description"], + "dims": dims, + "distance_metric": ds_info["distance_metric"], + "hnsw_m": HNSW_M, + "hnsw_ef_construction": HNSW_EF_CONSTRUCTION, + "ef_runtime": args.ef_runtime, + "top_k": args.top_k, + "recall_k_max": args.recall_k_max, + "recall_k_values": [ + k for k in RECALL_K_VALUES if k <= max(args.top_k, args.recall_k_max) + ], + "query_count": args.query_count, + "configs": [c["label"] for c in INDEX_CONFIGS], + "results": [], + } + + for size in sizes: + print(f"\n{'='*60}") + print(f" Size: {size:,} documents") + print(f"{'='*60}") + + client.flushdb() + + result = benchmark_scale( + client=client, + all_vectors=all_vectors, + size=size, + query_count=args.query_count, + dims=dims, + distance_metric=ds_info["distance_metric"], + top_k=args.top_k, + ef_runtime=args.ef_runtime, + load_batch_size=args.load_batch_size, + recall_k_max=args.recall_k_max, + ) + report["results"].append(result) + + # Print summary table for this size + print( + f"\n {'Config':<12} {'Load(s)':>8} {'Memory(MB)':>11} " + f"{'p50(ms)':>8} {'p95(ms)':>8} {'QPS':>7} {'Overlap@K':>10}" + ) + print(f" {'-'*12} {'-'*8} {'-'*11} {'-'*8} {'-'*8} {'-'*7} {'-'*10}") + for label, cfg in result["configs"].items(): + overlap = result["overlap_vs_ground_truth"][label] + print( + f" {label:<12} " + f"{cfg['load_duration_seconds']:>8.1f} " + f"{cfg['memory_mb']:>11.1f} " + f"{cfg['latency']['p50_ms']:>8.2f} " + f"{cfg['latency']['p95_ms']:>8.2f} " + f"{cfg['latency']['qps']:>7.1f} " + f"{overlap['mean_overlap_at_k']:>10.4f}" + ) + + # Print recall curve summary + recall_data = result.get("recall_vs_ground_truth", {}) + if recall_data: + first_label = next(iter(recall_data)) + k_keys = sorted( + recall_data[first_label].get("recall_at_k", {}).keys(), + key=lambda x: int(x.split("@")[1]), + ) + header = f" {'Config':<12} " + " ".join(f"{k:>10}" for k in k_keys) + print(f"\n Recall Curve:") + print(header) + print(f" {'-'*12} " + " ".join(f"{'-'*10}" for _ in k_keys)) + for label, rdata in recall_data.items(): + vals = " ".join( + f"{rdata['recall_at_k'].get(k, 0):>10.4f}" for k in k_keys + ) + print(f" {label:<12} {vals}") + + # Write report + output_path = Path(args.output).resolve() + with open(output_path, "w") as f: + json.dump(report, f, indent=2) + print(f"\nReport written to {output_path}") + + +if __name__ == "__main__": + main() diff --git a/tests/benchmarks/visualize_results.py b/tests/benchmarks/visualize_results.py new file mode 100644 index 000000000..8b282743a --- /dev/null +++ b/tests/benchmarks/visualize_results.py @@ -0,0 +1,529 @@ +#!/usr/bin/env python3 +""" +Visualization script for retrieval benchmark results. + +Generates charts replicating the style of the Redis SVS-VAMANA blog post: + 1. Memory footprint comparison (FP32 vs FP16, bar chart) + 2. Precision (Overlap@K) comparison (grouped bar chart) + 3. QPS comparison (grouped bar chart) + 4. Latency comparison (p50/p95, grouped bar chart) + 5. QPS vs Overlap@K curve (line chart) + +Usage: + python tests/benchmarks/visualize_results.py \ + --input tests/benchmarks/results_dbpedia.json \ + --output-dir tests/benchmarks/charts/ +""" + +import argparse +import json +import os +from typing import Any, Dict, List + +try: + import matplotlib.pyplot as plt + import matplotlib.ticker as mticker +except ImportError: + raise ImportError( + "matplotlib is required by this visualization script. " + "Install it with: pip install matplotlib" + ) +import numpy as np + +# Redis-inspired color palette +COLORS = { + "FLAT_FP32": "#1E3A5F", # dark navy + "FLAT_FP16": "#3B82F6", # bright blue + "HNSW_FP32": "#DC2626", # Redis red + "HNSW_FP16": "#F97316", # orange +} + +LABELS = { + "FLAT_FP32": "FLAT FP32", + "FLAT_FP16": "FLAT FP16", + "HNSW_FP32": "HNSW FP32", + "HNSW_FP16": "HNSW FP16", +} + + +def load_results(path: str) -> Dict[str, Any]: + with open(path) as f: + return json.load(f) + + +def setup_style(): + """Apply a clean, modern chart style.""" + plt.rcParams.update( + { + "figure.facecolor": "white", + "axes.facecolor": "#F8F9FA", + "axes.edgecolor": "#DEE2E6", + "axes.grid": True, + "grid.color": "#E9ECEF", + "grid.alpha": 0.7, + "font.family": "sans-serif", + "font.size": 11, + "axes.titlesize": 14, + "axes.titleweight": "bold", + "axes.labelsize": 12, + } + ) + + +def chart_memory(results: List[Dict], dataset: str, output_dir: str): + """Chart 1: Memory footprint comparison per size (grouped bar chart).""" + fig, ax = plt.subplots(figsize=(10, 6)) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + sizes = [r["size"] for r in results] + x = np.arange(len(sizes)) + width = 0.18 + + for i, cfg in enumerate(configs): + mem = [r["configs"][cfg]["memory_mb"] for r in results] + bars = ax.bar( + x + i * width, + mem, + width, + label=LABELS[cfg], + color=COLORS[cfg], + edgecolor="white", + linewidth=0.5, + ) + for bar, val in zip(bars, mem): + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + 1, + f"{val:.0f}", + ha="center", + va="bottom", + fontsize=8, + ) + + ax.set_xlabel("Corpus Size") + ax.set_ylabel("Total Memory (MB)") + ax.set_title(f"Memory Footprint: FP32 vs FP16 -- {dataset}") + ax.set_xticks(x + width * 1.5) + ax.set_xticklabels([f"{s:,}" for s in sizes]) + ax.legend(loc="upper left") + ax.set_ylim(bottom=0) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_memory.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_memory.png") + + +def chart_overlap(results: List[Dict], dataset: str, output_dir: str): + """Chart 2: Overlap@K (precision) comparison per size.""" + fig, ax = plt.subplots(figsize=(10, 6)) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + sizes = [r["size"] for r in results] + x = np.arange(len(sizes)) + width = 0.18 + + for i, cfg in enumerate(configs): + overlap = [ + r["overlap_vs_ground_truth"][cfg]["mean_overlap_at_k"] for r in results + ] + bars = ax.bar( + x + i * width, + overlap, + width, + label=LABELS[cfg], + color=COLORS[cfg], + edgecolor="white", + linewidth=0.5, + ) + for bar, val in zip(bars, overlap): + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + 0.005, + f"{val:.3f}", + ha="center", + va="bottom", + fontsize=8, + ) + + ax.set_xlabel("Corpus Size") + ax.set_ylabel("Overlap@K (Precision vs FLAT FP32)") + ax.set_title(f"Search Precision: FP32 vs FP16 -- {dataset}") + ax.set_xticks(x + width * 1.5) + ax.set_xticklabels([f"{s:,}" for s in sizes]) + ax.legend(loc="lower left") + ax.set_ylim(0, 1.1) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_overlap.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_overlap.png") + + +def chart_qps(results: List[Dict], dataset: str, output_dir: str): + """Chart 3: QPS comparison per size.""" + fig, ax = plt.subplots(figsize=(10, 6)) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + sizes = [r["size"] for r in results] + x = np.arange(len(sizes)) + width = 0.18 + + for i, cfg in enumerate(configs): + qps = [r["configs"][cfg]["latency"]["qps"] for r in results] + bars = ax.bar( + x + i * width, + qps, + width, + label=LABELS[cfg], + color=COLORS[cfg], + edgecolor="white", + linewidth=0.5, + ) + for bar, val in zip(bars, qps): + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + 10, + f"{val:.0f}", + ha="center", + va="bottom", + fontsize=7, + rotation=45, + ) + + ax.set_xlabel("Corpus Size") + ax.set_ylabel("Queries Per Second (QPS)") + ax.set_title(f"Query Throughput: FP32 vs FP16 -- {dataset}") + ax.set_xticks(x + width * 1.5) + ax.set_xticklabels([f"{s:,}" for s in sizes]) + ax.legend(loc="upper right") + ax.set_ylim(bottom=0) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_qps.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_qps.png") + + +def chart_latency(results: List[Dict], dataset: str, output_dir: str): + """Chart 4: p50 and p95 latency comparison per size.""" + fig, axes = plt.subplots(1, 2, figsize=(14, 6), sharey=True) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + sizes = [r["size"] for r in results] + x = np.arange(len(sizes)) + width = 0.18 + + for ax, metric, title in zip( + axes, ["p50_ms", "p95_ms"], ["p50 Latency", "p95 Latency"] + ): + for i, cfg in enumerate(configs): + vals = [r["configs"][cfg]["latency"][metric] for r in results] + bars = ax.bar( + x + i * width, + vals, + width, + label=LABELS[cfg], + color=COLORS[cfg], + edgecolor="white", + linewidth=0.5, + ) + for bar, val in zip(bars, vals): + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + 0.02, + f"{val:.2f}", + ha="center", + va="bottom", + fontsize=7, + ) + ax.set_xlabel("Corpus Size") + ax.set_ylabel("Latency (ms)") + ax.set_title(f"{title} -- {dataset}") + ax.set_xticks(x + width * 1.5) + ax.set_xticklabels([f"{s:,}" for s in sizes]) + ax.legend(loc="upper left", fontsize=9) + ax.set_ylim(bottom=0) + + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_latency.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_latency.png") + + +def chart_qps_vs_overlap(results: List[Dict], dataset: str, output_dir: str): + """Chart 5: QPS vs Overlap@K curve (Redis blog Chart 2 style).""" + fig, ax = plt.subplots(figsize=(10, 6)) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + markers = {"FLAT_FP32": "s", "FLAT_FP16": "D", "HNSW_FP32": "o", "HNSW_FP16": "^"} + + for cfg in configs: + overlaps = [] + qps_vals = [] + for r in results: + overlaps.append(r["overlap_vs_ground_truth"][cfg]["mean_overlap_at_k"]) + qps_vals.append(r["configs"][cfg]["latency"]["qps"]) + + ax.plot( + overlaps, + qps_vals, + marker=markers[cfg], + markersize=8, + linewidth=2, + label=LABELS[cfg], + color=COLORS[cfg], + ) + # Annotate points with size + for ov, qps, r in zip(overlaps, qps_vals, results): + ax.annotate( + f'{r["size"]//1000}K', + (ov, qps), + textcoords="offset points", + xytext=(5, 5), + fontsize=7, + color=COLORS[cfg], + ) + + ax.set_xlabel("Overlap@K (Precision)") + ax.set_ylabel("Queries Per Second (QPS)") + ax.set_title(f"Precision vs Throughput -- {dataset}") + ax.legend(loc="best") + ax.set_xlim(0, 1.05) + ax.set_ylim(bottom=0) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_qps_vs_overlap.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_qps_vs_overlap.png") + + +def chart_memory_savings(results: List[Dict], dataset: str, output_dir: str): + """Chart 6: Memory savings percentage (Redis blog Chart 1 style).""" + fig, ax = plt.subplots(figsize=(10, 6)) + sizes = [r["size"] for r in results] + + # Calculate savings: FP16 vs FP32 for both FLAT and HNSW + pairs = [ + ("FLAT", "FLAT_FP32", "FLAT_FP16", "#3B82F6"), + ("HNSW", "HNSW_FP32", "HNSW_FP16", "#F97316"), + ] + + x = np.arange(len(sizes)) + width = 0.3 + + for i, (label, fp32, fp16, color) in enumerate(pairs): + savings = [] + for r in results: + m32 = r["configs"][fp32]["memory_mb"] + m16 = r["configs"][fp16]["memory_mb"] + pct = (1 - m16 / m32) * 100 if m32 > 0 else 0.0 + savings.append(pct) + + bars = ax.bar( + x + i * width, + savings, + width, + label=f"{label} FP16 savings", + color=color, + edgecolor="white", + linewidth=0.5, + ) + for bar, val in zip(bars, savings): + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + 0.5, + f"{val:.1f}%", + ha="center", + va="bottom", + fontsize=9, + fontweight="bold", + ) + + ax.set_xlabel("Corpus Size") + ax.set_ylabel("Memory Savings (%)") + ax.set_title(f"FP16 Memory Savings vs FP32 -- {dataset}") + ax.set_xticks(x + width * 0.5) + ax.set_xticklabels([f"{s:,}" for s in sizes]) + ax.legend(loc="lower right") + ax.set_ylim(0, 60) + ax.yaxis.set_major_formatter(mticker.PercentFormatter()) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_memory_savings.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_memory_savings.png") + + +def chart_build_time(results: List[Dict], dataset: str, output_dir: str): + """Chart 7: Index build/load time comparison.""" + fig, ax = plt.subplots(figsize=(10, 6)) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + sizes = [r["size"] for r in results] + x = np.arange(len(sizes)) + width = 0.18 + + for i, cfg in enumerate(configs): + times = [r["configs"][cfg]["load_duration_seconds"] for r in results] + bars = ax.bar( + x + i * width, + times, + width, + label=LABELS[cfg], + color=COLORS[cfg], + edgecolor="white", + linewidth=0.5, + ) + for bar, val in zip(bars, times): + if val > 0.1: + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + 0.2, + f"{val:.1f}s", + ha="center", + va="bottom", + fontsize=7, + ) + + ax.set_xlabel("Corpus Size") + ax.set_ylabel("Build Time (seconds)") + ax.set_title(f"Index Build Time -- {dataset}") + ax.set_xticks(x + width * 1.5) + ax.set_xticklabels([f"{s:,}" for s in sizes]) + ax.legend(loc="upper left") + ax.set_ylim(bottom=0) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_build_time.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_build_time.png") + + +def chart_recall_curve(results: List[Dict], dataset: str, output_dir: str): + """Chart 8: Recall@K curve -- recall at multiple K values for the largest size.""" + # Use the largest corpus size for the recall curve + r = results[-1] + recall_data = r.get("recall_vs_ground_truth") + if not recall_data: + print(f" Skipping recall curve (no recall data in results)") + return + + fig, ax = plt.subplots(figsize=(10, 6)) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + markers = {"FLAT_FP32": "s", "FLAT_FP16": "D", "HNSW_FP32": "o", "HNSW_FP16": "^"} + linestyles = { + "FLAT_FP32": "-", + "FLAT_FP16": "--", + "HNSW_FP32": "-", + "HNSW_FP16": "--", + } + + for cfg in configs: + if cfg not in recall_data: + continue + recall_at_k = recall_data[cfg].get("recall_at_k", {}) + if not recall_at_k: + continue + k_vals = sorted([int(k.split("@")[1]) for k in recall_at_k.keys()]) + recalls = [recall_at_k[f"recall@{k}"] for k in k_vals] + + ax.plot( + k_vals, + recalls, + marker=markers[cfg], + markersize=7, + linewidth=2, + linestyle=linestyles[cfg], + label=LABELS[cfg], + color=COLORS[cfg], + ) + + ax.set_xlabel("K (number of results)") + ax.set_ylabel("Recall@K") + ax.set_title(f"Recall@K Curve at {r['size']:,} documents -- {dataset}") + ax.legend(loc="lower right") + ax.set_ylim(0, 1.05) + ax.set_xlim(left=0) + ax.grid(True, alpha=0.3) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_recall_curve.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_recall_curve.png") + + +def chart_recall_by_size(results: List[Dict], dataset: str, output_dir: str): + """Chart 9: Recall@10 comparison across corpus sizes (grouped bar chart).""" + # Check if recall data exists + if not results[0].get("recall_vs_ground_truth"): + print(f" Skipping recall by size (no recall data)") + return + + fig, ax = plt.subplots(figsize=(10, 6)) + configs = ["FLAT_FP32", "FLAT_FP16", "HNSW_FP32", "HNSW_FP16"] + sizes = [r["size"] for r in results] + x = np.arange(len(sizes)) + width = 0.18 + + for i, cfg in enumerate(configs): + recalls = [] + for r in results: + recall_data = r.get("recall_vs_ground_truth", {}).get(cfg, {}) + recall_at_k = recall_data.get("recall_at_k", {}) + recalls.append(recall_at_k.get("recall@10", 0)) + bars = ax.bar( + x + i * width, + recalls, + width, + label=LABELS[cfg], + color=COLORS[cfg], + edgecolor="white", + linewidth=0.5, + ) + for bar, val in zip(bars, recalls): + ax.text( + bar.get_x() + bar.get_width() / 2, + bar.get_height() + 0.005, + f"{val:.3f}", + ha="center", + va="bottom", + fontsize=8, + ) + + ax.set_xlabel("Corpus Size") + ax.set_ylabel("Recall@10") + ax.set_title(f"Recall@10: FP32 vs FP16 -- {dataset}") + ax.set_xticks(x + width * 1.5) + ax.set_xticklabels([f"{s:,}" for s in sizes]) + ax.legend(loc="lower left") + ax.set_ylim(0, 1.1) + fig.tight_layout() + fig.savefig(os.path.join(output_dir, f"{dataset}_recall.png"), dpi=150) + plt.close(fig) + print(f" Saved {dataset}_recall.png") + + +def main(): + parser = argparse.ArgumentParser(description="Visualize benchmark results.") + parser.add_argument( + "--input", nargs="+", required=True, help="One or more result JSON files." + ) + parser.add_argument( + "--output-dir", + default="tests/benchmarks/charts/", + help="Directory to save chart images.", + ) + args = parser.parse_args() + + os.makedirs(args.output_dir, exist_ok=True) + setup_style() + + for path in args.input: + data = load_results(path) + dataset = data["dataset"] + results = data["results"] + print(f"\nGenerating charts for {dataset} ({len(results)} sizes) ...") + + chart_memory(results, dataset, args.output_dir) + chart_overlap(results, dataset, args.output_dir) + chart_qps(results, dataset, args.output_dir) + chart_latency(results, dataset, args.output_dir) + chart_qps_vs_overlap(results, dataset, args.output_dir) + chart_memory_savings(results, dataset, args.output_dir) + chart_build_time(results, dataset, args.output_dir) + chart_recall_curve(results, dataset, args.output_dir) + chart_recall_by_size(results, dataset, args.output_dir) + + print(f"\nAll charts saved to {args.output_dir}") + + +if __name__ == "__main__": + main() From 0fce1d3f51e7f80a38f39a8518bee2218ba16e12 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Thu, 2 Apr 2026 16:26:22 -0400 Subject: [PATCH 2/8] fix: remove unrelated SQL notebook, fix emdashes in migration docs - Remove 13_sql_query_exercises.ipynb (unrelated to migration feature) - Replace ' -- ' emdashes with colons in crash-safe resume docs --- docs/user_guide/13_sql_query_exercises.ipynb | 1239 ----------------- .../how_to_guides/migrate-indexes.md | 10 +- 2 files changed, 5 insertions(+), 1244 deletions(-) delete mode 100644 docs/user_guide/13_sql_query_exercises.ipynb diff --git a/docs/user_guide/13_sql_query_exercises.ipynb b/docs/user_guide/13_sql_query_exercises.ipynb deleted file mode 100644 index c09af709f..000000000 --- a/docs/user_guide/13_sql_query_exercises.ipynb +++ /dev/null @@ -1,1239 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# SQL-to-Redis Query Translation: Hands-On Exercises\n", - "\n", - "This notebook provides hands-on exercises for learning the new **SQLQuery** feature in RedisVL, which allows you to write familiar SQL syntax that automatically translates to Redis Search commands.\n", - "\n", - "## What You'll Learn\n", - "\n", - "1. How to use the `SQLQuery` class to write SQL-like queries\n", - "2. Three equivalent approaches for the same queries:\n", - " - **RedisVL Python API** - Using native query classes (`FilterQuery`, `VectorQuery`, etc.)\n", - " - **RedisVL SQL** - Using the new `SQLQuery` class with SQL syntax\n", - " - **Raw Redis FT.SEARCH** - The equivalent Redis Search command\n", - "3. Various query types: filtering, numeric ranges, text search, aggregations, and vector similarity\n", - "\n", - "## Prerequisites\n", - "\n", - "- Redis Stack running locally (or Redis Cloud)\n", - "- RedisVL with SQL support: `pip install redisvl[sql-redis]`\n", - "\n", - "## Documentation References\n", - "\n", - "- [RedisVL Documentation](https://docs.redisvl.com)\n", - "- [Redis Search Query Syntax](https://redis.io/docs/latest/develop/ai/search-and-query/query/)\n", - "- [Redis Aggregations](https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/aggregations/)\n", - "- [sql-redis Package](https://pypi.org/project/sql-redis/)\n", - "\n", - "---" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Setup: Create Sample Dataset and Index\n", - "\n", - "We'll create a realistic e-commerce products dataset with multiple field types to demonstrate various query capabilities." - ] - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T22:52:37.820727Z", - "start_time": "2026-02-05T22:52:37.431065Z" - } - }, - "source": [ - "import numpy as np\n", - "from redis import Redis\n", - "from redisvl.index import SearchIndex\n", - "from redisvl.query import FilterQuery, VectorQuery, CountQuery, SQLQuery\n", - "from redisvl.query.filter import Tag, Num, Text\n", - "\n", - "# Redis connection\n", - "REDIS_URL = \"redis://localhost:6379\"\n", - "client = Redis.from_url(REDIS_URL)\n", - "\n", - "# Define schema with multiple field types\n", - "schema = {\n", - " \"index\": {\n", - " \"name\": \"products_exercise\",\n", - " \"prefix\": \"product_exercise\",\n", - " \"storage_type\": \"hash\",\n", - " },\n", - " \"fields\": [\n", - " {\"name\": \"name\", \"type\": \"text\", \"attrs\": {\"sortable\": True}},\n", - " {\"name\": \"description\", \"type\": \"text\"},\n", - " {\"name\": \"category\", \"type\": \"tag\", \"attrs\": {\"sortable\": True}},\n", - " {\"name\": \"brand\", \"type\": \"tag\"},\n", - " {\"name\": \"price\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", - " {\"name\": \"stock\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", - " {\"name\": \"rating\", \"type\": \"numeric\", \"attrs\": {\"sortable\": True}},\n", - " {\n", - " \"name\": \"embedding\",\n", - " \"type\": \"vector\",\n", - " \"attrs\": {\n", - " \"dims\": 4,\n", - " \"distance_metric\": \"cosine\",\n", - " \"algorithm\": \"flat\",\n", - " \"datatype\": \"float32\",\n", - " },\n", - " },\n", - " ],\n", - "}\n", - "\n", - "# Create the index\n", - "index = SearchIndex.from_dict(schema, redis_client=client)\n", - "index.create(overwrite=True, drop=True)\n", - "print(f\"Created index: {index.name}\")" - ], - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Created index: products_exercise\n" - ] - } - ], - "execution_count": 1 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T22:52:57.150152Z", - "start_time": "2026-02-05T22:52:57.105851Z" - } - }, - "source": [ - "# Sample product data with embeddings\n", - "products = [\n", - " {\"name\": \"Gaming Laptop Pro\", \"description\": \"High-performance laptop for gaming\", \"category\": \"electronics\", \"brand\": \"TechBrand\", \"price\": 1299, \"stock\": 15, \"rating\": 4.7, \"embedding\": np.array([0.9, 0.1, 0.2, 0.3], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Budget Laptop Basic\", \"description\": \"Affordable laptop for everyday tasks\", \"category\": \"electronics\", \"brand\": \"ValueTech\", \"price\": 499, \"stock\": 50, \"rating\": 4.0, \"embedding\": np.array([0.8, 0.2, 0.3, 0.4], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Wireless Mouse\", \"description\": \"Ergonomic wireless mouse\", \"category\": \"electronics\", \"brand\": \"TechBrand\", \"price\": 35, \"stock\": 200, \"rating\": 4.3, \"embedding\": np.array([0.7, 0.3, 0.4, 0.5], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Python Programming Guide\", \"description\": \"Comprehensive Python programming guide\", \"category\": \"books\", \"brand\": \"TechBooks\", \"price\": 45, \"stock\": 100, \"rating\": 4.8, \"embedding\": np.array([0.2, 0.8, 0.1, 0.3], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Redis in Action\", \"description\": \"Learn Redis with practical examples\", \"category\": \"books\", \"brand\": \"TechBooks\", \"price\": 55, \"stock\": 75, \"rating\": 4.6, \"embedding\": np.array([0.3, 0.7, 0.2, 0.4], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Data Science Handbook\", \"description\": \"Essential data science handbook\", \"category\": \"books\", \"brand\": \"DataPress\", \"price\": 65, \"stock\": 40, \"rating\": 4.5, \"embedding\": np.array([0.25, 0.75, 0.15, 0.35], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Mechanical Keyboard\", \"description\": \"Premium mechanical keyboard with RGB\", \"category\": \"electronics\", \"brand\": \"KeyMaster\", \"price\": 149, \"stock\": 80, \"rating\": 4.6, \"embedding\": np.array([0.6, 0.4, 0.5, 0.6], dtype=np.float32).tobytes()},\n", - " {\"name\": \"USB-C Hub\", \"description\": \"Multi-port USB-C hub\", \"category\": \"electronics\", \"brand\": \"TechBrand\", \"price\": 49, \"stock\": 150, \"rating\": 4.2, \"embedding\": np.array([0.65, 0.35, 0.45, 0.55], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Desk Lamp LED\", \"description\": \"Adjustable LED desk lamp\", \"category\": \"accessories\", \"brand\": \"LightCo\", \"price\": 39, \"stock\": 120, \"rating\": 4.1, \"embedding\": np.array([0.4, 0.5, 0.6, 0.7], dtype=np.float32).tobytes()},\n", - " {\"name\": \"Monitor Stand\", \"description\": \"Ergonomic monitor stand\", \"category\": \"accessories\", \"brand\": \"DeskPro\", \"price\": 79, \"stock\": 60, \"rating\": 4.4, \"embedding\": np.array([0.45, 0.55, 0.65, 0.75], dtype=np.float32).tobytes()},\n", - "]\n", - "\n", - "# Load data into Redis\n", - "keys = index.load(products)\n", - "print(f\"Loaded {len(keys)} products into Redis\")" - ], - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Loaded 10 products into Redis\n" - ] - } - ], - "execution_count": 2 - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Exercise 1: Simple Tag Filtering\n", - "\n", - "**Goal:** Find all products in the \"electronics\" category.\n", - "\n", - "### Do It Yourself\n", - "\n", - "**Documentation:**\n", - "- [RedisVL FilterQuery](https://docs.redisvl.com/en/latest/api/query.html#filterquery)\n", - "- [Redis Tag Queries](https://redis.io/docs/latest/develop/ai/search-and-query/query/exact-match/)" - ] - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T22:56:41.131491Z", - "start_time": "2026-02-05T22:56:41.113689Z" - } - }, - "source": [ - "# YOUR CODE HERE - Method 1: RedisVL Python API\n", - "# Hint: Use Tag(\"category\") == \"electronics\" with FilterQuery\n", - "q= FilterQuery(\n", - " filter_expression=Tag(\"category\") == \"electronics\",\n", - " return_fields=[\"name\", \"category\", \"price\"],\n", - " num_results=10\n", - ")\n", - "\n", - "q = index.query(q)\n", - "q" - ], - "outputs": [ - { - "data": { - "text/plain": [ - "[{'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GK',\n", - " 'name': 'Wireless Mouse',\n", - " 'category': 'electronics',\n", - " 'price': '35'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GQ',\n", - " 'name': 'Mechanical Keyboard',\n", - " 'category': 'electronics',\n", - " 'price': '149'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GH',\n", - " 'name': 'Gaming Laptop Pro',\n", - " 'category': 'electronics',\n", - " 'price': '1299'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GJ',\n", - " 'name': 'Budget Laptop Basic',\n", - " 'category': 'electronics',\n", - " 'price': '499'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", - " 'name': 'USB-C Hub',\n", - " 'category': 'electronics',\n", - " 'price': '49'}]" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "execution_count": 5 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T22:58:38.451826Z", - "start_time": "2026-02-05T22:58:38.269871Z" - } - }, - "source": [ - "# YOUR CODE HERE - Method 2: SQLQuery\n", - "# Hint: SELECT ... FROM products_exercise WHERE category = 'electronics'\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT name, category, price\n", - " FROM {index.name}\n", - " WHERE category = 'electronics'\n", - "\"\"\")\n", - "index.query(sql_query)" - ], - "outputs": [ - { - "data": { - "text/plain": [ - "[{'name': 'Wireless Mouse', 'category': 'electronics', 'price': '35'},\n", - " {'name': 'Mechanical Keyboard', 'category': 'electronics', 'price': '149'},\n", - " {'name': 'Gaming Laptop Pro', 'category': 'electronics', 'price': '1299'},\n", - " {'name': 'Budget Laptop Basic', 'category': 'electronics', 'price': '499'},\n", - " {'name': 'USB-C Hub', 'category': 'electronics', 'price': '49'}]" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "execution_count": 7 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T22:59:22.518910Z", - "start_time": "2026-02-05T22:59:22.495076Z" - } - }, - "source": [ - "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", - "# Hint: client.execute_command(\"FT.SEARCH\", index_name, \"@category:{electronics}\", ...)\n", - "client.execute_command(\"FT.Search\", index.name, \"@category:{electronics}\")" - ], - "outputs": [ - { - "data": { - "text/plain": [ - "[5,\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GK',\n", - " [b'name',\n", - " b'Wireless Mouse',\n", - " b'description',\n", - " b'Ergonomic wireless mouse',\n", - " b'category',\n", - " b'electronics',\n", - " b'brand',\n", - " b'TechBrand',\n", - " b'price',\n", - " b'35',\n", - " b'stock',\n", - " b'200',\n", - " b'rating',\n", - " b'4.3',\n", - " b'embedding',\n", - " b'333?\\x9a\\x99\\x99>\\xcd\\xcc\\xcc>\\x00\\x00\\x00?'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GQ',\n", - " [b'name',\n", - " b'Mechanical Keyboard',\n", - " b'description',\n", - " b'Premium mechanical keyboard with RGB',\n", - " b'category',\n", - " b'electronics',\n", - " b'brand',\n", - " b'KeyMaster',\n", - " b'price',\n", - " b'149',\n", - " b'stock',\n", - " b'80',\n", - " b'rating',\n", - " b'4.6',\n", - " b'embedding',\n", - " b'\\x9a\\x99\\x19?\\xcd\\xcc\\xcc>\\x00\\x00\\x00?\\x9a\\x99\\x19?'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GH',\n", - " [b'name',\n", - " b'Gaming Laptop Pro',\n", - " b'description',\n", - " b'High-performance laptop for gaming',\n", - " b'category',\n", - " b'electronics',\n", - " b'brand',\n", - " b'TechBrand',\n", - " b'price',\n", - " b'1299',\n", - " b'stock',\n", - " b'15',\n", - " b'rating',\n", - " b'4.7',\n", - " b'embedding',\n", - " b'fff?\\xcd\\xcc\\xcc=\\xcd\\xccL>\\x9a\\x99\\x99>'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GJ',\n", - " [b'name',\n", - " b'Budget Laptop Basic',\n", - " b'description',\n", - " b'Affordable laptop for everyday tasks',\n", - " b'category',\n", - " b'electronics',\n", - " b'brand',\n", - " b'ValueTech',\n", - " b'price',\n", - " b'499',\n", - " b'stock',\n", - " b'50',\n", - " b'rating',\n", - " b'4.0',\n", - " b'embedding',\n", - " b'\\xcd\\xccL?\\xcd\\xccL>\\x9a\\x99\\x99>\\xcd\\xcc\\xcc>'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", - " [b'name',\n", - " b'USB-C Hub',\n", - " b'description',\n", - " b'Multi-port USB-C hub',\n", - " b'category',\n", - " b'electronics',\n", - " b'brand',\n", - " b'TechBrand',\n", - " b'price',\n", - " b'49',\n", - " b'stock',\n", - " b'150',\n", - " b'rating',\n", - " b'4.2',\n", - " b'embedding',\n", - " b'ff&?33\\xb3>ff\\xe6>\\xcd\\xcc\\x0c?']]" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "execution_count": 9 - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Solution: Exercise 1" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 1: RedisVL Python API\n", - "filter_expr = Tag(\"category\") == \"electronics\"\n", - "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"category\", \"price\"], num_results=10)\n", - "results_api = index.query(query)\n", - "print(\"=== Method 1: RedisVL Python API ===\")\n", - "for r in results_api:\n", - " print(f\" {r['name']} - ${r['price']}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 2: RedisVL SQL\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT name, category, price\n", - " FROM {index.name}\n", - " WHERE category = 'electronics'\n", - "\"\"\")\n", - "results_sql = index.query(sql_query)\n", - "print(\"=== Method 2: RedisVL SQL ===\")\n", - "for r in results_sql:\n", - " print(f\" {r['name']} - ${r['price']}\")\n", - "\n", - "# Show the translated Redis command\n", - "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", - "print(f\"\\nTranslated Redis command: {redis_cmd}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 3: Raw Redis FT.SEARCH\n", - "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"@category:{electronics}\", \"RETURN\", \"3\", \"name\", \"category\", \"price\", \"LIMIT\", \"0\", \"10\")\n", - "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", - "total = raw_results[0]\n", - "print(f\"Total matches: {total}\")\n", - "for i in range(1, len(raw_results), 2):\n", - " if i + 1 < len(raw_results):\n", - " fields = raw_results[i + 1]\n", - " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", - " print(f\" {field_dict.get('name', 'N/A')} - ${field_dict.get('price', 'N/A')}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Exercise 2: Numeric Range Queries\n", - "\n", - "**Goal:** Find all products with price between $40 and $100.\n", - "\n", - "### Do It Yourself\n", - "\n", - "**Documentation:**\n", - "- [RedisVL Numeric Filters](https://docs.redisvl.com/en/latest/api/query.html#redisvl.query.filter.Num)\n", - "- [Redis Numeric Range Queries](https://redis.io/docs/latest/develop/ai/search-and-query/query/range/)" - ] - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:01:53.873244Z", - "start_time": "2026-02-05T23:01:53.847120Z" - } - }, - "source": [ - "# YOUR CODE HERE - Method 1: RedisVL Python API\n", - "# Hint: Use Num(\"price\").between(40, 100) with FilterQuery\n", - "q = FilterQuery(\n", - " filter_expression = Num(\"price\").between(40,100),\n", - " return_fields=[\"name\", \"price\"],\n", - " num_results=10\n", - ")\n", - "index.query(q)" - ], - "outputs": [ - { - "data": { - "text/plain": [ - "[{'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GM',\n", - " 'name': 'Python Programming Guide',\n", - " 'price': '45'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GN',\n", - " 'name': 'Redis in Action',\n", - " 'price': '55'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GP',\n", - " 'name': 'Data Science Handbook',\n", - " 'price': '65'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", - " 'name': 'USB-C Hub',\n", - " 'price': '49'},\n", - " {'id': 'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GT',\n", - " 'name': 'Monitor Stand',\n", - " 'price': '79'}]" - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "execution_count": 14 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:02:52.791662Z", - "start_time": "2026-02-05T23:02:52.770651Z" - } - }, - "source": [ - "# YOUR CODE HERE - Method 2: SQLQuery\n", - "# Hint: SELECT ... WHERE price BETWEEN 40 AND 100\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT name, price from {index.name} where price between 40 and 100\n", - "\"\"\")\n", - "\n", - "index.query(sql_query)\n" - ], - "outputs": [ - { - "data": { - "text/plain": [ - "[{'name': 'Python Programming Guide', 'price': '45'},\n", - " {'name': 'Redis in Action', 'price': '55'},\n", - " {'name': 'Data Science Handbook', 'price': '65'},\n", - " {'name': 'USB-C Hub', 'price': '49'},\n", - " {'name': 'Monitor Stand', 'price': '79'}]" - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], - "execution_count": 17 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:03:50.055289Z", - "start_time": "2026-02-05T23:03:50.029265Z" - } - }, - "source": [ - "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", - "# Hint: @price:[40 100]\n", - "client.execute_command(\"FT.SEARCH\", index.name, \"@price:[40 100]\", \"RETURN\", \"2\", \"name\", \"price\", \"LIMIT\", \"0\", \"10\")" - ], - "outputs": [ - { - "data": { - "text/plain": [ - "[5,\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GM',\n", - " [b'name', b'Python Programming Guide', b'price', b'45'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GN',\n", - " [b'name', b'Redis in Action', b'price', b'55'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GP',\n", - " [b'name', b'Data Science Handbook', b'price', b'65'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GR',\n", - " [b'name', b'USB-C Hub', b'price', b'49'],\n", - " b'product_exercise:01KGR04DCSDX9D0KSD7GAYC1GT',\n", - " [b'name', b'Monitor Stand', b'price', b'79']]" - ] - }, - "execution_count": 19, - "metadata": {}, - "output_type": "execute_result" - } - ], - "execution_count": 19 - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Solution: Exercise 2" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 1: RedisVL Python API\n", - "filter_expr = Num(\"price\").between(40, 100)\n", - "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"price\"], num_results=10)\n", - "results_api = index.query(query)\n", - "print(\"=== Method 1: RedisVL Python API ===\")\n", - "for r in results_api:\n", - " print(f\" {r['name']} - ${r['price']}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 2: RedisVL SQL\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT name, price\n", - " FROM {index.name}\n", - " WHERE price BETWEEN 40 AND 100\n", - "\"\"\")\n", - "results_sql = index.query(sql_query)\n", - "print(\"=== Method 2: RedisVL SQL ===\")\n", - "for r in results_sql:\n", - " print(f\" {r['name']} - ${r['price']}\")\n", - "\n", - "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", - "print(f\"\\nTranslated Redis command: {redis_cmd}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 3: Raw Redis FT.SEARCH\n", - "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"@price:[40 100]\", \"RETURN\", \"2\", \"name\", \"price\", \"LIMIT\", \"0\", \"10\")\n", - "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", - "for i in range(1, len(raw_results), 2):\n", - " if i + 1 < len(raw_results):\n", - " fields = raw_results[i + 1]\n", - " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", - " print(f\" {field_dict.get('name', 'N/A')} - ${field_dict.get('price', 'N/A')}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Exercise 3: Combined Filters (AND/OR)\n", - "\n", - "**Goal:** Find electronics products under $100.\n", - "\n", - "### Do It Yourself\n", - "\n", - "**Documentation:**\n", - "- [RedisVL Filter Expressions](https://docs.redisvl.com/en/latest/api/query.html#filter-expressions)\n", - "- [Redis Combined Queries](https://redis.io/docs/latest/develop/ai/search-and-query/query/)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 1: RedisVL Python API\n", - "# Hint: Combine filters with & operator: (Tag(\"category\") == \"electronics\") & (Num(\"price\") < 100)\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 2: SQLQuery\n", - "# Hint: WHERE category = 'electronics' AND price < 100\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", - "# Hint: (@category:{electronics} @price:[-inf 100])\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Solution: Exercise 3" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 1: RedisVL Python API\n", - "filter_expr = (Tag(\"category\") == \"electronics\") & (Num(\"price\") < 100)\n", - "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"category\", \"price\"], num_results=10)\n", - "results_api = index.query(query)\n", - "print(\"=== Method 1: RedisVL Python API ===\")\n", - "for r in results_api:\n", - " print(f\" {r['name']} ({r['category']}) - ${r['price']}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 2: RedisVL SQL\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT name, category, price\n", - " FROM {index.name}\n", - " WHERE category = 'electronics' AND price < 100\n", - "\"\"\")\n", - "results_sql = index.query(sql_query)\n", - "print(\"=== Method 2: RedisVL SQL ===\")\n", - "for r in results_sql:\n", - " print(f\" {r['name']} ({r['category']}) - ${r['price']}\")\n", - "\n", - "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", - "print(f\"\\nTranslated Redis command: {redis_cmd}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 3: Raw Redis FT.SEARCH\n", - "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"(@category:{electronics} @price:[-inf (100])\", \"RETURN\", \"3\", \"name\", \"category\", \"price\", \"LIMIT\", \"0\", \"10\")\n", - "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", - "for i in range(1, len(raw_results), 2):\n", - " if i + 1 < len(raw_results):\n", - " fields = raw_results[i + 1]\n", - " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", - " print(f\" {field_dict.get('name', 'N/A')} ({field_dict.get('category', 'N/A')}) - ${field_dict.get('price', 'N/A')}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Exercise 4: Text Search\n", - "\n", - "**Goal:** Find products with \"laptop\" in the name.\n", - "\n", - "### Do It Yourself\n", - "\n", - "**Documentation:**\n", - "- [RedisVL Text Filters](https://docs.redisvl.com/en/latest/api/query.html#redisvl.query.filter.Text)\n", - "- [Redis Full-Text Search](https://redis.io/docs/latest/develop/ai/search-and-query/query/full-text/)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 1: RedisVL Python API\n", - "# Hint: Use Text(\"name\") % \"laptop\" with FilterQuery\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 2: SQLQuery\n", - "# Hint: WHERE name = 'laptop'\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", - "# Hint: @name:laptop\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Solution: Exercise 4" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 1: RedisVL Python API\n", - "filter_expr = Text(\"name\") % \"laptop\"\n", - "query = FilterQuery(filter_expression=filter_expr, return_fields=[\"name\", \"description\", \"price\"], num_results=10)\n", - "results_api = index.query(query)\n", - "print(\"=== Method 1: RedisVL Python API ===\")\n", - "for r in results_api:\n", - " print(f\" {r['name']} - ${r['price']}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 2: RedisVL SQL\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT name, description, price\n", - " FROM {index.name}\n", - " WHERE name = 'laptop'\n", - "\"\"\")\n", - "results_sql = index.query(sql_query)\n", - "print(\"=== Method 2: RedisVL SQL ===\")\n", - "for r in results_sql:\n", - " print(f\" {r['name']} - ${r['price']}\")\n", - "\n", - "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", - "print(f\"\\nTranslated Redis command: {redis_cmd}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 3: Raw Redis FT.SEARCH\n", - "raw_results = client.execute_command(\"FT.SEARCH\", index.name, \"@name:laptop\", \"RETURN\", \"3\", \"name\", \"description\", \"price\", \"LIMIT\", \"0\", \"10\")\n", - "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", - "for i in range(1, len(raw_results), 2):\n", - " if i + 1 < len(raw_results):\n", - " fields = raw_results[i + 1]\n", - " field_dict = {fields[j].decode(): fields[j+1].decode() for j in range(0, len(fields), 2)}\n", - " print(f\" {field_dict.get('name', 'N/A')} - ${field_dict.get('price', 'N/A')}\")\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Exercise 5: Vector Similarity Search\n", - "\n", - "**Goal:** Find products most similar to a query vector (simulating a semantic search).\n", - "\n", - "### Do It Yourself\n", - "\n", - "**Documentation:**\n", - "- [RedisVL VectorQuery](https://docs.redisvl.com/en/latest/api/query.html#vectorquery)\n", - "- [Redis Vector Search](https://redis.io/docs/latest/develop/ai/search-and-query/vectors/)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 1: RedisVL Python API\n", - "# Hint: Use VectorQuery with a query vector\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 2: SQLQuery\n", - "# Hint: SELECT ... ORDER BY cosine_distance(embedding, ) LIMIT k\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 3: Raw FT.SEARCH\n", - "# Hint: FT.SEARCH with KNN and BLOB parameter\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Solution: Exercise 5" - ] - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:05:53.684001Z", - "start_time": "2026-02-05T23:05:53.656720Z" - } - }, - "source": [ - "# Query vector (similar to electronics products)\n", - "query_vector = np.array([0.85, 0.15, 0.25, 0.35], dtype=np.float32)\n", - "\n", - "# Method 1: RedisVL Python API\n", - "vector_query = VectorQuery(\n", - " vector=query_vector,\n", - " vector_field_name=\"embedding\",\n", - " return_fields=[\"name\", \"category\", \"price\"],\n", - " num_results=3\n", - ")\n", - "results_api = index.query(vector_query)\n", - "print(\"=== Method 1: RedisVL Python API ===\")\n", - "for r in results_api:\n", - " print(f\" {r['name']} ({r['category']}) - distance: {r.get('vector_distance', 'N/A')}\")" - ], - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=== Method 1: RedisVL Python API ===\n", - " Gaming Laptop Pro (electronics) - distance: 0.00526285171509\n", - " Budget Laptop Basic (electronics) - distance: 0.00537633895874\n", - " Wireless Mouse (electronics) - distance: 0.0464093089104\n" - ] - } - ], - "execution_count": 21 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:07:23.911432Z", - "start_time": "2026-02-05T23:07:23.889830Z" - } - }, - "source": [ - "# Method 2: RedisVL SQL\n", - "# Note: sql-redis uses cosine_distance() function for vector search\n", - "vector_bytes = query_vector.tobytes()\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT name, category, price\n", - " FROM {index.name}\n", - " ORDER BY cosine_distance(embedding, :vector)\n", - " LIMIT 3\n", - "\"\"\",\n", - "params={\"vector\": vector_bytes}\n", - " )\n", - "\n", - "\n", - "results_sql = index.query(sql_query)\n", - "print(\"=== Method 2: RedisVL SQL ===\")\n", - "for r in results_sql:\n", - " print(f\" {r['name']} ({r['category']})\")\n", - "\n", - "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", - "print(f\"\\nTranslated Redis command: {redis_cmd[:100]}...\")\n", - "\n" - ], - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=== Method 2: RedisVL SQL ===\n", - " Wireless Mouse (electronics)\n", - " Monitor Stand (accessories)\n", - " Redis in Action (books)\n", - "\n", - "Translated Redis command: FT.SEARCH products_exercise \"*\" RETURN 3 name category price LIMIT 0 3...\n" - ] - } - ], - "execution_count": 26 - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Method 3: Raw Redis FT.SEARCH with KNN\n", - "import struct\n", - "vector_blob = struct.pack(f'{len(query_vector)}f', *query_vector)\n", - "raw_results = client.execute_command(\n", - " \"FT.SEARCH\", index.name,\n", - " \"*=>[KNN 3 @embedding $vec AS vector_distance]\",\n", - " \"PARAMS\", \"2\", \"vec\", vector_blob,\n", - " \"RETURN\", \"4\", \"name\", \"category\", \"price\", \"vector_distance\",\n", - " \"SORTBY\", \"vector_distance\", \"ASC\",\n", - " \"DIALECT\", \"2\"\n", - ")\n", - "print(\"=== Method 3: Raw FT.SEARCH ===\")\n", - "for i in range(1, len(raw_results), 2):\n", - " if i + 1 < len(raw_results):\n", - " fields = raw_results[i + 1]\n", - " field_dict = {fields[j].decode() if isinstance(fields[j], bytes) else fields[j]:\n", - " fields[j+1].decode() if isinstance(fields[j+1], bytes) else fields[j+1]\n", - " for j in range(0, len(fields), 2)}\n", - " print(f\" {field_dict.get('name', 'N/A')} ({field_dict.get('category', 'N/A')}) - distance: {field_dict.get('vector_distance', 'N/A')}\")\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Exercise 6: Aggregations (COUNT, GROUP BY, AVG)\n", - "\n", - "**Goal:** Count products by category and calculate average prices.\n", - "\n", - "### Do It Yourself\n", - "\n", - "**Documentation:**\n", - "- [RedisVL CountQuery](https://docs.redisvl.com/en/latest/api/query.html#countquery)\n", - "- [Redis Aggregations](https://redis.io/docs/latest/develop/ai/search-and-query/advanced-concepts/aggregations/)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 1: RedisVL Python API\n", - "# Hint: Use CountQuery for counting\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 2: SQLQuery\n", - "# Hint: SELECT category, COUNT(*), AVG(price) FROM ... GROUP BY category\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# YOUR CODE HERE - Method 3: Raw FT.AGGREGATE\n", - "# Hint: FT.AGGREGATE with GROUPBY and REDUCE\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Solution: Exercise 6" - ] - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:08:26.684434Z", - "start_time": "2026-02-05T23:08:26.651689Z" - } - }, - "source": [ - "# Method 1: RedisVL Python API - Count total products\n", - "count_query = CountQuery(filter_expression=Tag(\"category\") == \"electronics\")\n", - "count_result = index.query(count_query)\n", - "print(\"=== Method 1: RedisVL Python API ===\")\n", - "print(f\" Electronics products count: {count_result}\")\n", - "\n", - "# Count for each category\n", - "for cat in [\"electronics\", \"books\", \"accessories\"]:\n", - " count_query = CountQuery(filter_expression=Tag(\"category\") == cat)\n", - " count = index.query(count_query)\n", - " print(f\" {cat}: {count} products\")" - ], - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=== Method 1: RedisVL Python API ===\n", - " Electronics products count: 5\n", - " electronics: 5 products\n", - " books: 3 products\n", - " accessories: 2 products\n" - ] - } - ], - "execution_count": 27 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:08:26.968246Z", - "start_time": "2026-02-05T23:08:26.948261Z" - } - }, - "source": [ - "# Method 2: RedisVL SQL - Group by with aggregations\n", - "sql_query = SQLQuery(f\"\"\"\n", - " SELECT category, COUNT(*) as count, AVG(price) as avg_price\n", - " FROM {index.name}\n", - " GROUP BY category\n", - "\"\"\")\n", - "results_sql = index.query(sql_query)\n", - "print(\"=== Method 2: RedisVL SQL ===\")\n", - "for r in results_sql:\n", - " print(f\" {r['category']}: {r['count']} products, avg price: ${float(r['avg_price']):.2f}\")\n", - "\n", - "redis_cmd = sql_query.redis_query_string(redis_client=client)\n", - "print(f\"\\nTranslated Redis command: {redis_cmd}\")" - ], - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=== Method 2: RedisVL SQL ===\n", - " books: 3 products, avg price: $55.00\n", - " accessories: 2 products, avg price: $59.00\n", - " electronics: 5 products, avg price: $406.20\n", - "\n", - "Translated Redis command: FT.AGGREGATE products_exercise \"*\" LOAD 2 category price GROUPBY 1 @category REDUCE COUNT 0 AS count REDUCE AVG 1 @price AS avg_price\n" - ] - } - ], - "execution_count": 28 - }, - { - "cell_type": "code", - "metadata": { - "ExecuteTime": { - "end_time": "2026-02-05T23:09:08.232955Z", - "start_time": "2026-02-05T23:09:08.206373Z" - } - }, - "source": [ - "# Method 3: Raw Redis FT.AGGREGATE\n", - "raw_results = client.execute_command(\n", - " \"FT.AGGREGATE\", index.name, \"*\",\n", - " \"GROUPBY\", \"1\", \"@category\",\n", - " \"REDUCE\", \"COUNT\", \"0\", \"AS\", \"count\",\n", - " \"REDUCE\", \"AVG\", \"1\", \"@price\", \"AS\", \"avg_price\"\n", - ")\n", - "print(\"=== Method 3: Raw FT.AGGREGATE ===\")\n", - "for i in range(1, len(raw_results)):\n", - " row = raw_results[i]\n", - " row_dict = {row[j].decode() if isinstance(row[j], bytes) else row[j]:\n", - " row[j+1].decode() if isinstance(row[j+1], bytes) else row[j+1]\n", - " for j in range(0, len(row), 2)}\n", - " cat = row_dict.get('category', 'N/A')\n", - " count = row_dict.get('count', 'N/A')\n", - " avg_price = float(row_dict.get('avg_price', 0))\n", - " print(f\" {cat}: {count} products, avg price: ${avg_price:.2f}\")\n", - "\n" - ], - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "=== Method 3: Raw FT.AGGREGATE ===\n", - " books: 3 products, avg price: $55.00\n", - " accessories: 2 products, avg price: $59.00\n", - " electronics: 5 products, avg price: $406.20\n" - ] - } - ], - "execution_count": 29 - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Comparison Summary\n", - "\n", - "| Approach | Pros | Cons | Best For |\n", - "|----------|------|------|----------|\n", - "| **RedisVL Python API** | Type-safe, IDE autocomplete, Pythonic | Learning curve for filter expressions | Production applications, complex queries |\n", - "| **RedisVL SQL** | Familiar SQL syntax, easy migration | Limited to SQL capabilities | SQL developers, quick prototyping |\n", - "| **Raw FT.SEARCH** | Full control, all Redis features | Verbose, error-prone | Advanced use cases, debugging |\n", - "\n", - "### Key Takeaways\n", - "\n", - "1. **SQLQuery** is great for developers familiar with SQL who want to quickly query Redis\n", - "2. **RedisVL Python API** provides the best developer experience with type safety\n", - "3. **Raw FT.SEARCH** gives you full control but requires deep Redis knowledge\n", - "4. All three methods can achieve the same results - choose based on your team's expertise\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Cleanup" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Delete the index and clean up\n", - "index.delete(drop=True)\n", - "print(f\"Deleted index: {index.name}\")\n", - "print(\"Cleanup complete!\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---\n", - "## Next Steps\n", - "\n", - "- Explore more complex queries in the [RedisVL documentation](https://docs.redisvl.com)\n", - "- Try the [sql-redis package](https://pypi.org/project/sql-redis/) for more SQL features\n", - "- Check out other RedisVL features like SemanticCache and SemanticRouter" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.6" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/docs/user_guide/how_to_guides/migrate-indexes.md b/docs/user_guide/how_to_guides/migrate-indexes.md index 505c70dff..ec1d84e53 100644 --- a/docs/user_guide/how_to_guides/migrate-indexes.md +++ b/docs/user_guide/how_to_guides/migrate-indexes.md @@ -396,11 +396,11 @@ When migrating large datasets with vector quantization (e.g. float32 to float16) #### How it works -1. **Pre-flight estimate** -- before any mutations, `apply` prints a disk space estimate showing RDB snapshot cost, AOF growth (if enabled), and post-migration memory savings. -2. **BGSAVE safety snapshot** -- the migrator triggers a Redis `BGSAVE` and waits for it to complete before modifying any data. This gives you a point-in-time snapshot to fall back on. -3. **Checkpoint file** -- when `--resume` is provided, the migrator writes a YAML checkpoint after every batch of 500 documents. The checkpoint records how many keys have been processed and the last batch of keys written. -4. **Batch undo buffer** -- if a single batch fails mid-write, original vector values are rolled back via pipeline before the error propagates. Only the current batch is held in memory. -5. **Idempotent skip** -- on resume, vectors that were already converted are detected by byte-width inspection and skipped automatically. +1. **Pre-flight estimate**: before any mutations, `apply` prints a disk space estimate showing RDB snapshot cost, AOF growth (if enabled), and post-migration memory savings. +2. **BGSAVE safety snapshot**: the migrator triggers a Redis `BGSAVE` and waits for it to complete before modifying any data. This gives you a point-in-time snapshot to fall back on. +3. **Checkpoint file**: when `--resume` is provided, the migrator writes a YAML checkpoint after every batch of 500 documents. The checkpoint records how many keys have been processed and the last batch of keys written. +4. **Batch undo buffer**: if a single batch fails mid-write, original vector values are rolled back via pipeline before the error propagates. Only the current batch is held in memory. +5. **Idempotent skip**: on resume, vectors that were already converted are detected by byte-width inspection and skipped automatically. #### Step-by-step: using crash-safe resume From 7f7f7464428e004d11d39ee15448ed23c180a034 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Thu, 2 Apr 2026 17:25:35 -0400 Subject: [PATCH 3/8] fix: address P1/P2 review issues across migrator stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix async executor readiness check to handle missing percent_indexed - Fix benchmark wait_for_index_ready masking percent_indexed=0 - Fix wizard showing dependent prompts when sortable explicitly False - Fix CLI docs: --patch→--schema-patch, --output/-o→--plan-out - Fix migration docs: field renames now listed as supported - Fix batch resume not forwarding batch_plan_path to apply() - Fix batch-resume CLI missing quantization safety gate --- docs/api/cli.rst | 10 ++++---- docs/concepts/index-migrations.md | 2 +- redisvl/cli/migrate.py | 23 ++++++++++++++++++- redisvl/migration/async_executor.py | 15 ++++++++---- redisvl/migration/batch_executor.py | 1 + redisvl/migration/wizard.py | 19 +++++++++++---- .../index_migrator_real_benchmark.py | 2 +- tests/benchmarks/retrieval_benchmark.py | 2 +- 8 files changed, 56 insertions(+), 18 deletions(-) diff --git a/docs/api/cli.rst b/docs/api/cli.rst index 4f651a38c..ffd8659f1 100644 --- a/docs/api/cli.rst +++ b/docs/api/cli.rst @@ -494,7 +494,7 @@ Generate a migration plan for a document-preserving drop/recreate migration. .. code-block:: bash - rvl migrate plan --index (--patch | --target-schema ) [OPTIONS] + rvl migrate plan --index (--schema-patch | --target-schema ) [OPTIONS] **Required Options** @@ -506,7 +506,7 @@ Generate a migration plan for a document-preserving drop/recreate migration. - Description * - ``--index``, ``-i`` - Name of the source index to migrate - * - ``--patch`` + * - ``--schema-patch`` - Path to a YAML schema patch file (mutually exclusive with ``--target-schema``) * - ``--target-schema`` - Path to a full target schema YAML file (mutually exclusive with ``--patch``) @@ -519,14 +519,14 @@ Generate a migration plan for a document-preserving drop/recreate migration. * - Option - Description - * - ``--output``, ``-o`` + * - ``--plan-out`` - Output path for the migration plan YAML (default: ``migration_plan.yaml``) **Example** .. code-block:: bash - rvl migrate plan -i my_index --patch changes.yaml -o plan.yaml + rvl migrate plan -i my_index --schema-patch changes.yaml --plan-out plan.yaml rvl migrate apply ^^^^^^^^^^^^^^^^^ @@ -587,7 +587,7 @@ Interactively build a schema patch and migration plan through a guided wizard. .. code-block:: bash - rvl migrate wizard -i my_index -o plan.yaml + rvl migrate wizard -i my_index --plan-out plan.yaml Exit Codes ========== diff --git a/docs/concepts/index-migrations.md b/docs/concepts/index-migrations.md index 065522d98..fb12d75ca 100644 --- a/docs/concepts/index-migrations.md +++ b/docs/concepts/index-migrations.md @@ -79,7 +79,7 @@ These changes complete quickly because they only require rebuilding the index. - Field renames: Stored field names must be updated in every document. - Dimension changes: Vectors must be re-embedded with a different model. -The migrator handles datatype changes automatically. Other document dependent changes are blocked because they require application level logic or external services. +The migrator handles datatype changes and field renames automatically. Dimension changes are blocked because they require re-embedding with a different model (application level logic). ## Vector quantization diff --git a/redisvl/cli/migrate.py b/redisvl/cli/migrate.py index 9b91f549c..c1c76331b 100644 --- a/redisvl/cli/migrate.py +++ b/redisvl/cli/migrate.py @@ -637,6 +637,11 @@ def batch_resume(self): help="Retry previously failed indexes", action="store_true", ) + parser.add_argument( + "--accept-data-loss", + help="Acknowledge vector quantization data loss", + action="store_true", + ) parser.add_argument( "--report-dir", help="Directory for per-index migration reports", @@ -645,8 +650,24 @@ def batch_resume(self): parser = add_redis_connection_options(parser) args = parser.parse_args(sys.argv[3:]) - redis_url = create_redis_url(args) + # Load the batch plan to check for quantization safety gate executor = BatchMigrationExecutor() + state = executor._load_state(args.state) + plan_path = args.plan or state.plan_path + if plan_path: + batch_plan = executor._load_batch_plan(plan_path) + if batch_plan.requires_quantization and not args.accept_data_loss: + print( + """WARNING: This batch migration includes quantization (e.g., float32 -> float16). + Vector data will be modified. Original precision cannot be recovered. + To proceed, add --accept-data-loss flag. + + If you need to preserve original vectors, backup your data first: + redis-cli BGSAVE""" + ) + sys.exit(1) + + redis_url = create_redis_url(args) def progress_callback( index_name: str, position: int, total: int, status: str diff --git a/redisvl/migration/async_executor.py b/redisvl/migration/async_executor.py index 22fd3a832..40843a35e 100644 --- a/redisvl/migration/async_executor.py +++ b/redisvl/migration/async_executor.py @@ -1073,12 +1073,19 @@ async def _async_wait_for_index_ready( percent_indexed = latest_info.get("percent_indexed") if percent_indexed is not None or indexing is not None: - ready = float(percent_indexed or 0) >= 1.0 and not bool(indexing) + pct = float(percent_indexed) if percent_indexed is not None else None + is_indexing = bool(indexing) + if pct is not None: + ready = pct >= 1.0 and not is_indexing + else: + # percent_indexed missing but indexing flag present: + # treat as ready when indexing flag is falsy (0 / False). + ready = not is_indexing if progress_callback: total_docs = int(latest_info.get("num_docs", 0)) - pct = float(percent_indexed or 0) - indexed_docs = int(total_docs * pct) - progress_callback(indexed_docs, total_docs, pct * 100) + display_pct = pct if pct is not None else (1.0 if ready else 0.0) + indexed_docs = int(total_docs * display_pct) + progress_callback(indexed_docs, total_docs, display_pct * 100) else: current_docs = latest_info.get("num_docs") if current_docs is None: diff --git a/redisvl/migration/batch_executor.py b/redisvl/migration/batch_executor.py index 62024d38e..038e0a2a3 100644 --- a/redisvl/migration/batch_executor.py +++ b/redisvl/migration/batch_executor.py @@ -190,6 +190,7 @@ def resume( # Re-run apply with the updated state return self.apply( batch_plan, + batch_plan_path=batch_plan_path, state_path=state_path, report_dir=report_dir, redis_url=redis_url, diff --git a/redisvl/migration/wizard.py b/redisvl/migration/wizard.py index 24ec15cde..ae646027f 100644 --- a/redisvl/migration/wizard.py +++ b/redisvl/migration/wizard.py @@ -564,8 +564,13 @@ def _prompt_common_attrs( # No index - only meaningful with sortable. # When updating (allow_blank), also check the existing field's sortable # state so we offer dependent prompts even if the user left sortable blank. + # But if sortable was explicitly set to False, skip dependent prompts. _existing_sortable = self._existing_sortable - if sortable or (allow_blank and (_existing_sortable or attrs.get("sortable"))): + if sortable or ( + sortable is None + and allow_blank + and (_existing_sortable or attrs.get("sortable")) + ): print(" No index: store field for sorting only, not searchable") no_index = self._prompt_bool("No index", allow_blank=allow_blank) if no_index is not None: @@ -604,8 +609,10 @@ def _prompt_text_attrs(self, attrs: Dict[str, Any], allow_blank: bool) -> None: if phonetic: attrs["phonetic_matcher"] = phonetic - # UNF (only if sortable) - if attrs.get("sortable") or self._existing_sortable: + # UNF (only if sortable – skip if sortable was explicitly set to False) + if attrs.get("sortable") or ( + attrs.get("sortable") is not False and self._existing_sortable + ): print(" UNF: preserve original form (no lowercasing) for sorting") unf = self._prompt_bool("UNF (un-normalized form)", allow_blank=allow_blank) if unf is not None: @@ -629,8 +636,10 @@ def _prompt_numeric_attrs( self, attrs: Dict[str, Any], allow_blank: bool, sortable: Optional[bool] ) -> None: """Prompt for numeric field specific attributes.""" - # UNF (only if sortable) - if sortable or attrs.get("sortable") or self._existing_sortable: + # UNF (only if sortable – skip if sortable was explicitly set to False) + if sortable or ( + sortable is not False and (attrs.get("sortable") or self._existing_sortable) + ): print(" UNF: preserve exact numeric representation for sorting") unf = self._prompt_bool("UNF (un-normalized form)", allow_blank=allow_blank) if unf is not None: diff --git a/tests/benchmarks/index_migrator_real_benchmark.py b/tests/benchmarks/index_migrator_real_benchmark.py index ad5e0b0de..c2a28bd1a 100644 --- a/tests/benchmarks/index_migrator_real_benchmark.py +++ b/tests/benchmarks/index_migrator_real_benchmark.py @@ -219,7 +219,7 @@ def wait_for_index_ready( latest_info = index.info() while time.perf_counter() < deadline: latest_info = index.info() - percent_indexed = float(latest_info.get("percent_indexed", 1) or 1) + percent_indexed = float(latest_info.get("percent_indexed", 1)) indexing = latest_info.get("indexing", 0) if percent_indexed >= 1.0 and not indexing: return latest_info diff --git a/tests/benchmarks/retrieval_benchmark.py b/tests/benchmarks/retrieval_benchmark.py index 6406dd9ad..584a098cf 100644 --- a/tests/benchmarks/retrieval_benchmark.py +++ b/tests/benchmarks/retrieval_benchmark.py @@ -233,7 +233,7 @@ def wait_for_index_ready( info = index.info() while time.perf_counter() < deadline: info = index.info() - pct = float(info.get("percent_indexed", 1) or 1) + pct = float(info.get("percent_indexed", 1)) indexing = info.get("indexing", 0) if pct >= 1.0 and not indexing: return info From e40e28d03cc1a1fe2c7bb4ece8e2a7a494eeb9e9 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Thu, 2 Apr 2026 17:40:47 -0400 Subject: [PATCH 4/8] fix: additional review fixes (docs, async parity, test data) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fix --query-check → --query-check-file in cli.rst - Fix --target-schema mutually-exclusive ref to --schema-patch in cli.rst - Fix async validation functional check to match sync (>0 not ==) - Fix async quantize pipeline to use transaction=False - Fix test checkpoint status 'succeeded' → 'success' --- docs/api/cli.rst | 4 ++-- redisvl/migration/async_executor.py | 2 +- redisvl/migration/async_validation.py | 7 ++++++- tests/integration/test_batch_migration_integration.py | 2 +- 4 files changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/api/cli.rst b/docs/api/cli.rst index ffd8659f1..d13ff5c9b 100644 --- a/docs/api/cli.rst +++ b/docs/api/cli.rst @@ -509,7 +509,7 @@ Generate a migration plan for a document-preserving drop/recreate migration. * - ``--schema-patch`` - Path to a YAML schema patch file (mutually exclusive with ``--target-schema``) * - ``--target-schema`` - - Path to a full target schema YAML file (mutually exclusive with ``--patch``) + - Path to a full target schema YAML file (mutually exclusive with ``--schema-patch``) **Optional Options** @@ -560,7 +560,7 @@ Execute a reviewed drop/recreate migration plan. Use ``--async`` for large migra - Description * - ``--async`` - Run migration asynchronously (recommended for large quantization jobs) - * - ``--query-check`` + * - ``--query-check-file`` - Path to a YAML file with post-migration query checks * - ``--resume`` - Path to a checkpoint file for crash-safe recovery diff --git a/redisvl/migration/async_executor.py b/redisvl/migration/async_executor.py index 40843a35e..8de6b9401 100644 --- a/redisvl/migration/async_executor.py +++ b/redisvl/migration/async_executor.py @@ -986,7 +986,7 @@ async def _async_quantize_vectors( for i in range(0, remaining_keys, batch_size): batch = keys[i : i + batch_size] - pipe = client.pipeline() + pipe = client.pipeline(transaction=False) undo = BatchUndoBuffer() keys_updated_in_batch: set[str] = set() diff --git a/redisvl/migration/async_validation.py b/redisvl/migration/async_validation.py index bd0dcf876..0f0ea66ae 100644 --- a/redisvl/migration/async_validation.py +++ b/redisvl/migration/async_validation.py @@ -167,7 +167,12 @@ async def _run_functional_checks( try: search_result = await target_index.search(Query("*").paging(0, 1)) total_found = search_result.total - passed = total_found == expected_doc_count + # When expected_doc_count is 0 (empty index), a successful + # search returning 0 docs is correct behaviour, not a failure. + if expected_doc_count == 0: + passed = total_found == 0 + else: + passed = total_found > 0 results.append( QueryCheckResult( name="functional:wildcard_search", diff --git a/tests/integration/test_batch_migration_integration.py b/tests/integration/test_batch_migration_integration.py index 976db0528..92ea7b94d 100644 --- a/tests/integration/test_batch_migration_integration.py +++ b/tests/integration/test_batch_migration_integration.py @@ -381,7 +381,7 @@ def test_resume_from_checkpoint(self, redis_url, worker_id, tmp_path): "completed": [ { "name": index_names[0], - "status": "succeeded", + "status": "success", "completed_at": "2026-03-20T10:00:30Z", } ], From 3902820b2f9b23f963736c218f6703e08c2e8ae4 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Thu, 2 Apr 2026 17:46:11 -0400 Subject: [PATCH 5/8] fix: handle empty plan_path in batch-resume safety gate --- redisvl/cli/migrate.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/redisvl/cli/migrate.py b/redisvl/cli/migrate.py index c1c76331b..95598eb04 100644 --- a/redisvl/cli/migrate.py +++ b/redisvl/cli/migrate.py @@ -653,7 +653,7 @@ def batch_resume(self): # Load the batch plan to check for quantization safety gate executor = BatchMigrationExecutor() state = executor._load_state(args.state) - plan_path = args.plan or state.plan_path + plan_path = args.plan or state.plan_path or None if plan_path: batch_plan = executor._load_batch_plan(plan_path) if batch_plan.requires_quantization and not args.accept_data_loss: From 55406a983db813934313fdd313a25a99777831d1 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Thu, 2 Apr 2026 18:16:07 -0400 Subject: [PATCH 6/8] fix: validation details message, doc status values, and geo coordinate order - Update wildcard_search details to say 'expected >0, source had N' instead of misleading 'expected N' - Change doc examples from 'succeeded' to 'success' matching BatchIndexState values - Add --accept-data-loss note for batch-resume in migration guide - Fix geo coordinate order (lon,lat) in test sample data --- CLAUDE.md | 3 + .../how_to_guides/migrate-indexes.md | 10 +- nitin_docs/index_migrator/00_index.md | 76 ++ nitin_docs/index_migrator/01_context.md | 103 +++ nitin_docs/index_migrator/02_architecture.md | 234 ++++++ nitin_docs/index_migrator/03_benchmarking.md | 248 +++++++ .../04_implementation_summary.md | 138 ++++ .../05_migration_benchmark_report.md | 200 +++++ .../10_v1_drop_recreate_spec.md | 326 ++++++++ .../11_v1_drop_recreate_tasks.md | 162 ++++ .../12_v1_drop_recreate_tests.md | 133 ++++ .../20_v2_iterative_shadow_spec.md | 213 ++++++ .../21_v2_iterative_shadow_tasks.md | 159 ++++ .../22_v2_iterative_shadow_tests.md | 102 +++ nitin_docs/index_migrator/90_prd.md | 599 +++++++++++++++ nitin_docs/index_migrator/99_tickets.md | 374 ++++++++++ nitin_docs/index_migrator/pr_comments.md | 700 ++++++++++++++++++ redisvl/migration/async_validation.py | 6 +- redisvl/migration/validation.py | 6 +- .../test_migration_comprehensive.py | 6 +- 20 files changed, 3789 insertions(+), 9 deletions(-) create mode 100644 nitin_docs/index_migrator/00_index.md create mode 100644 nitin_docs/index_migrator/01_context.md create mode 100644 nitin_docs/index_migrator/02_architecture.md create mode 100644 nitin_docs/index_migrator/03_benchmarking.md create mode 100644 nitin_docs/index_migrator/04_implementation_summary.md create mode 100644 nitin_docs/index_migrator/05_migration_benchmark_report.md create mode 100644 nitin_docs/index_migrator/10_v1_drop_recreate_spec.md create mode 100644 nitin_docs/index_migrator/11_v1_drop_recreate_tasks.md create mode 100644 nitin_docs/index_migrator/12_v1_drop_recreate_tests.md create mode 100644 nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md create mode 100644 nitin_docs/index_migrator/21_v2_iterative_shadow_tasks.md create mode 100644 nitin_docs/index_migrator/22_v2_iterative_shadow_tests.md create mode 100644 nitin_docs/index_migrator/90_prd.md create mode 100644 nitin_docs/index_migrator/99_tickets.md create mode 100644 nitin_docs/index_migrator/pr_comments.md diff --git a/CLAUDE.md b/CLAUDE.md index 09ab66439..4af36381f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -48,6 +48,9 @@ index = SearchIndex(schema, redis_url="redis://localhost:6379") token.strip().strip(",").replace(""", "").replace(""", "").lower() ``` +### Protected Directories +**CRITICAL**: NEVER delete the `nitin_docs/` directory or any files within it. + ### Git Operations **CRITICAL**: NEVER use `git push` or attempt to push to remote repositories. The user will handle all git push operations. diff --git a/docs/user_guide/how_to_guides/migrate-indexes.md b/docs/user_guide/how_to_guides/migrate-indexes.md index ec1d84e53..5795f6a0b 100644 --- a/docs/user_guide/how_to_guides/migrate-indexes.md +++ b/docs/user_guide/how_to_guides/migrate-indexes.md @@ -829,6 +829,8 @@ rvl migrate batch-resume \ --url redis://localhost:6379 ``` +**Note:** If the batch plan involves quantization (e.g., `float32` → `float16`), you must pass `--accept-data-loss` to `batch-resume`, just as with `batch-apply`. + ### Checking Batch Status ```bash @@ -844,7 +846,7 @@ Started: 2026-03-20T10:00:00Z Updated: 2026-03-20T10:25:00Z Completed: 2 - - products_idx: succeeded (10:02:30) + - products_idx: success (10:02:30) - users_idx: failed - Redis connection timeout (10:05:45) In Progress: inventory_idx @@ -867,16 +869,16 @@ summary: total_duration_seconds: 127.5 indexes: - name: products_idx - status: succeeded + status: success duration_seconds: 45.2 docs_migrated: 15000 report_path: ./reports/products_idx_report.yaml - name: users_idx - status: succeeded + status: success duration_seconds: 38.1 docs_migrated: 8500 - name: orders_idx - status: succeeded + status: success duration_seconds: 44.2 docs_migrated: 22000 completed_at: "2026-03-20T10:02:07Z" diff --git a/nitin_docs/index_migrator/00_index.md b/nitin_docs/index_migrator/00_index.md new file mode 100644 index 000000000..4512315d1 --- /dev/null +++ b/nitin_docs/index_migrator/00_index.md @@ -0,0 +1,76 @@ +# Index Migrator Workspace + +## Overview + +This directory contains the planning, design, and tracking documents for the RedisVL index migration feature. + +Phase 1 (`drop_recreate`) has been implemented and shipped across a 6-PR stack. The implementation went beyond the original MVP spec to include vector quantization, field/prefix/index renames, async execution, batch operations, crash-safe reliability, and disk space estimation. + +Phase 2 (`iterative_shadow`) remains planned and has not been started. + +This workspace is preserved as a historical planning record and as the foundation for Phase 2 design. + +## Guiding Principles + +- Prefer simple and safe over clever orchestration. +- Reuse existing RedisVL primitives before adding new abstractions. +- Migrate one index at a time (batch mode migrates sequentially). +- Keep cutover and platform scaling operator-owned. +- Fail closed on unsupported schema changes. + +## Phase Status + +| Phase | Mode | Status | Notes | +| --- | --- | --- | --- | +| Phase 1 | `drop_recreate` | **Done** | Shipped as PRs #567-#572 | +| Phase 1+ | Extensions (async, batch, reliability) | **Done** | Shipped alongside Phase 1 | +| Phase 2 | `iterative_shadow` | Planned | Not started | + +## Doc Map + +### Planning (pre-implementation) +- [01_context.md](./01_context.md): customer problem, constraints, and why the work is phased +- [02_architecture.md](./02_architecture.md): shared architecture, responsibilities, capacity model, and diagrams +- [03_benchmarking.md](./03_benchmarking.md): migration benchmarking goals, metrics, scenarios, and output artifacts +- [90_prd.md](./90_prd.md): final product requirements document for team review + +### Phase 1 (implemented) +- [04_implementation_summary.md](./04_implementation_summary.md): what was actually built, actual modules, actual CLI surface +- [05_migration_benchmark_report.md](./05_migration_benchmark_report.md): benchmark results (1K/10K/100K docs) +- [10_v1_drop_recreate_spec.md](./10_v1_drop_recreate_spec.md): original MVP spec (updated with implementation notes) +- [11_v1_drop_recreate_tasks.md](./11_v1_drop_recreate_tasks.md): task list (all completed) +- [12_v1_drop_recreate_tests.md](./12_v1_drop_recreate_tests.md): test plan + +### Phase 2 (planned, not started) +- [20_v2_iterative_shadow_spec.md](./20_v2_iterative_shadow_spec.md): future iterative shadow spec +- [21_v2_iterative_shadow_tasks.md](./21_v2_iterative_shadow_tasks.md): future iterative shadow tasks +- [22_v2_iterative_shadow_tests.md](./22_v2_iterative_shadow_tests.md): future iterative shadow test plan + +### Tracking +- [99_tickets.md](./99_tickets.md): all IM tickets with statuses +- [pr_comments.md](./pr_comments.md): collected PR review feedback + +## Current Truth + +Phase 1 is complete. The implementation source of truth is the code in `redisvl/migration/`. + +For Phase 2 planning: +- Spec: [20_v2_iterative_shadow_spec.md](./20_v2_iterative_shadow_spec.md) +- Tasks: [21_v2_iterative_shadow_tasks.md](./21_v2_iterative_shadow_tasks.md) +- Tests: [22_v2_iterative_shadow_tests.md](./22_v2_iterative_shadow_tests.md) + +## Next Actions + +- Phase 1 implementation is complete. No remaining Phase 1 tasks. +- Phase 2 design review should begin once Phase 1 learnings are documented. +- See [99_tickets.md](./99_tickets.md) for backlog items (IM-B1, IM-11 through IM-20). + +## Locked Decisions + +- The planning workspace lives entirely under `nitin_docs/index_migrator/`. +- The default artifact format for plans and reports is YAML. +- Benchmarking is built into migration reporting, not a separate subsystem. +- The default execution unit is a single index (batch mode runs indexes sequentially). +- The default operational model is operator-owned downtime, cutover, and scaling. +- Phase 2 owns shadow migrations for incompatible changes that require running old and new indexes in parallel. +- Vector quantization, field renames, prefix changes, and index renames were added to Phase 1 scope during implementation (originally planned for Phase 2). diff --git a/nitin_docs/index_migrator/01_context.md b/nitin_docs/index_migrator/01_context.md new file mode 100644 index 000000000..7e2e6af63 --- /dev/null +++ b/nitin_docs/index_migrator/01_context.md @@ -0,0 +1,103 @@ +# Index Migrator Context + +> **Status**: Phase 1 is complete and shipped (PRs #567-#572). This document is preserved as planning history. + +## Problem Statement + +RedisVL did not provide a first-class migration workflow for search index changes prior to this work. + +Teams could create indexes, delete indexes, inspect index info, and load documents, but they needed ad hoc scripts and operational runbooks to handle schema evolution. This was risky when the index was large, shared by multiple applications, or deployed on clustered Redis Cloud or Redis Software. + +The migration problem has three different shapes: + +- A simpler index rebuild that preserves existing documents and recreates the index definition in place. +- A shadow migration over the same documents when the target schema can still be built from the current stored payload. +- A shadow migration with transform or backfill when vector dimensions, datatypes, precision, algorithms, or payload shape change and a new target payload must be built. + +This workspace deliberately splits those shapes into phases. Phase 1 proved the plan-first migration workflow. Phase 2 exists to take on shadow migrations safely. + +## Customer Requirements + +The planning baseline for this work is: + +- preserve existing documents during migration +- capture the previous index configuration before making changes +- apply only the requested schema changes +- preview the migration plan before execution +- support advanced vector migrations such as `HNSW -> FLAT`, `FP32 -> FP16`, vector dimension changes, and payload-shape-changing model or algorithm swaps +- estimate migration timing, memory impact, and operational impact using simple benchmark artifacts +- benchmark source-versus-target memory and size changes, including peak overlap footprint during shadow migrations +- support both guided and scripted workflows +- make downtime and disruption explicit +- support large datasets without defaulting to full-keyspace audits or fleet-wide orchestration +- keep the implementation understandable enough that another team can operate it safely + +## Current RedisVL Capabilities + +RedisVL already has useful primitives that should be reused instead of replaced: + +- `SearchIndex.from_existing()` can reconstruct schema from a live index. +- `SearchIndex.delete(drop=False)` can remove the index structure without deleting documents. +- `SearchIndex.info()` can retrieve index stats used for planning and validation. +- Existing CLI commands already establish the connection and index lookup patterns the migrator can follow. + +Phase 1 added the following (originally listed as missing): + +- a migration planner (`MigrationPlanner`, `AsyncMigrationPlanner`, `BatchMigrationPlanner`) +- a schema diff classifier (in `planner.py`) +- a migration-specific CLI workflow (`rvl migrate` with 11 subcommands) +- a guided schema migration wizard (`MigrationWizard`) +- structured migration reports (`MigrationReport`, `MigrationValidation`, `MigrationBenchmarkSummary`) +- batch orchestration across indexes (`BatchMigrationExecutor`) +- vector quantization (e.g., FP32 -> FP16) with crash-safe reliability + +Still not built (Phase 2 or future): + +- capacity-aware orchestration with platform inventory +- transform or backfill planning for migrations that need new stored payloads + +## Why Phase 1 Came First + +Phase 1 was intentionally narrow because it gave the team an MVP that was both useful and low-risk: + +- It preserves documents while changing only the index definition. +- It reuses current RedisVL primitives instead of introducing a separate migration runtime. +- It keeps operational ownership clear: RedisVL handles planning, execution, and validation for a single index, while the operator handles the migration window and downstream application expectations. +- It avoids the hardest problems for now: target-payload generation, shadow overlap estimation, cutover automation, and cluster-wide scheduling. + +Phase 1 did not define the full migration goal. The harder shadow migrations are the reason Phase 2 exists. + +The MVP proved the planning model, CLI shape, plan artifact, and validation/reporting flow. Notably, vector quantization (originally scoped for Phase 2) was pulled forward into Phase 1 during implementation because it could be done safely as an in-place rewrite without shadow indexes. + +## Downtime and Disruption + +Phase 1 accepts downtime for the migrated index. + +Engineers need to plan for the following impacts: + +- Search on the target index is unavailable between index drop and recreated index readiness. +- Query results can be partial or unstable while the recreated index is still completing its initial indexing pass. +- Reindexing uses shared database resources and can increase CPU, memory, and indexing pressure on the deployment. +- Shadow migrations can temporarily duplicate index structures and sometimes duplicate payloads as well, increasing peak memory requirements. +- Downstream applications need either a maintenance window, a degraded mode, or a clear operational pause during the rebuild. + +The tooling does not hide these facts. The plan artifact and CLI output force the user to review the plan before applying a `drop_recreate` migration. (The original `--allow-downtime` flag was removed in favor of explicit plan review with `--accept-data-loss` only required for quantization.) + +## Non-Goals + +The following remain out of scope (not for the overall initiative, just for Phase 1): + +- a generic migration framework for every schema evolution case +- automatic platform scaling +- automatic traffic cutover +- full key manifest capture by default +- document transforms or backfills that require new embeddings +- payload relocation to a new keyspace (shadow migrations) +- fully managed Redis Cloud or Redis Software integration +- automatic transform inference or automatic re-embedding + +Note: Some items originally in this list were implemented during Phase 1: +- ~~concurrent migration of multiple large indexes~~ - batch mode was added (`rvl migrate batch-plan/batch-apply`) +- ~~field renames~~ - implemented via `rename_fields` in schema patch +- ~~prefix changes~~ - implemented via `index.prefix` in schema patch +- ~~vector datatype changes~~ - implemented as in-place quantization diff --git a/nitin_docs/index_migrator/02_architecture.md b/nitin_docs/index_migrator/02_architecture.md new file mode 100644 index 000000000..4056f8354 --- /dev/null +++ b/nitin_docs/index_migrator/02_architecture.md @@ -0,0 +1,234 @@ +# Index Migrator Architecture + +> **Status**: Phase 1 is implemented and shipped. This document describes the original architecture design. See [04_implementation_summary.md](./04_implementation_summary.md) for the actual module map. + +## System Boundaries + +The migration system has three boundaries: + +- RedisVL migration workflow: snapshot, diff, plan, apply, validate, report +- Redis deployment: Redis Cloud or Redis Software database that stores documents and indexes +- Operator and application boundary: maintenance window, scaling decisions, transform inputs, and application-level cutover behavior + +The implementation adds migration capabilities on top of existing RedisVL primitives instead of creating a separate control plane. + +```mermaid +flowchart LR + OP["Engineer / Operator"] --> CLI["RedisVL migrate CLI"] + APP["Applications"] --> DB["Redis database endpoint"] + + CLI --> SNAP["Snapshot + diff + mode selection"] + CLI --> EXEC["Executor"] + CLI --> VAL["Validator + report"] + + SNAP --> IDX["Live index metadata and stats"] + SNAP --> INV["Optional platform inventory"] + SNAP --> XFORM["Optional transform or backfill plan"] + + EXEC --> DB + VAL --> DB + EXEC --> TARGET["Optional target keyspace + shadow index"] + TARGET --> DB + + DB --> SHARDS["One logical index may span multiple shards"] + OP --> INV + OP --> XFORM +``` + +## Responsibilities + +RedisVL should own: + +- schema snapshot and source configuration capture +- schema diff classification +- migration mode selection +- migration plan generation +- guided wizard and scripted CLI entrypoints +- supported strategy execution +- validation and report generation + +The operator should own: + +- choosing the migration window +- accepting downtime or degraded behavior +- providing platform inventory when capacity planning matters +- providing transform or backfill inputs when payload shape changes +- scaling the Redis deployment +- application cutover and rollback decisions + +The platform should be treated as an external dependency, not as part of the MVP runtime. + +## Platform Model + +The migrator should reason about the deployment at the database level. + +For planning purposes: + +- treat the database endpoint as the unit of execution +- treat a search index as one logical index even if the deployment is sharded +- do not build logic that assumes an entire index lives on a single shard +- record where data lives in terms of database, prefixes, key separators, and target keyspace plans, not physical shard pinning + +This keeps the model compatible with both Redis Cloud and Redis Software without requiring the MVP to integrate directly with their platform APIs. + +## Migration Modes + +### `drop_recreate` + +This is the Phase 1 MVP. + +- Snapshot the current schema and index stats. +- Merge only the requested schema changes. +- Drop only the index structure, preserving documents. +- Recreate the index with the merged schema. +- Wait until indexing is complete. +- Validate and report. + +This mode is explicit about downtime and does not attempt to preserve uninterrupted query availability. + +### `iterative_shadow` + +This is the planned Phase 2 mode. + +- Work on one index at a time. +- Check database-level capacity before creating any shadow index. +- Choose between: + - `shadow_reindex` when the target schema can be built from the current stored payload. + - `shadow_rewrite` when vector datatype, precision, dimension, algorithm, or payload shape changes require a new target payload or keyspace. +- Create a shadow target for the current index only. +- Transform or backfill into a target keyspace when the migration changes payload shape. +- Validate the shadow target. +- Hand cutover to the operator. +- Retire the old index, and optionally the old target payload, only after cutover confirmation. + +This mode aims to reduce disruption without introducing automatic cutover or automatic scaling. This is the mode that should ultimately support migrations such as `HNSW -> FLAT`, `FP32 -> FP16`, vector dimension changes, and embedding-model-driven payload rewrites. + +## Capacity Model + +Phase 1 keeps capacity handling simple: + +- use source index stats for warnings and reports +- show expected downtime and indexing pressure +- do not block on a complex capacity estimator + +Phase 2 introduces a conservative capacity gate: + +- planner input is database-level, not shard-local +- one index at a time is the only supported execution unit +- estimate both source and target footprint +- separate document footprint from index footprint +- calculate peak overlap as the source footprint plus the target footprint that exists during migration +- capture memory savings or growth caused by algorithm, datatype, precision, dimension, and payload-shape changes +- the planner blocks if available headroom is below the estimated peak overlap plus reserve +- scaling stays operator-owned + +Default key-location capture is intentionally bounded: + +- store index name +- store storage type +- store prefixes +- store key separator +- store a bounded key sample + +Full key manifests are not part of the default path. + +## Benchmarking Model + +Benchmarking should be built into migration reporting, not treated as a separate system. + +The shared model is: + +- capture baseline metadata before migration +- capture timing and progress during migration +- capture validation and query-impact signals after migration +- persist simple YAML benchmark artifacts that can be compared across runs + +Benchmarking should focus on the operator questions that matter most: + +- total migration duration +- downtime or overlap duration +- document throughput +- query latency change during the migration window +- resource impact before, during, and after migration +- source-versus-target memory and size delta +- estimated versus actual peak overlap footprint + +The benchmark requirements are defined in [03_benchmarking.md](./03_benchmarking.md). + +## Failure Model + +The system should fail closed. + +- Unsupported schema diffs stop at `plan`. +- Missing transform inputs for a payload-shape-changing migration stop at `plan`. +- Missing source metadata stops at `plan`. +- `apply` never deletes documents in Phase 1. +- Validation failures produce a report and manual next steps. +- The tool does not attempt automatic rollback or automatic traffic switching. + +## `drop_recreate` Sequence + +```mermaid +sequenceDiagram + participant Op as Operator + participant CLI as RedisVL migrate CLI + participant Snap as Snapshot + diff + participant DB as Redis + participant Val as Validator + + Op->>CLI: migrate plan or wizard + CLI->>Snap: capture source schema and stats + Snap->>DB: read live index metadata + Snap-->>CLI: normalized source snapshot + CLI-->>Op: migration_plan.yaml with downtime warning + + Op->>CLI: migrate apply --plan migration_plan.yaml + CLI->>DB: quantize vectors (if needed) + CLI->>DB: rename fields/keys (if needed) + CLI->>DB: drop index only + CLI->>DB: recreate merged index + loop until indexing complete + CLI->>DB: poll index info + DB-->>CLI: indexing status and counts + end + CLI->>Val: run validation checks + Val->>DB: verify schema, counts, samples + Val-->>CLI: migration_report.yaml + CLI-->>Op: success or manual review +``` + +## `iterative_shadow` Sequence + +```mermaid +sequenceDiagram + participant Op as Operator + participant CLI as RedisVL migrate CLI + participant Plan as Shadow planner + participant DB as Redis + participant App as Application + + Op->>CLI: migrate plan --mode iterative_shadow + CLI->>Plan: capture source stats, target schema, transform plan, and platform inventory + Plan-->>CLI: READY, SCALE_REQUIRED, or MANUAL_REVIEW_REQUIRED + CLI-->>Op: plan and operator actions + + Op->>CLI: migrate apply --mode iterative_shadow + alt payload-compatible shadow + CLI->>DB: create shadow index for one source index + else payload rewrite required + CLI->>DB: create target keyspace and shadow index + loop backfill target payload + CLI->>DB: read source documents + CLI->>DB: write transformed target documents + end + end + loop until shadow ready + CLI->>DB: poll shadow index info + DB-->>CLI: readiness status + end + CLI->>DB: validate shadow index + CLI-->>Op: cutover runbook + Op->>App: switch to target index + Op->>CLI: confirm cutover + CLI->>DB: retire old index +``` diff --git a/nitin_docs/index_migrator/03_benchmarking.md b/nitin_docs/index_migrator/03_benchmarking.md new file mode 100644 index 000000000..67a637f18 --- /dev/null +++ b/nitin_docs/index_migrator/03_benchmarking.md @@ -0,0 +1,248 @@ +# Migration Benchmarking + +## Goals + +Migration benchmarking exists to answer practical operator questions: + +- how long will the migration take +- how long will search be degraded or unavailable +- how much shared Redis capacity will the migration consume +- how much the target schema or vector shape will increase or reduce memory usage +- how much query performance changes during the migration window +- whether future migrations can be estimated from previous runs + +The first benchmarking design should stay simple. It should collect structured measurements from real runs and manual rehearsals rather than introducing a separate performance framework before the migrator exists. + +## Core Benchmark Questions + +Every migration benchmark should answer: + +1. How long did planning take? +2. How long did `apply` take end-to-end? +3. How long was the index unavailable or in degraded indexing state? +4. What document throughput did the migration achieve? +5. What query latency and error-rate changes occurred during the migration? +6. How much memory, flash, or disk footprint changed before, during, and after migration? +7. How accurate was the peak-overlap estimate? +8. Did the final migrated index match the expected schema and document count? + +## Metrics + +### Timing Metrics + +- `plan_duration_seconds` +- `apply_duration_seconds` +- `validation_duration_seconds` +- `total_migration_duration_seconds` +- `drop_duration_seconds` +- `recreate_duration_seconds` +- `initial_indexing_duration_seconds` +- `downtime_duration_seconds` for `drop_recreate` +- `shadow_overlap_duration_seconds` for `iterative_shadow` +- `transform_duration_seconds` for payload rewrite work +- `backfill_duration_seconds` for target payload creation + +### Throughput Metrics + +- `source_num_docs` +- `target_num_docs` +- `documents_indexed_per_second` +- `documents_transformed_per_second` +- `bytes_rewritten_per_second` +- `progress_samples` captured during readiness polling + +### Query Impact Metrics + +- baseline query latency: `p50`, `p95`, `p99` +- during-migration query latency: `p50`, `p95`, `p99` +- post-migration query latency: `p50`, `p95`, `p99` +- query error rate during migration +- query result overlap or sample correctness checks + +### Resource Impact Metrics + +- source document footprint from live stats or sampling +- source index size from live stats +- target document footprint from live stats or sampling +- target index size from live stats +- total source footprint +- total target footprint +- footprint delta after migration +- estimated peak overlap footprint +- actual peak overlap footprint +- indexing failure delta +- memory headroom before migration +- memory headroom after migration +- peak memory headroom during overlap +- flash or disk footprint before and after when relevant +- source vector dimensions, datatype, precision, and algorithm +- target vector dimensions, datatype, precision, and algorithm +- source vector bytes per document +- target vector bytes per document + +### Correctness Metrics + +- schema match +- document count match +- indexing failure delta equals zero +- representative document fetch checks pass + +## Benchmark Inputs + +Each benchmark run should record the workload context, not just the raw timings. + +Required context: + +- migration mode +- dataset size +- storage type +- field mix +- whether vectors are present +- source and target vector configuration when vectors are present +- whether payload shape changes +- shard count +- replica count +- query load level during migration +- environment label such as `local`, `staging`, `redis_cloud`, or `redis_software` + +Useful optional context: + +- vector dimensions and datatype +- vector precision and algorithm +- auto-tiering enabled or disabled +- representative document size +- maintenance window target + +## Benchmark Scenarios + +Start with a small scenario matrix and expand only when needed. + +Minimum Phase 1 benchmark scenarios: + +- small index, low query load +- medium or large index, low query load +- medium or large index, representative read load + +Minimum Phase 2 benchmark scenarios: + +- one shadow migration on a sharded deployment with sufficient capacity +- one shadow migration that is blocked by the capacity gate +- one shadow migration under representative read load +- one algorithm migration such as `HNSW -> FLAT` +- one vector storage migration such as `FP32 -> FP16` +- one payload-shape-changing migration that requires transform or backfill + +Scale variables should be changed one at a time where possible: + +- document count +- shard count +- query concurrency +- storage type +- vector footprint + +## Benchmark Artifacts + +The benchmark workflow should use simple YAML artifacts. + +### `benchmark_manifest.yaml` + +```yaml +version: 1 +label: staging-large-read-load +mode: drop_recreate +environment: staging +dataset: + num_docs: 1000000 + storage_type: json + vector_fields_present: true +platform: + shard_count: 4 + replica_count: 1 +workload: + query_profile: representative-read + query_check_file: queries.yaml +notes: "" +``` + +### `benchmark_report.yaml` + +```yaml +version: 1 +label: staging-large-read-load +mode: drop_recreate +timings: + total_migration_duration_seconds: 540 + downtime_duration_seconds: 420 + validation_duration_seconds: 18 +throughput: + source_num_docs: 1000000 + documents_indexed_per_second: 2380.95 +query_impact: + baseline_p95_ms: 42 + during_migration_p95_ms: 95 + post_migration_p95_ms: 44 +resource_impact: + source_doc_footprint_mb: 6144 + source_index_size_mb: 8192 + target_doc_footprint_mb: 6144 + target_index_size_mb: 6144 + total_footprint_delta_mb: -2048 + estimated_peak_overlap_footprint_mb: 20480 + actual_peak_overlap_footprint_mb: 19840 + source_vector: + dimensions: 1536 + datatype: float32 + algorithm: hnsw + target_vector: + dimensions: 1536 + datatype: float16 + algorithm: flat +correctness: + schema_match: true + doc_count_match: true +``` + +These artifacts are planning and validation aids. They should not become a separate system before the migrator itself is implemented. + +## How Benchmarking Fits the Phases + +### Phase 1: `drop_recreate` + +Phase 1 should always record: + +- start time +- end time +- index downtime duration +- readiness wait duration +- source and target document counts +- source and target index stats +- observed source-versus-target index footprint delta + +Phase 1 should optionally record: + +- representative query latency before, during, and after migration +- query correctness checks using the same file as validation queries + +### Phase 2: `iterative_shadow` + +Phase 2 should always record: + +- source-to-shadow overlap duration +- planner estimate versus actual runtime +- capacity gate decision +- source and target document and index stats +- estimated versus actual peak overlap footprint +- observed memory savings or growth after the migration +- query impact during overlap + +Phase 2 should use benchmark history as advisory input for ETA and risk reporting, not as a hard execution dependency. + +## Exit Criteria + +Benchmarking is good enough for the first implementation when: + +- every migration report includes core timing and correctness metrics +- every shadow migration benchmark includes source-versus-target footprint deltas +- manual benchmark rehearsals can be run from a simple manifest +- the docs define what to collect before performance tuning begins +- benchmark requirements do not force a separate subsystem before the migrator ships diff --git a/nitin_docs/index_migrator/04_implementation_summary.md b/nitin_docs/index_migrator/04_implementation_summary.md new file mode 100644 index 000000000..9aa5f5110 --- /dev/null +++ b/nitin_docs/index_migrator/04_implementation_summary.md @@ -0,0 +1,138 @@ +# Phase 1 Implementation Summary + +## Overview + +Phase 1 of the RedisVL Index Migrator shipped across a 6-PR stack: + +| PR | Scope | +| --- | --- | +| #567 | Design: models, schema patch, planner | +| #568 | Core: executor, validation, reliability | +| #569 | Wizard: interactive migration builder | +| #570 | Async: AsyncMigrationExecutor, AsyncMigrationPlanner | +| #571 | Batch: BatchMigrationPlanner, BatchMigrationExecutor | +| #572 | Docs: user guide, CLI help | + +## Module Map + +``` +redisvl/migration/ + __init__.py Public API exports + models.py All Pydantic models (SchemaPatch, MigrationPlan, MigrationReport, + DiskSpaceEstimate, BatchPlan, BatchState, etc.) + planner.py MigrationPlanner - snapshot, patch merge, diff classification, + rename/quantization detection, plan generation + executor.py MigrationExecutor - sync drop/recreate with quantization, + field renames, key renames, readiness polling + validation.py MigrationValidator - schema match, doc count, key sample, + query checks + wizard.py MigrationWizard - interactive guided plan builder + reliability.py Crash-safe quantization: idempotent dtype detection, + checkpointing, BGSAVE safety, bounded undo buffering + utils.py Shared utilities: list_indexes, load/write YAML, + estimate_disk_space, detect_aof_enabled, timestamp_utc + async_planner.py AsyncMigrationPlanner - async version of planner + async_executor.py AsyncMigrationExecutor - async version of executor + async_validation.py AsyncMigrationValidator - async version of validator + batch_planner.py BatchMigrationPlanner - multi-index plan generation + batch_executor.py BatchMigrationExecutor - sequential multi-index execution + with checkpointing, resume, retry-failed + +redisvl/cli/ + migrate.py CLI entry point: 11 subcommands (see below) +``` + +## CLI Surface + +``` +rvl migrate + +Commands: + helper Show migration guidance and supported capabilities + list List all available indexes + wizard Interactively build a migration plan and schema patch + plan Generate a migration plan for a document-preserving drop/recreate migration + apply Execute a reviewed drop/recreate migration plan (use --async for large migrations) + estimate Estimate disk space required for a migration plan (dry-run, no mutations) + validate Validate a completed migration plan against the live index + batch-plan Generate a batch migration plan for multiple indexes + batch-apply Execute a batch migration plan with checkpointing + batch-resume Resume an interrupted batch migration + batch-status Show status of an in-progress or completed batch migration +``` + +## Key Features Beyond Original MVP Spec + +### Vector Quantization +In-place rewriting of vector data (e.g., FP32 -> FP16, FP32 -> INT8). Implemented with: +- Idempotent dtype detection (`detect_vector_dtype` in `reliability.py`) +- Crash-safe checkpointing to local YAML file +- BGSAVE safety checks +- `--accept-data-loss` flag for CLI acknowledgment +- Disk space estimation before migration + +### Rename Operations +- **Index rename**: Change the index name via `index.name` in schema patch +- **Prefix change**: Change key prefix via `index.prefix` in schema patch +- **Field renames**: Rename hash fields via `rename_fields` in schema patch + +### Async Execution +- `--async` flag on `rvl migrate apply` for large migrations +- Full async planner, executor, and validator classes + +### Batch Operations +- `batch-plan`: Generate plans for multiple indexes (by pattern, list, or file) +- `batch-apply`: Execute with per-index checkpointing and progress callbacks +- `batch-resume`: Resume interrupted batch with `--retry-failed` +- `batch-status`: Inspect checkpoint state +- Failure policies: `fail_fast` or `continue_on_error` + +### Disk Space Estimation +- Pre-migration estimate of RDB snapshot cost, AOF growth, and memory savings +- Per-vector-field breakdown with source/target dtype and byte calculations +- Available as `rvl migrate estimate` or automatically shown during `apply` + +## Pydantic Models (in `models.py`) + +| Model | Purpose | +| --- | --- | +| `SchemaPatch` / `SchemaPatchChanges` | Schema change request input | +| `FieldUpdate` / `FieldRename` | Individual field modifications | +| `SourceSnapshot` / `KeyspaceSnapshot` | Captured source state | +| `MigrationPlan` | Full plan artifact with diff classification | +| `RenameOperations` | Tracks index/prefix/field renames | +| `DiffClassification` | Supported/blocked with reasons | +| `ValidationPolicy` | What to check after migration | +| `MigrationReport` | Full execution report | +| `MigrationValidation` | Post-migration validation results | +| `MigrationTimings` | Duration breakdowns | +| `MigrationBenchmarkSummary` | Throughput and size metrics | +| `DiskSpaceEstimate` / `VectorFieldEstimate` | Pre-migration disk cost | +| `BatchPlan` / `BatchIndexPlan` | Multi-index plan | +| `BatchState` / `CompletedIndex` | Checkpoint state for batch | + +## Test Files + +| File | Type | +| --- | --- | +| `tests/unit/test_migration_planner.py` | Unit tests for planner, patch merge, diff classification | +| `tests/unit/test_batch_migration.py` | Unit tests for batch planner and executor | +| `tests/unit/test_migration_wizard.py` | Unit tests for wizard flow | +| `tests/integration/test_migration_comprehensive.py` | Integration tests with live Redis | + +Run all tests: +```bash +uv run python -m pytest tests/unit/test_migration_planner.py tests/unit/test_batch_migration.py tests/unit/test_migration_wizard.py tests/integration/test_migration_comprehensive.py +``` + +## Execution Flow (as implemented) + +1. **Plan**: Snapshot source -> merge patch -> classify diff -> detect renames/quantization -> emit `migration_plan.yaml` +2. **Apply**: Quantize vectors (if needed) -> rename fields (if needed) -> rename keys (if needed) -> drop index -> recreate index -> poll readiness -> validate -> emit report +3. **Validate**: Schema match + doc count + key sample + query checks -> emit report + +For quantization, the executor uses `reliability.py` for: +- Detecting current dtype of each key's vector (idempotent - skips already-processed) +- Checkpointing progress to disk for crash recovery +- BGSAVE coordination to avoid data loss + diff --git a/nitin_docs/index_migrator/05_migration_benchmark_report.md b/nitin_docs/index_migrator/05_migration_benchmark_report.md new file mode 100644 index 000000000..0e9636c95 --- /dev/null +++ b/nitin_docs/index_migrator/05_migration_benchmark_report.md @@ -0,0 +1,200 @@ +# Migration Benchmark Report + +## Overview + +This report presents the results of benchmarking the RedisVL index migration process at three scales (1K, 10K, 100K documents). The migration converts an HNSW FP32 index to a FLAT FP16 index. All runs use a realistic 16-field schema modeled after a production knowledge management index with 3072-dimensional COSINE vectors. + +The benchmark isolates **load time** (populating Redis with synthetic data) from **migrate time** (the actual migration operation). It also confirms which enumeration method the migrator used to discover document keys. + +## Environment + +All trials ran on a single local machine against a Docker-hosted Redis Stack instance. Each scale was tested 3 times. Results shown below are averages across the 3 trials. + +## Results Summary + +| Scale | Load Time | Migrate Time | Quantize | Reindex | Downtime | Vec Memory Before | Vec Memory After | Saved | +|---|---|---|---|---|---|---|---|---| +| 1K | 0.4s | 0.8s | 0.3s | 0.5s | 0.8s | 12.3 MB | 6.0 MB | 51% | +| 10K | 26.2s | 3.4s | 2.4s | 1.0s | 3.4s | 123.5 MB | 60.5 MB | 51% | +| 100K | 454s | 30.9s | 24.1s | 6.6s | 30.7s | 1,211 MB | 593 MB | 51% | + +All 9 trials used **FT.AGGREGATE** for key enumeration (never SCAN). All trials passed validation. + +## Enumeration Method + +The migrator discovers which keys belong to the index before starting the migration. It uses `FT.AGGREGATE ... WITHCURSOR` as the primary method, which returns only indexed keys without scanning the full keyspace. SCAN is a fallback reserved for cases where `hash_indexing_failures > 0` or `FT.AGGREGATE` errors out. In all 9 trials, enumeration used `FT.AGGREGATE` and completed in under 150ms even at 100K. + +## How Drop-Recreate Migration Works + +The migration executor follows this sequence: + +**STEP 1: Enumerate keys** (before any modifications) +- Uses FT.AGGREGATE WITHCURSOR to discover all document keys in the source index +- Fallback to SCAN if the index has hash_indexing_failures > 0 or FT.AGGREGATE fails +- Keys are stored in memory for the quantization step + +**STEP 2: Drop source index** +- Issues FT.DROPINDEX (without KEEPDOCS) to remove the index structure +- **The underlying documents remain in Redis** - only the index metadata is deleted +- At this point, the index is gone but all document hashes/JSON still exist with their FP32 vectors + +**STEP 3: Quantize vectors** (rewrite document payloads IN-PLACE) +- For each document in the enumerated key list: + - HGETALL to read the document (including FP32 vector) + - Convert FP32 → FP16 in Python + - HSET to write back the FP16 vector to the same document +- Processes documents in batches of 500 using Redis pipelines +- **Memory note**: The old index is already dropped, so there is no "double index" overhead. Only the document data exists in Redis during this phase. + +**STEP 4: Key renames** (if needed) +- If the migration changes the key prefix, RENAME each key from old prefix to new prefix +- Skipped if no prefix change + +**STEP 5: Create target index** +- Issues FT.CREATE with the new schema (FLAT, FP16, etc.) +- Redis begins background indexing of existing documents + +**STEP 6: Wait for re-indexing** +- Polls FT.INFO until indexing completes (num_docs == expected count) +- The index is unavailable for queries until this completes + +## Phase Breakdown + +Here is the average duration of each phase at 100K documents (the most representative scale). + +| Phase | Duration | Share of Migration | +|---|---|---| +| Enumerate | 0.12s | 0.4% | +| Drop | 0.00s | 0.0% | +| Quantize | 24.1s | 77.9% | +| Create | 0.002s | 0.0% | +| Reindex | 6.6s | 21.3% | +| Validate | 0.012s | 0.0% | + +Quantization dominates at every scale. This is the client-side step that reads each document's FP32 vector from Redis, converts it to FP16 in Python, and writes it back. It is inherently I/O-bound and proportional to document count. + +## Scaling Analysis + +The central question is whether migration time grows linearly with document count, which determines whether we can predict costs at 1M and 10M. + +### Per-Document Costs + +| Scale | Per-Doc Quantize | Per-Doc Reindex | +|---|---|---| +| 1K | 277 us | 511 us | +| 10K | 237 us | 102 us | +| 100K | 241 us | 66 us | + +**Quantize scales linearly.** The per-document cost stabilizes around 240 microseconds from 10K onward. This makes sense because each document requires one HGETALL and one HSET regardless of index size. There is no interaction between documents during quantization. + +**Reindex scales sub-linearly.** The per-document cost decreases as scale increases. This is expected for FLAT indexes where Redis performs a simple sequential scan to build the brute-force index. Fixed overhead (index creation, initial polling delay) is amortized over more documents. At 100K the reindex throughput reaches ~15K docs/sec. + +### Scaling Ratios (10x increments) + +| Metric | 1K to 10K (10x data) | 10K to 100K (10x data) | +|---|---|---| +| Quantize time | 8.5x | 10.2x | +| Reindex time | 2.0x | 6.5x | +| Total migrate | 4.3x | 9.1x | + +The 10K-to-100K ratio is the most reliable predictor since 1K has proportionally more fixed overhead. Quantize is essentially 10x for 10x data (linear). Reindex is growing faster than at small scale but still sub-linear. + +## Predictions for 1M and 10M + +Using the per-document rates observed at 100K (the most representative scale) and assuming linear scaling for quantize with a conservative linear assumption for reindex. + +### Per-Document Rates Used + +| Component | Rate | +|---|---| +| Quantize | 241 us/doc (from 100K average) | +| Reindex | 66 us/doc (from 100K, likely optimistic at larger scale) | + +### Projected Migration Times + +| Scale | Quantize | Reindex | Total Migrate | Downtime | +|---|---|---|---|---| +| **1M** | ~241s (~4 min) | ~66s (~1.1 min) | ~5.2 min | ~5.2 min | +| **10M** | ~2,410s (~40 min) | ~660s (~11 min) | ~51 min | ~51 min | + +### Caveats on These Predictions + +**1M is realistic.** The quantize step is pure per-document I/O with no cross-document dependencies, so linear extrapolation is well-founded. Reindex for a FLAT index at 1M should also remain close to linear. Memory requirement would be roughly 11.4 GB for FP32 vectors plus metadata, so a machine with 32 GB RAM should handle it. + +**10M carries significant risk factors.** + +1. **Memory requirement.** 10M documents at 3072 dimensions requires ~57 GB for FP32 vectors plus metadata overhead. During quantization, the source index has already been dropped, so there is no "double index" memory overhead. Each batch (500 docs) temporarily holds both FP32 and FP16 representations during HSET, but this is a small incremental cost. The main memory requirement is the baseline FP32 data (~57 GB), not 80+ GB. After quantization completes, memory drops to ~28.5 GB (FP16 vectors). A machine with 64-128 GB RAM should handle this comfortably. + +2. **Reindex may slow down.** FLAT index construction at 10M with 3072 dimensions means Redis must build a brute-force index over the FP16 vector data. Background indexing throughput may degrade at this scale, especially if Redis is under memory pressure or serving concurrent traffic. + +3. **Quantize could slow down.** At 10M, the pipeline batches (500 docs each) would execute 20,000 batch cycles. If Redis starts swapping or if network latency increases under load, per-batch cost could rise above the observed 241 us/doc average. + +4. **FLAT may not be the right target at 10M.** A 10M-document FLAT index would make every query a brute-force scan over 10M vectors, which is impractical for production. HNSW FP16 would be the appropriate target, and HNSW index construction is O(n log n) rather than O(n), which would increase the reindex phase significantly (potentially 2-3x longer). + +### Adjusted Predictions with Risk + +| Scale | Optimistic | Expected | Pessimistic | +|---|---|---|---| +| **1M** | 4.5 min | 5.5 min | 8 min | +| **10M (FLAT target)** | 50 min | 60 min | 90 min | +| **10M (HNSW target)** | 70 min | 90 min | 150+ min | + +The pessimistic 10M estimate accounts for HNSW rebuild cost (O(n log n) indexing) and potential per-batch slowdown at scale. A production 10M migration would require a machine with 64-128 GB RAM and should be empirically tested before deployment. The memory requirement is the baseline FP32 data size (~57 GB), not double that, because the source index is dropped before quantization begins. + +## Async vs Sync Executor Comparison + +A second set of 9 trials was run using `AsyncMigrationExecutor` instead of the sync `MigrationExecutor`. The async executor is what the CLI (`rvl migrate apply`) uses internally and was expected to show improved throughput through non-blocking I/O. + +### Async Results Summary + +| Scale | Migrate Time | Quantize | Reindex | Downtime | +|---|---|---|---|---| +| 1K | 0.8s | 0.30s | 0.51s | 0.81s | +| 10K | 4.3s | 3.21s | 1.01s | 4.23s | +| 100K | 35.8s | 29.6s | 6.06s | 35.7s | + +### Side-by-Side Comparison + +| Scale | Sync Migrate | Async Migrate | Sync Quantize | Async Quantize | Async Overhead | +|---|---|---|---|---|---| +| 1K | 0.8s | 0.8s | 0.3s | 0.3s | ~0% | +| 10K | 3.4s | 4.3s | 2.4s | 3.2s | +33% | +| 100K | 30.9s | 35.8s | 24.1s | 29.6s | +23% | + +### Why Async is Slower + +The async executor adds overhead without gaining parallelism for three reasons. + +**Single-connection I/O.** Both executors talk to Redis over a single TCP connection. The async event loop adds coroutine scheduling and context-switch overhead on every `await`, but cannot overlap commands because Redis processes them sequentially on one connection. + +**CPU-bound quantization.** The FP32 to FP16 conversion uses `struct.unpack` and `struct.pack` in Python. This is CPU-bound work that gets no benefit from `asyncio`. The event loop overhead adds roughly 50 microseconds per document (296 us/doc async vs 241 us/doc sync at 100K). + +**Identical batching strategy.** Both executors use the same `pipeline.execute()` pattern with batches of 500 documents. The async version does not overlap I/O across batches because each batch must complete before the next begins. + +### When Async Would Help + +The async executor exists for integration with async application code (the CLI, web frameworks, or other coroutine-based systems). It does not improve raw migration throughput. To actually speed up the quantize phase, the optimization path would be multi-connection parallelism (splitting the key list across N workers, each with its own Redis connection), not async/await on a single connection. + +### N-Worker Parallelism Considerations + +Multi-connection parallelism has production risks that should be weighed before enabling it. Redis is single-threaded for command processing, so N connections do not give N times server-side throughput. The client-side overlap of network round-trips provides the speedup, but the server processes commands sequentially from one queue. In production deployments with replicas, concurrent HSET writes from N workers increase replication backlog pressure. If the buffer fills, Redis disconnects the replica and triggers a full resync, which is catastrophic during migration. AOF persistence adds similar risk since N concurrent writers accelerate AOF buffer growth and could trigger an AOF rewrite that forks the process and temporarily doubles memory. Sharded deployments require shard-aware key partitioning to avoid hotspots, and Redis Cloud proxy layers add per-connection overhead that does not appear in local benchmarks. The safe default should remain N=1 with opt-in parallelism. See `nitin_docs/index_migrator/03_benchmarking.md` for the full risk analysis. + +## Key Takeaways + +1. **Migration is fast relative to data loading.** At 100K, loading took 7.5 minutes while migration took only 31 seconds. The migration operation itself is not the bottleneck in any deployment workflow. + +2. **Quantization dominates migration time at every scale** (~78% of total). Any optimization effort should focus on the quantize step, such as parallelizing the read-convert-write pipeline across multiple connections. + +3. **Sync executor is faster than async** for raw migration throughput. The async version adds ~23% overhead at scale due to event loop costs on CPU-bound work. + +4. **FT.AGGREGATE is the default enumeration path** and it works reliably. SCAN fallback exists but did not trigger in any trial. + +5. **Vector memory savings are exactly 51% at every scale.** FP16 cuts the vector index footprint in half with no variation. Non-vector index metadata is unchanged. + +6. **Linear extrapolation is valid up to 1M.** Beyond that, memory pressure and index algorithm choice (FLAT vs HNSW) introduce non-linear factors that require empirical validation. + +## Raw Data + +- Sync results: `tests/benchmarks/results_migration.json` +- Async results: `tests/benchmarks/results_migration_async.json` + diff --git a/nitin_docs/index_migrator/10_v1_drop_recreate_spec.md b/nitin_docs/index_migrator/10_v1_drop_recreate_spec.md new file mode 100644 index 000000000..255295f52 --- /dev/null +++ b/nitin_docs/index_migrator/10_v1_drop_recreate_spec.md @@ -0,0 +1,326 @@ +# Phase 1 Spec: `drop_recreate` + +> **Status**: Implemented and shipped (PRs #567-#572). This spec has been updated with implementation notes where the shipped code diverged from the original design. + +## Goal + +Build a simple RedisVL migration workflow that: + +- preserves existing documents +- captures the old index configuration before change +- applies only the user-requested schema changes +- generates a plan before any mutation +- supports both guided and scripted use +- explicitly surfaces downtime for the migrated index +- supports vector quantization (e.g., FP32 -> FP16) as an in-place rewrite *(added during implementation)* +- supports field renames, prefix changes, and index renames *(added during implementation)* +- supports async execution for large migrations *(added during implementation)* +- supports batch migration across multiple indexes *(added during implementation)* + +## Supported Changes + +The MVP supports schema changes that can be satisfied by rebuilding the index over the existing document set without rewriting or relocating stored documents. + +Supported categories: + +- add a new non-vector field that indexes data already present in stored documents +- remove an existing field from the index definition +- change index options on an existing non-vector field when the field name, field type, and storage path stay the same +- change index-level options that only affect index definition and do not relocate data + +Supported field types for MVP changes: + +- text +- tag +- numeric +- geo + +The MVP always recreates the same logical index name unless the user is only generating a plan. + +## Blocked Changes + +The following changes are classified as unsupported and stop before `apply`: + +- key separator changes +- storage type changes (hash <-> JSON) +- JSON path remodels +- vector dimension changes +- any change that requires a completely new stored payload shape + +> **Implementation note**: Several items originally blocked in this spec were implemented during Phase 1: +> - ~~key prefix changes~~ - now supported via `index.prefix` in schema patch +> - ~~field renames~~ - now supported via `rename_fields` in schema patch +> - ~~vector datatype changes~~ - now supported as in-place quantization (e.g., FP32 -> FP16) +> - ~~new index name~~ - now supported via `index.name` in schema patch +> +> These were feasible because they could be done as in-place document rewrites without shadow indexes. + +## Inputs + +The workflow accepts: + +- Redis connection parameters +- source index name +- one of: + - `schema_patch.yaml` + - `target_schema.yaml` + - interactive wizard answers + +Actual CLI surface (as shipped): + +```text +rvl migrate helper +rvl migrate list +rvl migrate wizard --index --plan-out +rvl migrate plan --index --schema-patch +rvl migrate plan --index --target-schema +rvl migrate apply --plan [--async] [--resume ] +rvl migrate estimate --plan +rvl migrate validate --plan +rvl migrate batch-plan --schema-patch (--pattern | --indexes ) +rvl migrate batch-apply --plan [--accept-data-loss] +rvl migrate batch-resume --state [--retry-failed] +rvl migrate batch-status --state +``` + +> **Implementation note**: The `--allow-downtime` flag was removed. Downtime is implicit in `drop_recreate` mode. The `--accept-data-loss` flag is used only for quantization (lossy operation). The `helper` and `list` subcommands were added for discoverability. The `estimate` subcommand provides pre-migration disk space estimates. + +Key optional flags: + +- `--plan-out` / `--report-out` / `--benchmark-out` +- `--key-sample-limit` +- `--query-check-file` +- `--async` (for large migrations with quantization) +- `--resume` (crash-safe checkpoint resume) +- `--accept-data-loss` (for quantization acknowledgment) + +### `schema_patch.yaml` + +This is the authoritative input model for requested changes. Unspecified source configuration is preserved by default. + +Example: + +```yaml +version: 1 +changes: + add_fields: + - name: category + type: tag + path: $.category + separator: "," + remove_fields: + - legacy_score + update_fields: + - name: title + options: + sortable: true +``` + +### `target_schema.yaml` + +This is a convenience input. The planner normalizes it into a schema patch by diffing it against the live source schema. + +## Outputs + +The workflow produces: + +- `migration_plan.yaml` +- `migration_report.yaml` +- optional `benchmark_report.yaml` +- console summaries for plan, apply, and validate + +### `migration_plan.yaml` + +Required fields: + +```yaml +version: 1 +mode: drop_recreate +source: + index_name: docs + schema_snapshot: {} + stats_snapshot: {} + keyspace: + storage_type: json + prefixes: ["docs"] + key_separator: ":" + key_sample: ["docs:1", "docs:2"] +requested_changes: {} +merged_target_schema: {} +diff_classification: + supported: true + blocked_reasons: [] +warnings: + - index downtime is required +validation: + require_doc_count_match: true + require_schema_match: true +``` + +### `migration_report.yaml` + +Required fields: + +```yaml +version: 1 +mode: drop_recreate +source_index: docs +result: succeeded +started_at: 2026-03-17T00:00:00Z +finished_at: 2026-03-17T00:05:00Z +timings: + total_migration_duration_seconds: 300 + drop_duration_seconds: 3 + recreate_duration_seconds: 12 + initial_indexing_duration_seconds: 270 + validation_duration_seconds: 15 + downtime_duration_seconds: 285 +validation: + schema_match: true + doc_count_match: true + indexing_failures_delta: 0 + query_checks: [] +benchmark_summary: + documents_indexed_per_second: 3703.7 + source_index_size_mb: 2048 + target_index_size_mb: 1984 + index_size_delta_mb: -64 + baseline_query_p95_ms: 42 + during_migration_query_p95_ms: 90 + post_migration_query_p95_ms: 44 +manual_actions: [] +``` + +## CLI UX + +### `plan` + +- Capture the source snapshot from the live index. +- Normalize requested changes. +- Classify the diff as supported or blocked. +- Emit `migration_plan.yaml`. +- Print a short risk summary that includes downtime. + +### `wizard` + +- Read the live source schema first. +- Walk the user through supported change categories only. +- Reject unsupported requests during the wizard instead of silently converting them. +- Explain when a blocked request belongs to a future `iterative_shadow` migration. +- Emit the same `migration_plan.yaml` shape as `plan`. + +### `apply` + +- Accept only `migration_plan.yaml` as input. +- Refuse to run if the plan contains blocked reasons. +- Refuse to run if the current live schema no longer matches the saved source snapshot. +- Require `--accept-data-loss` when quantization is involved *(replaces original `--allow-downtime`)*. +- Support `--async` for large migrations with quantization. +- Support `--resume` for crash-safe checkpoint recovery. + +### `validate` + +- Re-run validation checks from the plan against the current live index. +- Emit `migration_report.yaml`. +- Emit `benchmark_report.yaml` when benchmark fields were collected. + +## Execution Flow (as implemented) + +1. Snapshot source state. + - Load the live index schema using existing RedisVL introspection. + - Capture live stats from index info. + - Record storage type, prefixes, key separator, and a bounded key sample. +2. Normalize requested changes. + - If the input is `target_schema.yaml`, diff it against the source schema and convert it to a patch. + - If the input is wizard answers, convert them to the same patch model. +3. Merge and classify. + - Apply only requested changes to the source schema. + - Classify each diff as supported or blocked. + - Detect rename operations (index name, prefix, field names). + - Detect vector quantization operations. + - Stop if any blocked diff exists. +4. Generate the plan. + - Save source snapshot, requested changes, merged target schema, rename operations, validation policy, and warnings. +5. Apply the migration. + - Confirm current live schema still matches the source snapshot. + - **Quantize vectors** in-place if quantization is requested (crash-safe with checkpointing). + - **Rename hash fields** if field renames are requested. + - **Rename keys** if prefix change is requested. + - Drop only the index structure. + - Recreate the index (possibly with new name) using the merged target schema. +6. Wait for indexing completion. + - Poll live index info until `indexing` is false and `percent_indexed` is complete. + - Stop with timeout rather than waiting forever. +7. Validate. + - Compare live schema to merged target schema. + - Compare live doc count to source doc count. + - Check indexing failure delta. + - Check key sample existence. + - Run optional query checks. +8. Emit the report with timings, validation, and benchmark summary. +9. Optionally emit separate benchmark report. + +## Validation + +Required validation checks: + +- exact schema match against `merged_target_schema` +- live doc count equals source `num_docs` +- `hash_indexing_failures` does not increase +- key sample records still exist + +Optional validation checks: + +- query checks loaded from `--query-check-file` +- bounded sample fetch checks for representative document ids + +Benchmark fields that should be collected during Phase 1: + +- migration start and end timestamps +- index downtime duration +- readiness polling duration +- source and target document counts +- documents indexed per second +- source and target index footprint +- observed index footprint delta after recreate +- optional representative query latency before, during, and after migration + +Validation is a hard failure for `apply`. + +## Failure Handling + +The implementation fails closed. + +- Unsupported diff: stop at `plan`. +- Source snapshot mismatch at apply time: stop and ask the operator to regenerate the plan. +- Drop succeeds but recreate fails: documents remain; emit a failure report and a manual recovery command using the saved merged schema. +- Validation fails after recreate: leave the recreated index in place, emit a failure report, and stop. +- Interrupted quantization run: crash-safe checkpointing allows resume via `--resume ` *(added during implementation, replacing original "no checkpointing" stance)*. +- Pipeline errors during batch field renames or key renames: re-raised with context. + +The implementation does not implement automatic rollback. + +## Operational Guidance + +This mode is downtime-accepting by design. + +Engineers should assume: + +- the index is unavailable between drop and recreated index readiness +- search quality can be degraded while initial indexing completes +- large indexes can place measurable pressure on shared Redis resources +- off-peak execution is preferred +- application-level maintenance handling is outside RedisVL +- blocked vector and payload-shape changes should be rerouted to Phase 2 planning instead of being forced into this path + +Default key capture is intentionally small: + +- keyspace definition is always recorded +- a bounded key sample is recorded +- a full key manifest is not part of the default MVP path + +Benchmarking for Phase 1 should stay simple: + +- capture timing and correctness metrics in structured reports +- support manual benchmark rehearsals using [03_benchmarking.md](./03_benchmarking.md) +- avoid introducing a dedicated benchmarking subsystem before the migrator exists diff --git a/nitin_docs/index_migrator/11_v1_drop_recreate_tasks.md b/nitin_docs/index_migrator/11_v1_drop_recreate_tasks.md new file mode 100644 index 000000000..e8c98e57a --- /dev/null +++ b/nitin_docs/index_migrator/11_v1_drop_recreate_tasks.md @@ -0,0 +1,162 @@ +# Phase 1 Tasks: `drop_recreate` + +> **Status**: All tasks complete. Shipped as PRs #567-#572. + +## Task Template + +Every Phase 1 task documented: + +- `ID` +- `Status` +- `Goal` +- `Inputs` +- `Outputs` +- `Touchpoints` +- `Dependencies` +- `Acceptance Criteria` +- `Non-Goals` +- `Handoff Notes` + +## V1-T01 + +- `ID`: `V1-T01` +- `Status`: **Done** +- `Goal`: Build the source snapshot and artifact models used by `plan`, `apply`, and `validate`. +- `Inputs`: source index name, Redis connection settings, optional key sample limit +- `Outputs`: in-memory and YAML-serializable source snapshot, migration plan model, migration report model +- `Touchpoints`: `redisvl/migration/models.py` (SourceSnapshot, MigrationPlan, MigrationReport, SchemaPatch, DiskSpaceEstimate, etc.) +- `Dependencies`: none +- `Acceptance Criteria`: + - source schema can be loaded from a live index + - source stats needed by the spec are captured + - storage type, prefixes, key separator, and bounded key sample are recorded + - models serialize to YAML without losing required fields +- `Non-Goals`: + - full key manifest generation + - capacity estimation + - schema diff logic +- `Handoff Notes`: preserve the raw source schema as faithfully as possible so later diffing does not invent defaults. + +## V1-T02 + +- `ID`: `V1-T02` +- `Status`: **Done** +- `Goal`: Implement schema patch normalization, source-plus-patch merge, and supported-versus-blocked diff classification. +- `Inputs`: source schema snapshot, `schema_patch.yaml` or normalized target schema diff +- `Outputs`: merged target schema and diff classification result +- `Touchpoints`: `redisvl/migration/planner.py` (MigrationPlanner - handles patch merge, diff classification, rename detection, quantization detection) +- `Dependencies`: `V1-T01` +- `Acceptance Criteria`: + - unspecified source config is preserved by default + - blocked diff categories from the spec are rejected with actionable reasons + - supported changes produce a deterministic merged target schema + - `target_schema.yaml` input normalizes to the same patch model +- `Non-Goals`: + - document rewrite planning + - vector migration logic + - shadow migration planning +- `Handoff Notes`: prefer an explicit allowlist of supported diff categories over a generic schema merge engine. + +## V1-T03 + +- `ID`: `V1-T03` +- `Status`: **Done** +- `Goal`: Add the `plan` command and plan artifact generation. +- `Inputs`: source index, connection settings, patch or target schema input +- `Outputs`: `migration_plan.yaml`, console summary +- `Touchpoints`: `redisvl/cli/migrate.py` (Migrate.plan), `redisvl/cli/main.py`, `redisvl/migration/planner.py` +- `Dependencies`: `V1-T01`, `V1-T02` +- `Acceptance Criteria`: + - `plan` emits the required YAML shape + - blocked plans do not proceed to mutation + - the console summary includes downtime warnings + - the current plan format is stable enough for `apply` and `validate` +- `Non-Goals`: + - interactive wizard flow + - mutation against Redis + - advanced report rendering +- `Handoff Notes`: make the plan file human-readable so operators can review it before running `apply`. + +## V1-T04 + +- `ID`: `V1-T04` +- `Status`: **Done** +- `Goal`: Add the guided `wizard` flow that emits the same plan artifact as `plan`. +- `Inputs`: source index, connection settings, interactive answers +- `Outputs`: normalized schema patch and `migration_plan.yaml` +- `Touchpoints`: `redisvl/migration/wizard.py` (MigrationWizard), `redisvl/cli/migrate.py` (Migrate.wizard) +- `Dependencies`: `V1-T01`, `V1-T02`, `V1-T03` +- `Acceptance Criteria`: + - wizard starts from the live source schema + - wizard only offers supported MVP change categories + - wizard emits the same plan structure as `plan` + - unsupported requests are blocked during the flow +- `Non-Goals`: + - platform inventory collection + - free-form schema editing for blocked categories + - shadow migration support +- `Handoff Notes`: keep prompts simple and linear; this is a guided assistant, not a general schema builder. + +## V1-T05 + +- `ID`: `V1-T05` +- `Status`: **Done** +- `Goal`: Implement `apply` for the `drop_recreate` strategy. +- `Inputs`: reviewed `migration_plan.yaml` +- `Outputs`: recreated index, execution status, migration report +- `Touchpoints`: `redisvl/migration/executor.py` (MigrationExecutor), `redisvl/migration/async_executor.py` (AsyncMigrationExecutor), `redisvl/migration/reliability.py` (checkpointing, quantization), `redisvl/cli/migrate.py` (Migrate.apply) +- `Dependencies`: `V1-T03` +- `Acceptance Criteria`: + - source snapshot mismatch blocks execution + - index drop preserves documents + - recreated index uses the merged target schema + - readiness polling stops on success or timeout + - quantization is crash-safe with checkpointing + - async execution available for large migrations +- `Non-Goals`: + - automatic rollback + - cutover orchestration +- `Handoff Notes`: `--allow-downtime` was removed. `--accept-data-loss` is used only for quantization acknowledgment. Crash-safe checkpointing was added via `reliability.py`. + +## V1-T06 + +- `ID`: `V1-T06` +- `Status`: **Done** +- `Goal`: Implement `validate` and `migration_report.yaml`. +- `Inputs`: `migration_plan.yaml`, live index state, optional query checks +- `Outputs`: validation result, report artifact, console summary +- `Touchpoints`: `redisvl/migration/validation.py` (MigrationValidator), `redisvl/migration/async_validation.py` (AsyncMigrationValidator), `redisvl/cli/migrate.py` (Migrate.validate) +- `Dependencies`: `V1-T01`, `V1-T03`, `V1-T05` +- `Acceptance Criteria`: + - schema match is verified + - doc count match is verified + - indexing failure delta is captured + - core timing metrics are captured in the report + - optional query checks run deterministically + - report artifact is emitted for both success and failure +- `Non-Goals`: + - benchmark replay + - observability integrations + - automatic remediation +- `Handoff Notes`: keep the report format concise and stable so it can become the operator handoff artifact later. + +## V1-T07 + +- `ID`: `V1-T07` +- `Status`: **Done** +- `Goal`: Add Phase 1 tests and user-facing documentation for the new CLI flow. +- `Inputs`: completed planner, wizard, executor, and validator behavior +- `Outputs`: passing tests and concise usage docs +- `Touchpoints`: `tests/unit/test_migration_planner.py`, `tests/unit/test_batch_migration.py`, `tests/unit/test_migration_wizard.py`, `tests/integration/test_migration_comprehensive.py`, `docs/user_guide/` +- `Dependencies`: `V1-T03`, `V1-T04`, `V1-T05`, `V1-T06` +- `Acceptance Criteria`: + - CI-friendly happy-path and failure-path tests exist + - manual benchmark rehearsal guidance exists + - manual smoke test instructions are captured in the test doc + - help text matches the Phase 1 spec + - the docs directory still points to the active truth +- `Non-Goals`: + - Phase 2 implementation + - platform API integrations + - performance tuning beyond smoke coverage +- `Handoff Notes`: keep test coverage focused on correctness and operator safety, not on simulating every future migration shape. diff --git a/nitin_docs/index_migrator/12_v1_drop_recreate_tests.md b/nitin_docs/index_migrator/12_v1_drop_recreate_tests.md new file mode 100644 index 000000000..b00e4998c --- /dev/null +++ b/nitin_docs/index_migrator/12_v1_drop_recreate_tests.md @@ -0,0 +1,133 @@ +# Phase 1 Tests: `drop_recreate` + +> **Status**: All test scenarios covered. See actual test files below. + +## Actual Test Files + +| File | Coverage | +| --- | --- | +| `tests/unit/test_migration_planner.py` | Planner, patch merge, diff classification, rename detection, quantization detection | +| `tests/unit/test_batch_migration.py` | Batch planner, batch executor, batch state management | +| `tests/unit/test_migration_wizard.py` | Wizard flow, interactive prompts, plan generation | +| `tests/integration/test_migration_comprehensive.py` | End-to-end: plan -> apply -> validate with live Redis | + +## Original Test Matrix (mapped to implementation) + +| ID | Scenario | Type | Status | +| --- | --- | --- | --- | +| V1-CI-01 | source snapshot loads live schema and stats | CI | Done (test_migration_planner) | +| V1-CI-02 | patch merge preserves unspecified config | CI | Done (test_migration_planner) | +| V1-CI-03 | blocked diff categories stop at `plan` | CI | Done (test_migration_planner) | +| V1-CI-04 | `plan` emits valid YAML artifact | CI | Done (test_migration_planner) | +| V1-CI-05 | quantization requires `--accept-data-loss` | CI | Done (test_migration_comprehensive) | +| V1-CI-06 | drop and recreate preserves documents | CI | Done (test_migration_comprehensive) | +| V1-CI-07 | readiness polling completes or times out | CI | Done (executor tests) | +| V1-CI-08 | `validate` emits a report on success | CI | Done (test_migration_comprehensive) | +| V1-CI-09 | `validate` emits a report on failure | CI | Done (test_migration_comprehensive) | +| V1-CI-10 | timing metrics are captured in reports | CI | Done (test_migration_comprehensive) | +| V1-MAN-01 | guided wizard produces the same plan model | Unit | Done (test_migration_wizard) | +| V1-MAN-02 | realistic rebuild on larger dataset | Manual | Done (benchmark report in 05_migration_benchmark_report.md) | +| V1-MAN-03 | benchmark rehearsal on representative workload | Manual | Done (benchmark report in 05_migration_benchmark_report.md) | + +## Happy Path + +The minimum automated happy path should cover: + +- create a source index with existing documents +- generate `migration_plan.yaml` from `schema_patch.yaml` +- run `apply --plan migration_plan.yaml` +- wait for recreated index readiness +- run `validate` +- confirm schema match, doc count match, and zero indexing failure delta + +Representative happy-path schema changes: + +- add a tag field backed by existing JSON data +- remove a legacy numeric field from the index +- make an existing text field sortable + +## Failure Paths + +CI should cover at least: + +- blocked diff because of vector dimension change +- blocked diff because of storage type change +- source snapshot mismatch between `plan` and `apply` +- recreate failure after drop +- validation failure because doc counts diverge +- readiness timeout +- missing required plan fields + +Every failure path must prove: + +- documents are not intentionally deleted by the migrator +- an actionable error is surfaced +- blocked vector and payload-shape diffs point the user to the Phase 2 migration path +- a `migration_report.yaml` can still be produced when the failure happens after `apply` starts + +## Manual Smoke Test + +Run a manual smoke test on a non-production Redis deployment: + +1. Create an index with representative JSON documents. +2. Prepare a `schema_patch.yaml` that adds one non-vector field and removes one old field. +3. Run `rvl migrate plan`. +4. Confirm the plan includes the downtime warning and no blocked diffs. +5. Run `rvl migrate apply --plan migration_plan.yaml`. +6. Wait until readiness completes. +7. Run `rvl migrate validate --plan migration_plan.yaml`. +8. Confirm search behavior has resumed and the new schema is active. + +Manual smoke test success means: + +- the operator can understand the plan without reading code +- the index rebuild completes without deleting documents +- the report is sufficient to hand back to another operator + +## Scale Sanity Check + +Phase 1 does not need a cluster-wide stress harness, but it does need a basic scale sanity check. + +Manual checks: + +- run the flow on an index large enough to make polling and downtime visible +- confirm default key capture stays bounded +- confirm the tool does not attempt a full key manifest by default +- confirm console output still stays readable for a larger index + +This is not a benchmark. The goal is to catch accidental implementation choices that make the MVP operationally unsafe on larger datasets. + +## Benchmark Rehearsal + +Phase 1 benchmarking should be lightweight and operationally useful. + +Use a simple rehearsal driven by [03_benchmarking.md](./03_benchmarking.md): + +1. Record a benchmark label and workload context. +2. Measure baseline query latency on a representative query set. +3. Run the migration on a realistic non-production index. +4. Record total migration duration, downtime duration, and readiness duration. +5. Record source and target document counts and index stats. +6. Record the observed source-versus-target index footprint delta. +7. Re-run the representative query set after migration. +8. Save a `benchmark_report.yaml`. + +The first benchmark questions to answer are: + +- how long does the rebuild take end-to-end +- how long is the index unavailable +- how many documents per second can the rebuild sustain +- how much query latency changes during and after the rebuild +- how much the recreated index footprint changes even for schema-only rebuilds +- whether the observed runtime is predictable enough for a maintenance window + +## Release Gate + +Phase 1 should not be considered ready until all of the following are true: + +- all CI scenarios in the test matrix pass +- at least one manual smoke test passes +- at least one benchmark rehearsal has been documented on a representative dataset +- help text matches the spec +- the docs in `nitin_docs/index_migrator/` still match the shipped CLI behavior +- the release notes or implementation summary clearly state that `drop_recreate` is downtime-accepting diff --git a/nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md b/nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md new file mode 100644 index 000000000..c5b893a01 --- /dev/null +++ b/nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md @@ -0,0 +1,213 @@ +# Phase 2 Spec: `iterative_shadow` + +## Goal + +Add a conservative, capacity-aware shadow migration mode that works one index at a time and reduces disruption without attempting to automate cutover or platform scaling. + +This phase exists to support the migration cases that Phase 1 intentionally does not handle safely: + +- vector datatype changes such as `FP32 -> FP16` +- vector precision changes +- vector dimension changes +- vector algorithm changes such as `HNSW -> FLAT` +- payload-shape-changing model or algorithm migrations that require new stored fields or a new target keyspace + +The first Phase 2 implementation should stay simple in operation even though it handles harder migration shapes: + +- one index at a time +- capacity gate before each index +- operator-owned cutover +- no automatic scale-up or scale-down +- no multi-index concurrent shadowing +- explicit transform or backfill plan when the target payload shape changes + +## Why It Is Not MVP + +This mode is not the MVP because it introduces operational questions that Phase 1 does not need to solve: + +- database-level capacity inventory +- target-footprint estimation for old and new document and index shapes +- overlap estimation for old and new payloads +- transform or backfill planning for target payload creation +- operator handoff between validation and cutover +- cleanup sequencing after cutover +- larger-scale manual testing on clustered deployments + +Phase 1 should prove the core planning and reporting model first. + +## Planner Inputs + +The Phase 2 planner takes: + +- source index name +- Redis connection settings +- supported schema patch or target schema input +- `platform_inventory.yaml` +- optional `transform_plan.yaml` when the migration requires new target payloads + +### `platform_inventory.yaml` + +```yaml +version: 1 +platform: redis_cloud +database: + name: customer-a-prod + total_memory_mb: 131072 + available_memory_mb: 32768 + shard_count: 8 + replica_count: 1 + auto_tiering: false + notes: "" +policy: + reserve_percent: 15 +``` + +Required inventory fields: + +- platform +- total memory +- available memory +- shard count +- replica count +- reserve policy + +Optional inventory fields: + +- flash or disk notes +- environment labels +- operator comments +- benchmark history notes + +### `transform_plan.yaml` + +This file is required when the target schema cannot be built from the current stored payload. + +Example: + +```yaml +version: 1 +target_keyspace: + storage_type: json + prefixes: ["docs_v2"] + key_separator: ":" +transform: + mode: rewrite + vector_fields: + - name: embedding + source_path: $.embedding + target_path: $.embedding_v2 + source_dimensions: 1536 + target_dimensions: 1536 + source_datatype: float32 + target_datatype: float16 + source_algorithm: hnsw + target_algorithm: flat + payload_changes: + - source_path: $.body + target_path: $.body_v2 + strategy: copy +``` + +The first implementation should keep this model explicit and declarative. The migrator should not guess how to transform payloads. + +## Capacity Gate + +The first Phase 2 capacity gate should be intentionally conservative. + +Planner rules: + +1. Compute source document footprint from live stats or bounded sampling. +2. Compute source index footprint from live index stats. +3. Estimate target document footprint. + - For payload-compatible shadowing, this can be zero or near-zero additional document storage. + - For payload rewrite shadowing, this includes the duplicated target payload. +4. Estimate target index footprint. + - Use live source footprint as a baseline when the target is structurally similar. + - Adjust for vector dimension, datatype, precision, and algorithm changes when those are present. +5. Compute reserve headroom as `max(operator reserve, 15 percent of configured memory)` when no stricter operator value is provided. +6. Compute `estimated_peak_overlap_footprint` as: + - `source_docs + source_index + target_docs + target_index` +7. Return `READY` only if: + - the migration diff is supported for Phase 2 + - any required transform plan is present and valid + - available memory is greater than or equal to `estimated_peak_overlap_footprint + reserve` +8. Return `SCALE_REQUIRED` when the migration is supported but headroom is insufficient. +9. Return `MANUAL_REVIEW_REQUIRED` when the diff is ambiguous or live data is insufficient for a safe estimate. + +This keeps the first shadow planner understandable and safe. More sophisticated estimators can come later if Phase 1 and early Phase 2 learnings justify them. + +The planner should also report: + +- estimated migration window +- estimated peak overlap footprint +- expected source-versus-target footprint delta after cutover +- whether the migration is `shadow_reindex` or `shadow_rewrite` + +## Execution Flow + +1. Capture the source snapshot and normalize requested changes. +2. Classify the migration as either: + - `shadow_reindex` when the target schema can be built from the current payload + - `shadow_rewrite` when a transform or backfill is needed +3. Load `platform_inventory.yaml`. +4. Load `transform_plan.yaml` when `shadow_rewrite` is required. +5. Compute the capacity gate result. +6. Stop if the result is not `READY`. +7. Create the shadow target for the current index only. +8. If `shadow_rewrite` is selected: + - create the target keyspace + - transform or backfill source documents into the target keyspace +9. Wait until the shadow index is ready. +10. Validate the shadow target. +11. Emit an operator cutover runbook. +12. Wait for operator confirmation that cutover is complete. +13. Retire the old index. +14. Retire old source payloads only when the plan explicitly says they are no longer needed. +15. Move to the next index only after the current index is finished. + +The scheduler for Phase 2 is intentionally serial. + +## Operator Actions + +The operator is responsible for: + +- supplying platform inventory +- supplying the transform or backfill plan when payload shape changes +- choosing the migration window +- scaling the database if the plan returns `SCALE_REQUIRED` +- switching application traffic to the shadow target +- confirming cutover before old index retirement +- monitoring the deployment during overlap + +RedisVL should not attempt to perform these actions automatically in the first Phase 2 implementation. + +Phase 2 should still emit structured benchmark outputs so operators can compare: + +- estimated overlap duration versus actual overlap duration +- estimated capacity usage versus observed document and index stats +- memory savings or growth after algorithm, datatype, precision, dimension, or payload-shape changes +- query latency impact during shadow validation and overlap + +## Blocked Scenarios + +The initial Phase 2 plan still blocks: + +- automatic scaling +- automatic traffic switching +- concurrent shadowing of multiple large indexes +- in-place destructive rewrites without a shadow target +- payload-shape-changing migrations without an explicit transform or backfill plan +- transform plans that do not define a deterministic target keyspace +- Active-Active specific workflows +- platform API integrations as a hard requirement + +## Open Questions Deferred + +These questions should stay deferred until after Phase 1 implementation: + +- whether to add direct Redis Cloud or Redis Software API integrations +- whether to support checkpoint and resume across shadow runs +- whether alias-based cutover should be added later +- how transform hooks should be expressed beyond the initial declarative plan format +- whether re-embedding should be integrated directly or stay an operator-supplied preprocessing step +- how much historical benchmark data should influence ETA predictions diff --git a/nitin_docs/index_migrator/21_v2_iterative_shadow_tasks.md b/nitin_docs/index_migrator/21_v2_iterative_shadow_tasks.md new file mode 100644 index 000000000..4b00955e8 --- /dev/null +++ b/nitin_docs/index_migrator/21_v2_iterative_shadow_tasks.md @@ -0,0 +1,159 @@ +# Phase 2 Tasks: `iterative_shadow` + +## Task Template + +Every Phase 2 task must document: + +- `ID` +- `Status` +- `Goal` +- `Inputs` +- `Outputs` +- `Touchpoints` +- `Dependencies` +- `Acceptance Criteria` +- `Non-Goals` +- `Handoff Notes` + +Phase 2 tasks are planned work only. They should not start until Phase 1 implementation is complete and learnings are folded back into this workspace. + +## V2-T01 + +- `ID`: `V2-T01` +- `Status`: `Planned` +- `Goal`: Add the platform inventory model and parser used by the capacity-aware planner. +- `Inputs`: `platform_inventory.yaml` +- `Outputs`: validated inventory model +- `Touchpoints`: new `redisvl/migration/inventory.py`, `redisvl/migration/models.py`, `redisvl/cli/migrate.py` +- `Dependencies`: Phase 1 implementation complete +- `Acceptance Criteria`: + - required inventory fields are validated + - unsupported platform inventory shapes are rejected clearly + - inventory values are available to the planner without CLI-specific parsing logic +- `Non-Goals`: + - platform API calls + - capacity math + - shadow execution +- `Handoff Notes`: keep the inventory model platform-neutral enough to support both Redis Cloud and Redis Software. + +## V2-T02 + +- `ID`: `V2-T02` +- `Status`: `Planned` +- `Goal`: Add the transform or backfill plan model and classify whether a migration is `shadow_reindex` or `shadow_rewrite`. +- `Inputs`: normalized diff classification, optional `transform_plan.yaml` +- `Outputs`: validated transform model and execution-mode classification +- `Touchpoints`: new `redisvl/migration/transforms.py`, `redisvl/migration/models.py`, `redisvl/migration/planner.py` +- `Dependencies`: `V2-T01` +- `Acceptance Criteria`: + - payload-compatible migrations are classified as `shadow_reindex` + - vector or payload-shape-changing migrations require `shadow_rewrite` + - missing transform plans are rejected clearly when they are required + - transform plans remain declarative and deterministic +- `Non-Goals`: + - direct embedding generation + - platform API calls + - shadow execution +- `Handoff Notes`: keep the first transform model simple and explicit rather than inventing a generic transformation framework. + +## V2-T03 + +- `ID`: `V2-T03` +- `Status`: `Planned` +- `Goal`: Implement the conservative capacity estimator and gate result classification. +- `Inputs`: source index stats, source document footprint, inventory model, normalized diff classification, optional transform model +- `Outputs`: `READY`, `SCALE_REQUIRED`, or `MANUAL_REVIEW_REQUIRED` +- `Touchpoints`: new `redisvl/migration/capacity.py`, `redisvl/migration/planner.py` +- `Dependencies`: `V2-T01`, `V2-T02` +- `Acceptance Criteria`: + - source document and index footprint are computed consistently + - target footprint estimates account for vector datatype, precision, dimension, algorithm, and payload-shape changes when those are present + - reserve policy is applied consistently + - supported diffs can produce `READY` or `SCALE_REQUIRED` + - ambiguous inputs produce `MANUAL_REVIEW_REQUIRED` +- `Non-Goals`: + - fine-grained shard placement modeling + - automated scale actions + - performance benchmarking as a separate subsystem +- `Handoff Notes`: keep the first estimator intentionally conservative and easy to inspect. + +## V2-T04 + +- `ID`: `V2-T04` +- `Status`: `Planned` +- `Goal`: Extend the planner to support `iterative_shadow` for one index at a time. +- `Inputs`: source snapshot, normalized diff, inventory, transform model, capacity result +- `Outputs`: shadow migration plan and operator action list +- `Touchpoints`: `redisvl/migration/planner.py`, `redisvl/cli/migrate.py` +- `Dependencies`: `V2-T03` +- `Acceptance Criteria`: + - supported vector and payload-shape changes can produce a valid shadow plan + - non-`READY` capacity results block apply + - plan artifact clearly identifies source, shadow target, target keyspace when present, and operator actions + - plan artifact identifies whether the run is `shadow_reindex` or `shadow_rewrite` + - plan format stays readable and deterministic +- `Non-Goals`: + - multi-index concurrency + - automatic cleanup + - fleet scheduling +- `Handoff Notes`: preserve the same plan-first experience as Phase 1. + +## V2-T05 + +- `ID`: `V2-T05` +- `Status`: `Planned` +- `Goal`: Implement shadow target creation, optional transform or backfill execution, readiness waiting, and validation hooks. +- `Inputs`: approved shadow migration plan +- `Outputs`: ready shadow index and validation state +- `Touchpoints`: new `redisvl/migration/shadow.py`, `redisvl/migration/executor.py`, `redisvl/migration/validation.py` +- `Dependencies`: `V2-T04` +- `Acceptance Criteria`: + - only one index is processed at a time + - shadow target creation follows the plan artifact + - `shadow_rewrite` runs can build the target payload into the planned keyspace + - readiness polling behaves deterministically + - validation runs before cutover handoff +- `Non-Goals`: + - automatic cutover + - cross-index scheduling + - platform autoscaling +- `Handoff Notes`: do not generalize this into a fleet scheduler in the first Phase 2 implementation. + +## V2-T06 + +- `ID`: `V2-T06` +- `Status`: `Planned` +- `Goal`: Add validation reporting, benchmark reporting, operator handoff, cutover confirmation, and old-index retirement. +- `Inputs`: validated shadow plan and operator confirmation +- `Outputs`: post-cutover cleanup result and report +- `Touchpoints`: `redisvl/cli/migrate.py`, `redisvl/migration/reporting.py`, `redisvl/migration/executor.py` +- `Dependencies`: `V2-T05` +- `Acceptance Criteria`: + - cutover remains operator-owned + - cleanup does not run before operator confirmation + - report captures cutover handoff, cleanup outcome, and source-versus-target footprint deltas +- `Non-Goals`: + - alias management + - application config mutation + - rollback orchestration +- `Handoff Notes`: the CLI should guide the operator clearly, but it must not attempt to switch traffic itself. + +## V2-T07 + +- `ID`: `V2-T07` +- `Status`: `Planned` +- `Goal`: Add future-facing tests and docs for clustered shadow migration planning. +- `Inputs`: completed Phase 2 planner and executor behavior +- `Outputs`: test coverage, manual scale rehearsal instructions, and updated planning docs +- `Touchpoints`: `tests/`, `nitin_docs/index_migrator`, `redisvl/cli` +- `Dependencies`: `V2-T04`, `V2-T05`, `V2-T06` +- `Acceptance Criteria`: + - planner outcomes are covered in automated tests + - benchmark, ETA, and memory-delta guidance are documented for manual cluster rehearsals + - manual cluster rehearsal steps are documented + - docs still reflect the shipped Phase 2 behavior accurately +- `Non-Goals`: + - fleet-wide migration support + - performance tuning beyond safety validation + - platform-specific automation +- `Handoff Notes`: keep Phase 2 documentation grounded in the one-index-at-a-time rule. diff --git a/nitin_docs/index_migrator/22_v2_iterative_shadow_tests.md b/nitin_docs/index_migrator/22_v2_iterative_shadow_tests.md new file mode 100644 index 000000000..e7d2d5c27 --- /dev/null +++ b/nitin_docs/index_migrator/22_v2_iterative_shadow_tests.md @@ -0,0 +1,102 @@ +# Phase 2 Tests: `iterative_shadow` + +## Clustered Test Setup + +Phase 2 needs both automated planner coverage and manual clustered rehearsals. + +Minimum clustered rehearsal setup: + +- Redis Cloud or Redis Software deployment +- sharded database +- one source index large enough to make overlap meaningful +- representative application query set +- operator-supplied `platform_inventory.yaml` +- `transform_plan.yaml` for any vector or payload-shape-changing migration + +The first manual scale rehearsal should stay focused on a single index, not a fleet-wide migration. + +## Planner Acceptance + +Automated planner tests should cover: + +- supported shadow diff with sufficient headroom returns `READY` +- supported shadow diff with insufficient headroom returns `SCALE_REQUIRED` +- ambiguous or incomplete input returns `MANUAL_REVIEW_REQUIRED` +- vector datatype, precision, dimension, or algorithm changes require `shadow_rewrite` +- payload-shape-changing diffs stop before planning unless a valid transform plan is present + +Planner acceptance is successful when the result is deterministic and the operator action list is clear. + +## Unsafe Capacity Cases + +Manual and automated coverage should include: + +- insufficient available memory +- missing or invalid inventory fields +- conflicting operator reserve policy +- large source footprint with conservative reserve +- target footprint larger than source footprint because of dimension or payload expansion +- peak overlap estimate exceeds available headroom even when post-cutover memory would shrink + +Unsafe capacity handling is correct when: + +- the planner blocks the run +- no shadow index is created +- the report tells the operator what must change before retry + +## Shadow Validation + +Validation coverage should prove: + +- shadow target reaches readiness before handoff +- schema matches the planned target +- transformed payload fields match the declared target shape when `shadow_rewrite` is used +- query checks pass before cutover +- old index is not retired before operator confirmation + +This is the safety boundary for Phase 2. + +## Benchmark Rehearsal + +Phase 2 benchmarks should answer: + +- how accurate the planner ETA was +- how long the old and shadow indexes overlapped +- how much query latency changed during overlap +- whether the capacity reserve was conservative enough +- how much memory or size changed after datatype, precision, dimension, algorithm, or payload-shape changes +- whether estimated peak overlap footprint matched observed overlap closely enough + +Minimum manual benchmark coverage: + +- one run where the planner returns `READY` and the migration completes +- one run where the planner returns `SCALE_REQUIRED` +- one run with representative read traffic during overlap +- one vector-shape or algorithm change such as `HNSW -> FLAT` or `FP32 -> FP16` +- one payload-shape-changing migration that requires transform or backfill + +Every benchmark rehearsal should produce a structured benchmark report that can be compared against previous runs. + +## Resume/Retry + +The first Phase 2 implementation does not need fleet-grade checkpointing, but it does need basic retry behavior. + +Required checks: + +- planner can be rerun with the same inventory and produce the same decision +- failed shadow creation does not trigger cleanup of the old index +- operator can rerun the planned index only after fixing the blocking condition + +If stronger checkpointing is needed later, it should become its own scoped follow-up rather than being absorbed into the first shadow implementation. + +## Exit Criteria + +Phase 2 should not move from planned to ready until: + +- Phase 1 has been implemented and reviewed +- Phase 1 learnings have been written back into this workspace +- planner outcomes are covered by automated tests +- at least one manual clustered rehearsal has been designed in detail +- at least one benchmark rehearsal has been defined for a representative shadow migration +- at least one benchmark rehearsal has been defined for a vector or payload-shape-changing shadow migration +- the one-index-at-a-time execution rule is still preserved in the design diff --git a/nitin_docs/index_migrator/90_prd.md b/nitin_docs/index_migrator/90_prd.md new file mode 100644 index 000000000..b54b8f82c --- /dev/null +++ b/nitin_docs/index_migrator/90_prd.md @@ -0,0 +1,599 @@ +# Product Requirements Document: RedisVL Index Migrator + +> **Status**: Phase 1 is implemented and shipped (PRs #567-#572). This PRD has been updated with implementation notes where the shipped product diverged from the original requirements. + +## Summary + +RedisVL now has a migration workflow for search index changes that is safer and more operationally predictable than ad hoc scripts, while remaining simple enough to build, review, and operate without introducing an orchestration-heavy subsystem. + +This PRD defines a phased migration product: + +- Phase 1: `drop_recreate` - **Done** +- Phase 2: `iterative_shadow` - Planned + +The overall product goal is broader than the MVP. The migrator should eventually handle not only simple schema changes, but also vector datatype, precision, dimension, algorithm, and payload-shape-changing migrations such as: + +- `HNSW -> FLAT` +- `FP32 -> FP16` +- vector dimension changes +- embedding or payload-shape changes that require new stored fields or a new target keyspace + +Phase 1 stays intentionally narrow so the team can ship a plan-first, document-preserving migration tool quickly. Phase 2 is where those harder migrations are handled safely through one-index-at-a-time shadow migration and optional transform or backfill. + +This document is the review-oriented summary of the detailed planning docs in this workspace. + +## Problem + +RedisVL today gives users index lifecycle primitives, not a migration product. + +Users can: + +- create indexes +- delete indexes +- inspect index information +- load documents + +Phase 1 addressed the following gaps (users can now): + +- preview a schema migration in a structured way (`rvl migrate plan`) +- preserve current index configuration before change (`source` snapshot in plan) +- apply only a requested subset of schema changes (`schema_patch.yaml`) +- generate a repeatable migration plan artifact (`migration_plan.yaml`) +- validate a migration with a consistent report (`rvl migrate validate`) +- estimate migration duration, query impact, or memory impact from benchmarkable outputs (`migration_report.yaml`, `benchmark_report.yaml`) +- perform vector quantization (e.g., FP32 -> FP16) with crash-safe reliability + +Still not addressed (Phase 2): + +- shadow migrations that require running old and new indexes in parallel + +This gap is manageable for small experiments, but becomes painful for production workloads where: + +- indexes can contain millions of documents +- query latency matters during rebuild windows +- teams need predictable maintenance timelines +- migrations may change vector algorithms, datatypes, or payload shape +- Redis deployments may be clustered on Redis Cloud or Redis Software +- operators need a clean handoff between planning, execution, and validation + +## Users and Review Audience + +Primary users: + +- application engineers managing RedisVL-backed indexes +- platform engineers responsible for Redis operations +- support or solution engineers helping customers plan schema migrations + +Review audience: + +- RedisVL maintainers +- product and engineering stakeholders +- operators who will validate whether the workflow is practical in real environments + +## Goals + +- Provide a first-class migration workflow for RedisVL-managed indexes. +- Preserve existing documents during the Phase 1 path. +- Capture current schema and index configuration before any mutation. +- Apply only requested schema changes. +- Require a reviewed migration plan before execution. +- Support both scripted and guided user experiences. +- Make downtime and disruption explicit. +- Add structured reports and benchmarking outputs so migration windows become more predictable over time. +- Add benchmarking for memory and size deltas caused by schema, vector, and payload-shape changes. +- Keep the implementation simple enough that another engineer can understand and modify it quickly. + +## In Scope + +### Product-Wide Scope + +- schema-change migrations for RedisVL-managed indexes +- vector datatype, precision, dimension, and algorithm migrations +- payload-shape-changing migrations when the operator provides an explicit transform or backfill plan +- YAML artifacts for plans and reports +- operator-readable console output +- one-index-at-a-time execution +- benchmarking outputs for timing, query impact, and memory or size deltas + +### Phase 1 MVP + +- one index at a time +- document-preserving `drop_recreate` migrations +- source schema and stats snapshot +- patch-based schema change requests +- target-schema diff normalization into the same patch model +- guided wizard and scripted CLI flows +- plan generation before any mutation +- explicit downtime acknowledgment for `apply` +- readiness waiting after recreate +- validation and reporting +- benchmark-friendly timing, correctness, and index-footprint outputs + +### Phase 2 + +- one-index-at-a-time `iterative_shadow` migrations +- conservative capacity gating before each index +- optional platform inventory input +- shadow target creation and validation +- `shadow_reindex` for payload-compatible migrations +- `shadow_rewrite` for vector or payload-shape-changing migrations +- explicit transform or backfill plan input when payload shape changes +- operator handoff for cutover +- benchmark outputs for ETA, peak overlap, and source-versus-target size deltas + +## Out of Scope + +- automatic traffic cutover +- automatic platform scaling +- concurrent migration of multiple large indexes +- fully managed rollback orchestration +- full key manifest capture by default +- platform API integration as a hard requirement +- automatic transform inference +- automatic embedding generation or automatic re-embedding inside the migrator +- in-place destructive payload rewrites without a shadow target + +## Product Principles + +- Prefer simple and safe over fully automated. +- Reuse existing RedisVL primitives before adding new layers. +- Make the plan artifact the center of the workflow. +- Treat operator actions as first-class, not hidden implementation details. +- Fail closed when a migration request is ambiguous or unsupported for the selected phase. +- Measure migration behavior so future planning gets better with evidence. + +## Current State + +RedisVL already has building blocks that support a migration product: + +- `SearchIndex.from_existing()` for live schema reconstruction +- `SearchIndex.delete(drop=False)` for dropping the index structure while preserving documents +- `SearchIndex.info()` for live index stats that can inform planning, validation, and timing + +What is missing is the product layer on top: + +- a migration planner +- schema patch normalization and diff classification +- migration-specific CLI commands +- guided user flow +- structured migration and benchmark artifacts +- a capacity-aware future mode for larger production environments +- transform or backfill planning for migrations that change payload shape + +## Proposed Product + +### Phase 1 MVP: `drop_recreate` + +Scope: + +- one index at a time +- preserve documents +- snapshot source schema and stats +- accept `schema_patch.yaml`, `target_schema.yaml`, or wizard answers +- normalize all inputs into the same plan model +- classify requested changes as supported or blocked +- generate `migration_plan.yaml` +- require explicit downtime acknowledgment for `apply` +- drop only the index structure +- recreate the index using the merged schema +- wait for readiness +- validate and emit `migration_report.yaml` +- optionally emit `benchmark_report.yaml` + +Supported changes: + +- add non-vector fields backed by existing document data +- remove fields +- adjust supported non-vector index options where stored payload shape does not change +- adjust index-level options that do not relocate or rewrite data + +Blocked (still): + +- key separator changes +- storage type changes (hash <-> JSON) +- JSON path remodels +- vector dimension changes +- any change requiring a completely new stored payload shape + +> **Implementation note**: The following were originally blocked in this PRD but were implemented: +> - ~~key prefix changes~~ - supported via `index.prefix` in schema patch +> - ~~field renames~~ - supported via `rename_fields` in schema patch +> - ~~vector datatype changes~~ - supported as in-place quantization +> - ~~index name changes~~ - supported via `index.name` in schema patch + +### Phase 2: `iterative_shadow` + +Scope: + +- one index at a time +- conservative capacity gate before each index +- optional `platform_inventory.yaml` +- optional `transform_plan.yaml` when payload shape changes +- shadow target creation +- readiness waiting and validation +- operator-owned cutover +- old index retirement after operator confirmation +- optional old-payload retirement after operator confirmation +- structured benchmark outputs for overlap timing, ETA accuracy, and memory or size deltas + +Execution submodes: + +- `shadow_reindex` + - use when the new index can be built from the current stored payload + - still useful for lower-disruption rebuilds when the payload shape does not change +- `shadow_rewrite` + - use when vector datatype, precision, dimension, algorithm, or payload shape changes require a new target payload or keyspace + - examples: `HNSW -> FLAT`, `FP32 -> FP16`, dimension changes, new embedding schema + +Still intentionally excluded: + +- automatic cutover +- automatic scaling +- concurrent shadowing of multiple large indexes +- transform inference + +## Architecture + +The product should work as a plan-first migration workflow with explicit operator handoff and an optional transform path for harder migrations. + +```mermaid +flowchart LR + OP["Engineer / Operator"] --> CLI["RedisVL migrate CLI"] + APP["Applications"] --> DB["Redis database endpoint"] + + CLI --> PLAN["Planner
snapshot + diff + mode selection"] + CLI --> EXEC["Executor"] + CLI --> VAL["Validator + reports"] + + PLAN --> META["Live index schema + stats"] + PLAN --> INV["Optional platform inventory"] + PLAN --> XFORM["Optional transform or backfill plan"] + + EXEC --> DB + EXEC --> TARGET["Optional target keyspace + shadow index"] + TARGET --> DB + VAL --> DB + + OP --> CUT["Maintenance window / cutover decision"] + CUT --> APP +``` + +Architecture expectations: + +- RedisVL owns planning, execution, validation, and artifact generation. +- Redis remains the system of record for source documents and index state. +- The operator owns maintenance windows, scaling, transform inputs, and application cutover decisions. +- The product must stay compatible with single-node and clustered deployments without assuming the whole index lives on one shard. + +## Why the Work Is Phased + +The product is phased because the migration strategies solve different problems: + +- `drop_recreate` is the fastest path to a usable, understandable MVP +- `iterative_shadow` is the future path for tighter operational control and safer handling of vector or payload-shape changes + +Trying to ship everything as one fully mature product would push the team into: + +- premature capacity-estimation complexity +- premature transform-runtime design +- premature cutover abstractions +- premature platform-specific automation +- a larger QA and support surface before the MVP proves value + +Phase 1 is therefore the implementation target, while Phase 2 remains planned work informed by Phase 1 learnings. + +## User Experience + +### Scripted Flow (as shipped) + +```text +rvl migrate plan --index --schema-patch +rvl migrate plan --index --target-schema +rvl migrate apply --plan [--async] [--resume ] +rvl migrate validate --plan +rvl migrate estimate --plan +``` + +### Guided Flow (as shipped) + +```text +rvl migrate wizard --index --plan-out +``` + +### Batch Flow (as shipped) + +```text +rvl migrate batch-plan --schema-patch --pattern '*_idx' +rvl migrate batch-apply --plan --accept-data-loss +rvl migrate batch-resume --state --retry-failed +rvl migrate batch-status --state +``` + +User experience requirements (verified in implementation): + +- `plan` never mutates Redis +- `wizard` emits the same plan artifact shape as `plan` +- `apply` only accepts a reviewed plan file +- `apply` requires `--accept-data-loss` when quantization is involved +- `validate` is usable independently after `apply` +- console output is concise and operator-readable +- blocked requests tell the user what is not supported + +## Usage + +### Phase 1: `drop_recreate` + +Review-first workflow: + +```text +rvl migrate plan --index products --schema-patch patch.yaml --plan-out migration_plan.yaml +rvl migrate apply --plan migration_plan.yaml --report-out migration_report.yaml +rvl migrate validate --plan migration_plan.yaml --report-out migration_report.yaml +``` + +Guided workflow: + +```text +rvl migrate wizard --index products --plan-out migration_plan.yaml +rvl migrate apply --plan migration_plan.yaml +``` + +Expected usage pattern: + +1. Generate a plan from a live source index. +2. Review blocked diffs, warnings, downtime notice, and merged target schema. +3. Run `apply` only after the operator accepts the maintenance window. +4. Run `validate` and retain the report as the handoff artifact. + +### Phase 2: `iterative_shadow` + +Payload-compatible shadow workflow: + +```text +rvl migrate plan --mode iterative_shadow --index products --schema-patch patch.yaml --platform-inventory platform_inventory.yaml --plan-out migration_plan.yaml +rvl migrate apply --plan migration_plan.yaml --report-out migration_report.yaml +``` + +Payload-rewrite shadow workflow: + +```text +rvl migrate plan --mode iterative_shadow --index products --target-schema target_schema.yaml --platform-inventory platform_inventory.yaml --transform-plan transform_plan.yaml --plan-out migration_plan.yaml +rvl migrate apply --plan migration_plan.yaml --report-out migration_report.yaml +``` + +Expected usage pattern: + +1. Provide the schema request and platform inventory. +2. Provide `transform_plan.yaml` when the target payload shape changes. +3. Review the capacity-gate outcome, estimated migration window, and estimated peak overlap footprint. +4. Run the shadow migration for one index only. +5. Hand cutover to the operator. +6. Confirm cutover before retiring the old index and any obsolete payloads. + +## Artifacts + +Required artifacts: + +- `migration_plan.yaml` +- `migration_report.yaml` + +Optional or phase-dependent artifacts: + +- `benchmark_report.yaml` +- `platform_inventory.yaml` +- `transform_plan.yaml` +- `benchmark_manifest.yaml` + +Artifact requirements: + +- YAML-based +- stable enough for handoff and review +- readable by humans first +- structured enough for future automation + +## Operational Model + +RedisVL owns: + +- source snapshot +- schema diffing +- plan generation +- supported strategy execution +- readiness waiting +- validation +- reporting + +Operators own: + +- maintenance windows +- application behavior during migration +- platform scaling +- transform inputs for payload-shape changes +- cutover +- final go or no-go decisions in production + +The product should not imply that RedisVL is a full migration control plane. It is a migration toolset with explicit operator handoff. + +## Capacity and Scale + +Phase 1 keeps capacity handling simple: + +- use source index stats for warnings +- capture timing and impact for later planning +- avoid a complex estimator in the MVP + +Phase 2 introduces a conservative planner: + +- reason at the database level, not as “an index lives on one shard” +- treat each index as one logical distributed index even on sharded deployments +- estimate source document footprint and source index footprint separately +- estimate target document footprint and target index footprint separately +- compute peak overlap as the source footprint plus the target footprint that exists during migration +- require reserve headroom before apply +- return `READY`, `SCALE_REQUIRED`, or `MANUAL_REVIEW_REQUIRED` + +The execution rule stays simple across both phases: + +- one index at a time + +This is the core design choice that keeps the system understandable at production scale. + +## Downtime and Disruption + +Phase 1 explicitly accepts downtime. + +Expected impacts: + +- search on the affected index is unavailable between drop and recreated index readiness +- query quality may be degraded while initial indexing completes +- shared Redis resources are consumed during rebuild +- large indexes need maintenance windows or application-level degraded mode handling + +Phase 2 aims to reduce disruption, but it still has operational costs: + +- old and new index structures overlap during migration +- payload-rewrite migrations may also duplicate payloads temporarily +- memory and size can either grow or shrink depending on datatype, precision, dimension, algorithm, and payload-shape changes + +These are product facts and must be visible in the plan and report artifacts. + +## Benchmarking and Success Metrics + +Benchmarking is a product requirement, not an afterthought. + +The product should help answer: + +- how long planning takes +- how long apply takes +- how long downtime or overlap lasts +- how much document throughput the migration achieves +- how query latency changes during the migration window +- how much memory and size change between source and target +- how accurate the peak-overlap estimate was + +Core success metrics: + +- migration plan generation succeeds for supported diffs +- unsupported diffs are blocked before mutation +- Phase 1 preserves documents +- Phase 2 produces deterministic shadow plans for supported vector and payload-shape migrations +- schema match and document count match succeed after migration +- reports include stable timing, correctness, and memory-delta metrics +- benchmark rehearsals are good enough to estimate future maintenance windows and scaling decisions with confidence + +## Functional Requirements + +- plan generation from live index plus requested schema changes +- schema patch normalization +- supported-versus-blocked diff classification +- guided wizard for supported Phase 1 changes +- explicit downtime acknowledgment in Phase 1 +- structured plan, report, and benchmark outputs +- validation of schema, counts, and indexing-failure deltas +- one-index-at-a-time execution +- Phase 2 capacity-gated shadow planning +- Phase 2 support for vector and payload-shape migrations through explicit shadow planning + +## Non-Functional Requirements + +- deterministic plan outputs +- human-readable YAML artifacts +- clear failure modes +- conservative defaults +- no document deletion by the Phase 1 migrator path +- reasonable operation on large indexes without default full-key manifests +- documentation detailed enough for implementation handoff + +## Risks + +- Users may assume unsupported Phase 1 schema changes should “just work” unless the diff classifier clearly routes them to Phase 2. +- Operators may underestimate downtime for large indexes unless benchmark outputs become part of the review flow. +- Phase 2 can grow too complex if transform logic or platform-specific automation is pulled in too early. +- Capacity estimation may be wrong unless benchmark data and observed footprint deltas are captured consistently. +- Validation may be treated as optional unless the CLI and reports make it central to the workflow. + +## Rollout Plan + +### Phase 1 - **Done** + +- ~~finalize docs and task list~~ Done +- ~~implement the planner, diff classifier, CLI flow, executor, and validator~~ Done (PRs #567-#572) +- ~~add CI coverage for supported and blocked migration paths~~ Done +- ~~run at least one benchmark rehearsal~~ Done (see 05_migration_benchmark_report.md) + +### Phase 1.5 - **Done** + +- ~~review real implementation learnings~~ Done +- ~~update the planning workspace~~ Done (this update) +- Phase 2 assumptions still hold; shadow migrations remain the right approach for incompatible changes + +### Phase 2 + +- implement inventory parsing +- implement transform or backfill plan modeling +- implement conservative capacity gating +- implement one-index-at-a-time shadow planning and execution +- add benchmark rehearsals for overlap duration, ETA accuracy, and memory-delta accuracy + +## Review Questions for the Team + +- Is the Phase 1 boundary narrow enough to ship quickly, but useful enough to solve real user pain? +- Is Phase 2 scoped clearly enough to own vector datatype, precision, dimension, algorithm, and payload-shape changes? +- Is operator-owned cutover still the right long-term boundary? +- Is the benchmarking scope sufficient to make migration windows and scaling decisions predictable without overbuilding a measurement subsystem? +- Does the one-index-at-a-time rule provide the right balance of simplicity and scale? + +## Decision Summary + +- Build the migration product in phases. +- Implement Phase 1 first and keep it intentionally narrow. +- Treat vector and payload-shape migrations as a core product goal, delivered in Phase 2 rather than ignored. +- Keep the plan artifact central to the workflow. +- Keep the operational model explicit. +- Use evidence from benchmark outputs to shape later migration planning. + +## References + +Detailed supporting docs in this workspace: + +- [00_index.md](./00_index.md) +- [01_context.md](./01_context.md) +- [02_architecture.md](./02_architecture.md) +- [03_benchmarking.md](./03_benchmarking.md) +- [10_v1_drop_recreate_spec.md](./10_v1_drop_recreate_spec.md) +- [11_v1_drop_recreate_tasks.md](./11_v1_drop_recreate_tasks.md) +- [12_v1_drop_recreate_tests.md](./12_v1_drop_recreate_tests.md) +- [20_v2_iterative_shadow_spec.md](./20_v2_iterative_shadow_spec.md) +- [21_v2_iterative_shadow_tasks.md](./21_v2_iterative_shadow_tasks.md) +- [22_v2_iterative_shadow_tests.md](./22_v2_iterative_shadow_tests.md) + +## User Journeys + +### Journey 1: Application Engineer Running a Simple Schema Migration + +An application engineer needs to add a new filterable metadata field to an existing index without deleting documents. They run `plan`, review the merged target schema and downtime warning, schedule a maintenance window, run `apply`, then run `validate` and hand the migration report to the team. They do not need to understand Redis internals beyond the migration inputs and the reported downtime. + +### Journey 2: Platform Engineer Reviewing a Vector Precision Migration + +A platform engineer needs to review a planned `FP32 -> FP16` migration for a large production index. They supply platform inventory, review the planner’s peak-overlap estimate, compare the projected post-cutover memory savings to previous benchmark reports, and decide whether the current deployment can run the migration safely in the next window. + +### Journey 3: Engineer Migrating from `HNSW` to `FLAT` + +An engineer wants to switch vector search behavior from `HNSW` to `FLAT` to simplify runtime performance characteristics. The planner classifies the request as a Phase 2 shadow migration, estimates the target index footprint, and produces a one-index-at-a-time plan. The operator runs the migration, validates the shadow target, and cuts traffic over once the benchmark and validation reports look acceptable. + +### Journey 4: Solutions Engineer Validating a Payload-Shape Change + +A solutions engineer wants to understand how long a customer migration will take when a new embedding model changes the stored payload shape. They create a `transform_plan.yaml`, run a rehearsal in non-production, collect benchmark timing, throughput, query-latency, and source-versus-target memory outputs, and use those artifacts to advise on maintenance windows and scaling needs. + +## User Stories + +- As an application engineer, I want to generate a migration plan before any mutation so that I can review the exact schema changes and downtime implications. +- As an application engineer, I want the Phase 1 migrator to preserve documents so that I do not have to rebuild my dataset from another source. +- As an application engineer, I want blocked Phase 1 schema changes to fail early and point me to the correct Phase 2 path so that I do not start a migration the product cannot safely complete. +- As an operator, I want migration and validation reports in YAML so that I can review, archive, and share them with other teams. +- As an operator, I want the CLI to require explicit downtime acknowledgment in Phase 1 so that maintenance-window risk is never implicit. +- As a platform engineer, I want Phase 2 to process one index at a time so that capacity planning stays understandable and bounded. +- As a platform engineer, I want the planner to estimate peak overlap and post-cutover memory deltas so that I can decide whether a migration fits safely. +- As a platform engineer, I want the shadow planner to return `READY`, `SCALE_REQUIRED`, or `MANUAL_REVIEW_REQUIRED` so that I can make a clear operational decision before execution. +- As a solutions engineer, I want benchmark outputs for duration, throughput, query impact, and memory change so that I can estimate future migrations with real evidence. +- As a maintainer, I want the migration product to reuse existing RedisVL primitives so that implementation and long-term maintenance stay simple. diff --git a/nitin_docs/index_migrator/99_tickets.md b/nitin_docs/index_migrator/99_tickets.md new file mode 100644 index 000000000..6d7a31049 --- /dev/null +++ b/nitin_docs/index_migrator/99_tickets.md @@ -0,0 +1,374 @@ +# Index Migrator Tickets + +--- + +## Milestones + +| Milestone | Theme | Stories | +|-----------|-------|---------| +| M1 | Plan and Execute Single-Index Schema Migrations | IM-01, IM-05 | +| M2 | Interactive Migration Wizard | IM-02 | +| M3 | Rename Indexes, Prefixes, and Fields | IM-06 | +| M4 | Async Execution and Batch Operations | IM-03, IM-04 | +| M5 | Validation Fixes, Integration Tests, and Documentation | IM-07, IM-08, IM-09, IM-10 | + +--- + +## Completed + +### IM-01: Plan, Execute, and Validate Document-Preserving Index Schema Migrations + +**Status:** Done | **Commit:** `a3d534b` | **Milestone:** M1 + +**Story:** As a developer with an existing Redis index, I want to generate a reviewable migration plan, execute a safe drop-and-recreate, and validate the result, so that I can add/remove fields, change vector algorithms (FLAT/HNSW/SVS-VAMANA), change distance metrics (cosine/L2/IP), quantize vectors (float32 to float16/bfloat16/int8/uint8), and tune HNSW parameters (m, ef_construction, ef_runtime, epsilon) — all without losing documents. + +**What This Delivers:** +- **Discovery**: `rvl migrate list` shows all indexes, `rvl migrate helper` explains capabilities +- **Planning**: MigrationPlanner generates a plan from a schema patch or target schema. Captures source snapshot, target schema, classifies changes as supported or blocked. Incompatible changes (dimension, storage type) are rejected at plan time. +- **Execution**: MigrationExecutor drops the index definition (not documents), re-encodes vectors if quantization is needed, and recreates the index with the merged schema. +- **Validation**: MigrationValidator confirms schema match, doc count parity, key sample existence, and functional query correctness post-migration. +- **Reporting**: Structured `migration_report.yaml` with per-phase timings, counts, benchmark summary, and warnings. + +**Key Files:** `redisvl/migration/planner.py`, `executor.py`, `validation.py`, `models.py` + +--- + +### IM-02: Build Migration Plans Interactively via Guided Wizard + +**Status:** Done | **Commit:** `b06e949` | **Milestone:** M2 + +**Story:** As a developer unfamiliar with YAML schema syntax, I want a menu-driven wizard that walks me through adding, removing, updating, and renaming fields with real-time validation, so that I can build a correct migration plan without reading documentation. + +**What This Delivers:** +- `rvl migrate wizard --index ` launches an interactive session +- Menus for: add field (text/tag/numeric/geo), remove field (any type, with vector warning), rename field, update field attributes (sortable, weight, no_stem, phonetic_matcher, separator, case_sensitive, index_missing, index_empty), update vector settings (algorithm, datatype, distance metric, all HNSW and SVS-VAMANA params), rename index, change prefix +- Shows current schema and previews changes before generating plan +- Outputs both `schema_patch.yaml` and `migration_plan.yaml` +- Validates choices against what's actually supported + +**Key Files:** `redisvl/migration/wizard.py` + +--- + +### IM-03: Execute Migrations Asynchronously for Large Indexes + +**Status:** Done | **Commit:** `b559215` | **Milestone:** M4 + +**Story:** As a developer with a large index (1M+ vectors) in an async codebase, I want async migration planning, execution, and validation so that my application remains responsive and I don't block the event loop during long-running migrations. + +**What This Delivers:** +- `AsyncMigrationPlanner`, `AsyncMigrationExecutor`, `AsyncMigrationValidator` classes with full feature parity +- `rvl migrate apply --async` CLI flag +- Same `MigrationPlan` model works for both sync and async +- Same plan format works for both sync and async + +**Key Files:** `redisvl/migration/async_planner.py`, `async_executor.py`, `async_validation.py` + +--- + +### IM-04: Migrate Multiple Indexes in a Single Batch with Failure Isolation and Resume + +**Status:** Done | **Commit:** `61c6e80` | **Milestone:** M4 + +**Story:** As a platform operator with many indexes, I want to apply a shared schema patch to multiple indexes in one operation, choose whether to stop or continue on failure, and resume interrupted batches from a checkpoint, so that I can coordinate migrations during maintenance windows. + +**What This Delivers:** +- `BatchMigrationPlanner` generates per-index plans from a shared patch +- `BatchMigrationExecutor` runs migrations sequentially with state persistence +- Failure policies: `fail_fast` (stop on first error), `continue_on_error` (skip and continue) +- CLI: `batch-plan`, `batch-apply`, `batch-resume`, `batch-status` +- `batch_state.yaml` checkpoint file for resume capability +- `BatchReport` with per-index status and aggregate summary + +**Key Files:** `redisvl/migration/batch_planner.py`, `batch_executor.py` + +--- + +### IM-05: Optimize Document Enumeration Using FT.AGGREGATE Cursors + +**Status:** Done | **Commit:** `9561094` | **Milestone:** M1 + +**Story:** As a developer migrating a large index over a sparse keyspace, I want document enumeration to use the search index directly instead of SCAN, so that migration runs faster and only touches indexed keys. +``` +FT.AGGREGATE idx "*" + LOAD 1 __key # Get document key + WITHCURSOR COUNT 500 # Cursor-based pagination +``` + +**What This Delivers:** +- Executor uses `FT.AGGREGATE ... WITHCURSOR COUNT LOAD 0` for key enumeration +- Falls back to SCAN only when `hash_indexing_failures > 0` (those docs wouldn't appear in aggregate) +- Pre-enumerates all keys before dropping index for reliable re-indexing +- CLI simplified: removed `--allow-downtime` flag (plan review is the safety mechanism) + +**Key Files:** `redisvl/migration/executor.py`, `async_executor.py` + +--- + +### IM-06: Rename Indexes, Change Key Prefixes, and Rename Fields Across Documents + +**Status:** Done | **Commit:** pending | **Milestone:** M3 + +**Story:** As a developer, I want to rename my index, change its key prefix, or rename fields in my schema, so that I can refactor naming conventions without rebuilding from scratch. + +**What This Delivers:** +- Index rename: drop old index, create new with same prefix (no document changes) +- Prefix change: `RENAME` command on every key (single-prefix indexes only, multi-prefix blocked) +- Field rename: `HSET`/`HDEL` for hash, `JSON.SET`/`JSON.DEL` for JSON, on every document +- Execution order: field renames, then key renames, then drop, then recreate +- `RenameOperations` model in migration plan +- Timing fields: `field_rename_duration_seconds`, `key_rename_duration_seconds` +- Warnings issued for expensive operations + +**Key Files:** `redisvl/migration/models.py`, `planner.py`, `executor.py`, `async_executor.py` + +**Spec:** `nitin_docs/index_migrator/30_rename_operations_spec.md` + +--- + +### IM-07: Fix HNSW Parameter Parsing, Weight Normalization, and Algorithm Case Sensitivity + +**Status:** Done | **Commit:** `ab8a017` | **Milestone:** M5 + +**Story:** As a developer, I want post-migration validation to correctly handle HNSW-specific parameters, weight normalization, and algorithm case sensitivity, so that validation doesn't produce false failures. + +**What This Fixes:** +- HNSW-specific parameters (m, ef_construction) were not being parsed from `FT.INFO`, causing validation failures +- Weight int/float normalization mismatch (schema defines `1`, Redis returns `1.0`) +- Algorithm case sensitivity in wizard (schema stores `'hnsw'`, wizard compared to `'HNSW'`) + +**Key Files:** `redisvl/redis/connection.py`, `redisvl/migration/utils.py`, `redisvl/migration/wizard.py` + +--- + +### IM-08: Add Integration Tests for All Supported Migration Routes + +**Status:** Done | **Commit:** `b3d88a0` | **Milestone:** M5 + +**Story:** As a maintainer, I want integration tests covering algorithm changes, quantization, distance metrics, HNSW tuning, and combined migrations, so that regressions are caught before release. + +**What This Delivers:** +- 22 integration tests running full apply+validate against a live Redis instance +- Covers: 9 datatype routes, 4 distance metric routes, 5 HNSW tuning routes, 2 algorithm routes, 2 combined routes +- Tests require Redis 8.0+ for INT8/UINT8 datatypes +- Located in `tests/integration/test_migration_routes.py` + +--- + +### IM-09: Update Migration Documentation to Reflect Rename, Batch, and Redis 8.0 Support + +**Status:** Done | **Commit:** `d452eab` | **Milestone:** M5 + +**Story:** As a user, I want documentation that accurately reflects all supported migration operations, so that I can self-serve without guessing at capabilities. + +**What This Delivers:** +- Updated `docs/concepts/index-migrations.md` to reflect prefix/field rename support +- Updated `docs/user_guide/how_to_guides/migrate-indexes.md` with Redis 8.0 requirements +- Added batch migration commands to CLI reference in `docs/user_guide/cli.ipynb` +- Removed prefix/field rename from "blocked" lists + +--- + +### IM-10: Address PR Review Feedback for Correctness and Consistency + +**Status:** Done | **Commit:** pending | **Milestone:** M5 + +**Story:** As a maintainer, I want code review issues addressed so that the migration engine is correct, consistent, and production-ready. + +**What This Fixes:** +- `merge_patch()` now applies `rename_fields` to merged schema +- `BatchState.success_count` uses correct status string (`"succeeded"`) +- CLI helper text updated to show prefix/rename as supported +- Planner docstring updated to reflect current capabilities +- `batch_plan_path` stored in state for proper resume support +- Fixed `--output` to `--plan-out` in batch migration docs +- Fixed `--indexes` docs to use comma-separated format +- Added validation to block multi-prefix migrations +- Updated migration plan YAML example to match actual model +- Added `skipped_count` property and `[SKIP]` status display + +**Key Files:** `redisvl/migration/planner.py`, `models.py`, `batch_executor.py`, `redisvl/cli/migrate.py`, `docs/user_guide/how_to_guides/migrate-indexes.md` + +--- + +## Pending / Future + +### IM-R1: Add Crash-Safe Quantization with Checkpoint Resume and Pre-Migration Snapshot + +**Status:** Done | **Commit:** `30cc6c1` | **Priority:** High + +**Story:** As a developer running vector quantization on a production index, I want the migration to be resumable if it crashes mid-quantization, so that I don't end up with a partially quantized index and no rollback path. + +**Problem:** +The current quantization flow is: enumerate keys, drop index, quantize vectors in-place, recreate index, validate. If the process crashes during quantization, you're left with no index, a mix of float32 and float16 vectors, and no way to recover. + +**What This Delivers:** +A four-layer reliability model. A pre-migration `BGSAVE` (run sequentially, waited to completion) provides full disaster recovery by restoring the RDB to pre-migration state. A checkpoint file on disk tracks which keys have been quantized, enabling resume from the exact failure point on retry. Each key conversion detects the vector dtype before converting, making it idempotent so already-converted keys are safely skipped on resume. A bounded undo buffer stores originals for only the current in-flight batch, allowing rollback of the batch that was in progress at crash time. + +**Acceptance Criteria:** +1. Pre-migration `BGSAVE` is triggered and completes before any mutations begin +2. A checkpoint file records progress as each batch of keys is quantized +3. `rvl migrate apply --resume` picks up from the last checkpoint and completes the migration +4. Each key conversion is idempotent -- running the migration twice on the same key produces the correct result +5. If a batch fails mid-write, only that batch's vectors are rolled back using the bounded undo buffer +6. A disk space estimator function calculates projected RDB snapshot size, AOF growth, and total new disk required based on doc count, vector dimensions, source/target dtype, and AOF status. The estimator runs before any mutations and prints a human-readable summary. If available disk is below 80% of the estimate, the CLI prompts for confirmation. The estimator also supports a standalone dry-run mode via `rvl migrate estimate --plan plan.yaml`. See `nitin_docs/index_migrator/40_reliability_brainstorm.md` section "Pre-Migration Disk Space Estimator" for the full specification including inputs, outputs (DiskSpaceEstimate dataclass), calculation logic, CLI output format, integration points, and edge cases. + +**Alternatives Considered:** Undo log (WAL-style), new-field-then-swap (side-write), shadow index (blue-green), streaming with bounded undo buffer. See `nitin_docs/index_migrator/40_reliability_brainstorm.md` for full analysis. + +--- + +### IM-B1: Benchmark Float32 vs Float16 Quantization: Search Quality and Migration Performance at Scale + +**Status:** Planned | **Priority:** High + +**Story:** As a developer considering vector quantization to reduce memory, I want benchmarks measuring search quality degradation (precision, recall, F1) and migration performance (throughput, latency, memory savings) across realistic dataset sizes, so that I can make an informed decision about whether the memory-accuracy tradeoff is acceptable for my use case. + +**Problem:** +We tell users they can quantize float32 vectors to float16 to cut memory in half, but we don't have published data showing what they actually lose in search quality or what they can expect in migration performance at different scales. + +**What This Delivers:** +A benchmark script and published results using a real dataset (AG News with sentence-transformers embeddings) that measures two things across multiple dataset sizes (1K, 10K, 100K). For search quality: precision@K, recall@K, and F1@K comparing float32 (ground truth) vs float16 (post-migration) top-K nearest neighbor results. For migration performance: end-to-end duration, quantization throughput (vectors/second), index downtime, pre/post memory footprint, and query latency before and after (p50, p95, p99). + +**Acceptance Criteria:** +1. Benchmark runs end-to-end against a local Redis instance with a single command +2. Uses a real public dataset with real embeddings (not synthetic random vectors) +3. Reports precision@K, recall@K, and F1@K for float32 vs float16 search results +4. Reports per-query statistics (mean, p50, p95, min, max) not just aggregates +5. Runs at multiple dataset sizes (at minimum 1K, 10K, 100K) to show how quality and performance scale +6. Reports memory savings (index size delta in MB) and migration throughput (docs/second) +7. Reports query latency before and after migration +8. Outputs a structured JSON report that can be compared across runs + +**Note:** Benchmark script scaffolded at `tests/benchmarks/index_migrator_real_benchmark.py`. + +--- + +### IM-11: Run Old and New Indexes in Parallel for Incompatible Changes with Operator-Controlled Cutover + +**Status:** Future | **Priority:** Medium + +**Story:** As a developer changing vector dimensions or storage type, I want to run old and new indexes in parallel until I'm confident in the new one, so that I can migrate without downtime and rollback if needed. + +**Context:** +Some migrations cannot use `drop_recreate` because the stored data is incompatible (dimension changes, storage type changes, complex payload restructuring). Shadow migration creates a new index alongside the old one, copies/transforms documents, validates, then hands off cutover to the operator. + +**What This Requires:** +- Capacity estimation (can Redis hold both indexes?) +- Shadow index creation +- Document copy with optional transform +- Progress tracking with resume +- Validation gate before cutover +- Operator handoff for cutover decision +- Cleanup of old index/keys after cutover + +**Spec:** `nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md` + +--- + +### IM-12: Pipeline Vector Reads During Quantization to Reduce Round Trips on Large Datasets + +**Status:** Backlog | **Priority:** Low + +**Story:** As a developer migrating large datasets, I want quantization reads to be pipelined so that migration completes faster. + +**Context:** +Current quantization implementation does O(N) round trips for reads (one `HGET` per key/field) while only pipelining writes. For large datasets this is slow. + +**What This Requires:** +- Pipeline all reads in a batch before processing +- Use `transaction=False` for read pipeline +- Add JSON storage support (`JSON.GET`/`JSON.SET`) for JSON indexes + +--- + +### IM-13: Wire ValidationPolicy Enforcement into Validators or Remove the Unused Model + +**Status:** Backlog | **Priority:** Low + +**Story:** As a developer, I want to skip certain validation checks (e.g., doc count) when I know they'll fail due to expected conditions. + +**Context:** +`MigrationPlan.validation` (ValidationPolicy) exists in the model but is not enforced by validators. Schema/doc-count mismatches always produce errors. + +**What This Requires:** +- Wire `ValidationPolicy.require_doc_count_match` into validators +- Add CLI flag to set policy during plan creation +- Or remove unused ValidationPolicy model + +--- + +### IM-14: Clean Up Unused Imports and Linting Across the Codebase + +**Status:** Backlog | **Priority:** Low + +**Story:** As a maintainer, I want clean linting so that CI is reliable and code quality is consistent. + +**Context:** +During development, pyflakes identified unused imports across the codebase. These were fixed in migration files but not committed for non-migration files to keep the PR focused. + +**What This Requires:** +- Fix remaining unused imports (see `nitin_docs/issues/unused_imports_cleanup.md`) +- Update `.pylintrc` to remove deprecated Python 2/3 compat options +- Consider adding `check-lint` to the main `lint` target after cleanup + +--- + +### IM-15: Use RENAMENX for Prefix Migrations to Fail Fast on Key Collisions + +**Status:** Backlog | **Priority:** Low + +**Story:** As a developer changing key prefixes, I want the migration to fail fast if target keys already exist, so I don't end up with a partially renamed keyspace. + +**Context:** +Current implementation uses `RENAME` without checking if destination key exists. If a target key exists, RENAME will error and the pipeline may abort, leaving a partially-renamed keyspace. + +**What This Requires:** +- Preflight check for key collisions or use `RENAMENX` +- Surface hard error rather than warning +- Consider rollback strategy + +--- + +### IM-16: Auto-Detect AOF Status for Disk Space Estimation + +**Status:** Backlog | **Priority:** Low + +**Story:** As an operator running `rvl migrate estimate`, I want the disk space estimate to automatically detect whether AOF is enabled on the target Redis instance, so that AOF growth is included in the estimate without me needing to know or pass a flag. + +**Context:** +The disk space estimator (`estimate_disk_space`) is a pure calculation that accepts `aof_enabled` as a parameter (default `False`). In CLI usage, this means AOF growth is never estimated unless the caller explicitly passes `aof_enabled=True`. The summary currently prints "not estimated (pass aof_enabled=True if AOF is on)" which is accurate but requires the operator to know their Redis config. + +**What This Requires:** +- Add `--aof-enabled` flag to `rvl migrate estimate` CLI for offline/pure-calculation use +- During `rvl migrate apply`, read `CONFIG GET appendonly` from the live Redis connection and pass the result to `estimate_disk_space` +- Handle `CONFIG GET` failures gracefully (e.g. ACL restrictions) by falling back to the current "not estimated" behavior + +--- + +## Summary + +| Ticket | Title | Status | +|--------|-------|--------| +| IM-01 | Plan, Execute, and Validate Document-Preserving Index Schema Migrations | Done | +| IM-02 | Build Migration Plans Interactively via Guided Wizard | Done | +| IM-03 | Execute Migrations Asynchronously for Large Indexes | Done | +| IM-04 | Migrate Multiple Indexes in a Single Batch with Failure Isolation and Resume | Done | +| IM-05 | Optimize Document Enumeration Using FT.AGGREGATE Cursors | Done | +| IM-06 | Rename Indexes, Change Key Prefixes, and Rename Fields Across Documents | Done | +| IM-07 | Fix HNSW Parameter Parsing, Weight Normalization, and Algorithm Case Sensitivity | Done | +| IM-08 | Add Integration Tests for All Supported Migration Routes | Done | +| IM-09 | Update Migration Documentation to Reflect Rename, Batch, and Redis 8.0 Support | Done | +| IM-10 | Address PR Review Feedback for Correctness and Consistency | Done | +| IM-R1 | Add Crash-Safe Quantization with Checkpoint Resume and Pre-Migration Snapshot | Done | +| IM-B1 | Benchmark Float32 vs Float16 Quantization: Search Quality and Migration Performance at Scale | Planned | +| IM-11 | Run Old and New Indexes in Parallel for Incompatible Changes with Operator-Controlled Cutover | Future | +| IM-12 | Pipeline Vector Reads During Quantization to Reduce Round Trips on Large Datasets | Backlog | +| IM-13 | Wire ValidationPolicy Enforcement into Validators or Remove the Unused Model | Backlog | +| IM-14 | Clean Up Unused Imports and Linting Across the Codebase | Backlog | +| IM-15 | Use RENAMENX for Prefix Migrations to Fail Fast on Key Collisions | Backlog | +| IM-16 | Auto-Detect AOF Status for Disk Space Estimation | Backlog | +| IM-17 | Guard Against Connection Leaks in Long-Running Batch Migrations | Backlog | +| IM-18 | Optimize O(n^2) Checkpoint Serialization for Large Key Sets | Backlog | +| IM-19 | Add Redis Cluster Slot-Aware Key Distribution for Quantization Batches | Backlog | +| IM-20 | Add Pipelined Reads for Quantization to Reduce Per-Key Round Trips | Backlog | + diff --git a/nitin_docs/index_migrator/pr_comments.md b/nitin_docs/index_migrator/pr_comments.md new file mode 100644 index 000000000..db8f14544 --- /dev/null +++ b/nitin_docs/index_migrator/pr_comments.md @@ -0,0 +1,700 @@ +The case sensitivity - maybe use an enum or something else? + +Check these if they are true and useful as well. We dont want to implement everything, only if it makes sense. and it also makes errors, so go through them and before implementing give me an of how important and necessary they are: + +redisvl/migration/planner.py +Comment on lines +24 to +38 + """Migration planner for document-preserving drop/recreate flows. + + The `drop_recreate` mode drops the index definition and recreates it with + a new schema. Documents remain untouched in Redis. + + This means: + - Index-only changes work (algorithm, distance metric, tuning params) + - Document-dependent changes fail (the index expects data in a format + that doesn't match what's stored) + + Document-dependent changes (not supported): + - Vector dimensions: stored vectors have wrong number of dimensions + - Prefix/keyspace: documents are at keys the new index won't scan + - Field rename: documents store data under the old field name + - Storage type: documents are in hash format but index expects JSON +Copilot AI +27 minutes ago +The class docstring still claims prefix/keyspace changes and field renames are “not supported”, but the planner/executor now explicitly support prefix changes and rename_fields. Please update this docstring (and any related user-facing docs) to reflect the current supported operations so users don’t incorrectly avoid supported migrations. + +Suggested change + """Migration planner for document-preserving drop/recreate flows. + The `drop_recreate` mode drops the index definition and recreates it with + a new schema. Documents remain untouched in Redis. + This means: + - Index-only changes work (algorithm, distance metric, tuning params) + - Document-dependent changes fail (the index expects data in a format + that doesn't match what's stored) + Document-dependent changes (not supported): + - Vector dimensions: stored vectors have wrong number of dimensions + - Prefix/keyspace: documents are at keys the new index won't scan + - Field rename: documents store data under the old field name + - Storage type: documents are in hash format but index expects JSON + """Migration planner for drop/recreate-based index migrations. + The `drop_recreate` mode drops the index definition and recreates it with + a new schema. By default, documents are preserved in Redis, and when + possible the planner/executor can apply transformations so the preserved + documents remain compatible with the new index schema. + This means: + - Index-only changes are always safe (algorithm, distance metric, tuning + params, etc.) + - Some document-dependent changes are supported via explicit migration + operations in the migration plan + - Other document-dependent changes remain unsupported because existing + data cannot be safely or automatically adapted + Currently supported document-dependent changes include: + - Prefix/keyspace changes when specified in the target schema/patch + - Field renames via `rename_fields` operations in a schema patch + Document-dependent changes that remain unsupported: + - Vector dimensions: stored vectors have the wrong number of dimensions + for the new schema + - Storage type: documents are in hash format but the new index expects JSON +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/planner.py + fields_by_name = { + field["name"]: deepcopy(field) for field in schema_dict["fields"] + } + +Copilot AI +27 minutes ago +merge_patch() applies add/remove/update changes but ignores schema_patch.changes.rename_fields. The wizard adds renames via rename_fields, so the merged target schema will keep the old field name even though the executor renames document fields—leading to an index built with the wrong field name. Apply rename_fields during merge (with collision checks) or have the wizard encode renames as remove+add in the patch. + +Suggested change + # Apply field renames from the patch before other modifications. + # This ensures the merged schema's field names match the executor's + # renamed document fields. + for rename in getattr(changes, "rename_fields", []) or []: + # Be defensive about attribute names on FieldRename. + old_name = ( + getattr(rename, "from_name", None) + or getattr(rename, "old_name", None) + or getattr(rename, "source", None) + ) + new_name = ( + getattr(rename, "to_name", None) + or getattr(rename, "new_name", None) + or getattr(rename, "target", None) + ) + if not old_name or not new_name: + # As a fallback, inspect the model's dict representation. + rename_dict = ( + rename.dict() + if hasattr(rename, "dict") + else getattr(rename, "__dict__", {}) or {} + ) + old_name = old_name or rename_dict.get("from_name") or rename_dict.get("old_name") or rename_dict.get("source") + new_name = new_name or rename_dict.get("to_name") or rename_dict.get("new_name") or rename_dict.get("target") + if not old_name or not new_name: + raise ValueError(f"Invalid field rename specification: {rename!r}") + if old_name not in fields_by_name: + raise ValueError( + f"Cannot rename field '{old_name}' because it does not exist in the source schema" + ) + # If the new name already exists and is different from the source + # field, renaming would collide with another field. + if new_name in fields_by_name and new_name != old_name: + raise ValueError( + f"Cannot rename field '{old_name}' to '{new_name}' because a field with the new name already exists" + ) + if new_name == old_name: + # No-op rename; skip. + continue + field_def = fields_by_name.pop(old_name) + field_def["name"] = new_name + fields_by_name[new_name] = field_def +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/executor.py +Comment on lines +674 to +695 + total_keys = len(keys) + docs_processed = 0 + batch_size = 500 + + for i in range(0, total_keys, batch_size): + batch = keys[i : i + batch_size] + pipe = client.pipeline() + keys_updated_in_batch = set() + + for key in batch: + # Read all vector fields that need conversion + for field_name, change in datatype_changes.items(): + field_data: bytes | None = client.hget(key, field_name) # type: ignore[misc,assignment] + if field_data: + # Convert: source dtype -> array -> target dtype -> bytes + array = buffer_to_array(field_data, change["source"]) + new_bytes = array_to_buffer(array, change["target"]) + pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] + keys_updated_in_batch.add(key) + + if keys_updated_in_batch: + pipe.execute() +Copilot AI +27 minutes ago +Vector quantization reads each field via client.hget() inside the loop while also creating a pipeline for writes; this defeats pipelining and makes quantization O(N) round trips. Additionally, this assumes HASH storage; for JSON indexes the vector is not stored in a hash field, so quantization will be incorrect/no-op. Use pipelined HGET (or HMGET) for reads, set transaction=False, and branch on storage_type to use JSON.GET/SET for JSON storage. + +Suggested change + total_keys = len(keys) + docs_processed = 0 + batch_size = 500 + for i in range(0, total_keys, batch_size): + batch = keys[i : i + batch_size] + pipe = client.pipeline() + keys_updated_in_batch = set() + for key in batch: + # Read all vector fields that need conversion + for field_name, change in datatype_changes.items(): + field_data: bytes | None = client.hget(key, field_name) # type: ignore[misc,assignment] + if field_data: + # Convert: source dtype -> array -> target dtype -> bytes + array = buffer_to_array(field_data, change["source"]) + new_bytes = array_to_buffer(array, change["target"]) + pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] + keys_updated_in_batch.add(key) + if keys_updated_in_batch: + pipe.execute() + storage_type = getattr(source_index, "storage_type", "HASH").upper() + total_keys = len(keys) + docs_processed = 0 + batch_size = 500 + for i in range(0, total_keys, batch_size): + batch = keys[i : i + batch_size] + # First, pipeline all reads for this batch + read_pipe = client.pipeline(transaction=False) + key_field_pairs: list[tuple[str, str]] = [] + for key in batch: + # Read all vector fields that need conversion + for field_name in datatype_changes.keys(): + key_field_pairs.append((key, field_name)) + if storage_type == "JSON": + json_path = f"$.{field_name}" + read_pipe.execute_command("JSON.GET", key, json_path) + else: + # Default to HASH storage + read_pipe.hget(key, field_name) + if not key_field_pairs: + continue + read_results = read_pipe.execute() + # Now, process results and pipeline all writes for this batch + write_pipe = client.pipeline(transaction=False) + keys_updated_in_batch = set() + for (key, field_name), field_data in zip(key_field_pairs, read_results): + if not field_data: + continue + change = datatype_changes[field_name] + # Convert: source dtype -> array -> target dtype -> bytes + array = buffer_to_array(field_data, change["source"]) + new_bytes = array_to_buffer(array, change["target"]) + if storage_type == "JSON": + json_path = f"$.{field_name}" + write_pipe.execute_command("JSON.SET", key, json_path, new_bytes) + else: + write_pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] + keys_updated_in_batch.add(key) + if keys_updated_in_batch: + write_pipe.execute() +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/models.py +Comment on lines +203 to +206 + @property + def success_count(self) -> int: + return sum(1 for idx in self.completed if idx.status == "success") + +Copilot AI +27 minutes ago +BatchState.success_count counts entries with status == "success", but the executor records "succeeded" and the rest of the batch code checks for "succeeded". This makes CLI status reporting incorrect. Align the allowed status values (either change success_count to check "succeeded" or standardize statuses everywhere). + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/cli/migrate.py +Comment on lines +96 to +101 + +Not yet supported: + - Changing vector dimensions + - Changing key prefix or separator + - Changing storage type (hash to JSON) + - Renaming fields +Copilot AI +27 minutes ago +The CLI helper text lists prefix changes and field renames under “Not yet supported”, but the current planner/executor implement prefix changes and rename_fields. Update this help output (and any capability lists) to match the actual supported operations so users aren’t misled. + +Suggested change +Not yet supported: + - Changing vector dimensions + - Changing key prefix or separator + - Changing storage type (hash to JSON) + - Renaming fields + - Changing key prefix or separator + - Renaming fields +Not yet supported: + - Changing vector dimensions + - Changing storage type (hash to JSON) +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/concepts/index-migrations.md +Comment on lines +24 to +31 +| Tune algorithm parameters (M, EF_CONSTRUCTION) | Supported | +| Quantize vectors (float32 to float16) | Supported | +| Change vector dimensions | Blocked | +| Change key prefix | Blocked | +| Rename a field | Blocked | +| Change storage type (hash to JSON) | Blocked | +| Add a new vector field | Blocked | + +Copilot AI +27 minutes ago +This doc lists “Change key prefix” and “Rename a field” as blocked, but the implementation in this PR supports prefix changes and explicit field renames (with document rewriting). Please update the supported/blocked tables and the explanatory sections to match the actual behavior, or re-block these operations in code if they’re not intended to be supported yet. + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/user_guide/how_to_guides/migrate-indexes.md +rvl migrate batch-plan \ + --pattern "*_idx" \ + --schema-patch quantize_patch.yaml \ + --output batch_plan.yaml \ +Copilot AI +27 minutes ago +The batch CLI examples use flags/argument formats that don’t match the implemented argparse interface: batch-plan uses --plan-out (not --output), and --indexes is parsed as a single comma-separated string in code (space-separated values won’t work). Please adjust the examples to reflect the actual CLI syntax. + +Suggested change + --output batch_plan.yaml \ + --plan-out batch_plan.yaml \ +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/async_executor.py +Comment on lines +639 to +654 + pipe = client.pipeline() + keys_updated_in_batch: set[str] = set() + + for key in batch: + # Read all vector fields that need conversion + for field_name, change in datatype_changes.items(): + field_data: bytes | None = await client.hget(key, field_name) # type: ignore[misc,assignment] + if field_data: + # Convert: source dtype -> array -> target dtype -> bytes + array = buffer_to_array(field_data, change["source"]) + new_bytes = array_to_buffer(array, change["target"]) + pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] + keys_updated_in_batch.add(key) + + if keys_updated_in_batch: + await pipe.execute() +Copilot AI +27 minutes ago +Async quantization has the same issue as sync: it does await client.hget() per key/field (one round trip each) while only pipelining writes. This will be extremely slow for large datasets and still assumes HASH storage. Consider pipelining/batching reads as well (e.g., pipeline HGETs then a single execute), set transaction=False, and use JSON.GET/SET when storage_type is JSON. + +Suggested change + pipe = client.pipeline() + keys_updated_in_batch: set[str] = set() + for key in batch: + # Read all vector fields that need conversion + for field_name, change in datatype_changes.items(): + field_data: bytes | None = await client.hget(key, field_name) # type: ignore[misc,assignment] + if field_data: + # Convert: source dtype -> array -> target dtype -> bytes + array = buffer_to_array(field_data, change["source"]) + new_bytes = array_to_buffer(array, change["target"]) + pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] + keys_updated_in_batch.add(key) + if keys_updated_in_batch: + await pipe.execute() + # First, batch all reads using a non-transactional pipeline to avoid + # one round trip per HGET. + read_pipe = client.pipeline(transaction=False) + tasks_meta: List[tuple[str, str, Dict[str, Any]]] = [] + for key in batch: + # Read all vector fields that need conversion + for field_name, change in datatype_changes.items(): + read_pipe.hget(key, field_name) + tasks_meta.append((key, field_name, change)) + if not tasks_meta: + continue + read_results = await read_pipe.execute() + # Now, prepare a write pipeline for the updated fields. + write_pipe = client.pipeline() + keys_updated_in_batch: set[str] = set() + for (key, field_name, change), field_data in zip(tasks_meta, read_results): + # field_data may be None if the field is missing + if field_data: + # Convert: source dtype -> array -> target dtype -> bytes + array = buffer_to_array(field_data, change["source"]) + new_bytes = array_to_buffer(array, change["target"]) + write_pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] + keys_updated_in_batch.add(key) + if keys_updated_in_batch: + await write_pipe.execute() +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/executor.py +Comment on lines +220 to +237 + for key in batch: + # Compute new key name + if key.startswith(old_prefix): + new_key = new_prefix + key[len(old_prefix) :] + else: + # Key doesn't match expected prefix, skip + logger.warning( + f"Key '{key}' does not start with prefix '{old_prefix}'" + ) + continue + pipe.rename(key, new_key) + + try: + results = pipe.execute() + renamed += sum(1 for r in results if r is True or r == "OK") + except Exception as e: + logger.warning(f"Error in rename batch: {e}") + +Copilot AI +27 minutes ago +Key prefix migration uses RENAME without checking whether the destination key already exists. If a target key exists, RENAME will error and the pipeline will likely abort, leaving a partially-renamed keyspace. Consider preflight-checking for collisions (or using a safer strategy like RENAMENX + fail-fast), and surface a hard error rather than a warning to avoid inconsistent migrations. + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/validation.py +Comment on lines +42 to +45 + source_num_docs = int(plan.source.stats_snapshot.get("num_docs", 0) or 0) + target_num_docs = int(target_info.get("num_docs", 0) or 0) + validation.doc_count_match = source_num_docs == target_num_docs + +Copilot AI +27 minutes ago +MigrationPlan.validation (ValidationPolicy) isn’t used by MigrationValidator/AsyncMigrationValidator: schema/doc-count mismatches always produce errors regardless of the policy fields. This also makes docs like “skip count check” hard to implement. Either wire these flags into validation/error generation or remove the policy from the model until it’s enforced. + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/async_validation.py +Comment on lines +49 to +52 + source_num_docs = int(plan.source.stats_snapshot.get("num_docs", 0) or 0) + target_num_docs = int(target_info.get("num_docs", 0) or 0) + validation.doc_count_match = source_num_docs == target_num_docs + +Copilot AI +27 minutes ago +Like the sync validator, this ignores plan.validation (ValidationPolicy) and always treats schema/doc-count mismatches as errors. If ValidationPolicy is part of the plan contract, the async validator should respect it too (or the policy should be removed until enforced). + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/user_guide/how_to_guides/migrate-indexes.md +Outdated +Comment on lines +385 to +392 + +## What's Blocked + +| Change | Why | Workaround | +|--------|-----|------------| +| Change vector dimensions | Requires re-embedding | Re-embed with new model, reload data | +| Change prefix/keyspace | Documents at wrong keys | Create new index, reload data | +| Rename a field | Stored data uses old name | Create new index, reload data | +Copilot AI +27 minutes ago +This guide’s “What’s Blocked” table still lists prefix changes and field renames as blocked, but the implementation in this PR supports both (prefix changes via key renames; field renames via document updates). Please update this table (and any surrounding narrative) to reflect the current supported operations, including the operational risks/caveats. + +Suggested change +## What's Blocked +| Change | Why | Workaround | +|--------|-----|------------| +| Change vector dimensions | Requires re-embedding | Re-embed with new model, reload data | +| Change prefix/keyspace | Documents at wrong keys | Create new index, reload data | +| Rename a field | Stored data uses old name | Create new index, reload data | +| Change prefix/keyspace | ✅ | Uses key renames; coordinate carefully to avoid clients writing during migration | +| Rename a field | ✅ | Uses document updates; may be slow and resource-intensive on large datasets | +## What's Blocked +| Change | Why | Workaround | +|--------|-----|------------| +| Change vector dimensions | Requires re-embedding | Re-embed with new model, reload data | +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +tests/unit/test_async_migration_executor.py +Comment on lines +115 to +125 +@pytest.mark.asyncio +async def test_async_executor_validates_redis_url(): + """Test executor requires redis_url or redis_client.""" + plan = _make_basic_plan() + executor = AsyncMigrationExecutor() + + # The executor should raise an error internally when trying to connect + # but let's verify it doesn't crash before it tries to apply + # For a proper test, we'd need to mock AsyncSearchIndex.from_existing + # For now, we just verify the executor is created + assert executor is not None +Copilot AI +27 minutes ago +test_async_executor_validates_redis_url doesn’t actually validate behavior (it only asserts the executor is instantiated). Either mock the Redis connection layer and assert the expected error/report when neither redis_url nor redis_client is provided, or rename/remove this test to avoid giving a false sense of coverage. + +Copilot uses AI. Check for mistakes. + +redisvl/migration/models.py +Comment on lines +71 to +77 +class RenameOperations(BaseModel): + """Tracks which rename operations are required for a migration.""" + + rename_index: Optional[str] = None # New index name if renaming + change_prefix: Optional[str] = None # New prefix if changing + rename_fields: List[FieldRename] = Field(default_factory=list) + +Copilot AI +13 minutes ago +RenameOperations.change_prefix is typed as Optional[str], but IndexSchema.index.prefix supports Union[str, List[str]]. As implemented elsewhere (e.g., using prefixes[0]), prefix-change migrations for multi-prefix indexes will behave incorrectly or partially. Either explicitly block list-prefix migrations in the planner/validator, or extend rename_operations/change_prefix handling to support lists end-to-end. + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/batch_executor.py +Comment on lines +72 to +77 + # Initialize or load state + state = self._init_or_load_state(batch_plan, state_path) + started_at = state.started_at + batch_start_time = time.perf_counter() + + # Get applicable indexes +Copilot AI +13 minutes ago +_init_or_load_state initializes BatchState.plan_path to an empty string, but resume() defaults to using state.plan_path when batch_plan_path is not provided. This breaks the advertised "resume without extra args" flow and causes FileNotFoundError unless users always pass --plan. Store the batch plan path in state when initializing (e.g., add a batch_plan_path parameter to apply() and persist it). + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/migration/async_validation.py +Comment on lines +67 to +78 + # Handle prefix change: transform key_sample to use new prefix + keys_to_check = key_sample + if plan.rename_operations.change_prefix: + old_prefix = plan.source.keyspace.prefixes[0] + new_prefix = plan.rename_operations.change_prefix + keys_to_check = [ + new_prefix + k[len(old_prefix) :] if k.startswith(old_prefix) else k + for k in key_sample + ] + existing_count = await client.exists(*keys_to_check) + validation.key_sample_exists = existing_count == len(keys_to_check) + +Copilot AI +13 minutes ago +Key-sample validation assumes a single old prefix (prefixes[0]) when applying a prefix change. For multi-prefix indexes (IndexSchema supports prefix as a list), this check can produce false failures or miss missing keys. Either block list-prefix migrations or update the rewrite logic to handle all prefixes. + +Suggested change + # Handle prefix change: transform key_sample to use new prefix + keys_to_check = key_sample + if plan.rename_operations.change_prefix: + old_prefix = plan.source.keyspace.prefixes[0] + new_prefix = plan.rename_operations.change_prefix + keys_to_check = [ + new_prefix + k[len(old_prefix) :] if k.startswith(old_prefix) else k + for k in key_sample + ] + existing_count = await client.exists(*keys_to_check) + validation.key_sample_exists = existing_count == len(keys_to_check) + # Default to False; will be updated only if the check runs successfully. + validation.key_sample_exists = False + # Handle prefix change: transform key_sample to use new prefix + keys_to_check = key_sample + multi_prefix_unsupported = False + if plan.rename_operations.change_prefix: + prefixes = plan.source.keyspace.prefixes or [] + if len(prefixes) == 1: + old_prefix = prefixes[0] + new_prefix = plan.rename_operations.change_prefix + keys_to_check = [ + new_prefix + k[len(old_prefix) :] + if k.startswith(old_prefix) + else k + for k in key_sample + ] + else: + # IndexSchema supports multiple prefixes, but change_prefix assumes a single + # old prefix. To avoid incorrect key rewriting, we skip this check. + multi_prefix_unsupported = True + validation.errors.append( + "Key-sample existence check does not support change_prefix when " + "source.keyspace.prefixes has multiple entries." + ) + if not multi_prefix_unsupported: + existing_count = await client.exists(*keys_to_check) + validation.key_sample_exists = existing_count == len(keys_to_check) +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/user_guide/how_to_guides/migrate-indexes.md +rvl migrate batch-plan \ + --pattern "*_idx" \ + --schema-patch quantize_patch.yaml \ + --output batch_plan.yaml \ +Copilot AI +13 minutes ago +The batch migration examples use a non-existent --output flag. The CLI implements --plan-out for batch-plan (and uses --plan for batch-apply). Update the docs to match the actual CLI flags to avoid copy/paste failures. + +Suggested change + --output batch_plan.yaml \ + --plan-out batch_plan.yaml \ +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/user_guide/how_to_guides/migrate-indexes.md +Comment on lines +557 to +564 +**Select indexes by explicit list:** +```bash +rvl migrate batch-plan \ + --indexes products_idx users_idx orders_idx \ + --schema-patch quantize_patch.yaml \ + --output batch_plan.yaml \ + --url redis://localhost:6379 +``` +Copilot AI +13 minutes ago +The docs show --indexes products_idx users_idx orders_idx, but the CLI parses --indexes as a comma-separated string (args.indexes.split(",")). Either update the CLI to accept nargs-style lists, or update the docs to use comma-separated values (e.g., --indexes "products_idx,users_idx,orders_idx"). + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/user_guide/how_to_guides/migrate-indexes.md +Comment on lines +209 to +214 +```yaml +# migration_plan.yaml (example) +version: 1 +index_name: myindex +migration_mode: drop_recreate + +Copilot AI +13 minutes ago +The migration plan YAML example does not match the actual MigrationPlan model emitted by planner.write_plan (e.g., it uses index_name/migration_mode/changes keys that aren't present). Replace the example with a real plan structure (mode/source/requested_changes/merged_target_schema/diff_classification/rename_operations/warnings/validation) to avoid confusing users. + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/user_guide/how_to_guides/migrate-indexes.md + +### Validation failed: "document count mismatch" + +Documents were added or removed between plan and apply. This is expected if your application is actively writing. Re-run validation with `--skip-count-check` if acceptable. +Copilot AI +13 minutes ago +The troubleshooting section references a --skip-count-check flag, but the CLI validate command doesn't define this option. Either add the flag and wire it to ValidationPolicy.require_doc_count_match, or update the docs to the supported behavior. + +Suggested change +Documents were added or removed between plan and apply. This is expected if your application is actively writing. Re-run validation with `--skip-count-check` if acceptable. +Documents were added or removed between plan and apply. This is expected if your application is actively writing. Wait until writes have quiesced, then re-run `rvl migrate validate`, or regenerate the migration plan and run `apply` and `validate` again. +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +docs/user_guide/how_to_guides/migrate-indexes.md +Comment on lines +548 to +555 +**Select indexes by pattern:** +```bash +rvl migrate batch-plan \ + --pattern "*_idx" \ + --schema-patch quantize_patch.yaml \ + --output batch_plan.yaml \ + --url redis://localhost:6379 +``` +Copilot AI +13 minutes ago +This section also uses --output for batch-plan, but the CLI flag is --plan-out. Update these examples (and the other --output occurrences in this doc) to match the implemented CLI options. + +Copilot uses AI. Check for mistakes. +@nkanu17 Reply... +redisvl/cli/migrate.py +Comment on lines +638 to +645 + print(f"Completed: {len(state.completed)}") + print(f" - Succeeded: {state.success_count}") + print(f" - Failed: {state.failed_count}") + + if state.completed: + print("\nCompleted indexes:") + for idx in state.completed: + status_icon = "[OK]" if idx.status == "succeeded" else "[FAIL]" +Copilot AI +13 minutes ago +batch-status output relies on BatchState.success_count and also treats anything other than status == "succeeded" as a failure. With the current model/executor status strings, skipped indexes will display as [FAIL], and success_count is also inconsistent (it counts "success"). Consider normalizing status enums and rendering skipped separately (e.g., [SKIP]). + +Suggested change + print(f"Completed: {len(state.completed)}") + print(f" - Succeeded: {state.success_count}") + print(f" - Failed: {state.failed_count}") + if state.completed: + print("\nCompleted indexes:") + for idx in state.completed: + status_icon = "[OK]" if idx.status == "succeeded" else "[FAIL]" + # Normalize and summarize completed index statuses locally to ensure + # consistent handling of success/skip/failure states, regardless of + # how BatchState aggregates counts. + success_statuses = {"succeeded", "success"} + skipped_statuses = {"skipped"} + succeeded = 0 + failed = 0 + skipped = 0 + for idx in state.completed: + if idx.status in success_statuses: + succeeded += 1 + elif idx.status in skipped_statuses: + skipped += 1 + else: + failed += 1 + print(f"Completed: {len(state.completed)}") + print(f" - Succeeded: {succeeded}") + print(f" - Failed: {failed}") + print(f" - Skipped: {skipped}") + if state.completed: + print("\nCompleted indexes:") + for idx in state.completed: + if idx.status in success_statuses: + status_icon = "[OK]" + elif idx.status in skipped_statuses: + status_icon = "[SKIP]" + else: + status_icon = "[FAIL]" +Copilot uses AI. Check for mistakes. + + +--- + +## Deferred Items from Review Rounds (Latest Triage) + +### Performance +- **Pipelined reads for quantization**: Both sync and async executors do per-key HGET reads. Should batch with pipelines for large datasets. +- **keys_to_process materialization**: `_enumerate_indexed_keys()` materializes full key list in memory. Consider streaming for large indexes. +- **Double snapshot in planner**: `create_plan()` snapshots source, then `create_plan_from_patch()` snapshots again. Reuse the initial snapshot. +- **Async executor duplicates readiness polling**: Has its own implementation instead of calling `async_wait_for_index_ready`. Consolidate. + +### Edge Cases +- **SCAN vs FT.AGGREGATE key mismatch on resume**: Initial run may enumerate via FT.AGGREGATE but resume uses SCAN, potentially producing different key sets. +- **Redis Cluster slot issues with prefix rename**: RENAME/RENAMENX only works when source/dest hash to same slot. +- **Empty prefix SCAN fallback**: When FT.INFO yields empty prefix, async executor scans entire keyspace without warning (sync version warns). +- **JSON path assumption in field renames**: Renames assume `$.` but non-default paths could exist. +- **DiskSpaceEstimate negative values**: Up-quantization (float16->float32) produces negative "savings" - `_format_bytes()` doesn't handle gracefully. +- **Batch plan stale state reuse**: `_init_or_load_state` doesn't validate state matches current batch plan on resume. +- **Batch duplicate index names**: No dedup in batch plan, later reports overwrite earlier ones. +- **list-valued prefix in wizard**: Wizard assumes single string prefix but schemas can have multiple. + +### UX/CLI +- **`_prompt_from_choices` block_message on any invalid input**: Shows vector-specific message on typos. +- **`_prompt_bool` empty input returns False**: Could be confusing with `[y/n]` prompt. +- **PR description API names don't match implementation**: `plan()` vs `create_plan()`, `execute()` vs `apply()`. +- **CLI `--ssl` URL construction bug**: Appends `rediss://` incorrectly. +- **CLI `rvl migrate` not in API docs**: `docs/api/cli.rst` doesn't document the new migrate command group. +- **Exit code contract inconsistency**: Some commands call `exit(0)` on exceptions. +- **Wizard `run()` doesn't validate Redis connection upfront**: Fails late in `snapshot_source()`. +- **Wizard numeric validation inconsistent**: `ef_runtime` requires `> 0` but `m`, `ef_construction` accept `0`. + +### Documentation +- **Migration plan YAML example doesn't match model**: Shows `mode`/`warnings` under `diff_classification` but model only has `supported`/`blocked_reasons`. +- **Source section missing fields in plan example**: Missing `index_name`, `storage_type`, `key_separator`. +- **"Other document dependent changes are blocked" text outdated**: Field renames and prefix renames are now supported. +- **Recovery guidance doesn't mention `--resume`**: Should document checkpoint-based resume workflow. + +### Testing +- **No e2e tests for executor rename/prefix mutation paths**: Integration tests cover basic drop/recreate but not data-mutation paths. +- **test_migration_routes doesn't load actual vector docs**: Quantization tests pass vacuously with empty indexes. +- **Wizard `run()` not unit tested**: Only `_build_patch()` is tested. +- **`create_mock_executor` `succeed_on` param unused**: Test helper is misleading. +- **Batch resume progress position restarts at 1**: Should compute position from completed count. +- **JSON schema `path` in test_comprehensive**: `path` is a top-level field property, not inside `attrs` - test may not validate actual JSON path behavior. + +### Code Quality +- **`ValidationPolicy` flags not respected**: `validate()` always errors on mismatches regardless of policy. +- **Checkpoint JSON storage not marked completed**: JSON storage creates checkpoint but never marks it completed. +- **`index_empty` prompt for non-applicable field types**: Collected for numeric/geo fields where it's not part of schema. + +### Round N+2 Codex Comments (2026-04-01) +- **executor.py:325 - Abort on hash field rename batch failure (P1)**: Currently logs warning and continues on `pipe.execute()` failure. Codex wants abort/raise. Deferred: current "best-effort" is intentional -- aborting mid-batch leaves partial renames with no rollback. Caller can compare `renamed` count vs expected. A proper fix needs retry/partial-rollback tracking. +- **batch_planner.py:174 - Resolve renamed fields before update-field checks (P2)**: Check rejects `update_fields` whose name is missing from source schema. Codex says this misses valid patches that rename A->B then update B. Near-false-positive: the standard pattern is to update A (by current name) AND rename A->B separately. The edge case of referencing post-rename name in `update_fields` is unlikely in practice. Deferred. \ No newline at end of file diff --git a/redisvl/migration/async_validation.py b/redisvl/migration/async_validation.py index 0f0ea66ae..a76d9e0ea 100644 --- a/redisvl/migration/async_validation.py +++ b/redisvl/migration/async_validation.py @@ -173,13 +173,17 @@ async def _run_functional_checks( passed = total_found == 0 else: passed = total_found > 0 + if expected_doc_count == 0: + detail_expectation = "expected 0" + else: + detail_expectation = f"expected >0, source had {expected_doc_count}" results.append( QueryCheckResult( name="functional:wildcard_search", passed=passed, details=( f"Wildcard search returned {total_found} docs " - f"(expected {expected_doc_count})" + f"({detail_expectation})" ), ) ) diff --git a/redisvl/migration/validation.py b/redisvl/migration/validation.py index 192216434..fcea94aac 100644 --- a/redisvl/migration/validation.py +++ b/redisvl/migration/validation.py @@ -155,13 +155,17 @@ def _run_functional_checks( passed = total_found == 0 else: passed = total_found > 0 + if expected_doc_count == 0: + detail_expectation = "expected 0" + else: + detail_expectation = f"expected >0, source had {expected_doc_count}" results.append( QueryCheckResult( name="functional:wildcard_search", passed=passed, details=( f"Wildcard search returned {total_found} docs " - f"(expected {expected_doc_count})" + f"({detail_expectation})" ), ) ) diff --git a/tests/integration/test_migration_comprehensive.py b/tests/integration/test_migration_comprehensive.py index 370546c68..759a1f626 100644 --- a/tests/integration/test_migration_comprehensive.py +++ b/tests/integration/test_migration_comprehensive.py @@ -93,7 +93,7 @@ def sample_docs(): "description": "First product description", "category": "electronics", "price": 99.99, - "location": "37.7749,-122.4194", # SF coordinates (lat,lon) + "location": "-122.4194,37.7749", # SF coordinates (lon,lat) "embedding": array_to_buffer([0.1, 0.2, 0.3, 0.4], "float32"), }, { @@ -102,7 +102,7 @@ def sample_docs(): "description": "Second service description", "category": "software", "price": 149.99, - "location": "40.7484,-73.9857", # NYC coordinates (lat,lon) + "location": "-73.9857,40.7484", # NYC coordinates (lon,lat) "embedding": array_to_buffer([0.2, 0.3, 0.4, 0.5], "float32"), }, { @@ -111,7 +111,7 @@ def sample_docs(): "description": "", # Empty for index_empty tests "category": "", # Empty for index_empty tests "price": 0, - "location": "34.0522,-118.2437", # LA coordinates (lat,lon) + "location": "-118.2437,34.0522", # LA coordinates (lon,lat) "embedding": array_to_buffer([0.3, 0.4, 0.5, 0.6], "float32"), }, ] From 0ecd5c9b946c67dd603e8eea1188a96d67b5cb83 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Thu, 2 Apr 2026 18:17:36 -0400 Subject: [PATCH 7/8] rename nitin_docs to local_docs --- CLAUDE.md | 2 +- {nitin_docs => local_docs}/index_migrator/00_index.md | 2 +- .../index_migrator/01_context.md | 0 .../index_migrator/02_architecture.md | 0 .../index_migrator/03_benchmarking.md | 0 .../index_migrator/04_implementation_summary.md | 0 .../index_migrator/05_migration_benchmark_report.md | 2 +- .../index_migrator/10_v1_drop_recreate_spec.md | 0 .../index_migrator/11_v1_drop_recreate_tasks.md | 0 .../index_migrator/12_v1_drop_recreate_tests.md | 2 +- .../index_migrator/20_v2_iterative_shadow_spec.md | 0 .../index_migrator/21_v2_iterative_shadow_tasks.md | 2 +- .../index_migrator/22_v2_iterative_shadow_tests.md | 0 {nitin_docs => local_docs}/index_migrator/90_prd.md | 0 .../index_migrator/99_tickets.md | 10 +++++----- .../index_migrator/pr_comments.md | 0 tests/integration/test_migration_comprehensive.py | 2 +- 17 files changed, 11 insertions(+), 11 deletions(-) rename {nitin_docs => local_docs}/index_migrator/00_index.md (98%) rename {nitin_docs => local_docs}/index_migrator/01_context.md (100%) rename {nitin_docs => local_docs}/index_migrator/02_architecture.md (100%) rename {nitin_docs => local_docs}/index_migrator/03_benchmarking.md (100%) rename {nitin_docs => local_docs}/index_migrator/04_implementation_summary.md (100%) rename {nitin_docs => local_docs}/index_migrator/05_migration_benchmark_report.md (99%) rename {nitin_docs => local_docs}/index_migrator/10_v1_drop_recreate_spec.md (100%) rename {nitin_docs => local_docs}/index_migrator/11_v1_drop_recreate_tasks.md (100%) rename {nitin_docs => local_docs}/index_migrator/12_v1_drop_recreate_tests.md (98%) rename {nitin_docs => local_docs}/index_migrator/20_v2_iterative_shadow_spec.md (100%) rename {nitin_docs => local_docs}/index_migrator/21_v2_iterative_shadow_tasks.md (99%) rename {nitin_docs => local_docs}/index_migrator/22_v2_iterative_shadow_tests.md (100%) rename {nitin_docs => local_docs}/index_migrator/90_prd.md (100%) rename {nitin_docs => local_docs}/index_migrator/99_tickets.md (98%) rename {nitin_docs => local_docs}/index_migrator/pr_comments.md (100%) diff --git a/CLAUDE.md b/CLAUDE.md index 4af36381f..9f22e9b93 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -49,7 +49,7 @@ index = SearchIndex(schema, redis_url="redis://localhost:6379") ``` ### Protected Directories -**CRITICAL**: NEVER delete the `nitin_docs/` directory or any files within it. +**CRITICAL**: NEVER delete the `local_docs/` directory or any files within it. ### Git Operations **CRITICAL**: NEVER use `git push` or attempt to push to remote repositories. The user will handle all git push operations. diff --git a/nitin_docs/index_migrator/00_index.md b/local_docs/index_migrator/00_index.md similarity index 98% rename from nitin_docs/index_migrator/00_index.md rename to local_docs/index_migrator/00_index.md index 4512315d1..6703cf99a 100644 --- a/nitin_docs/index_migrator/00_index.md +++ b/local_docs/index_migrator/00_index.md @@ -67,7 +67,7 @@ For Phase 2 planning: ## Locked Decisions -- The planning workspace lives entirely under `nitin_docs/index_migrator/`. +- The planning workspace lives entirely under `local_docs/index_migrator/`. - The default artifact format for plans and reports is YAML. - Benchmarking is built into migration reporting, not a separate subsystem. - The default execution unit is a single index (batch mode runs indexes sequentially). diff --git a/nitin_docs/index_migrator/01_context.md b/local_docs/index_migrator/01_context.md similarity index 100% rename from nitin_docs/index_migrator/01_context.md rename to local_docs/index_migrator/01_context.md diff --git a/nitin_docs/index_migrator/02_architecture.md b/local_docs/index_migrator/02_architecture.md similarity index 100% rename from nitin_docs/index_migrator/02_architecture.md rename to local_docs/index_migrator/02_architecture.md diff --git a/nitin_docs/index_migrator/03_benchmarking.md b/local_docs/index_migrator/03_benchmarking.md similarity index 100% rename from nitin_docs/index_migrator/03_benchmarking.md rename to local_docs/index_migrator/03_benchmarking.md diff --git a/nitin_docs/index_migrator/04_implementation_summary.md b/local_docs/index_migrator/04_implementation_summary.md similarity index 100% rename from nitin_docs/index_migrator/04_implementation_summary.md rename to local_docs/index_migrator/04_implementation_summary.md diff --git a/nitin_docs/index_migrator/05_migration_benchmark_report.md b/local_docs/index_migrator/05_migration_benchmark_report.md similarity index 99% rename from nitin_docs/index_migrator/05_migration_benchmark_report.md rename to local_docs/index_migrator/05_migration_benchmark_report.md index 0e9636c95..76e52f134 100644 --- a/nitin_docs/index_migrator/05_migration_benchmark_report.md +++ b/local_docs/index_migrator/05_migration_benchmark_report.md @@ -177,7 +177,7 @@ The async executor exists for integration with async application code (the CLI, ### N-Worker Parallelism Considerations -Multi-connection parallelism has production risks that should be weighed before enabling it. Redis is single-threaded for command processing, so N connections do not give N times server-side throughput. The client-side overlap of network round-trips provides the speedup, but the server processes commands sequentially from one queue. In production deployments with replicas, concurrent HSET writes from N workers increase replication backlog pressure. If the buffer fills, Redis disconnects the replica and triggers a full resync, which is catastrophic during migration. AOF persistence adds similar risk since N concurrent writers accelerate AOF buffer growth and could trigger an AOF rewrite that forks the process and temporarily doubles memory. Sharded deployments require shard-aware key partitioning to avoid hotspots, and Redis Cloud proxy layers add per-connection overhead that does not appear in local benchmarks. The safe default should remain N=1 with opt-in parallelism. See `nitin_docs/index_migrator/03_benchmarking.md` for the full risk analysis. +Multi-connection parallelism has production risks that should be weighed before enabling it. Redis is single-threaded for command processing, so N connections do not give N times server-side throughput. The client-side overlap of network round-trips provides the speedup, but the server processes commands sequentially from one queue. In production deployments with replicas, concurrent HSET writes from N workers increase replication backlog pressure. If the buffer fills, Redis disconnects the replica and triggers a full resync, which is catastrophic during migration. AOF persistence adds similar risk since N concurrent writers accelerate AOF buffer growth and could trigger an AOF rewrite that forks the process and temporarily doubles memory. Sharded deployments require shard-aware key partitioning to avoid hotspots, and Redis Cloud proxy layers add per-connection overhead that does not appear in local benchmarks. The safe default should remain N=1 with opt-in parallelism. See `local_docs/index_migrator/03_benchmarking.md` for the full risk analysis. ## Key Takeaways diff --git a/nitin_docs/index_migrator/10_v1_drop_recreate_spec.md b/local_docs/index_migrator/10_v1_drop_recreate_spec.md similarity index 100% rename from nitin_docs/index_migrator/10_v1_drop_recreate_spec.md rename to local_docs/index_migrator/10_v1_drop_recreate_spec.md diff --git a/nitin_docs/index_migrator/11_v1_drop_recreate_tasks.md b/local_docs/index_migrator/11_v1_drop_recreate_tasks.md similarity index 100% rename from nitin_docs/index_migrator/11_v1_drop_recreate_tasks.md rename to local_docs/index_migrator/11_v1_drop_recreate_tasks.md diff --git a/nitin_docs/index_migrator/12_v1_drop_recreate_tests.md b/local_docs/index_migrator/12_v1_drop_recreate_tests.md similarity index 98% rename from nitin_docs/index_migrator/12_v1_drop_recreate_tests.md rename to local_docs/index_migrator/12_v1_drop_recreate_tests.md index b00e4998c..87aaa497a 100644 --- a/nitin_docs/index_migrator/12_v1_drop_recreate_tests.md +++ b/local_docs/index_migrator/12_v1_drop_recreate_tests.md @@ -129,5 +129,5 @@ Phase 1 should not be considered ready until all of the following are true: - at least one manual smoke test passes - at least one benchmark rehearsal has been documented on a representative dataset - help text matches the spec -- the docs in `nitin_docs/index_migrator/` still match the shipped CLI behavior +- the docs in `local_docs/index_migrator/` still match the shipped CLI behavior - the release notes or implementation summary clearly state that `drop_recreate` is downtime-accepting diff --git a/nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md b/local_docs/index_migrator/20_v2_iterative_shadow_spec.md similarity index 100% rename from nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md rename to local_docs/index_migrator/20_v2_iterative_shadow_spec.md diff --git a/nitin_docs/index_migrator/21_v2_iterative_shadow_tasks.md b/local_docs/index_migrator/21_v2_iterative_shadow_tasks.md similarity index 99% rename from nitin_docs/index_migrator/21_v2_iterative_shadow_tasks.md rename to local_docs/index_migrator/21_v2_iterative_shadow_tasks.md index 4b00955e8..e77dc4b98 100644 --- a/nitin_docs/index_migrator/21_v2_iterative_shadow_tasks.md +++ b/local_docs/index_migrator/21_v2_iterative_shadow_tasks.md @@ -145,7 +145,7 @@ Phase 2 tasks are planned work only. They should not start until Phase 1 impleme - `Goal`: Add future-facing tests and docs for clustered shadow migration planning. - `Inputs`: completed Phase 2 planner and executor behavior - `Outputs`: test coverage, manual scale rehearsal instructions, and updated planning docs -- `Touchpoints`: `tests/`, `nitin_docs/index_migrator`, `redisvl/cli` +- `Touchpoints`: `tests/`, `local_docs/index_migrator`, `redisvl/cli` - `Dependencies`: `V2-T04`, `V2-T05`, `V2-T06` - `Acceptance Criteria`: - planner outcomes are covered in automated tests diff --git a/nitin_docs/index_migrator/22_v2_iterative_shadow_tests.md b/local_docs/index_migrator/22_v2_iterative_shadow_tests.md similarity index 100% rename from nitin_docs/index_migrator/22_v2_iterative_shadow_tests.md rename to local_docs/index_migrator/22_v2_iterative_shadow_tests.md diff --git a/nitin_docs/index_migrator/90_prd.md b/local_docs/index_migrator/90_prd.md similarity index 100% rename from nitin_docs/index_migrator/90_prd.md rename to local_docs/index_migrator/90_prd.md diff --git a/nitin_docs/index_migrator/99_tickets.md b/local_docs/index_migrator/99_tickets.md similarity index 98% rename from nitin_docs/index_migrator/99_tickets.md rename to local_docs/index_migrator/99_tickets.md index 6d7a31049..6d959aa9b 100644 --- a/nitin_docs/index_migrator/99_tickets.md +++ b/local_docs/index_migrator/99_tickets.md @@ -122,7 +122,7 @@ FT.AGGREGATE idx "*" **Key Files:** `redisvl/migration/models.py`, `planner.py`, `executor.py`, `async_executor.py` -**Spec:** `nitin_docs/index_migrator/30_rename_operations_spec.md` +**Spec:** `local_docs/index_migrator/30_rename_operations_spec.md` --- @@ -211,9 +211,9 @@ A four-layer reliability model. A pre-migration `BGSAVE` (run sequentially, wait 3. `rvl migrate apply --resume` picks up from the last checkpoint and completes the migration 4. Each key conversion is idempotent -- running the migration twice on the same key produces the correct result 5. If a batch fails mid-write, only that batch's vectors are rolled back using the bounded undo buffer -6. A disk space estimator function calculates projected RDB snapshot size, AOF growth, and total new disk required based on doc count, vector dimensions, source/target dtype, and AOF status. The estimator runs before any mutations and prints a human-readable summary. If available disk is below 80% of the estimate, the CLI prompts for confirmation. The estimator also supports a standalone dry-run mode via `rvl migrate estimate --plan plan.yaml`. See `nitin_docs/index_migrator/40_reliability_brainstorm.md` section "Pre-Migration Disk Space Estimator" for the full specification including inputs, outputs (DiskSpaceEstimate dataclass), calculation logic, CLI output format, integration points, and edge cases. +6. A disk space estimator function calculates projected RDB snapshot size, AOF growth, and total new disk required based on doc count, vector dimensions, source/target dtype, and AOF status. The estimator runs before any mutations and prints a human-readable summary. If available disk is below 80% of the estimate, the CLI prompts for confirmation. The estimator also supports a standalone dry-run mode via `rvl migrate estimate --plan plan.yaml`. See `local_docs/index_migrator/40_reliability_brainstorm.md` section "Pre-Migration Disk Space Estimator" for the full specification including inputs, outputs (DiskSpaceEstimate dataclass), calculation logic, CLI output format, integration points, and edge cases. -**Alternatives Considered:** Undo log (WAL-style), new-field-then-swap (side-write), shadow index (blue-green), streaming with bounded undo buffer. See `nitin_docs/index_migrator/40_reliability_brainstorm.md` for full analysis. +**Alternatives Considered:** Undo log (WAL-style), new-field-then-swap (side-write), shadow index (blue-green), streaming with bounded undo buffer. See `local_docs/index_migrator/40_reliability_brainstorm.md` for full analysis. --- @@ -261,7 +261,7 @@ Some migrations cannot use `drop_recreate` because the stored data is incompatib - Operator handoff for cutover decision - Cleanup of old index/keys after cutover -**Spec:** `nitin_docs/index_migrator/20_v2_iterative_shadow_spec.md` +**Spec:** `local_docs/index_migrator/20_v2_iterative_shadow_spec.md` --- @@ -307,7 +307,7 @@ Current quantization implementation does O(N) round trips for reads (one `HGET` During development, pyflakes identified unused imports across the codebase. These were fixed in migration files but not committed for non-migration files to keep the PR focused. **What This Requires:** -- Fix remaining unused imports (see `nitin_docs/issues/unused_imports_cleanup.md`) +- Fix remaining unused imports (see `local_docs/issues/unused_imports_cleanup.md`) - Update `.pylintrc` to remove deprecated Python 2/3 compat options - Consider adding `check-lint` to the main `lint` target after cleanup diff --git a/nitin_docs/index_migrator/pr_comments.md b/local_docs/index_migrator/pr_comments.md similarity index 100% rename from nitin_docs/index_migrator/pr_comments.md rename to local_docs/index_migrator/pr_comments.md diff --git a/tests/integration/test_migration_comprehensive.py b/tests/integration/test_migration_comprehensive.py index 759a1f626..3fef6acbe 100644 --- a/tests/integration/test_migration_comprehensive.py +++ b/tests/integration/test_migration_comprehensive.py @@ -24,7 +24,7 @@ - prefix change (keys renamed), HNSW params, initial_cap, phonetic_matcher, numeric unf Run: pytest tests/integration/test_migration_comprehensive.py -v -Spec: nitin_docs/index_migrator/32_integration_test_spec.md +Spec: local_docs/index_migrator/32_integration_test_spec.md """ import uuid From 9c6b57bebd220fa65749a42ea4d9a41916e24e10 Mon Sep 17 00:00:00 2001 From: Nitin Kanukolanu Date: Fri, 3 Apr 2026 10:31:06 -0400 Subject: [PATCH 8/8] gitignore: add local_docs/ and stop tracking it --- .gitignore | 1 + local_docs/index_migrator/00_index.md | 76 -- local_docs/index_migrator/01_context.md | 103 --- local_docs/index_migrator/02_architecture.md | 234 ------ local_docs/index_migrator/03_benchmarking.md | 248 ------- .../04_implementation_summary.md | 138 ---- .../05_migration_benchmark_report.md | 200 ----- .../10_v1_drop_recreate_spec.md | 326 -------- .../11_v1_drop_recreate_tasks.md | 162 ---- .../12_v1_drop_recreate_tests.md | 133 ---- .../20_v2_iterative_shadow_spec.md | 213 ------ .../21_v2_iterative_shadow_tasks.md | 159 ---- .../22_v2_iterative_shadow_tests.md | 102 --- local_docs/index_migrator/90_prd.md | 599 --------------- local_docs/index_migrator/99_tickets.md | 374 ---------- local_docs/index_migrator/pr_comments.md | 700 ------------------ 16 files changed, 1 insertion(+), 3767 deletions(-) delete mode 100644 local_docs/index_migrator/00_index.md delete mode 100644 local_docs/index_migrator/01_context.md delete mode 100644 local_docs/index_migrator/02_architecture.md delete mode 100644 local_docs/index_migrator/03_benchmarking.md delete mode 100644 local_docs/index_migrator/04_implementation_summary.md delete mode 100644 local_docs/index_migrator/05_migration_benchmark_report.md delete mode 100644 local_docs/index_migrator/10_v1_drop_recreate_spec.md delete mode 100644 local_docs/index_migrator/11_v1_drop_recreate_tasks.md delete mode 100644 local_docs/index_migrator/12_v1_drop_recreate_tests.md delete mode 100644 local_docs/index_migrator/20_v2_iterative_shadow_spec.md delete mode 100644 local_docs/index_migrator/21_v2_iterative_shadow_tasks.md delete mode 100644 local_docs/index_migrator/22_v2_iterative_shadow_tests.md delete mode 100644 local_docs/index_migrator/90_prd.md delete mode 100644 local_docs/index_migrator/99_tickets.md delete mode 100644 local_docs/index_migrator/pr_comments.md diff --git a/.gitignore b/.gitignore index cd8005815..274ab5e13 100644 --- a/.gitignore +++ b/.gitignore @@ -234,3 +234,4 @@ tests/data # Local working directory (personal scripts, docs, tools) local/ +local_docs/ diff --git a/local_docs/index_migrator/00_index.md b/local_docs/index_migrator/00_index.md deleted file mode 100644 index 6703cf99a..000000000 --- a/local_docs/index_migrator/00_index.md +++ /dev/null @@ -1,76 +0,0 @@ -# Index Migrator Workspace - -## Overview - -This directory contains the planning, design, and tracking documents for the RedisVL index migration feature. - -Phase 1 (`drop_recreate`) has been implemented and shipped across a 6-PR stack. The implementation went beyond the original MVP spec to include vector quantization, field/prefix/index renames, async execution, batch operations, crash-safe reliability, and disk space estimation. - -Phase 2 (`iterative_shadow`) remains planned and has not been started. - -This workspace is preserved as a historical planning record and as the foundation for Phase 2 design. - -## Guiding Principles - -- Prefer simple and safe over clever orchestration. -- Reuse existing RedisVL primitives before adding new abstractions. -- Migrate one index at a time (batch mode migrates sequentially). -- Keep cutover and platform scaling operator-owned. -- Fail closed on unsupported schema changes. - -## Phase Status - -| Phase | Mode | Status | Notes | -| --- | --- | --- | --- | -| Phase 1 | `drop_recreate` | **Done** | Shipped as PRs #567-#572 | -| Phase 1+ | Extensions (async, batch, reliability) | **Done** | Shipped alongside Phase 1 | -| Phase 2 | `iterative_shadow` | Planned | Not started | - -## Doc Map - -### Planning (pre-implementation) -- [01_context.md](./01_context.md): customer problem, constraints, and why the work is phased -- [02_architecture.md](./02_architecture.md): shared architecture, responsibilities, capacity model, and diagrams -- [03_benchmarking.md](./03_benchmarking.md): migration benchmarking goals, metrics, scenarios, and output artifacts -- [90_prd.md](./90_prd.md): final product requirements document for team review - -### Phase 1 (implemented) -- [04_implementation_summary.md](./04_implementation_summary.md): what was actually built, actual modules, actual CLI surface -- [05_migration_benchmark_report.md](./05_migration_benchmark_report.md): benchmark results (1K/10K/100K docs) -- [10_v1_drop_recreate_spec.md](./10_v1_drop_recreate_spec.md): original MVP spec (updated with implementation notes) -- [11_v1_drop_recreate_tasks.md](./11_v1_drop_recreate_tasks.md): task list (all completed) -- [12_v1_drop_recreate_tests.md](./12_v1_drop_recreate_tests.md): test plan - -### Phase 2 (planned, not started) -- [20_v2_iterative_shadow_spec.md](./20_v2_iterative_shadow_spec.md): future iterative shadow spec -- [21_v2_iterative_shadow_tasks.md](./21_v2_iterative_shadow_tasks.md): future iterative shadow tasks -- [22_v2_iterative_shadow_tests.md](./22_v2_iterative_shadow_tests.md): future iterative shadow test plan - -### Tracking -- [99_tickets.md](./99_tickets.md): all IM tickets with statuses -- [pr_comments.md](./pr_comments.md): collected PR review feedback - -## Current Truth - -Phase 1 is complete. The implementation source of truth is the code in `redisvl/migration/`. - -For Phase 2 planning: -- Spec: [20_v2_iterative_shadow_spec.md](./20_v2_iterative_shadow_spec.md) -- Tasks: [21_v2_iterative_shadow_tasks.md](./21_v2_iterative_shadow_tasks.md) -- Tests: [22_v2_iterative_shadow_tests.md](./22_v2_iterative_shadow_tests.md) - -## Next Actions - -- Phase 1 implementation is complete. No remaining Phase 1 tasks. -- Phase 2 design review should begin once Phase 1 learnings are documented. -- See [99_tickets.md](./99_tickets.md) for backlog items (IM-B1, IM-11 through IM-20). - -## Locked Decisions - -- The planning workspace lives entirely under `local_docs/index_migrator/`. -- The default artifact format for plans and reports is YAML. -- Benchmarking is built into migration reporting, not a separate subsystem. -- The default execution unit is a single index (batch mode runs indexes sequentially). -- The default operational model is operator-owned downtime, cutover, and scaling. -- Phase 2 owns shadow migrations for incompatible changes that require running old and new indexes in parallel. -- Vector quantization, field renames, prefix changes, and index renames were added to Phase 1 scope during implementation (originally planned for Phase 2). diff --git a/local_docs/index_migrator/01_context.md b/local_docs/index_migrator/01_context.md deleted file mode 100644 index 7e2e6af63..000000000 --- a/local_docs/index_migrator/01_context.md +++ /dev/null @@ -1,103 +0,0 @@ -# Index Migrator Context - -> **Status**: Phase 1 is complete and shipped (PRs #567-#572). This document is preserved as planning history. - -## Problem Statement - -RedisVL did not provide a first-class migration workflow for search index changes prior to this work. - -Teams could create indexes, delete indexes, inspect index info, and load documents, but they needed ad hoc scripts and operational runbooks to handle schema evolution. This was risky when the index was large, shared by multiple applications, or deployed on clustered Redis Cloud or Redis Software. - -The migration problem has three different shapes: - -- A simpler index rebuild that preserves existing documents and recreates the index definition in place. -- A shadow migration over the same documents when the target schema can still be built from the current stored payload. -- A shadow migration with transform or backfill when vector dimensions, datatypes, precision, algorithms, or payload shape change and a new target payload must be built. - -This workspace deliberately splits those shapes into phases. Phase 1 proved the plan-first migration workflow. Phase 2 exists to take on shadow migrations safely. - -## Customer Requirements - -The planning baseline for this work is: - -- preserve existing documents during migration -- capture the previous index configuration before making changes -- apply only the requested schema changes -- preview the migration plan before execution -- support advanced vector migrations such as `HNSW -> FLAT`, `FP32 -> FP16`, vector dimension changes, and payload-shape-changing model or algorithm swaps -- estimate migration timing, memory impact, and operational impact using simple benchmark artifacts -- benchmark source-versus-target memory and size changes, including peak overlap footprint during shadow migrations -- support both guided and scripted workflows -- make downtime and disruption explicit -- support large datasets without defaulting to full-keyspace audits or fleet-wide orchestration -- keep the implementation understandable enough that another team can operate it safely - -## Current RedisVL Capabilities - -RedisVL already has useful primitives that should be reused instead of replaced: - -- `SearchIndex.from_existing()` can reconstruct schema from a live index. -- `SearchIndex.delete(drop=False)` can remove the index structure without deleting documents. -- `SearchIndex.info()` can retrieve index stats used for planning and validation. -- Existing CLI commands already establish the connection and index lookup patterns the migrator can follow. - -Phase 1 added the following (originally listed as missing): - -- a migration planner (`MigrationPlanner`, `AsyncMigrationPlanner`, `BatchMigrationPlanner`) -- a schema diff classifier (in `planner.py`) -- a migration-specific CLI workflow (`rvl migrate` with 11 subcommands) -- a guided schema migration wizard (`MigrationWizard`) -- structured migration reports (`MigrationReport`, `MigrationValidation`, `MigrationBenchmarkSummary`) -- batch orchestration across indexes (`BatchMigrationExecutor`) -- vector quantization (e.g., FP32 -> FP16) with crash-safe reliability - -Still not built (Phase 2 or future): - -- capacity-aware orchestration with platform inventory -- transform or backfill planning for migrations that need new stored payloads - -## Why Phase 1 Came First - -Phase 1 was intentionally narrow because it gave the team an MVP that was both useful and low-risk: - -- It preserves documents while changing only the index definition. -- It reuses current RedisVL primitives instead of introducing a separate migration runtime. -- It keeps operational ownership clear: RedisVL handles planning, execution, and validation for a single index, while the operator handles the migration window and downstream application expectations. -- It avoids the hardest problems for now: target-payload generation, shadow overlap estimation, cutover automation, and cluster-wide scheduling. - -Phase 1 did not define the full migration goal. The harder shadow migrations are the reason Phase 2 exists. - -The MVP proved the planning model, CLI shape, plan artifact, and validation/reporting flow. Notably, vector quantization (originally scoped for Phase 2) was pulled forward into Phase 1 during implementation because it could be done safely as an in-place rewrite without shadow indexes. - -## Downtime and Disruption - -Phase 1 accepts downtime for the migrated index. - -Engineers need to plan for the following impacts: - -- Search on the target index is unavailable between index drop and recreated index readiness. -- Query results can be partial or unstable while the recreated index is still completing its initial indexing pass. -- Reindexing uses shared database resources and can increase CPU, memory, and indexing pressure on the deployment. -- Shadow migrations can temporarily duplicate index structures and sometimes duplicate payloads as well, increasing peak memory requirements. -- Downstream applications need either a maintenance window, a degraded mode, or a clear operational pause during the rebuild. - -The tooling does not hide these facts. The plan artifact and CLI output force the user to review the plan before applying a `drop_recreate` migration. (The original `--allow-downtime` flag was removed in favor of explicit plan review with `--accept-data-loss` only required for quantization.) - -## Non-Goals - -The following remain out of scope (not for the overall initiative, just for Phase 1): - -- a generic migration framework for every schema evolution case -- automatic platform scaling -- automatic traffic cutover -- full key manifest capture by default -- document transforms or backfills that require new embeddings -- payload relocation to a new keyspace (shadow migrations) -- fully managed Redis Cloud or Redis Software integration -- automatic transform inference or automatic re-embedding - -Note: Some items originally in this list were implemented during Phase 1: -- ~~concurrent migration of multiple large indexes~~ - batch mode was added (`rvl migrate batch-plan/batch-apply`) -- ~~field renames~~ - implemented via `rename_fields` in schema patch -- ~~prefix changes~~ - implemented via `index.prefix` in schema patch -- ~~vector datatype changes~~ - implemented as in-place quantization diff --git a/local_docs/index_migrator/02_architecture.md b/local_docs/index_migrator/02_architecture.md deleted file mode 100644 index 4056f8354..000000000 --- a/local_docs/index_migrator/02_architecture.md +++ /dev/null @@ -1,234 +0,0 @@ -# Index Migrator Architecture - -> **Status**: Phase 1 is implemented and shipped. This document describes the original architecture design. See [04_implementation_summary.md](./04_implementation_summary.md) for the actual module map. - -## System Boundaries - -The migration system has three boundaries: - -- RedisVL migration workflow: snapshot, diff, plan, apply, validate, report -- Redis deployment: Redis Cloud or Redis Software database that stores documents and indexes -- Operator and application boundary: maintenance window, scaling decisions, transform inputs, and application-level cutover behavior - -The implementation adds migration capabilities on top of existing RedisVL primitives instead of creating a separate control plane. - -```mermaid -flowchart LR - OP["Engineer / Operator"] --> CLI["RedisVL migrate CLI"] - APP["Applications"] --> DB["Redis database endpoint"] - - CLI --> SNAP["Snapshot + diff + mode selection"] - CLI --> EXEC["Executor"] - CLI --> VAL["Validator + report"] - - SNAP --> IDX["Live index metadata and stats"] - SNAP --> INV["Optional platform inventory"] - SNAP --> XFORM["Optional transform or backfill plan"] - - EXEC --> DB - VAL --> DB - EXEC --> TARGET["Optional target keyspace + shadow index"] - TARGET --> DB - - DB --> SHARDS["One logical index may span multiple shards"] - OP --> INV - OP --> XFORM -``` - -## Responsibilities - -RedisVL should own: - -- schema snapshot and source configuration capture -- schema diff classification -- migration mode selection -- migration plan generation -- guided wizard and scripted CLI entrypoints -- supported strategy execution -- validation and report generation - -The operator should own: - -- choosing the migration window -- accepting downtime or degraded behavior -- providing platform inventory when capacity planning matters -- providing transform or backfill inputs when payload shape changes -- scaling the Redis deployment -- application cutover and rollback decisions - -The platform should be treated as an external dependency, not as part of the MVP runtime. - -## Platform Model - -The migrator should reason about the deployment at the database level. - -For planning purposes: - -- treat the database endpoint as the unit of execution -- treat a search index as one logical index even if the deployment is sharded -- do not build logic that assumes an entire index lives on a single shard -- record where data lives in terms of database, prefixes, key separators, and target keyspace plans, not physical shard pinning - -This keeps the model compatible with both Redis Cloud and Redis Software without requiring the MVP to integrate directly with their platform APIs. - -## Migration Modes - -### `drop_recreate` - -This is the Phase 1 MVP. - -- Snapshot the current schema and index stats. -- Merge only the requested schema changes. -- Drop only the index structure, preserving documents. -- Recreate the index with the merged schema. -- Wait until indexing is complete. -- Validate and report. - -This mode is explicit about downtime and does not attempt to preserve uninterrupted query availability. - -### `iterative_shadow` - -This is the planned Phase 2 mode. - -- Work on one index at a time. -- Check database-level capacity before creating any shadow index. -- Choose between: - - `shadow_reindex` when the target schema can be built from the current stored payload. - - `shadow_rewrite` when vector datatype, precision, dimension, algorithm, or payload shape changes require a new target payload or keyspace. -- Create a shadow target for the current index only. -- Transform or backfill into a target keyspace when the migration changes payload shape. -- Validate the shadow target. -- Hand cutover to the operator. -- Retire the old index, and optionally the old target payload, only after cutover confirmation. - -This mode aims to reduce disruption without introducing automatic cutover or automatic scaling. This is the mode that should ultimately support migrations such as `HNSW -> FLAT`, `FP32 -> FP16`, vector dimension changes, and embedding-model-driven payload rewrites. - -## Capacity Model - -Phase 1 keeps capacity handling simple: - -- use source index stats for warnings and reports -- show expected downtime and indexing pressure -- do not block on a complex capacity estimator - -Phase 2 introduces a conservative capacity gate: - -- planner input is database-level, not shard-local -- one index at a time is the only supported execution unit -- estimate both source and target footprint -- separate document footprint from index footprint -- calculate peak overlap as the source footprint plus the target footprint that exists during migration -- capture memory savings or growth caused by algorithm, datatype, precision, dimension, and payload-shape changes -- the planner blocks if available headroom is below the estimated peak overlap plus reserve -- scaling stays operator-owned - -Default key-location capture is intentionally bounded: - -- store index name -- store storage type -- store prefixes -- store key separator -- store a bounded key sample - -Full key manifests are not part of the default path. - -## Benchmarking Model - -Benchmarking should be built into migration reporting, not treated as a separate system. - -The shared model is: - -- capture baseline metadata before migration -- capture timing and progress during migration -- capture validation and query-impact signals after migration -- persist simple YAML benchmark artifacts that can be compared across runs - -Benchmarking should focus on the operator questions that matter most: - -- total migration duration -- downtime or overlap duration -- document throughput -- query latency change during the migration window -- resource impact before, during, and after migration -- source-versus-target memory and size delta -- estimated versus actual peak overlap footprint - -The benchmark requirements are defined in [03_benchmarking.md](./03_benchmarking.md). - -## Failure Model - -The system should fail closed. - -- Unsupported schema diffs stop at `plan`. -- Missing transform inputs for a payload-shape-changing migration stop at `plan`. -- Missing source metadata stops at `plan`. -- `apply` never deletes documents in Phase 1. -- Validation failures produce a report and manual next steps. -- The tool does not attempt automatic rollback or automatic traffic switching. - -## `drop_recreate` Sequence - -```mermaid -sequenceDiagram - participant Op as Operator - participant CLI as RedisVL migrate CLI - participant Snap as Snapshot + diff - participant DB as Redis - participant Val as Validator - - Op->>CLI: migrate plan or wizard - CLI->>Snap: capture source schema and stats - Snap->>DB: read live index metadata - Snap-->>CLI: normalized source snapshot - CLI-->>Op: migration_plan.yaml with downtime warning - - Op->>CLI: migrate apply --plan migration_plan.yaml - CLI->>DB: quantize vectors (if needed) - CLI->>DB: rename fields/keys (if needed) - CLI->>DB: drop index only - CLI->>DB: recreate merged index - loop until indexing complete - CLI->>DB: poll index info - DB-->>CLI: indexing status and counts - end - CLI->>Val: run validation checks - Val->>DB: verify schema, counts, samples - Val-->>CLI: migration_report.yaml - CLI-->>Op: success or manual review -``` - -## `iterative_shadow` Sequence - -```mermaid -sequenceDiagram - participant Op as Operator - participant CLI as RedisVL migrate CLI - participant Plan as Shadow planner - participant DB as Redis - participant App as Application - - Op->>CLI: migrate plan --mode iterative_shadow - CLI->>Plan: capture source stats, target schema, transform plan, and platform inventory - Plan-->>CLI: READY, SCALE_REQUIRED, or MANUAL_REVIEW_REQUIRED - CLI-->>Op: plan and operator actions - - Op->>CLI: migrate apply --mode iterative_shadow - alt payload-compatible shadow - CLI->>DB: create shadow index for one source index - else payload rewrite required - CLI->>DB: create target keyspace and shadow index - loop backfill target payload - CLI->>DB: read source documents - CLI->>DB: write transformed target documents - end - end - loop until shadow ready - CLI->>DB: poll shadow index info - DB-->>CLI: readiness status - end - CLI->>DB: validate shadow index - CLI-->>Op: cutover runbook - Op->>App: switch to target index - Op->>CLI: confirm cutover - CLI->>DB: retire old index -``` diff --git a/local_docs/index_migrator/03_benchmarking.md b/local_docs/index_migrator/03_benchmarking.md deleted file mode 100644 index 67a637f18..000000000 --- a/local_docs/index_migrator/03_benchmarking.md +++ /dev/null @@ -1,248 +0,0 @@ -# Migration Benchmarking - -## Goals - -Migration benchmarking exists to answer practical operator questions: - -- how long will the migration take -- how long will search be degraded or unavailable -- how much shared Redis capacity will the migration consume -- how much the target schema or vector shape will increase or reduce memory usage -- how much query performance changes during the migration window -- whether future migrations can be estimated from previous runs - -The first benchmarking design should stay simple. It should collect structured measurements from real runs and manual rehearsals rather than introducing a separate performance framework before the migrator exists. - -## Core Benchmark Questions - -Every migration benchmark should answer: - -1. How long did planning take? -2. How long did `apply` take end-to-end? -3. How long was the index unavailable or in degraded indexing state? -4. What document throughput did the migration achieve? -5. What query latency and error-rate changes occurred during the migration? -6. How much memory, flash, or disk footprint changed before, during, and after migration? -7. How accurate was the peak-overlap estimate? -8. Did the final migrated index match the expected schema and document count? - -## Metrics - -### Timing Metrics - -- `plan_duration_seconds` -- `apply_duration_seconds` -- `validation_duration_seconds` -- `total_migration_duration_seconds` -- `drop_duration_seconds` -- `recreate_duration_seconds` -- `initial_indexing_duration_seconds` -- `downtime_duration_seconds` for `drop_recreate` -- `shadow_overlap_duration_seconds` for `iterative_shadow` -- `transform_duration_seconds` for payload rewrite work -- `backfill_duration_seconds` for target payload creation - -### Throughput Metrics - -- `source_num_docs` -- `target_num_docs` -- `documents_indexed_per_second` -- `documents_transformed_per_second` -- `bytes_rewritten_per_second` -- `progress_samples` captured during readiness polling - -### Query Impact Metrics - -- baseline query latency: `p50`, `p95`, `p99` -- during-migration query latency: `p50`, `p95`, `p99` -- post-migration query latency: `p50`, `p95`, `p99` -- query error rate during migration -- query result overlap or sample correctness checks - -### Resource Impact Metrics - -- source document footprint from live stats or sampling -- source index size from live stats -- target document footprint from live stats or sampling -- target index size from live stats -- total source footprint -- total target footprint -- footprint delta after migration -- estimated peak overlap footprint -- actual peak overlap footprint -- indexing failure delta -- memory headroom before migration -- memory headroom after migration -- peak memory headroom during overlap -- flash or disk footprint before and after when relevant -- source vector dimensions, datatype, precision, and algorithm -- target vector dimensions, datatype, precision, and algorithm -- source vector bytes per document -- target vector bytes per document - -### Correctness Metrics - -- schema match -- document count match -- indexing failure delta equals zero -- representative document fetch checks pass - -## Benchmark Inputs - -Each benchmark run should record the workload context, not just the raw timings. - -Required context: - -- migration mode -- dataset size -- storage type -- field mix -- whether vectors are present -- source and target vector configuration when vectors are present -- whether payload shape changes -- shard count -- replica count -- query load level during migration -- environment label such as `local`, `staging`, `redis_cloud`, or `redis_software` - -Useful optional context: - -- vector dimensions and datatype -- vector precision and algorithm -- auto-tiering enabled or disabled -- representative document size -- maintenance window target - -## Benchmark Scenarios - -Start with a small scenario matrix and expand only when needed. - -Minimum Phase 1 benchmark scenarios: - -- small index, low query load -- medium or large index, low query load -- medium or large index, representative read load - -Minimum Phase 2 benchmark scenarios: - -- one shadow migration on a sharded deployment with sufficient capacity -- one shadow migration that is blocked by the capacity gate -- one shadow migration under representative read load -- one algorithm migration such as `HNSW -> FLAT` -- one vector storage migration such as `FP32 -> FP16` -- one payload-shape-changing migration that requires transform or backfill - -Scale variables should be changed one at a time where possible: - -- document count -- shard count -- query concurrency -- storage type -- vector footprint - -## Benchmark Artifacts - -The benchmark workflow should use simple YAML artifacts. - -### `benchmark_manifest.yaml` - -```yaml -version: 1 -label: staging-large-read-load -mode: drop_recreate -environment: staging -dataset: - num_docs: 1000000 - storage_type: json - vector_fields_present: true -platform: - shard_count: 4 - replica_count: 1 -workload: - query_profile: representative-read - query_check_file: queries.yaml -notes: "" -``` - -### `benchmark_report.yaml` - -```yaml -version: 1 -label: staging-large-read-load -mode: drop_recreate -timings: - total_migration_duration_seconds: 540 - downtime_duration_seconds: 420 - validation_duration_seconds: 18 -throughput: - source_num_docs: 1000000 - documents_indexed_per_second: 2380.95 -query_impact: - baseline_p95_ms: 42 - during_migration_p95_ms: 95 - post_migration_p95_ms: 44 -resource_impact: - source_doc_footprint_mb: 6144 - source_index_size_mb: 8192 - target_doc_footprint_mb: 6144 - target_index_size_mb: 6144 - total_footprint_delta_mb: -2048 - estimated_peak_overlap_footprint_mb: 20480 - actual_peak_overlap_footprint_mb: 19840 - source_vector: - dimensions: 1536 - datatype: float32 - algorithm: hnsw - target_vector: - dimensions: 1536 - datatype: float16 - algorithm: flat -correctness: - schema_match: true - doc_count_match: true -``` - -These artifacts are planning and validation aids. They should not become a separate system before the migrator itself is implemented. - -## How Benchmarking Fits the Phases - -### Phase 1: `drop_recreate` - -Phase 1 should always record: - -- start time -- end time -- index downtime duration -- readiness wait duration -- source and target document counts -- source and target index stats -- observed source-versus-target index footprint delta - -Phase 1 should optionally record: - -- representative query latency before, during, and after migration -- query correctness checks using the same file as validation queries - -### Phase 2: `iterative_shadow` - -Phase 2 should always record: - -- source-to-shadow overlap duration -- planner estimate versus actual runtime -- capacity gate decision -- source and target document and index stats -- estimated versus actual peak overlap footprint -- observed memory savings or growth after the migration -- query impact during overlap - -Phase 2 should use benchmark history as advisory input for ETA and risk reporting, not as a hard execution dependency. - -## Exit Criteria - -Benchmarking is good enough for the first implementation when: - -- every migration report includes core timing and correctness metrics -- every shadow migration benchmark includes source-versus-target footprint deltas -- manual benchmark rehearsals can be run from a simple manifest -- the docs define what to collect before performance tuning begins -- benchmark requirements do not force a separate subsystem before the migrator ships diff --git a/local_docs/index_migrator/04_implementation_summary.md b/local_docs/index_migrator/04_implementation_summary.md deleted file mode 100644 index 9aa5f5110..000000000 --- a/local_docs/index_migrator/04_implementation_summary.md +++ /dev/null @@ -1,138 +0,0 @@ -# Phase 1 Implementation Summary - -## Overview - -Phase 1 of the RedisVL Index Migrator shipped across a 6-PR stack: - -| PR | Scope | -| --- | --- | -| #567 | Design: models, schema patch, planner | -| #568 | Core: executor, validation, reliability | -| #569 | Wizard: interactive migration builder | -| #570 | Async: AsyncMigrationExecutor, AsyncMigrationPlanner | -| #571 | Batch: BatchMigrationPlanner, BatchMigrationExecutor | -| #572 | Docs: user guide, CLI help | - -## Module Map - -``` -redisvl/migration/ - __init__.py Public API exports - models.py All Pydantic models (SchemaPatch, MigrationPlan, MigrationReport, - DiskSpaceEstimate, BatchPlan, BatchState, etc.) - planner.py MigrationPlanner - snapshot, patch merge, diff classification, - rename/quantization detection, plan generation - executor.py MigrationExecutor - sync drop/recreate with quantization, - field renames, key renames, readiness polling - validation.py MigrationValidator - schema match, doc count, key sample, - query checks - wizard.py MigrationWizard - interactive guided plan builder - reliability.py Crash-safe quantization: idempotent dtype detection, - checkpointing, BGSAVE safety, bounded undo buffering - utils.py Shared utilities: list_indexes, load/write YAML, - estimate_disk_space, detect_aof_enabled, timestamp_utc - async_planner.py AsyncMigrationPlanner - async version of planner - async_executor.py AsyncMigrationExecutor - async version of executor - async_validation.py AsyncMigrationValidator - async version of validator - batch_planner.py BatchMigrationPlanner - multi-index plan generation - batch_executor.py BatchMigrationExecutor - sequential multi-index execution - with checkpointing, resume, retry-failed - -redisvl/cli/ - migrate.py CLI entry point: 11 subcommands (see below) -``` - -## CLI Surface - -``` -rvl migrate - -Commands: - helper Show migration guidance and supported capabilities - list List all available indexes - wizard Interactively build a migration plan and schema patch - plan Generate a migration plan for a document-preserving drop/recreate migration - apply Execute a reviewed drop/recreate migration plan (use --async for large migrations) - estimate Estimate disk space required for a migration plan (dry-run, no mutations) - validate Validate a completed migration plan against the live index - batch-plan Generate a batch migration plan for multiple indexes - batch-apply Execute a batch migration plan with checkpointing - batch-resume Resume an interrupted batch migration - batch-status Show status of an in-progress or completed batch migration -``` - -## Key Features Beyond Original MVP Spec - -### Vector Quantization -In-place rewriting of vector data (e.g., FP32 -> FP16, FP32 -> INT8). Implemented with: -- Idempotent dtype detection (`detect_vector_dtype` in `reliability.py`) -- Crash-safe checkpointing to local YAML file -- BGSAVE safety checks -- `--accept-data-loss` flag for CLI acknowledgment -- Disk space estimation before migration - -### Rename Operations -- **Index rename**: Change the index name via `index.name` in schema patch -- **Prefix change**: Change key prefix via `index.prefix` in schema patch -- **Field renames**: Rename hash fields via `rename_fields` in schema patch - -### Async Execution -- `--async` flag on `rvl migrate apply` for large migrations -- Full async planner, executor, and validator classes - -### Batch Operations -- `batch-plan`: Generate plans for multiple indexes (by pattern, list, or file) -- `batch-apply`: Execute with per-index checkpointing and progress callbacks -- `batch-resume`: Resume interrupted batch with `--retry-failed` -- `batch-status`: Inspect checkpoint state -- Failure policies: `fail_fast` or `continue_on_error` - -### Disk Space Estimation -- Pre-migration estimate of RDB snapshot cost, AOF growth, and memory savings -- Per-vector-field breakdown with source/target dtype and byte calculations -- Available as `rvl migrate estimate` or automatically shown during `apply` - -## Pydantic Models (in `models.py`) - -| Model | Purpose | -| --- | --- | -| `SchemaPatch` / `SchemaPatchChanges` | Schema change request input | -| `FieldUpdate` / `FieldRename` | Individual field modifications | -| `SourceSnapshot` / `KeyspaceSnapshot` | Captured source state | -| `MigrationPlan` | Full plan artifact with diff classification | -| `RenameOperations` | Tracks index/prefix/field renames | -| `DiffClassification` | Supported/blocked with reasons | -| `ValidationPolicy` | What to check after migration | -| `MigrationReport` | Full execution report | -| `MigrationValidation` | Post-migration validation results | -| `MigrationTimings` | Duration breakdowns | -| `MigrationBenchmarkSummary` | Throughput and size metrics | -| `DiskSpaceEstimate` / `VectorFieldEstimate` | Pre-migration disk cost | -| `BatchPlan` / `BatchIndexPlan` | Multi-index plan | -| `BatchState` / `CompletedIndex` | Checkpoint state for batch | - -## Test Files - -| File | Type | -| --- | --- | -| `tests/unit/test_migration_planner.py` | Unit tests for planner, patch merge, diff classification | -| `tests/unit/test_batch_migration.py` | Unit tests for batch planner and executor | -| `tests/unit/test_migration_wizard.py` | Unit tests for wizard flow | -| `tests/integration/test_migration_comprehensive.py` | Integration tests with live Redis | - -Run all tests: -```bash -uv run python -m pytest tests/unit/test_migration_planner.py tests/unit/test_batch_migration.py tests/unit/test_migration_wizard.py tests/integration/test_migration_comprehensive.py -``` - -## Execution Flow (as implemented) - -1. **Plan**: Snapshot source -> merge patch -> classify diff -> detect renames/quantization -> emit `migration_plan.yaml` -2. **Apply**: Quantize vectors (if needed) -> rename fields (if needed) -> rename keys (if needed) -> drop index -> recreate index -> poll readiness -> validate -> emit report -3. **Validate**: Schema match + doc count + key sample + query checks -> emit report - -For quantization, the executor uses `reliability.py` for: -- Detecting current dtype of each key's vector (idempotent - skips already-processed) -- Checkpointing progress to disk for crash recovery -- BGSAVE coordination to avoid data loss - diff --git a/local_docs/index_migrator/05_migration_benchmark_report.md b/local_docs/index_migrator/05_migration_benchmark_report.md deleted file mode 100644 index 76e52f134..000000000 --- a/local_docs/index_migrator/05_migration_benchmark_report.md +++ /dev/null @@ -1,200 +0,0 @@ -# Migration Benchmark Report - -## Overview - -This report presents the results of benchmarking the RedisVL index migration process at three scales (1K, 10K, 100K documents). The migration converts an HNSW FP32 index to a FLAT FP16 index. All runs use a realistic 16-field schema modeled after a production knowledge management index with 3072-dimensional COSINE vectors. - -The benchmark isolates **load time** (populating Redis with synthetic data) from **migrate time** (the actual migration operation). It also confirms which enumeration method the migrator used to discover document keys. - -## Environment - -All trials ran on a single local machine against a Docker-hosted Redis Stack instance. Each scale was tested 3 times. Results shown below are averages across the 3 trials. - -## Results Summary - -| Scale | Load Time | Migrate Time | Quantize | Reindex | Downtime | Vec Memory Before | Vec Memory After | Saved | -|---|---|---|---|---|---|---|---|---| -| 1K | 0.4s | 0.8s | 0.3s | 0.5s | 0.8s | 12.3 MB | 6.0 MB | 51% | -| 10K | 26.2s | 3.4s | 2.4s | 1.0s | 3.4s | 123.5 MB | 60.5 MB | 51% | -| 100K | 454s | 30.9s | 24.1s | 6.6s | 30.7s | 1,211 MB | 593 MB | 51% | - -All 9 trials used **FT.AGGREGATE** for key enumeration (never SCAN). All trials passed validation. - -## Enumeration Method - -The migrator discovers which keys belong to the index before starting the migration. It uses `FT.AGGREGATE ... WITHCURSOR` as the primary method, which returns only indexed keys without scanning the full keyspace. SCAN is a fallback reserved for cases where `hash_indexing_failures > 0` or `FT.AGGREGATE` errors out. In all 9 trials, enumeration used `FT.AGGREGATE` and completed in under 150ms even at 100K. - -## How Drop-Recreate Migration Works - -The migration executor follows this sequence: - -**STEP 1: Enumerate keys** (before any modifications) -- Uses FT.AGGREGATE WITHCURSOR to discover all document keys in the source index -- Fallback to SCAN if the index has hash_indexing_failures > 0 or FT.AGGREGATE fails -- Keys are stored in memory for the quantization step - -**STEP 2: Drop source index** -- Issues FT.DROPINDEX (without KEEPDOCS) to remove the index structure -- **The underlying documents remain in Redis** - only the index metadata is deleted -- At this point, the index is gone but all document hashes/JSON still exist with their FP32 vectors - -**STEP 3: Quantize vectors** (rewrite document payloads IN-PLACE) -- For each document in the enumerated key list: - - HGETALL to read the document (including FP32 vector) - - Convert FP32 → FP16 in Python - - HSET to write back the FP16 vector to the same document -- Processes documents in batches of 500 using Redis pipelines -- **Memory note**: The old index is already dropped, so there is no "double index" overhead. Only the document data exists in Redis during this phase. - -**STEP 4: Key renames** (if needed) -- If the migration changes the key prefix, RENAME each key from old prefix to new prefix -- Skipped if no prefix change - -**STEP 5: Create target index** -- Issues FT.CREATE with the new schema (FLAT, FP16, etc.) -- Redis begins background indexing of existing documents - -**STEP 6: Wait for re-indexing** -- Polls FT.INFO until indexing completes (num_docs == expected count) -- The index is unavailable for queries until this completes - -## Phase Breakdown - -Here is the average duration of each phase at 100K documents (the most representative scale). - -| Phase | Duration | Share of Migration | -|---|---|---| -| Enumerate | 0.12s | 0.4% | -| Drop | 0.00s | 0.0% | -| Quantize | 24.1s | 77.9% | -| Create | 0.002s | 0.0% | -| Reindex | 6.6s | 21.3% | -| Validate | 0.012s | 0.0% | - -Quantization dominates at every scale. This is the client-side step that reads each document's FP32 vector from Redis, converts it to FP16 in Python, and writes it back. It is inherently I/O-bound and proportional to document count. - -## Scaling Analysis - -The central question is whether migration time grows linearly with document count, which determines whether we can predict costs at 1M and 10M. - -### Per-Document Costs - -| Scale | Per-Doc Quantize | Per-Doc Reindex | -|---|---|---| -| 1K | 277 us | 511 us | -| 10K | 237 us | 102 us | -| 100K | 241 us | 66 us | - -**Quantize scales linearly.** The per-document cost stabilizes around 240 microseconds from 10K onward. This makes sense because each document requires one HGETALL and one HSET regardless of index size. There is no interaction between documents during quantization. - -**Reindex scales sub-linearly.** The per-document cost decreases as scale increases. This is expected for FLAT indexes where Redis performs a simple sequential scan to build the brute-force index. Fixed overhead (index creation, initial polling delay) is amortized over more documents. At 100K the reindex throughput reaches ~15K docs/sec. - -### Scaling Ratios (10x increments) - -| Metric | 1K to 10K (10x data) | 10K to 100K (10x data) | -|---|---|---| -| Quantize time | 8.5x | 10.2x | -| Reindex time | 2.0x | 6.5x | -| Total migrate | 4.3x | 9.1x | - -The 10K-to-100K ratio is the most reliable predictor since 1K has proportionally more fixed overhead. Quantize is essentially 10x for 10x data (linear). Reindex is growing faster than at small scale but still sub-linear. - -## Predictions for 1M and 10M - -Using the per-document rates observed at 100K (the most representative scale) and assuming linear scaling for quantize with a conservative linear assumption for reindex. - -### Per-Document Rates Used - -| Component | Rate | -|---|---| -| Quantize | 241 us/doc (from 100K average) | -| Reindex | 66 us/doc (from 100K, likely optimistic at larger scale) | - -### Projected Migration Times - -| Scale | Quantize | Reindex | Total Migrate | Downtime | -|---|---|---|---|---| -| **1M** | ~241s (~4 min) | ~66s (~1.1 min) | ~5.2 min | ~5.2 min | -| **10M** | ~2,410s (~40 min) | ~660s (~11 min) | ~51 min | ~51 min | - -### Caveats on These Predictions - -**1M is realistic.** The quantize step is pure per-document I/O with no cross-document dependencies, so linear extrapolation is well-founded. Reindex for a FLAT index at 1M should also remain close to linear. Memory requirement would be roughly 11.4 GB for FP32 vectors plus metadata, so a machine with 32 GB RAM should handle it. - -**10M carries significant risk factors.** - -1. **Memory requirement.** 10M documents at 3072 dimensions requires ~57 GB for FP32 vectors plus metadata overhead. During quantization, the source index has already been dropped, so there is no "double index" memory overhead. Each batch (500 docs) temporarily holds both FP32 and FP16 representations during HSET, but this is a small incremental cost. The main memory requirement is the baseline FP32 data (~57 GB), not 80+ GB. After quantization completes, memory drops to ~28.5 GB (FP16 vectors). A machine with 64-128 GB RAM should handle this comfortably. - -2. **Reindex may slow down.** FLAT index construction at 10M with 3072 dimensions means Redis must build a brute-force index over the FP16 vector data. Background indexing throughput may degrade at this scale, especially if Redis is under memory pressure or serving concurrent traffic. - -3. **Quantize could slow down.** At 10M, the pipeline batches (500 docs each) would execute 20,000 batch cycles. If Redis starts swapping or if network latency increases under load, per-batch cost could rise above the observed 241 us/doc average. - -4. **FLAT may not be the right target at 10M.** A 10M-document FLAT index would make every query a brute-force scan over 10M vectors, which is impractical for production. HNSW FP16 would be the appropriate target, and HNSW index construction is O(n log n) rather than O(n), which would increase the reindex phase significantly (potentially 2-3x longer). - -### Adjusted Predictions with Risk - -| Scale | Optimistic | Expected | Pessimistic | -|---|---|---|---| -| **1M** | 4.5 min | 5.5 min | 8 min | -| **10M (FLAT target)** | 50 min | 60 min | 90 min | -| **10M (HNSW target)** | 70 min | 90 min | 150+ min | - -The pessimistic 10M estimate accounts for HNSW rebuild cost (O(n log n) indexing) and potential per-batch slowdown at scale. A production 10M migration would require a machine with 64-128 GB RAM and should be empirically tested before deployment. The memory requirement is the baseline FP32 data size (~57 GB), not double that, because the source index is dropped before quantization begins. - -## Async vs Sync Executor Comparison - -A second set of 9 trials was run using `AsyncMigrationExecutor` instead of the sync `MigrationExecutor`. The async executor is what the CLI (`rvl migrate apply`) uses internally and was expected to show improved throughput through non-blocking I/O. - -### Async Results Summary - -| Scale | Migrate Time | Quantize | Reindex | Downtime | -|---|---|---|---|---| -| 1K | 0.8s | 0.30s | 0.51s | 0.81s | -| 10K | 4.3s | 3.21s | 1.01s | 4.23s | -| 100K | 35.8s | 29.6s | 6.06s | 35.7s | - -### Side-by-Side Comparison - -| Scale | Sync Migrate | Async Migrate | Sync Quantize | Async Quantize | Async Overhead | -|---|---|---|---|---|---| -| 1K | 0.8s | 0.8s | 0.3s | 0.3s | ~0% | -| 10K | 3.4s | 4.3s | 2.4s | 3.2s | +33% | -| 100K | 30.9s | 35.8s | 24.1s | 29.6s | +23% | - -### Why Async is Slower - -The async executor adds overhead without gaining parallelism for three reasons. - -**Single-connection I/O.** Both executors talk to Redis over a single TCP connection. The async event loop adds coroutine scheduling and context-switch overhead on every `await`, but cannot overlap commands because Redis processes them sequentially on one connection. - -**CPU-bound quantization.** The FP32 to FP16 conversion uses `struct.unpack` and `struct.pack` in Python. This is CPU-bound work that gets no benefit from `asyncio`. The event loop overhead adds roughly 50 microseconds per document (296 us/doc async vs 241 us/doc sync at 100K). - -**Identical batching strategy.** Both executors use the same `pipeline.execute()` pattern with batches of 500 documents. The async version does not overlap I/O across batches because each batch must complete before the next begins. - -### When Async Would Help - -The async executor exists for integration with async application code (the CLI, web frameworks, or other coroutine-based systems). It does not improve raw migration throughput. To actually speed up the quantize phase, the optimization path would be multi-connection parallelism (splitting the key list across N workers, each with its own Redis connection), not async/await on a single connection. - -### N-Worker Parallelism Considerations - -Multi-connection parallelism has production risks that should be weighed before enabling it. Redis is single-threaded for command processing, so N connections do not give N times server-side throughput. The client-side overlap of network round-trips provides the speedup, but the server processes commands sequentially from one queue. In production deployments with replicas, concurrent HSET writes from N workers increase replication backlog pressure. If the buffer fills, Redis disconnects the replica and triggers a full resync, which is catastrophic during migration. AOF persistence adds similar risk since N concurrent writers accelerate AOF buffer growth and could trigger an AOF rewrite that forks the process and temporarily doubles memory. Sharded deployments require shard-aware key partitioning to avoid hotspots, and Redis Cloud proxy layers add per-connection overhead that does not appear in local benchmarks. The safe default should remain N=1 with opt-in parallelism. See `local_docs/index_migrator/03_benchmarking.md` for the full risk analysis. - -## Key Takeaways - -1. **Migration is fast relative to data loading.** At 100K, loading took 7.5 minutes while migration took only 31 seconds. The migration operation itself is not the bottleneck in any deployment workflow. - -2. **Quantization dominates migration time at every scale** (~78% of total). Any optimization effort should focus on the quantize step, such as parallelizing the read-convert-write pipeline across multiple connections. - -3. **Sync executor is faster than async** for raw migration throughput. The async version adds ~23% overhead at scale due to event loop costs on CPU-bound work. - -4. **FT.AGGREGATE is the default enumeration path** and it works reliably. SCAN fallback exists but did not trigger in any trial. - -5. **Vector memory savings are exactly 51% at every scale.** FP16 cuts the vector index footprint in half with no variation. Non-vector index metadata is unchanged. - -6. **Linear extrapolation is valid up to 1M.** Beyond that, memory pressure and index algorithm choice (FLAT vs HNSW) introduce non-linear factors that require empirical validation. - -## Raw Data - -- Sync results: `tests/benchmarks/results_migration.json` -- Async results: `tests/benchmarks/results_migration_async.json` - diff --git a/local_docs/index_migrator/10_v1_drop_recreate_spec.md b/local_docs/index_migrator/10_v1_drop_recreate_spec.md deleted file mode 100644 index 255295f52..000000000 --- a/local_docs/index_migrator/10_v1_drop_recreate_spec.md +++ /dev/null @@ -1,326 +0,0 @@ -# Phase 1 Spec: `drop_recreate` - -> **Status**: Implemented and shipped (PRs #567-#572). This spec has been updated with implementation notes where the shipped code diverged from the original design. - -## Goal - -Build a simple RedisVL migration workflow that: - -- preserves existing documents -- captures the old index configuration before change -- applies only the user-requested schema changes -- generates a plan before any mutation -- supports both guided and scripted use -- explicitly surfaces downtime for the migrated index -- supports vector quantization (e.g., FP32 -> FP16) as an in-place rewrite *(added during implementation)* -- supports field renames, prefix changes, and index renames *(added during implementation)* -- supports async execution for large migrations *(added during implementation)* -- supports batch migration across multiple indexes *(added during implementation)* - -## Supported Changes - -The MVP supports schema changes that can be satisfied by rebuilding the index over the existing document set without rewriting or relocating stored documents. - -Supported categories: - -- add a new non-vector field that indexes data already present in stored documents -- remove an existing field from the index definition -- change index options on an existing non-vector field when the field name, field type, and storage path stay the same -- change index-level options that only affect index definition and do not relocate data - -Supported field types for MVP changes: - -- text -- tag -- numeric -- geo - -The MVP always recreates the same logical index name unless the user is only generating a plan. - -## Blocked Changes - -The following changes are classified as unsupported and stop before `apply`: - -- key separator changes -- storage type changes (hash <-> JSON) -- JSON path remodels -- vector dimension changes -- any change that requires a completely new stored payload shape - -> **Implementation note**: Several items originally blocked in this spec were implemented during Phase 1: -> - ~~key prefix changes~~ - now supported via `index.prefix` in schema patch -> - ~~field renames~~ - now supported via `rename_fields` in schema patch -> - ~~vector datatype changes~~ - now supported as in-place quantization (e.g., FP32 -> FP16) -> - ~~new index name~~ - now supported via `index.name` in schema patch -> -> These were feasible because they could be done as in-place document rewrites without shadow indexes. - -## Inputs - -The workflow accepts: - -- Redis connection parameters -- source index name -- one of: - - `schema_patch.yaml` - - `target_schema.yaml` - - interactive wizard answers - -Actual CLI surface (as shipped): - -```text -rvl migrate helper -rvl migrate list -rvl migrate wizard --index --plan-out -rvl migrate plan --index --schema-patch -rvl migrate plan --index --target-schema -rvl migrate apply --plan [--async] [--resume ] -rvl migrate estimate --plan -rvl migrate validate --plan -rvl migrate batch-plan --schema-patch (--pattern | --indexes ) -rvl migrate batch-apply --plan [--accept-data-loss] -rvl migrate batch-resume --state [--retry-failed] -rvl migrate batch-status --state -``` - -> **Implementation note**: The `--allow-downtime` flag was removed. Downtime is implicit in `drop_recreate` mode. The `--accept-data-loss` flag is used only for quantization (lossy operation). The `helper` and `list` subcommands were added for discoverability. The `estimate` subcommand provides pre-migration disk space estimates. - -Key optional flags: - -- `--plan-out` / `--report-out` / `--benchmark-out` -- `--key-sample-limit` -- `--query-check-file` -- `--async` (for large migrations with quantization) -- `--resume` (crash-safe checkpoint resume) -- `--accept-data-loss` (for quantization acknowledgment) - -### `schema_patch.yaml` - -This is the authoritative input model for requested changes. Unspecified source configuration is preserved by default. - -Example: - -```yaml -version: 1 -changes: - add_fields: - - name: category - type: tag - path: $.category - separator: "," - remove_fields: - - legacy_score - update_fields: - - name: title - options: - sortable: true -``` - -### `target_schema.yaml` - -This is a convenience input. The planner normalizes it into a schema patch by diffing it against the live source schema. - -## Outputs - -The workflow produces: - -- `migration_plan.yaml` -- `migration_report.yaml` -- optional `benchmark_report.yaml` -- console summaries for plan, apply, and validate - -### `migration_plan.yaml` - -Required fields: - -```yaml -version: 1 -mode: drop_recreate -source: - index_name: docs - schema_snapshot: {} - stats_snapshot: {} - keyspace: - storage_type: json - prefixes: ["docs"] - key_separator: ":" - key_sample: ["docs:1", "docs:2"] -requested_changes: {} -merged_target_schema: {} -diff_classification: - supported: true - blocked_reasons: [] -warnings: - - index downtime is required -validation: - require_doc_count_match: true - require_schema_match: true -``` - -### `migration_report.yaml` - -Required fields: - -```yaml -version: 1 -mode: drop_recreate -source_index: docs -result: succeeded -started_at: 2026-03-17T00:00:00Z -finished_at: 2026-03-17T00:05:00Z -timings: - total_migration_duration_seconds: 300 - drop_duration_seconds: 3 - recreate_duration_seconds: 12 - initial_indexing_duration_seconds: 270 - validation_duration_seconds: 15 - downtime_duration_seconds: 285 -validation: - schema_match: true - doc_count_match: true - indexing_failures_delta: 0 - query_checks: [] -benchmark_summary: - documents_indexed_per_second: 3703.7 - source_index_size_mb: 2048 - target_index_size_mb: 1984 - index_size_delta_mb: -64 - baseline_query_p95_ms: 42 - during_migration_query_p95_ms: 90 - post_migration_query_p95_ms: 44 -manual_actions: [] -``` - -## CLI UX - -### `plan` - -- Capture the source snapshot from the live index. -- Normalize requested changes. -- Classify the diff as supported or blocked. -- Emit `migration_plan.yaml`. -- Print a short risk summary that includes downtime. - -### `wizard` - -- Read the live source schema first. -- Walk the user through supported change categories only. -- Reject unsupported requests during the wizard instead of silently converting them. -- Explain when a blocked request belongs to a future `iterative_shadow` migration. -- Emit the same `migration_plan.yaml` shape as `plan`. - -### `apply` - -- Accept only `migration_plan.yaml` as input. -- Refuse to run if the plan contains blocked reasons. -- Refuse to run if the current live schema no longer matches the saved source snapshot. -- Require `--accept-data-loss` when quantization is involved *(replaces original `--allow-downtime`)*. -- Support `--async` for large migrations with quantization. -- Support `--resume` for crash-safe checkpoint recovery. - -### `validate` - -- Re-run validation checks from the plan against the current live index. -- Emit `migration_report.yaml`. -- Emit `benchmark_report.yaml` when benchmark fields were collected. - -## Execution Flow (as implemented) - -1. Snapshot source state. - - Load the live index schema using existing RedisVL introspection. - - Capture live stats from index info. - - Record storage type, prefixes, key separator, and a bounded key sample. -2. Normalize requested changes. - - If the input is `target_schema.yaml`, diff it against the source schema and convert it to a patch. - - If the input is wizard answers, convert them to the same patch model. -3. Merge and classify. - - Apply only requested changes to the source schema. - - Classify each diff as supported or blocked. - - Detect rename operations (index name, prefix, field names). - - Detect vector quantization operations. - - Stop if any blocked diff exists. -4. Generate the plan. - - Save source snapshot, requested changes, merged target schema, rename operations, validation policy, and warnings. -5. Apply the migration. - - Confirm current live schema still matches the source snapshot. - - **Quantize vectors** in-place if quantization is requested (crash-safe with checkpointing). - - **Rename hash fields** if field renames are requested. - - **Rename keys** if prefix change is requested. - - Drop only the index structure. - - Recreate the index (possibly with new name) using the merged target schema. -6. Wait for indexing completion. - - Poll live index info until `indexing` is false and `percent_indexed` is complete. - - Stop with timeout rather than waiting forever. -7. Validate. - - Compare live schema to merged target schema. - - Compare live doc count to source doc count. - - Check indexing failure delta. - - Check key sample existence. - - Run optional query checks. -8. Emit the report with timings, validation, and benchmark summary. -9. Optionally emit separate benchmark report. - -## Validation - -Required validation checks: - -- exact schema match against `merged_target_schema` -- live doc count equals source `num_docs` -- `hash_indexing_failures` does not increase -- key sample records still exist - -Optional validation checks: - -- query checks loaded from `--query-check-file` -- bounded sample fetch checks for representative document ids - -Benchmark fields that should be collected during Phase 1: - -- migration start and end timestamps -- index downtime duration -- readiness polling duration -- source and target document counts -- documents indexed per second -- source and target index footprint -- observed index footprint delta after recreate -- optional representative query latency before, during, and after migration - -Validation is a hard failure for `apply`. - -## Failure Handling - -The implementation fails closed. - -- Unsupported diff: stop at `plan`. -- Source snapshot mismatch at apply time: stop and ask the operator to regenerate the plan. -- Drop succeeds but recreate fails: documents remain; emit a failure report and a manual recovery command using the saved merged schema. -- Validation fails after recreate: leave the recreated index in place, emit a failure report, and stop. -- Interrupted quantization run: crash-safe checkpointing allows resume via `--resume ` *(added during implementation, replacing original "no checkpointing" stance)*. -- Pipeline errors during batch field renames or key renames: re-raised with context. - -The implementation does not implement automatic rollback. - -## Operational Guidance - -This mode is downtime-accepting by design. - -Engineers should assume: - -- the index is unavailable between drop and recreated index readiness -- search quality can be degraded while initial indexing completes -- large indexes can place measurable pressure on shared Redis resources -- off-peak execution is preferred -- application-level maintenance handling is outside RedisVL -- blocked vector and payload-shape changes should be rerouted to Phase 2 planning instead of being forced into this path - -Default key capture is intentionally small: - -- keyspace definition is always recorded -- a bounded key sample is recorded -- a full key manifest is not part of the default MVP path - -Benchmarking for Phase 1 should stay simple: - -- capture timing and correctness metrics in structured reports -- support manual benchmark rehearsals using [03_benchmarking.md](./03_benchmarking.md) -- avoid introducing a dedicated benchmarking subsystem before the migrator exists diff --git a/local_docs/index_migrator/11_v1_drop_recreate_tasks.md b/local_docs/index_migrator/11_v1_drop_recreate_tasks.md deleted file mode 100644 index e8c98e57a..000000000 --- a/local_docs/index_migrator/11_v1_drop_recreate_tasks.md +++ /dev/null @@ -1,162 +0,0 @@ -# Phase 1 Tasks: `drop_recreate` - -> **Status**: All tasks complete. Shipped as PRs #567-#572. - -## Task Template - -Every Phase 1 task documented: - -- `ID` -- `Status` -- `Goal` -- `Inputs` -- `Outputs` -- `Touchpoints` -- `Dependencies` -- `Acceptance Criteria` -- `Non-Goals` -- `Handoff Notes` - -## V1-T01 - -- `ID`: `V1-T01` -- `Status`: **Done** -- `Goal`: Build the source snapshot and artifact models used by `plan`, `apply`, and `validate`. -- `Inputs`: source index name, Redis connection settings, optional key sample limit -- `Outputs`: in-memory and YAML-serializable source snapshot, migration plan model, migration report model -- `Touchpoints`: `redisvl/migration/models.py` (SourceSnapshot, MigrationPlan, MigrationReport, SchemaPatch, DiskSpaceEstimate, etc.) -- `Dependencies`: none -- `Acceptance Criteria`: - - source schema can be loaded from a live index - - source stats needed by the spec are captured - - storage type, prefixes, key separator, and bounded key sample are recorded - - models serialize to YAML without losing required fields -- `Non-Goals`: - - full key manifest generation - - capacity estimation - - schema diff logic -- `Handoff Notes`: preserve the raw source schema as faithfully as possible so later diffing does not invent defaults. - -## V1-T02 - -- `ID`: `V1-T02` -- `Status`: **Done** -- `Goal`: Implement schema patch normalization, source-plus-patch merge, and supported-versus-blocked diff classification. -- `Inputs`: source schema snapshot, `schema_patch.yaml` or normalized target schema diff -- `Outputs`: merged target schema and diff classification result -- `Touchpoints`: `redisvl/migration/planner.py` (MigrationPlanner - handles patch merge, diff classification, rename detection, quantization detection) -- `Dependencies`: `V1-T01` -- `Acceptance Criteria`: - - unspecified source config is preserved by default - - blocked diff categories from the spec are rejected with actionable reasons - - supported changes produce a deterministic merged target schema - - `target_schema.yaml` input normalizes to the same patch model -- `Non-Goals`: - - document rewrite planning - - vector migration logic - - shadow migration planning -- `Handoff Notes`: prefer an explicit allowlist of supported diff categories over a generic schema merge engine. - -## V1-T03 - -- `ID`: `V1-T03` -- `Status`: **Done** -- `Goal`: Add the `plan` command and plan artifact generation. -- `Inputs`: source index, connection settings, patch or target schema input -- `Outputs`: `migration_plan.yaml`, console summary -- `Touchpoints`: `redisvl/cli/migrate.py` (Migrate.plan), `redisvl/cli/main.py`, `redisvl/migration/planner.py` -- `Dependencies`: `V1-T01`, `V1-T02` -- `Acceptance Criteria`: - - `plan` emits the required YAML shape - - blocked plans do not proceed to mutation - - the console summary includes downtime warnings - - the current plan format is stable enough for `apply` and `validate` -- `Non-Goals`: - - interactive wizard flow - - mutation against Redis - - advanced report rendering -- `Handoff Notes`: make the plan file human-readable so operators can review it before running `apply`. - -## V1-T04 - -- `ID`: `V1-T04` -- `Status`: **Done** -- `Goal`: Add the guided `wizard` flow that emits the same plan artifact as `plan`. -- `Inputs`: source index, connection settings, interactive answers -- `Outputs`: normalized schema patch and `migration_plan.yaml` -- `Touchpoints`: `redisvl/migration/wizard.py` (MigrationWizard), `redisvl/cli/migrate.py` (Migrate.wizard) -- `Dependencies`: `V1-T01`, `V1-T02`, `V1-T03` -- `Acceptance Criteria`: - - wizard starts from the live source schema - - wizard only offers supported MVP change categories - - wizard emits the same plan structure as `plan` - - unsupported requests are blocked during the flow -- `Non-Goals`: - - platform inventory collection - - free-form schema editing for blocked categories - - shadow migration support -- `Handoff Notes`: keep prompts simple and linear; this is a guided assistant, not a general schema builder. - -## V1-T05 - -- `ID`: `V1-T05` -- `Status`: **Done** -- `Goal`: Implement `apply` for the `drop_recreate` strategy. -- `Inputs`: reviewed `migration_plan.yaml` -- `Outputs`: recreated index, execution status, migration report -- `Touchpoints`: `redisvl/migration/executor.py` (MigrationExecutor), `redisvl/migration/async_executor.py` (AsyncMigrationExecutor), `redisvl/migration/reliability.py` (checkpointing, quantization), `redisvl/cli/migrate.py` (Migrate.apply) -- `Dependencies`: `V1-T03` -- `Acceptance Criteria`: - - source snapshot mismatch blocks execution - - index drop preserves documents - - recreated index uses the merged target schema - - readiness polling stops on success or timeout - - quantization is crash-safe with checkpointing - - async execution available for large migrations -- `Non-Goals`: - - automatic rollback - - cutover orchestration -- `Handoff Notes`: `--allow-downtime` was removed. `--accept-data-loss` is used only for quantization acknowledgment. Crash-safe checkpointing was added via `reliability.py`. - -## V1-T06 - -- `ID`: `V1-T06` -- `Status`: **Done** -- `Goal`: Implement `validate` and `migration_report.yaml`. -- `Inputs`: `migration_plan.yaml`, live index state, optional query checks -- `Outputs`: validation result, report artifact, console summary -- `Touchpoints`: `redisvl/migration/validation.py` (MigrationValidator), `redisvl/migration/async_validation.py` (AsyncMigrationValidator), `redisvl/cli/migrate.py` (Migrate.validate) -- `Dependencies`: `V1-T01`, `V1-T03`, `V1-T05` -- `Acceptance Criteria`: - - schema match is verified - - doc count match is verified - - indexing failure delta is captured - - core timing metrics are captured in the report - - optional query checks run deterministically - - report artifact is emitted for both success and failure -- `Non-Goals`: - - benchmark replay - - observability integrations - - automatic remediation -- `Handoff Notes`: keep the report format concise and stable so it can become the operator handoff artifact later. - -## V1-T07 - -- `ID`: `V1-T07` -- `Status`: **Done** -- `Goal`: Add Phase 1 tests and user-facing documentation for the new CLI flow. -- `Inputs`: completed planner, wizard, executor, and validator behavior -- `Outputs`: passing tests and concise usage docs -- `Touchpoints`: `tests/unit/test_migration_planner.py`, `tests/unit/test_batch_migration.py`, `tests/unit/test_migration_wizard.py`, `tests/integration/test_migration_comprehensive.py`, `docs/user_guide/` -- `Dependencies`: `V1-T03`, `V1-T04`, `V1-T05`, `V1-T06` -- `Acceptance Criteria`: - - CI-friendly happy-path and failure-path tests exist - - manual benchmark rehearsal guidance exists - - manual smoke test instructions are captured in the test doc - - help text matches the Phase 1 spec - - the docs directory still points to the active truth -- `Non-Goals`: - - Phase 2 implementation - - platform API integrations - - performance tuning beyond smoke coverage -- `Handoff Notes`: keep test coverage focused on correctness and operator safety, not on simulating every future migration shape. diff --git a/local_docs/index_migrator/12_v1_drop_recreate_tests.md b/local_docs/index_migrator/12_v1_drop_recreate_tests.md deleted file mode 100644 index 87aaa497a..000000000 --- a/local_docs/index_migrator/12_v1_drop_recreate_tests.md +++ /dev/null @@ -1,133 +0,0 @@ -# Phase 1 Tests: `drop_recreate` - -> **Status**: All test scenarios covered. See actual test files below. - -## Actual Test Files - -| File | Coverage | -| --- | --- | -| `tests/unit/test_migration_planner.py` | Planner, patch merge, diff classification, rename detection, quantization detection | -| `tests/unit/test_batch_migration.py` | Batch planner, batch executor, batch state management | -| `tests/unit/test_migration_wizard.py` | Wizard flow, interactive prompts, plan generation | -| `tests/integration/test_migration_comprehensive.py` | End-to-end: plan -> apply -> validate with live Redis | - -## Original Test Matrix (mapped to implementation) - -| ID | Scenario | Type | Status | -| --- | --- | --- | --- | -| V1-CI-01 | source snapshot loads live schema and stats | CI | Done (test_migration_planner) | -| V1-CI-02 | patch merge preserves unspecified config | CI | Done (test_migration_planner) | -| V1-CI-03 | blocked diff categories stop at `plan` | CI | Done (test_migration_planner) | -| V1-CI-04 | `plan` emits valid YAML artifact | CI | Done (test_migration_planner) | -| V1-CI-05 | quantization requires `--accept-data-loss` | CI | Done (test_migration_comprehensive) | -| V1-CI-06 | drop and recreate preserves documents | CI | Done (test_migration_comprehensive) | -| V1-CI-07 | readiness polling completes or times out | CI | Done (executor tests) | -| V1-CI-08 | `validate` emits a report on success | CI | Done (test_migration_comprehensive) | -| V1-CI-09 | `validate` emits a report on failure | CI | Done (test_migration_comprehensive) | -| V1-CI-10 | timing metrics are captured in reports | CI | Done (test_migration_comprehensive) | -| V1-MAN-01 | guided wizard produces the same plan model | Unit | Done (test_migration_wizard) | -| V1-MAN-02 | realistic rebuild on larger dataset | Manual | Done (benchmark report in 05_migration_benchmark_report.md) | -| V1-MAN-03 | benchmark rehearsal on representative workload | Manual | Done (benchmark report in 05_migration_benchmark_report.md) | - -## Happy Path - -The minimum automated happy path should cover: - -- create a source index with existing documents -- generate `migration_plan.yaml` from `schema_patch.yaml` -- run `apply --plan migration_plan.yaml` -- wait for recreated index readiness -- run `validate` -- confirm schema match, doc count match, and zero indexing failure delta - -Representative happy-path schema changes: - -- add a tag field backed by existing JSON data -- remove a legacy numeric field from the index -- make an existing text field sortable - -## Failure Paths - -CI should cover at least: - -- blocked diff because of vector dimension change -- blocked diff because of storage type change -- source snapshot mismatch between `plan` and `apply` -- recreate failure after drop -- validation failure because doc counts diverge -- readiness timeout -- missing required plan fields - -Every failure path must prove: - -- documents are not intentionally deleted by the migrator -- an actionable error is surfaced -- blocked vector and payload-shape diffs point the user to the Phase 2 migration path -- a `migration_report.yaml` can still be produced when the failure happens after `apply` starts - -## Manual Smoke Test - -Run a manual smoke test on a non-production Redis deployment: - -1. Create an index with representative JSON documents. -2. Prepare a `schema_patch.yaml` that adds one non-vector field and removes one old field. -3. Run `rvl migrate plan`. -4. Confirm the plan includes the downtime warning and no blocked diffs. -5. Run `rvl migrate apply --plan migration_plan.yaml`. -6. Wait until readiness completes. -7. Run `rvl migrate validate --plan migration_plan.yaml`. -8. Confirm search behavior has resumed and the new schema is active. - -Manual smoke test success means: - -- the operator can understand the plan without reading code -- the index rebuild completes without deleting documents -- the report is sufficient to hand back to another operator - -## Scale Sanity Check - -Phase 1 does not need a cluster-wide stress harness, but it does need a basic scale sanity check. - -Manual checks: - -- run the flow on an index large enough to make polling and downtime visible -- confirm default key capture stays bounded -- confirm the tool does not attempt a full key manifest by default -- confirm console output still stays readable for a larger index - -This is not a benchmark. The goal is to catch accidental implementation choices that make the MVP operationally unsafe on larger datasets. - -## Benchmark Rehearsal - -Phase 1 benchmarking should be lightweight and operationally useful. - -Use a simple rehearsal driven by [03_benchmarking.md](./03_benchmarking.md): - -1. Record a benchmark label and workload context. -2. Measure baseline query latency on a representative query set. -3. Run the migration on a realistic non-production index. -4. Record total migration duration, downtime duration, and readiness duration. -5. Record source and target document counts and index stats. -6. Record the observed source-versus-target index footprint delta. -7. Re-run the representative query set after migration. -8. Save a `benchmark_report.yaml`. - -The first benchmark questions to answer are: - -- how long does the rebuild take end-to-end -- how long is the index unavailable -- how many documents per second can the rebuild sustain -- how much query latency changes during and after the rebuild -- how much the recreated index footprint changes even for schema-only rebuilds -- whether the observed runtime is predictable enough for a maintenance window - -## Release Gate - -Phase 1 should not be considered ready until all of the following are true: - -- all CI scenarios in the test matrix pass -- at least one manual smoke test passes -- at least one benchmark rehearsal has been documented on a representative dataset -- help text matches the spec -- the docs in `local_docs/index_migrator/` still match the shipped CLI behavior -- the release notes or implementation summary clearly state that `drop_recreate` is downtime-accepting diff --git a/local_docs/index_migrator/20_v2_iterative_shadow_spec.md b/local_docs/index_migrator/20_v2_iterative_shadow_spec.md deleted file mode 100644 index c5b893a01..000000000 --- a/local_docs/index_migrator/20_v2_iterative_shadow_spec.md +++ /dev/null @@ -1,213 +0,0 @@ -# Phase 2 Spec: `iterative_shadow` - -## Goal - -Add a conservative, capacity-aware shadow migration mode that works one index at a time and reduces disruption without attempting to automate cutover or platform scaling. - -This phase exists to support the migration cases that Phase 1 intentionally does not handle safely: - -- vector datatype changes such as `FP32 -> FP16` -- vector precision changes -- vector dimension changes -- vector algorithm changes such as `HNSW -> FLAT` -- payload-shape-changing model or algorithm migrations that require new stored fields or a new target keyspace - -The first Phase 2 implementation should stay simple in operation even though it handles harder migration shapes: - -- one index at a time -- capacity gate before each index -- operator-owned cutover -- no automatic scale-up or scale-down -- no multi-index concurrent shadowing -- explicit transform or backfill plan when the target payload shape changes - -## Why It Is Not MVP - -This mode is not the MVP because it introduces operational questions that Phase 1 does not need to solve: - -- database-level capacity inventory -- target-footprint estimation for old and new document and index shapes -- overlap estimation for old and new payloads -- transform or backfill planning for target payload creation -- operator handoff between validation and cutover -- cleanup sequencing after cutover -- larger-scale manual testing on clustered deployments - -Phase 1 should prove the core planning and reporting model first. - -## Planner Inputs - -The Phase 2 planner takes: - -- source index name -- Redis connection settings -- supported schema patch or target schema input -- `platform_inventory.yaml` -- optional `transform_plan.yaml` when the migration requires new target payloads - -### `platform_inventory.yaml` - -```yaml -version: 1 -platform: redis_cloud -database: - name: customer-a-prod - total_memory_mb: 131072 - available_memory_mb: 32768 - shard_count: 8 - replica_count: 1 - auto_tiering: false - notes: "" -policy: - reserve_percent: 15 -``` - -Required inventory fields: - -- platform -- total memory -- available memory -- shard count -- replica count -- reserve policy - -Optional inventory fields: - -- flash or disk notes -- environment labels -- operator comments -- benchmark history notes - -### `transform_plan.yaml` - -This file is required when the target schema cannot be built from the current stored payload. - -Example: - -```yaml -version: 1 -target_keyspace: - storage_type: json - prefixes: ["docs_v2"] - key_separator: ":" -transform: - mode: rewrite - vector_fields: - - name: embedding - source_path: $.embedding - target_path: $.embedding_v2 - source_dimensions: 1536 - target_dimensions: 1536 - source_datatype: float32 - target_datatype: float16 - source_algorithm: hnsw - target_algorithm: flat - payload_changes: - - source_path: $.body - target_path: $.body_v2 - strategy: copy -``` - -The first implementation should keep this model explicit and declarative. The migrator should not guess how to transform payloads. - -## Capacity Gate - -The first Phase 2 capacity gate should be intentionally conservative. - -Planner rules: - -1. Compute source document footprint from live stats or bounded sampling. -2. Compute source index footprint from live index stats. -3. Estimate target document footprint. - - For payload-compatible shadowing, this can be zero or near-zero additional document storage. - - For payload rewrite shadowing, this includes the duplicated target payload. -4. Estimate target index footprint. - - Use live source footprint as a baseline when the target is structurally similar. - - Adjust for vector dimension, datatype, precision, and algorithm changes when those are present. -5. Compute reserve headroom as `max(operator reserve, 15 percent of configured memory)` when no stricter operator value is provided. -6. Compute `estimated_peak_overlap_footprint` as: - - `source_docs + source_index + target_docs + target_index` -7. Return `READY` only if: - - the migration diff is supported for Phase 2 - - any required transform plan is present and valid - - available memory is greater than or equal to `estimated_peak_overlap_footprint + reserve` -8. Return `SCALE_REQUIRED` when the migration is supported but headroom is insufficient. -9. Return `MANUAL_REVIEW_REQUIRED` when the diff is ambiguous or live data is insufficient for a safe estimate. - -This keeps the first shadow planner understandable and safe. More sophisticated estimators can come later if Phase 1 and early Phase 2 learnings justify them. - -The planner should also report: - -- estimated migration window -- estimated peak overlap footprint -- expected source-versus-target footprint delta after cutover -- whether the migration is `shadow_reindex` or `shadow_rewrite` - -## Execution Flow - -1. Capture the source snapshot and normalize requested changes. -2. Classify the migration as either: - - `shadow_reindex` when the target schema can be built from the current payload - - `shadow_rewrite` when a transform or backfill is needed -3. Load `platform_inventory.yaml`. -4. Load `transform_plan.yaml` when `shadow_rewrite` is required. -5. Compute the capacity gate result. -6. Stop if the result is not `READY`. -7. Create the shadow target for the current index only. -8. If `shadow_rewrite` is selected: - - create the target keyspace - - transform or backfill source documents into the target keyspace -9. Wait until the shadow index is ready. -10. Validate the shadow target. -11. Emit an operator cutover runbook. -12. Wait for operator confirmation that cutover is complete. -13. Retire the old index. -14. Retire old source payloads only when the plan explicitly says they are no longer needed. -15. Move to the next index only after the current index is finished. - -The scheduler for Phase 2 is intentionally serial. - -## Operator Actions - -The operator is responsible for: - -- supplying platform inventory -- supplying the transform or backfill plan when payload shape changes -- choosing the migration window -- scaling the database if the plan returns `SCALE_REQUIRED` -- switching application traffic to the shadow target -- confirming cutover before old index retirement -- monitoring the deployment during overlap - -RedisVL should not attempt to perform these actions automatically in the first Phase 2 implementation. - -Phase 2 should still emit structured benchmark outputs so operators can compare: - -- estimated overlap duration versus actual overlap duration -- estimated capacity usage versus observed document and index stats -- memory savings or growth after algorithm, datatype, precision, dimension, or payload-shape changes -- query latency impact during shadow validation and overlap - -## Blocked Scenarios - -The initial Phase 2 plan still blocks: - -- automatic scaling -- automatic traffic switching -- concurrent shadowing of multiple large indexes -- in-place destructive rewrites without a shadow target -- payload-shape-changing migrations without an explicit transform or backfill plan -- transform plans that do not define a deterministic target keyspace -- Active-Active specific workflows -- platform API integrations as a hard requirement - -## Open Questions Deferred - -These questions should stay deferred until after Phase 1 implementation: - -- whether to add direct Redis Cloud or Redis Software API integrations -- whether to support checkpoint and resume across shadow runs -- whether alias-based cutover should be added later -- how transform hooks should be expressed beyond the initial declarative plan format -- whether re-embedding should be integrated directly or stay an operator-supplied preprocessing step -- how much historical benchmark data should influence ETA predictions diff --git a/local_docs/index_migrator/21_v2_iterative_shadow_tasks.md b/local_docs/index_migrator/21_v2_iterative_shadow_tasks.md deleted file mode 100644 index e77dc4b98..000000000 --- a/local_docs/index_migrator/21_v2_iterative_shadow_tasks.md +++ /dev/null @@ -1,159 +0,0 @@ -# Phase 2 Tasks: `iterative_shadow` - -## Task Template - -Every Phase 2 task must document: - -- `ID` -- `Status` -- `Goal` -- `Inputs` -- `Outputs` -- `Touchpoints` -- `Dependencies` -- `Acceptance Criteria` -- `Non-Goals` -- `Handoff Notes` - -Phase 2 tasks are planned work only. They should not start until Phase 1 implementation is complete and learnings are folded back into this workspace. - -## V2-T01 - -- `ID`: `V2-T01` -- `Status`: `Planned` -- `Goal`: Add the platform inventory model and parser used by the capacity-aware planner. -- `Inputs`: `platform_inventory.yaml` -- `Outputs`: validated inventory model -- `Touchpoints`: new `redisvl/migration/inventory.py`, `redisvl/migration/models.py`, `redisvl/cli/migrate.py` -- `Dependencies`: Phase 1 implementation complete -- `Acceptance Criteria`: - - required inventory fields are validated - - unsupported platform inventory shapes are rejected clearly - - inventory values are available to the planner without CLI-specific parsing logic -- `Non-Goals`: - - platform API calls - - capacity math - - shadow execution -- `Handoff Notes`: keep the inventory model platform-neutral enough to support both Redis Cloud and Redis Software. - -## V2-T02 - -- `ID`: `V2-T02` -- `Status`: `Planned` -- `Goal`: Add the transform or backfill plan model and classify whether a migration is `shadow_reindex` or `shadow_rewrite`. -- `Inputs`: normalized diff classification, optional `transform_plan.yaml` -- `Outputs`: validated transform model and execution-mode classification -- `Touchpoints`: new `redisvl/migration/transforms.py`, `redisvl/migration/models.py`, `redisvl/migration/planner.py` -- `Dependencies`: `V2-T01` -- `Acceptance Criteria`: - - payload-compatible migrations are classified as `shadow_reindex` - - vector or payload-shape-changing migrations require `shadow_rewrite` - - missing transform plans are rejected clearly when they are required - - transform plans remain declarative and deterministic -- `Non-Goals`: - - direct embedding generation - - platform API calls - - shadow execution -- `Handoff Notes`: keep the first transform model simple and explicit rather than inventing a generic transformation framework. - -## V2-T03 - -- `ID`: `V2-T03` -- `Status`: `Planned` -- `Goal`: Implement the conservative capacity estimator and gate result classification. -- `Inputs`: source index stats, source document footprint, inventory model, normalized diff classification, optional transform model -- `Outputs`: `READY`, `SCALE_REQUIRED`, or `MANUAL_REVIEW_REQUIRED` -- `Touchpoints`: new `redisvl/migration/capacity.py`, `redisvl/migration/planner.py` -- `Dependencies`: `V2-T01`, `V2-T02` -- `Acceptance Criteria`: - - source document and index footprint are computed consistently - - target footprint estimates account for vector datatype, precision, dimension, algorithm, and payload-shape changes when those are present - - reserve policy is applied consistently - - supported diffs can produce `READY` or `SCALE_REQUIRED` - - ambiguous inputs produce `MANUAL_REVIEW_REQUIRED` -- `Non-Goals`: - - fine-grained shard placement modeling - - automated scale actions - - performance benchmarking as a separate subsystem -- `Handoff Notes`: keep the first estimator intentionally conservative and easy to inspect. - -## V2-T04 - -- `ID`: `V2-T04` -- `Status`: `Planned` -- `Goal`: Extend the planner to support `iterative_shadow` for one index at a time. -- `Inputs`: source snapshot, normalized diff, inventory, transform model, capacity result -- `Outputs`: shadow migration plan and operator action list -- `Touchpoints`: `redisvl/migration/planner.py`, `redisvl/cli/migrate.py` -- `Dependencies`: `V2-T03` -- `Acceptance Criteria`: - - supported vector and payload-shape changes can produce a valid shadow plan - - non-`READY` capacity results block apply - - plan artifact clearly identifies source, shadow target, target keyspace when present, and operator actions - - plan artifact identifies whether the run is `shadow_reindex` or `shadow_rewrite` - - plan format stays readable and deterministic -- `Non-Goals`: - - multi-index concurrency - - automatic cleanup - - fleet scheduling -- `Handoff Notes`: preserve the same plan-first experience as Phase 1. - -## V2-T05 - -- `ID`: `V2-T05` -- `Status`: `Planned` -- `Goal`: Implement shadow target creation, optional transform or backfill execution, readiness waiting, and validation hooks. -- `Inputs`: approved shadow migration plan -- `Outputs`: ready shadow index and validation state -- `Touchpoints`: new `redisvl/migration/shadow.py`, `redisvl/migration/executor.py`, `redisvl/migration/validation.py` -- `Dependencies`: `V2-T04` -- `Acceptance Criteria`: - - only one index is processed at a time - - shadow target creation follows the plan artifact - - `shadow_rewrite` runs can build the target payload into the planned keyspace - - readiness polling behaves deterministically - - validation runs before cutover handoff -- `Non-Goals`: - - automatic cutover - - cross-index scheduling - - platform autoscaling -- `Handoff Notes`: do not generalize this into a fleet scheduler in the first Phase 2 implementation. - -## V2-T06 - -- `ID`: `V2-T06` -- `Status`: `Planned` -- `Goal`: Add validation reporting, benchmark reporting, operator handoff, cutover confirmation, and old-index retirement. -- `Inputs`: validated shadow plan and operator confirmation -- `Outputs`: post-cutover cleanup result and report -- `Touchpoints`: `redisvl/cli/migrate.py`, `redisvl/migration/reporting.py`, `redisvl/migration/executor.py` -- `Dependencies`: `V2-T05` -- `Acceptance Criteria`: - - cutover remains operator-owned - - cleanup does not run before operator confirmation - - report captures cutover handoff, cleanup outcome, and source-versus-target footprint deltas -- `Non-Goals`: - - alias management - - application config mutation - - rollback orchestration -- `Handoff Notes`: the CLI should guide the operator clearly, but it must not attempt to switch traffic itself. - -## V2-T07 - -- `ID`: `V2-T07` -- `Status`: `Planned` -- `Goal`: Add future-facing tests and docs for clustered shadow migration planning. -- `Inputs`: completed Phase 2 planner and executor behavior -- `Outputs`: test coverage, manual scale rehearsal instructions, and updated planning docs -- `Touchpoints`: `tests/`, `local_docs/index_migrator`, `redisvl/cli` -- `Dependencies`: `V2-T04`, `V2-T05`, `V2-T06` -- `Acceptance Criteria`: - - planner outcomes are covered in automated tests - - benchmark, ETA, and memory-delta guidance are documented for manual cluster rehearsals - - manual cluster rehearsal steps are documented - - docs still reflect the shipped Phase 2 behavior accurately -- `Non-Goals`: - - fleet-wide migration support - - performance tuning beyond safety validation - - platform-specific automation -- `Handoff Notes`: keep Phase 2 documentation grounded in the one-index-at-a-time rule. diff --git a/local_docs/index_migrator/22_v2_iterative_shadow_tests.md b/local_docs/index_migrator/22_v2_iterative_shadow_tests.md deleted file mode 100644 index e7d2d5c27..000000000 --- a/local_docs/index_migrator/22_v2_iterative_shadow_tests.md +++ /dev/null @@ -1,102 +0,0 @@ -# Phase 2 Tests: `iterative_shadow` - -## Clustered Test Setup - -Phase 2 needs both automated planner coverage and manual clustered rehearsals. - -Minimum clustered rehearsal setup: - -- Redis Cloud or Redis Software deployment -- sharded database -- one source index large enough to make overlap meaningful -- representative application query set -- operator-supplied `platform_inventory.yaml` -- `transform_plan.yaml` for any vector or payload-shape-changing migration - -The first manual scale rehearsal should stay focused on a single index, not a fleet-wide migration. - -## Planner Acceptance - -Automated planner tests should cover: - -- supported shadow diff with sufficient headroom returns `READY` -- supported shadow diff with insufficient headroom returns `SCALE_REQUIRED` -- ambiguous or incomplete input returns `MANUAL_REVIEW_REQUIRED` -- vector datatype, precision, dimension, or algorithm changes require `shadow_rewrite` -- payload-shape-changing diffs stop before planning unless a valid transform plan is present - -Planner acceptance is successful when the result is deterministic and the operator action list is clear. - -## Unsafe Capacity Cases - -Manual and automated coverage should include: - -- insufficient available memory -- missing or invalid inventory fields -- conflicting operator reserve policy -- large source footprint with conservative reserve -- target footprint larger than source footprint because of dimension or payload expansion -- peak overlap estimate exceeds available headroom even when post-cutover memory would shrink - -Unsafe capacity handling is correct when: - -- the planner blocks the run -- no shadow index is created -- the report tells the operator what must change before retry - -## Shadow Validation - -Validation coverage should prove: - -- shadow target reaches readiness before handoff -- schema matches the planned target -- transformed payload fields match the declared target shape when `shadow_rewrite` is used -- query checks pass before cutover -- old index is not retired before operator confirmation - -This is the safety boundary for Phase 2. - -## Benchmark Rehearsal - -Phase 2 benchmarks should answer: - -- how accurate the planner ETA was -- how long the old and shadow indexes overlapped -- how much query latency changed during overlap -- whether the capacity reserve was conservative enough -- how much memory or size changed after datatype, precision, dimension, algorithm, or payload-shape changes -- whether estimated peak overlap footprint matched observed overlap closely enough - -Minimum manual benchmark coverage: - -- one run where the planner returns `READY` and the migration completes -- one run where the planner returns `SCALE_REQUIRED` -- one run with representative read traffic during overlap -- one vector-shape or algorithm change such as `HNSW -> FLAT` or `FP32 -> FP16` -- one payload-shape-changing migration that requires transform or backfill - -Every benchmark rehearsal should produce a structured benchmark report that can be compared against previous runs. - -## Resume/Retry - -The first Phase 2 implementation does not need fleet-grade checkpointing, but it does need basic retry behavior. - -Required checks: - -- planner can be rerun with the same inventory and produce the same decision -- failed shadow creation does not trigger cleanup of the old index -- operator can rerun the planned index only after fixing the blocking condition - -If stronger checkpointing is needed later, it should become its own scoped follow-up rather than being absorbed into the first shadow implementation. - -## Exit Criteria - -Phase 2 should not move from planned to ready until: - -- Phase 1 has been implemented and reviewed -- Phase 1 learnings have been written back into this workspace -- planner outcomes are covered by automated tests -- at least one manual clustered rehearsal has been designed in detail -- at least one benchmark rehearsal has been defined for a representative shadow migration -- at least one benchmark rehearsal has been defined for a vector or payload-shape-changing shadow migration -- the one-index-at-a-time execution rule is still preserved in the design diff --git a/local_docs/index_migrator/90_prd.md b/local_docs/index_migrator/90_prd.md deleted file mode 100644 index b54b8f82c..000000000 --- a/local_docs/index_migrator/90_prd.md +++ /dev/null @@ -1,599 +0,0 @@ -# Product Requirements Document: RedisVL Index Migrator - -> **Status**: Phase 1 is implemented and shipped (PRs #567-#572). This PRD has been updated with implementation notes where the shipped product diverged from the original requirements. - -## Summary - -RedisVL now has a migration workflow for search index changes that is safer and more operationally predictable than ad hoc scripts, while remaining simple enough to build, review, and operate without introducing an orchestration-heavy subsystem. - -This PRD defines a phased migration product: - -- Phase 1: `drop_recreate` - **Done** -- Phase 2: `iterative_shadow` - Planned - -The overall product goal is broader than the MVP. The migrator should eventually handle not only simple schema changes, but also vector datatype, precision, dimension, algorithm, and payload-shape-changing migrations such as: - -- `HNSW -> FLAT` -- `FP32 -> FP16` -- vector dimension changes -- embedding or payload-shape changes that require new stored fields or a new target keyspace - -Phase 1 stays intentionally narrow so the team can ship a plan-first, document-preserving migration tool quickly. Phase 2 is where those harder migrations are handled safely through one-index-at-a-time shadow migration and optional transform or backfill. - -This document is the review-oriented summary of the detailed planning docs in this workspace. - -## Problem - -RedisVL today gives users index lifecycle primitives, not a migration product. - -Users can: - -- create indexes -- delete indexes -- inspect index information -- load documents - -Phase 1 addressed the following gaps (users can now): - -- preview a schema migration in a structured way (`rvl migrate plan`) -- preserve current index configuration before change (`source` snapshot in plan) -- apply only a requested subset of schema changes (`schema_patch.yaml`) -- generate a repeatable migration plan artifact (`migration_plan.yaml`) -- validate a migration with a consistent report (`rvl migrate validate`) -- estimate migration duration, query impact, or memory impact from benchmarkable outputs (`migration_report.yaml`, `benchmark_report.yaml`) -- perform vector quantization (e.g., FP32 -> FP16) with crash-safe reliability - -Still not addressed (Phase 2): - -- shadow migrations that require running old and new indexes in parallel - -This gap is manageable for small experiments, but becomes painful for production workloads where: - -- indexes can contain millions of documents -- query latency matters during rebuild windows -- teams need predictable maintenance timelines -- migrations may change vector algorithms, datatypes, or payload shape -- Redis deployments may be clustered on Redis Cloud or Redis Software -- operators need a clean handoff between planning, execution, and validation - -## Users and Review Audience - -Primary users: - -- application engineers managing RedisVL-backed indexes -- platform engineers responsible for Redis operations -- support or solution engineers helping customers plan schema migrations - -Review audience: - -- RedisVL maintainers -- product and engineering stakeholders -- operators who will validate whether the workflow is practical in real environments - -## Goals - -- Provide a first-class migration workflow for RedisVL-managed indexes. -- Preserve existing documents during the Phase 1 path. -- Capture current schema and index configuration before any mutation. -- Apply only requested schema changes. -- Require a reviewed migration plan before execution. -- Support both scripted and guided user experiences. -- Make downtime and disruption explicit. -- Add structured reports and benchmarking outputs so migration windows become more predictable over time. -- Add benchmarking for memory and size deltas caused by schema, vector, and payload-shape changes. -- Keep the implementation simple enough that another engineer can understand and modify it quickly. - -## In Scope - -### Product-Wide Scope - -- schema-change migrations for RedisVL-managed indexes -- vector datatype, precision, dimension, and algorithm migrations -- payload-shape-changing migrations when the operator provides an explicit transform or backfill plan -- YAML artifacts for plans and reports -- operator-readable console output -- one-index-at-a-time execution -- benchmarking outputs for timing, query impact, and memory or size deltas - -### Phase 1 MVP - -- one index at a time -- document-preserving `drop_recreate` migrations -- source schema and stats snapshot -- patch-based schema change requests -- target-schema diff normalization into the same patch model -- guided wizard and scripted CLI flows -- plan generation before any mutation -- explicit downtime acknowledgment for `apply` -- readiness waiting after recreate -- validation and reporting -- benchmark-friendly timing, correctness, and index-footprint outputs - -### Phase 2 - -- one-index-at-a-time `iterative_shadow` migrations -- conservative capacity gating before each index -- optional platform inventory input -- shadow target creation and validation -- `shadow_reindex` for payload-compatible migrations -- `shadow_rewrite` for vector or payload-shape-changing migrations -- explicit transform or backfill plan input when payload shape changes -- operator handoff for cutover -- benchmark outputs for ETA, peak overlap, and source-versus-target size deltas - -## Out of Scope - -- automatic traffic cutover -- automatic platform scaling -- concurrent migration of multiple large indexes -- fully managed rollback orchestration -- full key manifest capture by default -- platform API integration as a hard requirement -- automatic transform inference -- automatic embedding generation or automatic re-embedding inside the migrator -- in-place destructive payload rewrites without a shadow target - -## Product Principles - -- Prefer simple and safe over fully automated. -- Reuse existing RedisVL primitives before adding new layers. -- Make the plan artifact the center of the workflow. -- Treat operator actions as first-class, not hidden implementation details. -- Fail closed when a migration request is ambiguous or unsupported for the selected phase. -- Measure migration behavior so future planning gets better with evidence. - -## Current State - -RedisVL already has building blocks that support a migration product: - -- `SearchIndex.from_existing()` for live schema reconstruction -- `SearchIndex.delete(drop=False)` for dropping the index structure while preserving documents -- `SearchIndex.info()` for live index stats that can inform planning, validation, and timing - -What is missing is the product layer on top: - -- a migration planner -- schema patch normalization and diff classification -- migration-specific CLI commands -- guided user flow -- structured migration and benchmark artifacts -- a capacity-aware future mode for larger production environments -- transform or backfill planning for migrations that change payload shape - -## Proposed Product - -### Phase 1 MVP: `drop_recreate` - -Scope: - -- one index at a time -- preserve documents -- snapshot source schema and stats -- accept `schema_patch.yaml`, `target_schema.yaml`, or wizard answers -- normalize all inputs into the same plan model -- classify requested changes as supported or blocked -- generate `migration_plan.yaml` -- require explicit downtime acknowledgment for `apply` -- drop only the index structure -- recreate the index using the merged schema -- wait for readiness -- validate and emit `migration_report.yaml` -- optionally emit `benchmark_report.yaml` - -Supported changes: - -- add non-vector fields backed by existing document data -- remove fields -- adjust supported non-vector index options where stored payload shape does not change -- adjust index-level options that do not relocate or rewrite data - -Blocked (still): - -- key separator changes -- storage type changes (hash <-> JSON) -- JSON path remodels -- vector dimension changes -- any change requiring a completely new stored payload shape - -> **Implementation note**: The following were originally blocked in this PRD but were implemented: -> - ~~key prefix changes~~ - supported via `index.prefix` in schema patch -> - ~~field renames~~ - supported via `rename_fields` in schema patch -> - ~~vector datatype changes~~ - supported as in-place quantization -> - ~~index name changes~~ - supported via `index.name` in schema patch - -### Phase 2: `iterative_shadow` - -Scope: - -- one index at a time -- conservative capacity gate before each index -- optional `platform_inventory.yaml` -- optional `transform_plan.yaml` when payload shape changes -- shadow target creation -- readiness waiting and validation -- operator-owned cutover -- old index retirement after operator confirmation -- optional old-payload retirement after operator confirmation -- structured benchmark outputs for overlap timing, ETA accuracy, and memory or size deltas - -Execution submodes: - -- `shadow_reindex` - - use when the new index can be built from the current stored payload - - still useful for lower-disruption rebuilds when the payload shape does not change -- `shadow_rewrite` - - use when vector datatype, precision, dimension, algorithm, or payload shape changes require a new target payload or keyspace - - examples: `HNSW -> FLAT`, `FP32 -> FP16`, dimension changes, new embedding schema - -Still intentionally excluded: - -- automatic cutover -- automatic scaling -- concurrent shadowing of multiple large indexes -- transform inference - -## Architecture - -The product should work as a plan-first migration workflow with explicit operator handoff and an optional transform path for harder migrations. - -```mermaid -flowchart LR - OP["Engineer / Operator"] --> CLI["RedisVL migrate CLI"] - APP["Applications"] --> DB["Redis database endpoint"] - - CLI --> PLAN["Planner
snapshot + diff + mode selection"] - CLI --> EXEC["Executor"] - CLI --> VAL["Validator + reports"] - - PLAN --> META["Live index schema + stats"] - PLAN --> INV["Optional platform inventory"] - PLAN --> XFORM["Optional transform or backfill plan"] - - EXEC --> DB - EXEC --> TARGET["Optional target keyspace + shadow index"] - TARGET --> DB - VAL --> DB - - OP --> CUT["Maintenance window / cutover decision"] - CUT --> APP -``` - -Architecture expectations: - -- RedisVL owns planning, execution, validation, and artifact generation. -- Redis remains the system of record for source documents and index state. -- The operator owns maintenance windows, scaling, transform inputs, and application cutover decisions. -- The product must stay compatible with single-node and clustered deployments without assuming the whole index lives on one shard. - -## Why the Work Is Phased - -The product is phased because the migration strategies solve different problems: - -- `drop_recreate` is the fastest path to a usable, understandable MVP -- `iterative_shadow` is the future path for tighter operational control and safer handling of vector or payload-shape changes - -Trying to ship everything as one fully mature product would push the team into: - -- premature capacity-estimation complexity -- premature transform-runtime design -- premature cutover abstractions -- premature platform-specific automation -- a larger QA and support surface before the MVP proves value - -Phase 1 is therefore the implementation target, while Phase 2 remains planned work informed by Phase 1 learnings. - -## User Experience - -### Scripted Flow (as shipped) - -```text -rvl migrate plan --index --schema-patch -rvl migrate plan --index --target-schema -rvl migrate apply --plan [--async] [--resume ] -rvl migrate validate --plan -rvl migrate estimate --plan -``` - -### Guided Flow (as shipped) - -```text -rvl migrate wizard --index --plan-out -``` - -### Batch Flow (as shipped) - -```text -rvl migrate batch-plan --schema-patch --pattern '*_idx' -rvl migrate batch-apply --plan --accept-data-loss -rvl migrate batch-resume --state --retry-failed -rvl migrate batch-status --state -``` - -User experience requirements (verified in implementation): - -- `plan` never mutates Redis -- `wizard` emits the same plan artifact shape as `plan` -- `apply` only accepts a reviewed plan file -- `apply` requires `--accept-data-loss` when quantization is involved -- `validate` is usable independently after `apply` -- console output is concise and operator-readable -- blocked requests tell the user what is not supported - -## Usage - -### Phase 1: `drop_recreate` - -Review-first workflow: - -```text -rvl migrate plan --index products --schema-patch patch.yaml --plan-out migration_plan.yaml -rvl migrate apply --plan migration_plan.yaml --report-out migration_report.yaml -rvl migrate validate --plan migration_plan.yaml --report-out migration_report.yaml -``` - -Guided workflow: - -```text -rvl migrate wizard --index products --plan-out migration_plan.yaml -rvl migrate apply --plan migration_plan.yaml -``` - -Expected usage pattern: - -1. Generate a plan from a live source index. -2. Review blocked diffs, warnings, downtime notice, and merged target schema. -3. Run `apply` only after the operator accepts the maintenance window. -4. Run `validate` and retain the report as the handoff artifact. - -### Phase 2: `iterative_shadow` - -Payload-compatible shadow workflow: - -```text -rvl migrate plan --mode iterative_shadow --index products --schema-patch patch.yaml --platform-inventory platform_inventory.yaml --plan-out migration_plan.yaml -rvl migrate apply --plan migration_plan.yaml --report-out migration_report.yaml -``` - -Payload-rewrite shadow workflow: - -```text -rvl migrate plan --mode iterative_shadow --index products --target-schema target_schema.yaml --platform-inventory platform_inventory.yaml --transform-plan transform_plan.yaml --plan-out migration_plan.yaml -rvl migrate apply --plan migration_plan.yaml --report-out migration_report.yaml -``` - -Expected usage pattern: - -1. Provide the schema request and platform inventory. -2. Provide `transform_plan.yaml` when the target payload shape changes. -3. Review the capacity-gate outcome, estimated migration window, and estimated peak overlap footprint. -4. Run the shadow migration for one index only. -5. Hand cutover to the operator. -6. Confirm cutover before retiring the old index and any obsolete payloads. - -## Artifacts - -Required artifacts: - -- `migration_plan.yaml` -- `migration_report.yaml` - -Optional or phase-dependent artifacts: - -- `benchmark_report.yaml` -- `platform_inventory.yaml` -- `transform_plan.yaml` -- `benchmark_manifest.yaml` - -Artifact requirements: - -- YAML-based -- stable enough for handoff and review -- readable by humans first -- structured enough for future automation - -## Operational Model - -RedisVL owns: - -- source snapshot -- schema diffing -- plan generation -- supported strategy execution -- readiness waiting -- validation -- reporting - -Operators own: - -- maintenance windows -- application behavior during migration -- platform scaling -- transform inputs for payload-shape changes -- cutover -- final go or no-go decisions in production - -The product should not imply that RedisVL is a full migration control plane. It is a migration toolset with explicit operator handoff. - -## Capacity and Scale - -Phase 1 keeps capacity handling simple: - -- use source index stats for warnings -- capture timing and impact for later planning -- avoid a complex estimator in the MVP - -Phase 2 introduces a conservative planner: - -- reason at the database level, not as “an index lives on one shard” -- treat each index as one logical distributed index even on sharded deployments -- estimate source document footprint and source index footprint separately -- estimate target document footprint and target index footprint separately -- compute peak overlap as the source footprint plus the target footprint that exists during migration -- require reserve headroom before apply -- return `READY`, `SCALE_REQUIRED`, or `MANUAL_REVIEW_REQUIRED` - -The execution rule stays simple across both phases: - -- one index at a time - -This is the core design choice that keeps the system understandable at production scale. - -## Downtime and Disruption - -Phase 1 explicitly accepts downtime. - -Expected impacts: - -- search on the affected index is unavailable between drop and recreated index readiness -- query quality may be degraded while initial indexing completes -- shared Redis resources are consumed during rebuild -- large indexes need maintenance windows or application-level degraded mode handling - -Phase 2 aims to reduce disruption, but it still has operational costs: - -- old and new index structures overlap during migration -- payload-rewrite migrations may also duplicate payloads temporarily -- memory and size can either grow or shrink depending on datatype, precision, dimension, algorithm, and payload-shape changes - -These are product facts and must be visible in the plan and report artifacts. - -## Benchmarking and Success Metrics - -Benchmarking is a product requirement, not an afterthought. - -The product should help answer: - -- how long planning takes -- how long apply takes -- how long downtime or overlap lasts -- how much document throughput the migration achieves -- how query latency changes during the migration window -- how much memory and size change between source and target -- how accurate the peak-overlap estimate was - -Core success metrics: - -- migration plan generation succeeds for supported diffs -- unsupported diffs are blocked before mutation -- Phase 1 preserves documents -- Phase 2 produces deterministic shadow plans for supported vector and payload-shape migrations -- schema match and document count match succeed after migration -- reports include stable timing, correctness, and memory-delta metrics -- benchmark rehearsals are good enough to estimate future maintenance windows and scaling decisions with confidence - -## Functional Requirements - -- plan generation from live index plus requested schema changes -- schema patch normalization -- supported-versus-blocked diff classification -- guided wizard for supported Phase 1 changes -- explicit downtime acknowledgment in Phase 1 -- structured plan, report, and benchmark outputs -- validation of schema, counts, and indexing-failure deltas -- one-index-at-a-time execution -- Phase 2 capacity-gated shadow planning -- Phase 2 support for vector and payload-shape migrations through explicit shadow planning - -## Non-Functional Requirements - -- deterministic plan outputs -- human-readable YAML artifacts -- clear failure modes -- conservative defaults -- no document deletion by the Phase 1 migrator path -- reasonable operation on large indexes without default full-key manifests -- documentation detailed enough for implementation handoff - -## Risks - -- Users may assume unsupported Phase 1 schema changes should “just work” unless the diff classifier clearly routes them to Phase 2. -- Operators may underestimate downtime for large indexes unless benchmark outputs become part of the review flow. -- Phase 2 can grow too complex if transform logic or platform-specific automation is pulled in too early. -- Capacity estimation may be wrong unless benchmark data and observed footprint deltas are captured consistently. -- Validation may be treated as optional unless the CLI and reports make it central to the workflow. - -## Rollout Plan - -### Phase 1 - **Done** - -- ~~finalize docs and task list~~ Done -- ~~implement the planner, diff classifier, CLI flow, executor, and validator~~ Done (PRs #567-#572) -- ~~add CI coverage for supported and blocked migration paths~~ Done -- ~~run at least one benchmark rehearsal~~ Done (see 05_migration_benchmark_report.md) - -### Phase 1.5 - **Done** - -- ~~review real implementation learnings~~ Done -- ~~update the planning workspace~~ Done (this update) -- Phase 2 assumptions still hold; shadow migrations remain the right approach for incompatible changes - -### Phase 2 - -- implement inventory parsing -- implement transform or backfill plan modeling -- implement conservative capacity gating -- implement one-index-at-a-time shadow planning and execution -- add benchmark rehearsals for overlap duration, ETA accuracy, and memory-delta accuracy - -## Review Questions for the Team - -- Is the Phase 1 boundary narrow enough to ship quickly, but useful enough to solve real user pain? -- Is Phase 2 scoped clearly enough to own vector datatype, precision, dimension, algorithm, and payload-shape changes? -- Is operator-owned cutover still the right long-term boundary? -- Is the benchmarking scope sufficient to make migration windows and scaling decisions predictable without overbuilding a measurement subsystem? -- Does the one-index-at-a-time rule provide the right balance of simplicity and scale? - -## Decision Summary - -- Build the migration product in phases. -- Implement Phase 1 first and keep it intentionally narrow. -- Treat vector and payload-shape migrations as a core product goal, delivered in Phase 2 rather than ignored. -- Keep the plan artifact central to the workflow. -- Keep the operational model explicit. -- Use evidence from benchmark outputs to shape later migration planning. - -## References - -Detailed supporting docs in this workspace: - -- [00_index.md](./00_index.md) -- [01_context.md](./01_context.md) -- [02_architecture.md](./02_architecture.md) -- [03_benchmarking.md](./03_benchmarking.md) -- [10_v1_drop_recreate_spec.md](./10_v1_drop_recreate_spec.md) -- [11_v1_drop_recreate_tasks.md](./11_v1_drop_recreate_tasks.md) -- [12_v1_drop_recreate_tests.md](./12_v1_drop_recreate_tests.md) -- [20_v2_iterative_shadow_spec.md](./20_v2_iterative_shadow_spec.md) -- [21_v2_iterative_shadow_tasks.md](./21_v2_iterative_shadow_tasks.md) -- [22_v2_iterative_shadow_tests.md](./22_v2_iterative_shadow_tests.md) - -## User Journeys - -### Journey 1: Application Engineer Running a Simple Schema Migration - -An application engineer needs to add a new filterable metadata field to an existing index without deleting documents. They run `plan`, review the merged target schema and downtime warning, schedule a maintenance window, run `apply`, then run `validate` and hand the migration report to the team. They do not need to understand Redis internals beyond the migration inputs and the reported downtime. - -### Journey 2: Platform Engineer Reviewing a Vector Precision Migration - -A platform engineer needs to review a planned `FP32 -> FP16` migration for a large production index. They supply platform inventory, review the planner’s peak-overlap estimate, compare the projected post-cutover memory savings to previous benchmark reports, and decide whether the current deployment can run the migration safely in the next window. - -### Journey 3: Engineer Migrating from `HNSW` to `FLAT` - -An engineer wants to switch vector search behavior from `HNSW` to `FLAT` to simplify runtime performance characteristics. The planner classifies the request as a Phase 2 shadow migration, estimates the target index footprint, and produces a one-index-at-a-time plan. The operator runs the migration, validates the shadow target, and cuts traffic over once the benchmark and validation reports look acceptable. - -### Journey 4: Solutions Engineer Validating a Payload-Shape Change - -A solutions engineer wants to understand how long a customer migration will take when a new embedding model changes the stored payload shape. They create a `transform_plan.yaml`, run a rehearsal in non-production, collect benchmark timing, throughput, query-latency, and source-versus-target memory outputs, and use those artifacts to advise on maintenance windows and scaling needs. - -## User Stories - -- As an application engineer, I want to generate a migration plan before any mutation so that I can review the exact schema changes and downtime implications. -- As an application engineer, I want the Phase 1 migrator to preserve documents so that I do not have to rebuild my dataset from another source. -- As an application engineer, I want blocked Phase 1 schema changes to fail early and point me to the correct Phase 2 path so that I do not start a migration the product cannot safely complete. -- As an operator, I want migration and validation reports in YAML so that I can review, archive, and share them with other teams. -- As an operator, I want the CLI to require explicit downtime acknowledgment in Phase 1 so that maintenance-window risk is never implicit. -- As a platform engineer, I want Phase 2 to process one index at a time so that capacity planning stays understandable and bounded. -- As a platform engineer, I want the planner to estimate peak overlap and post-cutover memory deltas so that I can decide whether a migration fits safely. -- As a platform engineer, I want the shadow planner to return `READY`, `SCALE_REQUIRED`, or `MANUAL_REVIEW_REQUIRED` so that I can make a clear operational decision before execution. -- As a solutions engineer, I want benchmark outputs for duration, throughput, query impact, and memory change so that I can estimate future migrations with real evidence. -- As a maintainer, I want the migration product to reuse existing RedisVL primitives so that implementation and long-term maintenance stay simple. diff --git a/local_docs/index_migrator/99_tickets.md b/local_docs/index_migrator/99_tickets.md deleted file mode 100644 index 6d959aa9b..000000000 --- a/local_docs/index_migrator/99_tickets.md +++ /dev/null @@ -1,374 +0,0 @@ -# Index Migrator Tickets - ---- - -## Milestones - -| Milestone | Theme | Stories | -|-----------|-------|---------| -| M1 | Plan and Execute Single-Index Schema Migrations | IM-01, IM-05 | -| M2 | Interactive Migration Wizard | IM-02 | -| M3 | Rename Indexes, Prefixes, and Fields | IM-06 | -| M4 | Async Execution and Batch Operations | IM-03, IM-04 | -| M5 | Validation Fixes, Integration Tests, and Documentation | IM-07, IM-08, IM-09, IM-10 | - ---- - -## Completed - -### IM-01: Plan, Execute, and Validate Document-Preserving Index Schema Migrations - -**Status:** Done | **Commit:** `a3d534b` | **Milestone:** M1 - -**Story:** As a developer with an existing Redis index, I want to generate a reviewable migration plan, execute a safe drop-and-recreate, and validate the result, so that I can add/remove fields, change vector algorithms (FLAT/HNSW/SVS-VAMANA), change distance metrics (cosine/L2/IP), quantize vectors (float32 to float16/bfloat16/int8/uint8), and tune HNSW parameters (m, ef_construction, ef_runtime, epsilon) — all without losing documents. - -**What This Delivers:** -- **Discovery**: `rvl migrate list` shows all indexes, `rvl migrate helper` explains capabilities -- **Planning**: MigrationPlanner generates a plan from a schema patch or target schema. Captures source snapshot, target schema, classifies changes as supported or blocked. Incompatible changes (dimension, storage type) are rejected at plan time. -- **Execution**: MigrationExecutor drops the index definition (not documents), re-encodes vectors if quantization is needed, and recreates the index with the merged schema. -- **Validation**: MigrationValidator confirms schema match, doc count parity, key sample existence, and functional query correctness post-migration. -- **Reporting**: Structured `migration_report.yaml` with per-phase timings, counts, benchmark summary, and warnings. - -**Key Files:** `redisvl/migration/planner.py`, `executor.py`, `validation.py`, `models.py` - ---- - -### IM-02: Build Migration Plans Interactively via Guided Wizard - -**Status:** Done | **Commit:** `b06e949` | **Milestone:** M2 - -**Story:** As a developer unfamiliar with YAML schema syntax, I want a menu-driven wizard that walks me through adding, removing, updating, and renaming fields with real-time validation, so that I can build a correct migration plan without reading documentation. - -**What This Delivers:** -- `rvl migrate wizard --index ` launches an interactive session -- Menus for: add field (text/tag/numeric/geo), remove field (any type, with vector warning), rename field, update field attributes (sortable, weight, no_stem, phonetic_matcher, separator, case_sensitive, index_missing, index_empty), update vector settings (algorithm, datatype, distance metric, all HNSW and SVS-VAMANA params), rename index, change prefix -- Shows current schema and previews changes before generating plan -- Outputs both `schema_patch.yaml` and `migration_plan.yaml` -- Validates choices against what's actually supported - -**Key Files:** `redisvl/migration/wizard.py` - ---- - -### IM-03: Execute Migrations Asynchronously for Large Indexes - -**Status:** Done | **Commit:** `b559215` | **Milestone:** M4 - -**Story:** As a developer with a large index (1M+ vectors) in an async codebase, I want async migration planning, execution, and validation so that my application remains responsive and I don't block the event loop during long-running migrations. - -**What This Delivers:** -- `AsyncMigrationPlanner`, `AsyncMigrationExecutor`, `AsyncMigrationValidator` classes with full feature parity -- `rvl migrate apply --async` CLI flag -- Same `MigrationPlan` model works for both sync and async -- Same plan format works for both sync and async - -**Key Files:** `redisvl/migration/async_planner.py`, `async_executor.py`, `async_validation.py` - ---- - -### IM-04: Migrate Multiple Indexes in a Single Batch with Failure Isolation and Resume - -**Status:** Done | **Commit:** `61c6e80` | **Milestone:** M4 - -**Story:** As a platform operator with many indexes, I want to apply a shared schema patch to multiple indexes in one operation, choose whether to stop or continue on failure, and resume interrupted batches from a checkpoint, so that I can coordinate migrations during maintenance windows. - -**What This Delivers:** -- `BatchMigrationPlanner` generates per-index plans from a shared patch -- `BatchMigrationExecutor` runs migrations sequentially with state persistence -- Failure policies: `fail_fast` (stop on first error), `continue_on_error` (skip and continue) -- CLI: `batch-plan`, `batch-apply`, `batch-resume`, `batch-status` -- `batch_state.yaml` checkpoint file for resume capability -- `BatchReport` with per-index status and aggregate summary - -**Key Files:** `redisvl/migration/batch_planner.py`, `batch_executor.py` - ---- - -### IM-05: Optimize Document Enumeration Using FT.AGGREGATE Cursors - -**Status:** Done | **Commit:** `9561094` | **Milestone:** M1 - -**Story:** As a developer migrating a large index over a sparse keyspace, I want document enumeration to use the search index directly instead of SCAN, so that migration runs faster and only touches indexed keys. -``` -FT.AGGREGATE idx "*" - LOAD 1 __key # Get document key - WITHCURSOR COUNT 500 # Cursor-based pagination -``` - -**What This Delivers:** -- Executor uses `FT.AGGREGATE ... WITHCURSOR COUNT LOAD 0` for key enumeration -- Falls back to SCAN only when `hash_indexing_failures > 0` (those docs wouldn't appear in aggregate) -- Pre-enumerates all keys before dropping index for reliable re-indexing -- CLI simplified: removed `--allow-downtime` flag (plan review is the safety mechanism) - -**Key Files:** `redisvl/migration/executor.py`, `async_executor.py` - ---- - -### IM-06: Rename Indexes, Change Key Prefixes, and Rename Fields Across Documents - -**Status:** Done | **Commit:** pending | **Milestone:** M3 - -**Story:** As a developer, I want to rename my index, change its key prefix, or rename fields in my schema, so that I can refactor naming conventions without rebuilding from scratch. - -**What This Delivers:** -- Index rename: drop old index, create new with same prefix (no document changes) -- Prefix change: `RENAME` command on every key (single-prefix indexes only, multi-prefix blocked) -- Field rename: `HSET`/`HDEL` for hash, `JSON.SET`/`JSON.DEL` for JSON, on every document -- Execution order: field renames, then key renames, then drop, then recreate -- `RenameOperations` model in migration plan -- Timing fields: `field_rename_duration_seconds`, `key_rename_duration_seconds` -- Warnings issued for expensive operations - -**Key Files:** `redisvl/migration/models.py`, `planner.py`, `executor.py`, `async_executor.py` - -**Spec:** `local_docs/index_migrator/30_rename_operations_spec.md` - ---- - -### IM-07: Fix HNSW Parameter Parsing, Weight Normalization, and Algorithm Case Sensitivity - -**Status:** Done | **Commit:** `ab8a017` | **Milestone:** M5 - -**Story:** As a developer, I want post-migration validation to correctly handle HNSW-specific parameters, weight normalization, and algorithm case sensitivity, so that validation doesn't produce false failures. - -**What This Fixes:** -- HNSW-specific parameters (m, ef_construction) were not being parsed from `FT.INFO`, causing validation failures -- Weight int/float normalization mismatch (schema defines `1`, Redis returns `1.0`) -- Algorithm case sensitivity in wizard (schema stores `'hnsw'`, wizard compared to `'HNSW'`) - -**Key Files:** `redisvl/redis/connection.py`, `redisvl/migration/utils.py`, `redisvl/migration/wizard.py` - ---- - -### IM-08: Add Integration Tests for All Supported Migration Routes - -**Status:** Done | **Commit:** `b3d88a0` | **Milestone:** M5 - -**Story:** As a maintainer, I want integration tests covering algorithm changes, quantization, distance metrics, HNSW tuning, and combined migrations, so that regressions are caught before release. - -**What This Delivers:** -- 22 integration tests running full apply+validate against a live Redis instance -- Covers: 9 datatype routes, 4 distance metric routes, 5 HNSW tuning routes, 2 algorithm routes, 2 combined routes -- Tests require Redis 8.0+ for INT8/UINT8 datatypes -- Located in `tests/integration/test_migration_routes.py` - ---- - -### IM-09: Update Migration Documentation to Reflect Rename, Batch, and Redis 8.0 Support - -**Status:** Done | **Commit:** `d452eab` | **Milestone:** M5 - -**Story:** As a user, I want documentation that accurately reflects all supported migration operations, so that I can self-serve without guessing at capabilities. - -**What This Delivers:** -- Updated `docs/concepts/index-migrations.md` to reflect prefix/field rename support -- Updated `docs/user_guide/how_to_guides/migrate-indexes.md` with Redis 8.0 requirements -- Added batch migration commands to CLI reference in `docs/user_guide/cli.ipynb` -- Removed prefix/field rename from "blocked" lists - ---- - -### IM-10: Address PR Review Feedback for Correctness and Consistency - -**Status:** Done | **Commit:** pending | **Milestone:** M5 - -**Story:** As a maintainer, I want code review issues addressed so that the migration engine is correct, consistent, and production-ready. - -**What This Fixes:** -- `merge_patch()` now applies `rename_fields` to merged schema -- `BatchState.success_count` uses correct status string (`"succeeded"`) -- CLI helper text updated to show prefix/rename as supported -- Planner docstring updated to reflect current capabilities -- `batch_plan_path` stored in state for proper resume support -- Fixed `--output` to `--plan-out` in batch migration docs -- Fixed `--indexes` docs to use comma-separated format -- Added validation to block multi-prefix migrations -- Updated migration plan YAML example to match actual model -- Added `skipped_count` property and `[SKIP]` status display - -**Key Files:** `redisvl/migration/planner.py`, `models.py`, `batch_executor.py`, `redisvl/cli/migrate.py`, `docs/user_guide/how_to_guides/migrate-indexes.md` - ---- - -## Pending / Future - -### IM-R1: Add Crash-Safe Quantization with Checkpoint Resume and Pre-Migration Snapshot - -**Status:** Done | **Commit:** `30cc6c1` | **Priority:** High - -**Story:** As a developer running vector quantization on a production index, I want the migration to be resumable if it crashes mid-quantization, so that I don't end up with a partially quantized index and no rollback path. - -**Problem:** -The current quantization flow is: enumerate keys, drop index, quantize vectors in-place, recreate index, validate. If the process crashes during quantization, you're left with no index, a mix of float32 and float16 vectors, and no way to recover. - -**What This Delivers:** -A four-layer reliability model. A pre-migration `BGSAVE` (run sequentially, waited to completion) provides full disaster recovery by restoring the RDB to pre-migration state. A checkpoint file on disk tracks which keys have been quantized, enabling resume from the exact failure point on retry. Each key conversion detects the vector dtype before converting, making it idempotent so already-converted keys are safely skipped on resume. A bounded undo buffer stores originals for only the current in-flight batch, allowing rollback of the batch that was in progress at crash time. - -**Acceptance Criteria:** -1. Pre-migration `BGSAVE` is triggered and completes before any mutations begin -2. A checkpoint file records progress as each batch of keys is quantized -3. `rvl migrate apply --resume` picks up from the last checkpoint and completes the migration -4. Each key conversion is idempotent -- running the migration twice on the same key produces the correct result -5. If a batch fails mid-write, only that batch's vectors are rolled back using the bounded undo buffer -6. A disk space estimator function calculates projected RDB snapshot size, AOF growth, and total new disk required based on doc count, vector dimensions, source/target dtype, and AOF status. The estimator runs before any mutations and prints a human-readable summary. If available disk is below 80% of the estimate, the CLI prompts for confirmation. The estimator also supports a standalone dry-run mode via `rvl migrate estimate --plan plan.yaml`. See `local_docs/index_migrator/40_reliability_brainstorm.md` section "Pre-Migration Disk Space Estimator" for the full specification including inputs, outputs (DiskSpaceEstimate dataclass), calculation logic, CLI output format, integration points, and edge cases. - -**Alternatives Considered:** Undo log (WAL-style), new-field-then-swap (side-write), shadow index (blue-green), streaming with bounded undo buffer. See `local_docs/index_migrator/40_reliability_brainstorm.md` for full analysis. - ---- - -### IM-B1: Benchmark Float32 vs Float16 Quantization: Search Quality and Migration Performance at Scale - -**Status:** Planned | **Priority:** High - -**Story:** As a developer considering vector quantization to reduce memory, I want benchmarks measuring search quality degradation (precision, recall, F1) and migration performance (throughput, latency, memory savings) across realistic dataset sizes, so that I can make an informed decision about whether the memory-accuracy tradeoff is acceptable for my use case. - -**Problem:** -We tell users they can quantize float32 vectors to float16 to cut memory in half, but we don't have published data showing what they actually lose in search quality or what they can expect in migration performance at different scales. - -**What This Delivers:** -A benchmark script and published results using a real dataset (AG News with sentence-transformers embeddings) that measures two things across multiple dataset sizes (1K, 10K, 100K). For search quality: precision@K, recall@K, and F1@K comparing float32 (ground truth) vs float16 (post-migration) top-K nearest neighbor results. For migration performance: end-to-end duration, quantization throughput (vectors/second), index downtime, pre/post memory footprint, and query latency before and after (p50, p95, p99). - -**Acceptance Criteria:** -1. Benchmark runs end-to-end against a local Redis instance with a single command -2. Uses a real public dataset with real embeddings (not synthetic random vectors) -3. Reports precision@K, recall@K, and F1@K for float32 vs float16 search results -4. Reports per-query statistics (mean, p50, p95, min, max) not just aggregates -5. Runs at multiple dataset sizes (at minimum 1K, 10K, 100K) to show how quality and performance scale -6. Reports memory savings (index size delta in MB) and migration throughput (docs/second) -7. Reports query latency before and after migration -8. Outputs a structured JSON report that can be compared across runs - -**Note:** Benchmark script scaffolded at `tests/benchmarks/index_migrator_real_benchmark.py`. - ---- - -### IM-11: Run Old and New Indexes in Parallel for Incompatible Changes with Operator-Controlled Cutover - -**Status:** Future | **Priority:** Medium - -**Story:** As a developer changing vector dimensions or storage type, I want to run old and new indexes in parallel until I'm confident in the new one, so that I can migrate without downtime and rollback if needed. - -**Context:** -Some migrations cannot use `drop_recreate` because the stored data is incompatible (dimension changes, storage type changes, complex payload restructuring). Shadow migration creates a new index alongside the old one, copies/transforms documents, validates, then hands off cutover to the operator. - -**What This Requires:** -- Capacity estimation (can Redis hold both indexes?) -- Shadow index creation -- Document copy with optional transform -- Progress tracking with resume -- Validation gate before cutover -- Operator handoff for cutover decision -- Cleanup of old index/keys after cutover - -**Spec:** `local_docs/index_migrator/20_v2_iterative_shadow_spec.md` - ---- - -### IM-12: Pipeline Vector Reads During Quantization to Reduce Round Trips on Large Datasets - -**Status:** Backlog | **Priority:** Low - -**Story:** As a developer migrating large datasets, I want quantization reads to be pipelined so that migration completes faster. - -**Context:** -Current quantization implementation does O(N) round trips for reads (one `HGET` per key/field) while only pipelining writes. For large datasets this is slow. - -**What This Requires:** -- Pipeline all reads in a batch before processing -- Use `transaction=False` for read pipeline -- Add JSON storage support (`JSON.GET`/`JSON.SET`) for JSON indexes - ---- - -### IM-13: Wire ValidationPolicy Enforcement into Validators or Remove the Unused Model - -**Status:** Backlog | **Priority:** Low - -**Story:** As a developer, I want to skip certain validation checks (e.g., doc count) when I know they'll fail due to expected conditions. - -**Context:** -`MigrationPlan.validation` (ValidationPolicy) exists in the model but is not enforced by validators. Schema/doc-count mismatches always produce errors. - -**What This Requires:** -- Wire `ValidationPolicy.require_doc_count_match` into validators -- Add CLI flag to set policy during plan creation -- Or remove unused ValidationPolicy model - ---- - -### IM-14: Clean Up Unused Imports and Linting Across the Codebase - -**Status:** Backlog | **Priority:** Low - -**Story:** As a maintainer, I want clean linting so that CI is reliable and code quality is consistent. - -**Context:** -During development, pyflakes identified unused imports across the codebase. These were fixed in migration files but not committed for non-migration files to keep the PR focused. - -**What This Requires:** -- Fix remaining unused imports (see `local_docs/issues/unused_imports_cleanup.md`) -- Update `.pylintrc` to remove deprecated Python 2/3 compat options -- Consider adding `check-lint` to the main `lint` target after cleanup - ---- - -### IM-15: Use RENAMENX for Prefix Migrations to Fail Fast on Key Collisions - -**Status:** Backlog | **Priority:** Low - -**Story:** As a developer changing key prefixes, I want the migration to fail fast if target keys already exist, so I don't end up with a partially renamed keyspace. - -**Context:** -Current implementation uses `RENAME` without checking if destination key exists. If a target key exists, RENAME will error and the pipeline may abort, leaving a partially-renamed keyspace. - -**What This Requires:** -- Preflight check for key collisions or use `RENAMENX` -- Surface hard error rather than warning -- Consider rollback strategy - ---- - -### IM-16: Auto-Detect AOF Status for Disk Space Estimation - -**Status:** Backlog | **Priority:** Low - -**Story:** As an operator running `rvl migrate estimate`, I want the disk space estimate to automatically detect whether AOF is enabled on the target Redis instance, so that AOF growth is included in the estimate without me needing to know or pass a flag. - -**Context:** -The disk space estimator (`estimate_disk_space`) is a pure calculation that accepts `aof_enabled` as a parameter (default `False`). In CLI usage, this means AOF growth is never estimated unless the caller explicitly passes `aof_enabled=True`. The summary currently prints "not estimated (pass aof_enabled=True if AOF is on)" which is accurate but requires the operator to know their Redis config. - -**What This Requires:** -- Add `--aof-enabled` flag to `rvl migrate estimate` CLI for offline/pure-calculation use -- During `rvl migrate apply`, read `CONFIG GET appendonly` from the live Redis connection and pass the result to `estimate_disk_space` -- Handle `CONFIG GET` failures gracefully (e.g. ACL restrictions) by falling back to the current "not estimated" behavior - ---- - -## Summary - -| Ticket | Title | Status | -|--------|-------|--------| -| IM-01 | Plan, Execute, and Validate Document-Preserving Index Schema Migrations | Done | -| IM-02 | Build Migration Plans Interactively via Guided Wizard | Done | -| IM-03 | Execute Migrations Asynchronously for Large Indexes | Done | -| IM-04 | Migrate Multiple Indexes in a Single Batch with Failure Isolation and Resume | Done | -| IM-05 | Optimize Document Enumeration Using FT.AGGREGATE Cursors | Done | -| IM-06 | Rename Indexes, Change Key Prefixes, and Rename Fields Across Documents | Done | -| IM-07 | Fix HNSW Parameter Parsing, Weight Normalization, and Algorithm Case Sensitivity | Done | -| IM-08 | Add Integration Tests for All Supported Migration Routes | Done | -| IM-09 | Update Migration Documentation to Reflect Rename, Batch, and Redis 8.0 Support | Done | -| IM-10 | Address PR Review Feedback for Correctness and Consistency | Done | -| IM-R1 | Add Crash-Safe Quantization with Checkpoint Resume and Pre-Migration Snapshot | Done | -| IM-B1 | Benchmark Float32 vs Float16 Quantization: Search Quality and Migration Performance at Scale | Planned | -| IM-11 | Run Old and New Indexes in Parallel for Incompatible Changes with Operator-Controlled Cutover | Future | -| IM-12 | Pipeline Vector Reads During Quantization to Reduce Round Trips on Large Datasets | Backlog | -| IM-13 | Wire ValidationPolicy Enforcement into Validators or Remove the Unused Model | Backlog | -| IM-14 | Clean Up Unused Imports and Linting Across the Codebase | Backlog | -| IM-15 | Use RENAMENX for Prefix Migrations to Fail Fast on Key Collisions | Backlog | -| IM-16 | Auto-Detect AOF Status for Disk Space Estimation | Backlog | -| IM-17 | Guard Against Connection Leaks in Long-Running Batch Migrations | Backlog | -| IM-18 | Optimize O(n^2) Checkpoint Serialization for Large Key Sets | Backlog | -| IM-19 | Add Redis Cluster Slot-Aware Key Distribution for Quantization Batches | Backlog | -| IM-20 | Add Pipelined Reads for Quantization to Reduce Per-Key Round Trips | Backlog | - diff --git a/local_docs/index_migrator/pr_comments.md b/local_docs/index_migrator/pr_comments.md deleted file mode 100644 index db8f14544..000000000 --- a/local_docs/index_migrator/pr_comments.md +++ /dev/null @@ -1,700 +0,0 @@ -The case sensitivity - maybe use an enum or something else? - -Check these if they are true and useful as well. We dont want to implement everything, only if it makes sense. and it also makes errors, so go through them and before implementing give me an of how important and necessary they are: - -redisvl/migration/planner.py -Comment on lines +24 to +38 - """Migration planner for document-preserving drop/recreate flows. - - The `drop_recreate` mode drops the index definition and recreates it with - a new schema. Documents remain untouched in Redis. - - This means: - - Index-only changes work (algorithm, distance metric, tuning params) - - Document-dependent changes fail (the index expects data in a format - that doesn't match what's stored) - - Document-dependent changes (not supported): - - Vector dimensions: stored vectors have wrong number of dimensions - - Prefix/keyspace: documents are at keys the new index won't scan - - Field rename: documents store data under the old field name - - Storage type: documents are in hash format but index expects JSON -Copilot AI -27 minutes ago -The class docstring still claims prefix/keyspace changes and field renames are “not supported”, but the planner/executor now explicitly support prefix changes and rename_fields. Please update this docstring (and any related user-facing docs) to reflect the current supported operations so users don’t incorrectly avoid supported migrations. - -Suggested change - """Migration planner for document-preserving drop/recreate flows. - The `drop_recreate` mode drops the index definition and recreates it with - a new schema. Documents remain untouched in Redis. - This means: - - Index-only changes work (algorithm, distance metric, tuning params) - - Document-dependent changes fail (the index expects data in a format - that doesn't match what's stored) - Document-dependent changes (not supported): - - Vector dimensions: stored vectors have wrong number of dimensions - - Prefix/keyspace: documents are at keys the new index won't scan - - Field rename: documents store data under the old field name - - Storage type: documents are in hash format but index expects JSON - """Migration planner for drop/recreate-based index migrations. - The `drop_recreate` mode drops the index definition and recreates it with - a new schema. By default, documents are preserved in Redis, and when - possible the planner/executor can apply transformations so the preserved - documents remain compatible with the new index schema. - This means: - - Index-only changes are always safe (algorithm, distance metric, tuning - params, etc.) - - Some document-dependent changes are supported via explicit migration - operations in the migration plan - - Other document-dependent changes remain unsupported because existing - data cannot be safely or automatically adapted - Currently supported document-dependent changes include: - - Prefix/keyspace changes when specified in the target schema/patch - - Field renames via `rename_fields` operations in a schema patch - Document-dependent changes that remain unsupported: - - Vector dimensions: stored vectors have the wrong number of dimensions - for the new schema - - Storage type: documents are in hash format but the new index expects JSON -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/planner.py - fields_by_name = { - field["name"]: deepcopy(field) for field in schema_dict["fields"] - } - -Copilot AI -27 minutes ago -merge_patch() applies add/remove/update changes but ignores schema_patch.changes.rename_fields. The wizard adds renames via rename_fields, so the merged target schema will keep the old field name even though the executor renames document fields—leading to an index built with the wrong field name. Apply rename_fields during merge (with collision checks) or have the wizard encode renames as remove+add in the patch. - -Suggested change - # Apply field renames from the patch before other modifications. - # This ensures the merged schema's field names match the executor's - # renamed document fields. - for rename in getattr(changes, "rename_fields", []) or []: - # Be defensive about attribute names on FieldRename. - old_name = ( - getattr(rename, "from_name", None) - or getattr(rename, "old_name", None) - or getattr(rename, "source", None) - ) - new_name = ( - getattr(rename, "to_name", None) - or getattr(rename, "new_name", None) - or getattr(rename, "target", None) - ) - if not old_name or not new_name: - # As a fallback, inspect the model's dict representation. - rename_dict = ( - rename.dict() - if hasattr(rename, "dict") - else getattr(rename, "__dict__", {}) or {} - ) - old_name = old_name or rename_dict.get("from_name") or rename_dict.get("old_name") or rename_dict.get("source") - new_name = new_name or rename_dict.get("to_name") or rename_dict.get("new_name") or rename_dict.get("target") - if not old_name or not new_name: - raise ValueError(f"Invalid field rename specification: {rename!r}") - if old_name not in fields_by_name: - raise ValueError( - f"Cannot rename field '{old_name}' because it does not exist in the source schema" - ) - # If the new name already exists and is different from the source - # field, renaming would collide with another field. - if new_name in fields_by_name and new_name != old_name: - raise ValueError( - f"Cannot rename field '{old_name}' to '{new_name}' because a field with the new name already exists" - ) - if new_name == old_name: - # No-op rename; skip. - continue - field_def = fields_by_name.pop(old_name) - field_def["name"] = new_name - fields_by_name[new_name] = field_def -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/executor.py -Comment on lines +674 to +695 - total_keys = len(keys) - docs_processed = 0 - batch_size = 500 - - for i in range(0, total_keys, batch_size): - batch = keys[i : i + batch_size] - pipe = client.pipeline() - keys_updated_in_batch = set() - - for key in batch: - # Read all vector fields that need conversion - for field_name, change in datatype_changes.items(): - field_data: bytes | None = client.hget(key, field_name) # type: ignore[misc,assignment] - if field_data: - # Convert: source dtype -> array -> target dtype -> bytes - array = buffer_to_array(field_data, change["source"]) - new_bytes = array_to_buffer(array, change["target"]) - pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] - keys_updated_in_batch.add(key) - - if keys_updated_in_batch: - pipe.execute() -Copilot AI -27 minutes ago -Vector quantization reads each field via client.hget() inside the loop while also creating a pipeline for writes; this defeats pipelining and makes quantization O(N) round trips. Additionally, this assumes HASH storage; for JSON indexes the vector is not stored in a hash field, so quantization will be incorrect/no-op. Use pipelined HGET (or HMGET) for reads, set transaction=False, and branch on storage_type to use JSON.GET/SET for JSON storage. - -Suggested change - total_keys = len(keys) - docs_processed = 0 - batch_size = 500 - for i in range(0, total_keys, batch_size): - batch = keys[i : i + batch_size] - pipe = client.pipeline() - keys_updated_in_batch = set() - for key in batch: - # Read all vector fields that need conversion - for field_name, change in datatype_changes.items(): - field_data: bytes | None = client.hget(key, field_name) # type: ignore[misc,assignment] - if field_data: - # Convert: source dtype -> array -> target dtype -> bytes - array = buffer_to_array(field_data, change["source"]) - new_bytes = array_to_buffer(array, change["target"]) - pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] - keys_updated_in_batch.add(key) - if keys_updated_in_batch: - pipe.execute() - storage_type = getattr(source_index, "storage_type", "HASH").upper() - total_keys = len(keys) - docs_processed = 0 - batch_size = 500 - for i in range(0, total_keys, batch_size): - batch = keys[i : i + batch_size] - # First, pipeline all reads for this batch - read_pipe = client.pipeline(transaction=False) - key_field_pairs: list[tuple[str, str]] = [] - for key in batch: - # Read all vector fields that need conversion - for field_name in datatype_changes.keys(): - key_field_pairs.append((key, field_name)) - if storage_type == "JSON": - json_path = f"$.{field_name}" - read_pipe.execute_command("JSON.GET", key, json_path) - else: - # Default to HASH storage - read_pipe.hget(key, field_name) - if not key_field_pairs: - continue - read_results = read_pipe.execute() - # Now, process results and pipeline all writes for this batch - write_pipe = client.pipeline(transaction=False) - keys_updated_in_batch = set() - for (key, field_name), field_data in zip(key_field_pairs, read_results): - if not field_data: - continue - change = datatype_changes[field_name] - # Convert: source dtype -> array -> target dtype -> bytes - array = buffer_to_array(field_data, change["source"]) - new_bytes = array_to_buffer(array, change["target"]) - if storage_type == "JSON": - json_path = f"$.{field_name}" - write_pipe.execute_command("JSON.SET", key, json_path, new_bytes) - else: - write_pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] - keys_updated_in_batch.add(key) - if keys_updated_in_batch: - write_pipe.execute() -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/models.py -Comment on lines +203 to +206 - @property - def success_count(self) -> int: - return sum(1 for idx in self.completed if idx.status == "success") - -Copilot AI -27 minutes ago -BatchState.success_count counts entries with status == "success", but the executor records "succeeded" and the rest of the batch code checks for "succeeded". This makes CLI status reporting incorrect. Align the allowed status values (either change success_count to check "succeeded" or standardize statuses everywhere). - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/cli/migrate.py -Comment on lines +96 to +101 - -Not yet supported: - - Changing vector dimensions - - Changing key prefix or separator - - Changing storage type (hash to JSON) - - Renaming fields -Copilot AI -27 minutes ago -The CLI helper text lists prefix changes and field renames under “Not yet supported”, but the current planner/executor implement prefix changes and rename_fields. Update this help output (and any capability lists) to match the actual supported operations so users aren’t misled. - -Suggested change -Not yet supported: - - Changing vector dimensions - - Changing key prefix or separator - - Changing storage type (hash to JSON) - - Renaming fields - - Changing key prefix or separator - - Renaming fields -Not yet supported: - - Changing vector dimensions - - Changing storage type (hash to JSON) -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/concepts/index-migrations.md -Comment on lines +24 to +31 -| Tune algorithm parameters (M, EF_CONSTRUCTION) | Supported | -| Quantize vectors (float32 to float16) | Supported | -| Change vector dimensions | Blocked | -| Change key prefix | Blocked | -| Rename a field | Blocked | -| Change storage type (hash to JSON) | Blocked | -| Add a new vector field | Blocked | - -Copilot AI -27 minutes ago -This doc lists “Change key prefix” and “Rename a field” as blocked, but the implementation in this PR supports prefix changes and explicit field renames (with document rewriting). Please update the supported/blocked tables and the explanatory sections to match the actual behavior, or re-block these operations in code if they’re not intended to be supported yet. - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/user_guide/how_to_guides/migrate-indexes.md -rvl migrate batch-plan \ - --pattern "*_idx" \ - --schema-patch quantize_patch.yaml \ - --output batch_plan.yaml \ -Copilot AI -27 minutes ago -The batch CLI examples use flags/argument formats that don’t match the implemented argparse interface: batch-plan uses --plan-out (not --output), and --indexes is parsed as a single comma-separated string in code (space-separated values won’t work). Please adjust the examples to reflect the actual CLI syntax. - -Suggested change - --output batch_plan.yaml \ - --plan-out batch_plan.yaml \ -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/async_executor.py -Comment on lines +639 to +654 - pipe = client.pipeline() - keys_updated_in_batch: set[str] = set() - - for key in batch: - # Read all vector fields that need conversion - for field_name, change in datatype_changes.items(): - field_data: bytes | None = await client.hget(key, field_name) # type: ignore[misc,assignment] - if field_data: - # Convert: source dtype -> array -> target dtype -> bytes - array = buffer_to_array(field_data, change["source"]) - new_bytes = array_to_buffer(array, change["target"]) - pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] - keys_updated_in_batch.add(key) - - if keys_updated_in_batch: - await pipe.execute() -Copilot AI -27 minutes ago -Async quantization has the same issue as sync: it does await client.hget() per key/field (one round trip each) while only pipelining writes. This will be extremely slow for large datasets and still assumes HASH storage. Consider pipelining/batching reads as well (e.g., pipeline HGETs then a single execute), set transaction=False, and use JSON.GET/SET when storage_type is JSON. - -Suggested change - pipe = client.pipeline() - keys_updated_in_batch: set[str] = set() - for key in batch: - # Read all vector fields that need conversion - for field_name, change in datatype_changes.items(): - field_data: bytes | None = await client.hget(key, field_name) # type: ignore[misc,assignment] - if field_data: - # Convert: source dtype -> array -> target dtype -> bytes - array = buffer_to_array(field_data, change["source"]) - new_bytes = array_to_buffer(array, change["target"]) - pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] - keys_updated_in_batch.add(key) - if keys_updated_in_batch: - await pipe.execute() - # First, batch all reads using a non-transactional pipeline to avoid - # one round trip per HGET. - read_pipe = client.pipeline(transaction=False) - tasks_meta: List[tuple[str, str, Dict[str, Any]]] = [] - for key in batch: - # Read all vector fields that need conversion - for field_name, change in datatype_changes.items(): - read_pipe.hget(key, field_name) - tasks_meta.append((key, field_name, change)) - if not tasks_meta: - continue - read_results = await read_pipe.execute() - # Now, prepare a write pipeline for the updated fields. - write_pipe = client.pipeline() - keys_updated_in_batch: set[str] = set() - for (key, field_name, change), field_data in zip(tasks_meta, read_results): - # field_data may be None if the field is missing - if field_data: - # Convert: source dtype -> array -> target dtype -> bytes - array = buffer_to_array(field_data, change["source"]) - new_bytes = array_to_buffer(array, change["target"]) - write_pipe.hset(key, field_name, new_bytes) # type: ignore[arg-type] - keys_updated_in_batch.add(key) - if keys_updated_in_batch: - await write_pipe.execute() -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/executor.py -Comment on lines +220 to +237 - for key in batch: - # Compute new key name - if key.startswith(old_prefix): - new_key = new_prefix + key[len(old_prefix) :] - else: - # Key doesn't match expected prefix, skip - logger.warning( - f"Key '{key}' does not start with prefix '{old_prefix}'" - ) - continue - pipe.rename(key, new_key) - - try: - results = pipe.execute() - renamed += sum(1 for r in results if r is True or r == "OK") - except Exception as e: - logger.warning(f"Error in rename batch: {e}") - -Copilot AI -27 minutes ago -Key prefix migration uses RENAME without checking whether the destination key already exists. If a target key exists, RENAME will error and the pipeline will likely abort, leaving a partially-renamed keyspace. Consider preflight-checking for collisions (or using a safer strategy like RENAMENX + fail-fast), and surface a hard error rather than a warning to avoid inconsistent migrations. - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/validation.py -Comment on lines +42 to +45 - source_num_docs = int(plan.source.stats_snapshot.get("num_docs", 0) or 0) - target_num_docs = int(target_info.get("num_docs", 0) or 0) - validation.doc_count_match = source_num_docs == target_num_docs - -Copilot AI -27 minutes ago -MigrationPlan.validation (ValidationPolicy) isn’t used by MigrationValidator/AsyncMigrationValidator: schema/doc-count mismatches always produce errors regardless of the policy fields. This also makes docs like “skip count check” hard to implement. Either wire these flags into validation/error generation or remove the policy from the model until it’s enforced. - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/async_validation.py -Comment on lines +49 to +52 - source_num_docs = int(plan.source.stats_snapshot.get("num_docs", 0) or 0) - target_num_docs = int(target_info.get("num_docs", 0) or 0) - validation.doc_count_match = source_num_docs == target_num_docs - -Copilot AI -27 minutes ago -Like the sync validator, this ignores plan.validation (ValidationPolicy) and always treats schema/doc-count mismatches as errors. If ValidationPolicy is part of the plan contract, the async validator should respect it too (or the policy should be removed until enforced). - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/user_guide/how_to_guides/migrate-indexes.md -Outdated -Comment on lines +385 to +392 - -## What's Blocked - -| Change | Why | Workaround | -|--------|-----|------------| -| Change vector dimensions | Requires re-embedding | Re-embed with new model, reload data | -| Change prefix/keyspace | Documents at wrong keys | Create new index, reload data | -| Rename a field | Stored data uses old name | Create new index, reload data | -Copilot AI -27 minutes ago -This guide’s “What’s Blocked” table still lists prefix changes and field renames as blocked, but the implementation in this PR supports both (prefix changes via key renames; field renames via document updates). Please update this table (and any surrounding narrative) to reflect the current supported operations, including the operational risks/caveats. - -Suggested change -## What's Blocked -| Change | Why | Workaround | -|--------|-----|------------| -| Change vector dimensions | Requires re-embedding | Re-embed with new model, reload data | -| Change prefix/keyspace | Documents at wrong keys | Create new index, reload data | -| Rename a field | Stored data uses old name | Create new index, reload data | -| Change prefix/keyspace | ✅ | Uses key renames; coordinate carefully to avoid clients writing during migration | -| Rename a field | ✅ | Uses document updates; may be slow and resource-intensive on large datasets | -## What's Blocked -| Change | Why | Workaround | -|--------|-----|------------| -| Change vector dimensions | Requires re-embedding | Re-embed with new model, reload data | -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -tests/unit/test_async_migration_executor.py -Comment on lines +115 to +125 -@pytest.mark.asyncio -async def test_async_executor_validates_redis_url(): - """Test executor requires redis_url or redis_client.""" - plan = _make_basic_plan() - executor = AsyncMigrationExecutor() - - # The executor should raise an error internally when trying to connect - # but let's verify it doesn't crash before it tries to apply - # For a proper test, we'd need to mock AsyncSearchIndex.from_existing - # For now, we just verify the executor is created - assert executor is not None -Copilot AI -27 minutes ago -test_async_executor_validates_redis_url doesn’t actually validate behavior (it only asserts the executor is instantiated). Either mock the Redis connection layer and assert the expected error/report when neither redis_url nor redis_client is provided, or rename/remove this test to avoid giving a false sense of coverage. - -Copilot uses AI. Check for mistakes. - -redisvl/migration/models.py -Comment on lines +71 to +77 -class RenameOperations(BaseModel): - """Tracks which rename operations are required for a migration.""" - - rename_index: Optional[str] = None # New index name if renaming - change_prefix: Optional[str] = None # New prefix if changing - rename_fields: List[FieldRename] = Field(default_factory=list) - -Copilot AI -13 minutes ago -RenameOperations.change_prefix is typed as Optional[str], but IndexSchema.index.prefix supports Union[str, List[str]]. As implemented elsewhere (e.g., using prefixes[0]), prefix-change migrations for multi-prefix indexes will behave incorrectly or partially. Either explicitly block list-prefix migrations in the planner/validator, or extend rename_operations/change_prefix handling to support lists end-to-end. - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/batch_executor.py -Comment on lines +72 to +77 - # Initialize or load state - state = self._init_or_load_state(batch_plan, state_path) - started_at = state.started_at - batch_start_time = time.perf_counter() - - # Get applicable indexes -Copilot AI -13 minutes ago -_init_or_load_state initializes BatchState.plan_path to an empty string, but resume() defaults to using state.plan_path when batch_plan_path is not provided. This breaks the advertised "resume without extra args" flow and causes FileNotFoundError unless users always pass --plan. Store the batch plan path in state when initializing (e.g., add a batch_plan_path parameter to apply() and persist it). - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/migration/async_validation.py -Comment on lines +67 to +78 - # Handle prefix change: transform key_sample to use new prefix - keys_to_check = key_sample - if plan.rename_operations.change_prefix: - old_prefix = plan.source.keyspace.prefixes[0] - new_prefix = plan.rename_operations.change_prefix - keys_to_check = [ - new_prefix + k[len(old_prefix) :] if k.startswith(old_prefix) else k - for k in key_sample - ] - existing_count = await client.exists(*keys_to_check) - validation.key_sample_exists = existing_count == len(keys_to_check) - -Copilot AI -13 minutes ago -Key-sample validation assumes a single old prefix (prefixes[0]) when applying a prefix change. For multi-prefix indexes (IndexSchema supports prefix as a list), this check can produce false failures or miss missing keys. Either block list-prefix migrations or update the rewrite logic to handle all prefixes. - -Suggested change - # Handle prefix change: transform key_sample to use new prefix - keys_to_check = key_sample - if plan.rename_operations.change_prefix: - old_prefix = plan.source.keyspace.prefixes[0] - new_prefix = plan.rename_operations.change_prefix - keys_to_check = [ - new_prefix + k[len(old_prefix) :] if k.startswith(old_prefix) else k - for k in key_sample - ] - existing_count = await client.exists(*keys_to_check) - validation.key_sample_exists = existing_count == len(keys_to_check) - # Default to False; will be updated only if the check runs successfully. - validation.key_sample_exists = False - # Handle prefix change: transform key_sample to use new prefix - keys_to_check = key_sample - multi_prefix_unsupported = False - if plan.rename_operations.change_prefix: - prefixes = plan.source.keyspace.prefixes or [] - if len(prefixes) == 1: - old_prefix = prefixes[0] - new_prefix = plan.rename_operations.change_prefix - keys_to_check = [ - new_prefix + k[len(old_prefix) :] - if k.startswith(old_prefix) - else k - for k in key_sample - ] - else: - # IndexSchema supports multiple prefixes, but change_prefix assumes a single - # old prefix. To avoid incorrect key rewriting, we skip this check. - multi_prefix_unsupported = True - validation.errors.append( - "Key-sample existence check does not support change_prefix when " - "source.keyspace.prefixes has multiple entries." - ) - if not multi_prefix_unsupported: - existing_count = await client.exists(*keys_to_check) - validation.key_sample_exists = existing_count == len(keys_to_check) -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/user_guide/how_to_guides/migrate-indexes.md -rvl migrate batch-plan \ - --pattern "*_idx" \ - --schema-patch quantize_patch.yaml \ - --output batch_plan.yaml \ -Copilot AI -13 minutes ago -The batch migration examples use a non-existent --output flag. The CLI implements --plan-out for batch-plan (and uses --plan for batch-apply). Update the docs to match the actual CLI flags to avoid copy/paste failures. - -Suggested change - --output batch_plan.yaml \ - --plan-out batch_plan.yaml \ -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/user_guide/how_to_guides/migrate-indexes.md -Comment on lines +557 to +564 -**Select indexes by explicit list:** -```bash -rvl migrate batch-plan \ - --indexes products_idx users_idx orders_idx \ - --schema-patch quantize_patch.yaml \ - --output batch_plan.yaml \ - --url redis://localhost:6379 -``` -Copilot AI -13 minutes ago -The docs show --indexes products_idx users_idx orders_idx, but the CLI parses --indexes as a comma-separated string (args.indexes.split(",")). Either update the CLI to accept nargs-style lists, or update the docs to use comma-separated values (e.g., --indexes "products_idx,users_idx,orders_idx"). - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/user_guide/how_to_guides/migrate-indexes.md -Comment on lines +209 to +214 -```yaml -# migration_plan.yaml (example) -version: 1 -index_name: myindex -migration_mode: drop_recreate - -Copilot AI -13 minutes ago -The migration plan YAML example does not match the actual MigrationPlan model emitted by planner.write_plan (e.g., it uses index_name/migration_mode/changes keys that aren't present). Replace the example with a real plan structure (mode/source/requested_changes/merged_target_schema/diff_classification/rename_operations/warnings/validation) to avoid confusing users. - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/user_guide/how_to_guides/migrate-indexes.md - -### Validation failed: "document count mismatch" - -Documents were added or removed between plan and apply. This is expected if your application is actively writing. Re-run validation with `--skip-count-check` if acceptable. -Copilot AI -13 minutes ago -The troubleshooting section references a --skip-count-check flag, but the CLI validate command doesn't define this option. Either add the flag and wire it to ValidationPolicy.require_doc_count_match, or update the docs to the supported behavior. - -Suggested change -Documents were added or removed between plan and apply. This is expected if your application is actively writing. Re-run validation with `--skip-count-check` if acceptable. -Documents were added or removed between plan and apply. This is expected if your application is actively writing. Wait until writes have quiesced, then re-run `rvl migrate validate`, or regenerate the migration plan and run `apply` and `validate` again. -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -docs/user_guide/how_to_guides/migrate-indexes.md -Comment on lines +548 to +555 -**Select indexes by pattern:** -```bash -rvl migrate batch-plan \ - --pattern "*_idx" \ - --schema-patch quantize_patch.yaml \ - --output batch_plan.yaml \ - --url redis://localhost:6379 -``` -Copilot AI -13 minutes ago -This section also uses --output for batch-plan, but the CLI flag is --plan-out. Update these examples (and the other --output occurrences in this doc) to match the implemented CLI options. - -Copilot uses AI. Check for mistakes. -@nkanu17 Reply... -redisvl/cli/migrate.py -Comment on lines +638 to +645 - print(f"Completed: {len(state.completed)}") - print(f" - Succeeded: {state.success_count}") - print(f" - Failed: {state.failed_count}") - - if state.completed: - print("\nCompleted indexes:") - for idx in state.completed: - status_icon = "[OK]" if idx.status == "succeeded" else "[FAIL]" -Copilot AI -13 minutes ago -batch-status output relies on BatchState.success_count and also treats anything other than status == "succeeded" as a failure. With the current model/executor status strings, skipped indexes will display as [FAIL], and success_count is also inconsistent (it counts "success"). Consider normalizing status enums and rendering skipped separately (e.g., [SKIP]). - -Suggested change - print(f"Completed: {len(state.completed)}") - print(f" - Succeeded: {state.success_count}") - print(f" - Failed: {state.failed_count}") - if state.completed: - print("\nCompleted indexes:") - for idx in state.completed: - status_icon = "[OK]" if idx.status == "succeeded" else "[FAIL]" - # Normalize and summarize completed index statuses locally to ensure - # consistent handling of success/skip/failure states, regardless of - # how BatchState aggregates counts. - success_statuses = {"succeeded", "success"} - skipped_statuses = {"skipped"} - succeeded = 0 - failed = 0 - skipped = 0 - for idx in state.completed: - if idx.status in success_statuses: - succeeded += 1 - elif idx.status in skipped_statuses: - skipped += 1 - else: - failed += 1 - print(f"Completed: {len(state.completed)}") - print(f" - Succeeded: {succeeded}") - print(f" - Failed: {failed}") - print(f" - Skipped: {skipped}") - if state.completed: - print("\nCompleted indexes:") - for idx in state.completed: - if idx.status in success_statuses: - status_icon = "[OK]" - elif idx.status in skipped_statuses: - status_icon = "[SKIP]" - else: - status_icon = "[FAIL]" -Copilot uses AI. Check for mistakes. - - ---- - -## Deferred Items from Review Rounds (Latest Triage) - -### Performance -- **Pipelined reads for quantization**: Both sync and async executors do per-key HGET reads. Should batch with pipelines for large datasets. -- **keys_to_process materialization**: `_enumerate_indexed_keys()` materializes full key list in memory. Consider streaming for large indexes. -- **Double snapshot in planner**: `create_plan()` snapshots source, then `create_plan_from_patch()` snapshots again. Reuse the initial snapshot. -- **Async executor duplicates readiness polling**: Has its own implementation instead of calling `async_wait_for_index_ready`. Consolidate. - -### Edge Cases -- **SCAN vs FT.AGGREGATE key mismatch on resume**: Initial run may enumerate via FT.AGGREGATE but resume uses SCAN, potentially producing different key sets. -- **Redis Cluster slot issues with prefix rename**: RENAME/RENAMENX only works when source/dest hash to same slot. -- **Empty prefix SCAN fallback**: When FT.INFO yields empty prefix, async executor scans entire keyspace without warning (sync version warns). -- **JSON path assumption in field renames**: Renames assume `$.` but non-default paths could exist. -- **DiskSpaceEstimate negative values**: Up-quantization (float16->float32) produces negative "savings" - `_format_bytes()` doesn't handle gracefully. -- **Batch plan stale state reuse**: `_init_or_load_state` doesn't validate state matches current batch plan on resume. -- **Batch duplicate index names**: No dedup in batch plan, later reports overwrite earlier ones. -- **list-valued prefix in wizard**: Wizard assumes single string prefix but schemas can have multiple. - -### UX/CLI -- **`_prompt_from_choices` block_message on any invalid input**: Shows vector-specific message on typos. -- **`_prompt_bool` empty input returns False**: Could be confusing with `[y/n]` prompt. -- **PR description API names don't match implementation**: `plan()` vs `create_plan()`, `execute()` vs `apply()`. -- **CLI `--ssl` URL construction bug**: Appends `rediss://` incorrectly. -- **CLI `rvl migrate` not in API docs**: `docs/api/cli.rst` doesn't document the new migrate command group. -- **Exit code contract inconsistency**: Some commands call `exit(0)` on exceptions. -- **Wizard `run()` doesn't validate Redis connection upfront**: Fails late in `snapshot_source()`. -- **Wizard numeric validation inconsistent**: `ef_runtime` requires `> 0` but `m`, `ef_construction` accept `0`. - -### Documentation -- **Migration plan YAML example doesn't match model**: Shows `mode`/`warnings` under `diff_classification` but model only has `supported`/`blocked_reasons`. -- **Source section missing fields in plan example**: Missing `index_name`, `storage_type`, `key_separator`. -- **"Other document dependent changes are blocked" text outdated**: Field renames and prefix renames are now supported. -- **Recovery guidance doesn't mention `--resume`**: Should document checkpoint-based resume workflow. - -### Testing -- **No e2e tests for executor rename/prefix mutation paths**: Integration tests cover basic drop/recreate but not data-mutation paths. -- **test_migration_routes doesn't load actual vector docs**: Quantization tests pass vacuously with empty indexes. -- **Wizard `run()` not unit tested**: Only `_build_patch()` is tested. -- **`create_mock_executor` `succeed_on` param unused**: Test helper is misleading. -- **Batch resume progress position restarts at 1**: Should compute position from completed count. -- **JSON schema `path` in test_comprehensive**: `path` is a top-level field property, not inside `attrs` - test may not validate actual JSON path behavior. - -### Code Quality -- **`ValidationPolicy` flags not respected**: `validate()` always errors on mismatches regardless of policy. -- **Checkpoint JSON storage not marked completed**: JSON storage creates checkpoint but never marks it completed. -- **`index_empty` prompt for non-applicable field types**: Collected for numeric/geo fields where it's not part of schema. - -### Round N+2 Codex Comments (2026-04-01) -- **executor.py:325 - Abort on hash field rename batch failure (P1)**: Currently logs warning and continues on `pipe.execute()` failure. Codex wants abort/raise. Deferred: current "best-effort" is intentional -- aborting mid-batch leaves partial renames with no rollback. Caller can compare `renamed` count vs expected. A proper fix needs retry/partial-rollback tracking. -- **batch_planner.py:174 - Resolve renamed fields before update-field checks (P2)**: Check rejects `update_fields` whose name is missing from source schema. Codex says this misses valid patches that rename A->B then update B. Near-false-positive: the standard pattern is to update A (by current name) AND rename A->B separately. The edge case of referencing post-rename name in `update_fields` is unlikely in practice. Deferred. \ No newline at end of file