Skip to content

feat(diskann): Add Windows DiskANN support with asynchronous I/O - #625

Open
iaojnh wants to merge 142 commits into
alibaba:mainfrom
iaojnh:feat/diskann-support-windows
Open

feat(diskann): Add Windows DiskANN support with asynchronous I/O#625
iaojnh wants to merge 142 commits into
alibaba:mainfrom
iaojnh:feat/diskann-support-windows

Conversation

@iaojnh

@iaojnh iaojnh commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR enables DiskANN on 64-bit Windows using aligned, unbuffered asynchronous
reads backed by Windows I/O completion ports (IOCP).

It also fixes a Windows-specific file-handle interaction that reduced random
read throughput to about 13% of the expected level, improves hot-node cache
preloading, and exposes a byte-oriented cache budget through the existing C++,
C, Python, protobuf, and JSON parameter paths.

Root cause and fix

DiskANN issues many aligned 4 KiB random reads. The standalone IOCP reader
reached about 29K IOPS, but the same reader inside a loaded searcher reached
only about 3.8K IOPS.

An ABBA test isolated the cause: FileReadStorage kept an ordinary buffered
file handle open beside DiskANN's
FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED handles.

Condition IOPS geometric mean Retention
Buffered handle closed 29,299 100%
Buffered handle held open 3,809 13.0%

The fix loads the metadata required by search, closes the shared buffered
native handle, releases the storage segments, and then opens DiskANN's aligned
IOCP reader. Vector fetching and streamer providers now use that reader as
well, while providers and iterators remain valid after the source streamer is
closed.

Main changes

  • Add a Windows IOCP reader using
    FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED.
  • Use an independent file handle and completion port for each I/O context.
  • Validate 4 KiB alignment, completion identity, completed length, duplicate
    completions, stale batches, and short reads.
  • Cancel and drain outstanding requests before releasing their contexts or
    destination buffers.
  • Support UTF-8/Unicode Windows index paths.
  • Add cache_node_budget_bytes as an alternative to cache_node_num, with a
    conservative per-node estimate and the existing 10% node cap.
  • Batch cache preloading and reuse nodes already read during BFS.
  • Report windows_overlapped through the existing I/O backend APIs.
  • Make Recall tooling portable on Windows and propagate worker failures to the
    process exit status.

Performance validation

Cohere 1M, 768 dimensions, cosine distance, FP32, 1,000 queries, Windows Server
2022, with a 128 MiB hot-node cache budget:

Metric Before handle fix After fix
Cache preload 10.36 s 1.06 s
Average latency 26.3 ms 4.3 ms
QPS 38.0 235.1

Both runs loaded the same 40,088 cache nodes with zero failures. The full
FP32/FP16 parameter sweep retained the expected Recall range and reached about
27.5K process read operations/s. These are process I/O counters, not
device-wide hardware IOPS.

Compatibility and tests

  • Existing DiskANN indexes do not require rebuilding.
  • cache_node_budget_bytes defaults to 0; existing cache_node_num
    configurations remain supported.
  • Old protobuf, JSON, and Python pickle data default the new field to 0.
  • The added field changes the binary layout of exported C++ parameter classes;
    applications using the C++ shared-library ABI must rebuild against the new
    headers.

Tests cover concurrent IOCP contexts, outstanding-request destruction,
misaligned and short reads, Unicode paths, retained storage references,
provider lifetime after streamer close, cache loading, parameter
serialization/validation, and Recall worker failure propagation.

Known limitation

cache_node_budget_bytes is a nominal budget per physical DiskANN index. It is
not yet charged against the process-wide memory_limit_mb; the default remains
disabled (0).

@iaojnh iaojnh changed the title feat(diskann): support windows feat(diskann): Add Windows DiskANN support with asynchronous I/O Aug 19, 2026
@iaojnh
iaojnh marked this pull request as ready for review August 19, 2026 07:48
@iaojnh
iaojnh marked this pull request as draft August 20, 2026 09:02
@iaojnh
iaojnh marked this pull request as ready for review August 20, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants