Skip to content

[C API] Implement memory estimation functions for index builders and tests - #364

Draft
rfsaliev wants to merge 2 commits into
dev/c-apifrom
rfsaliev/c-api-estimate-index-size
Draft

[C API] Implement memory estimation functions for index builders and tests#364
rfsaliev wants to merge 2 commits into
dev/c-apifrom
rfsaliev/c-api-estimate-index-size

Conversation

@rfsaliev

Copy link
Copy Markdown
Member

This pull request adds support for estimating the memory usage of an index (both static and dynamic Vamana) based on the builder configuration and number of vectors, and implements the necessary logic and API functions in the C bindings. It introduces new estimation methods in the C API, adds implementations for memory estimation in the builder and dispatcher classes, and updates data builders to support size estimation for various data types.

API Additions:

  • Added two new C API functions: svs_index_builder_estimate_memory and svs_index_builder_estimate_memory_dynamic to estimate the memory usage of an index and a dynamic index, respectively, based on the builder configuration and number of vectors.

Core Logic and Implementation:

  • Implemented estimate_memory_breakdown and estimate_memory_breakdown_dynamic methods in the IndexBuilder class to provide memory usage estimates for static and dynamic Vamana indexes.
  • Added dispatcher functions dispatch_vamana_memory_estimate and dispatch_dynamic_vamana_memory_estimate to calculate the memory breakdown for static and dynamic Vamana indexes, including graph, data, and metadata sizes. [1] [2]
  • Updated CMake configuration to include the new data_builder.cpp source file.

Data Builder Enhancements:

  • Added estimate_size methods to all data builder classes (SimpleDataBuilder, LVQDataBuilder, LeanVecDataBuilder, SQDataBuilder) to support size estimation for different data types. [1] [2] [3] [4] [5]
  • Created a new data_builder.cpp implementation file with logic for dispatching and estimating data sizes for various data builder specializations, including handling for blocked and non-blocked allocators.

Header and Dependency Updates:

  • Updated headers and includes to wire up the new estimation logic and expose the new API and dispatcher functions. [1] [2] [3] [4] [5] [6]

These changes collectively provide a robust mechanism for clients to estimate index memory requirements before actually building an index, improving usability and resource planning.

@rfsaliev

Copy link
Copy Markdown
Member Author

This PR contains number of workarounds/TODOs which should be resolved out-of the bindings/c directory.
To avoid potential conflicts, keeping the PR as 'Draft' until getting bindings/c on the main branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant