refactor(lance-index): cache tokenize_and_count's constant schema fie… - #8684
refactor(lance-index): cache tokenize_and_count's constant schema fie…#8684jagrutipatilp wants to merge 1 commit into
Conversation
…lds via LazyLock The output_fields for FLAT_ALL_TOKENS_COL, the FixedSizeList item field, and FLAT_PHRASE_MATCH_COL are identical on every call. Build them once via LazyLock instead of reallocating Field/DataType on each invocation. Fixes lance-format#8244
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
The schema behavior is preserved and the nested fixed-size-list item field is now genuinely shared. The allocation reduction is narrower than the PR description because the two top-level Field values are still deep-cloned per query; using shared field references would realize the full optimization. This is bounded to flat-query setup and does not affect correctness.
| const FLAT_QUERY_TOKEN_COUNTS_COL: &str = "query_token_counts"; | ||
| const FLAT_PHRASE_MATCH_COL: &str = "phrase_match"; | ||
|
|
||
| static FLAT_ALL_TOKENS_FIELD: LazyLock<Field> = |
There was a problem hiding this comment.
LazyLock<Field> does not share this field allocation: Arrow 58.4 derives Clone for Field, so cloning deep-copies its String, and Schema::new(Vec<Field>) then wraps the clone in a fresh Arc. The same applies to FLAT_PHRASE_MATCH_FIELD; only the nested LazyLock<Arc<Field>> is actually shared. An allocator-counting check against arrow-schema 58.4.0 recorded 1,000 allocations for 1,000 direct constructions, 1,000 for 1,000 LazyLock<Field> clones, and 0 for LazyLock<Arc<Field>> clones. Using Arc<Field> statics and a Vec<FieldRef> would remove these remaining allocations. This is non-blocking because behavior is unchanged and the nested item allocation is eliminated.
lance-gatekeeper
Bot
added
K-approved
1 participant
…lds via LazyLock
The output_fields for FLAT_ALL_TOKENS_COL, the FixedSizeList item field, and FLAT_PHRASE_MATCH_COL are identical on every call. Build them once via LazyLock instead of reallocating Field/DataType on each invocation.
Fixes #8244