feat: Implement standard tokenizer based on UAX29#547
Open
egolearner wants to merge 1 commit into
Open
Conversation
Summary: - Replace the previous general-category tokenizer with Unicode 17 UAX alibaba#29 word-boundary tables generated from official Unicode data. - Add the Unicode table generator and generated C++ lookup table. - Implement Lucene-style token selection for alphanumeric, numeric, ideographic, hiragana, katakana, hangul, southeast Asian, regional indicator, and common emoji sequences. - Keep the public Token contract as text, UTF-8 byte offset, and consecutive position, and validate max_token_length in the Lucene range [1, 1048576]. - Optimize hot paths with ASCII classification tables, page-indexed Unicode range lookup, lightweight UTF-8 decoding, a thread-local codepoint property cache, compact word-break classes, and non-ASCII dispatch fast paths. - Extend tokenizer tests and keep the Unicode WordBreakTest token projection exact. Differences from Elasticsearch standard tokenizer: - zvec does not expose Lucene token type attributes; token type is internal only. - zvec offsets are UTF-8 byte offsets, while Elasticsearch/Lucene offsets follow Java character offset semantics. - Unicode data is pinned to 17.0.0, so behavior can differ from Elasticsearch versions built with another Lucene or Unicode data version. - The emoji handling covers common keycap, modifier, regional-indicator, and Extended_Pictographic ZWJ sequences, but it is not a full copy of Lucene's TR51 emoji grammar. - Invalid UTF-8 is skipped at the tokenizer layer; Elasticsearch normally receives already-decoded Java strings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Replace the previous general-category tokenizer with Unicode 17 UAX
#29word-boundary tables generated from official Unicode data.Add the Unicode table generator and generated C++ lookup table.
Implement Lucene-style token selection for alphanumeric, numeric, ideographic, hiragana, katakana, hangul, southeast Asian, regional indicator, and common emoji sequences.
Keep the public Token contract as text, UTF-8 byte offset, and consecutive position, and validate max_token_length in the Lucene range [1, 1048576].
Optimize hot paths with ASCII classification tables, page-indexed Unicode range lookup, lightweight UTF-8 decoding, a thread-local codepoint property cache, compact word-break classes, and non-ASCII dispatch fast paths.
Extend tokenizer tests and keep the Unicode WordBreakTest token projection exact.
Differences from Elasticsearch standard tokenizer:
zvec does not expose Lucene token type attributes; token type is internal only.
zvec offsets are UTF-8 byte offsets, while Elasticsearch/Lucene offsets follow Java character offset semantics.
Unicode data is pinned to 17.0.0, so behavior can differ from Elasticsearch versions built with another Lucene or Unicode data version.
The emoji handling covers common keycap, modifier, regional-indicator, and Extended_Pictographic ZWJ sequences, but it is not a full copy of Lucene's TR51 emoji grammar.
Invalid UTF-8 is skipped at the tokenizer layer; Elasticsearch normally receives already-decoded Java strings.