Repository navigation
Conversation
…r loop can stop early
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues block the text-chunker early-termination change.
No open issues 🎉
What changed in this PR
Fixes #14490 by allowing Python’s text chunker to stop trying weaker separator tiers once every chunk fits the token budget, avoiding redundant processing while continuing to split oversized remainders.
Changes:
- Redefines the split flag to track whether all returned chunks fit.
- Combines flags with AND and enables the separator loop’s early exit.
- Adds regression tests for flag behavior, early termination, and oversized remainders.
| File | Description |
|---|---|
python/tests/unit/text/test_text_chunker.py |
Adds three regression tests for safe early termination. |
python/semantic_kernel/text/text_chunker.py |
Updates flag propagation and stops processing once all chunks fit. |
Review effort: Balanced
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Fixes #14490
python/semantic_kernel/text/text_chunker.pythreads a "was split" flag (input_was_split/was_split) through_split_str,_split_listand_split_str_lineswith the intent of short-circuiting the ordered separator-tier loop (TEXT_SPLIT_OPTIONS/MD_SPLIT_OPTIONS) once an earlier tier has resolved the input. As the issue describes, the flag can never becomeTrue: every base case returnsFalse, and the recursive cases only OR thoseFalsevalues together. The maintainers' own# pragma: no coveron thebreakalready flagged the branch as never executed. As a result every tier (newline,.,?!,;,:,,, brackets, space,-, and the final hard split) always re-scans every already-resolved chunk, doing up to ~10x more splitting work than the tiered design intends (e.g. 240_split_strcalls instead of 48 on a 40-sentence input in my measurements).Description
The fix makes the flag report what the loop actually needs to know to stop safely: whether every chunk returned so far fits within
max_tokens._split_strnow returnsTruewhen its result is fully within budget (a fitting chunk, or a split whose parts all fit). It returnsFalsewhen a chunk could not be split with the current tier, so weaker tiers keep running. A chunk that trims down to""is still reported as unresolved, so the next tier drops it exactly as before._split_listANDs the per-line flags instead of ORing them, so one fitting line no longer masks another oversized line._split_str_linesbreaks when the flag isTrue; the# pragma: no coveris removed because the branch is now reachable and covered by the new tests.Note why the flag must be AND (all-chunks-fit) rather than the issue's suggested "a split happened here" (True on any cutpoint): a strong separator can split most of a text while leaving one oversized chunk that contains no occurrence of that separator. Breaking on "a split happened" would leave that chunk over budget; weaker tiers must keep running while any oversized chunk remains. This matches the correctness concern raised in the issue discussion. The third new test (
test_split_text_lines_keeps_splitting_oversized_remainders) locks this behavior in.Outputs are unchanged by this fix: I verified byte-identical chunk output for 24 corpus cases (all existing unit-test inputs, the issue's repro, mixed-separator inputs with oversized remainders, whitespace-only text, CRLF/CR handling, hard-split cases) before and after the change — the early exit only fires when the remaining tiers are provably no-ops (they only act on chunks that exceed
max_tokens). Efficiency on the corpus improved from e.g. 43→7_split_str+ 9→0_split_listcalls on the issue's 30-line repro, and 240→48 on a 40-sentence input.Contribution Checklist
ruff checkreports only a pre-existingISCfinding in untouched test lines, present onmainwith current ruff)Validation
python/tests/unit/text/test_text_chunker.py(fail onmain, pass with the fix):test_split_str_flags_fully_resolved_input— the flag isTruefor fully resolved inputs andFalsewhen a tier cannot split a chunk;test_split_text_lines_stops_after_fully_resolved_tier— no weaker tier runs once one tier resolved every chunk (onmainthis records 9_split_listcalls; with the fix, 0);test_split_text_lines_keeps_splitting_oversized_remainders— oversized chunks without the current tier's separators are still split by weaker tiers.pytest tests/unit/text/— 32 passed (29 existing + 3 new).pytest tests/unit/core_plugins/— 467 passed, 3 skipped (coversconversation_summary_plugin, the other in-repo consumer of the chunker).AI assistance
Transparency note: this fix and its tests were prepared with the help of an AI coding agent (branch/commit authored under the DZH166 account, which reviewed and validated the change locally). No generated bulk content is included; the change is limited to the two files in the diff.