Skip to content

Python: fix text_chunker's never-true split flag so the separator-tier loop can stop early - #14795

Open
DZH166 wants to merge 1 commit into
microsoft:mainfrom
DZH166:codex/textchunker-wassplit-20261112
Open

DZH166 wants to merge 1 commit into
microsoft:mainfrom
DZH166:codex/textchunker-wassplit-20261112

Conversation

@DZH166

@DZH166 DZH166 commented Nov 14, 2026

Copy link
Copy Markdown

Motivation and Context

Fixes #14490

python/semantic_kernel/text/text_chunker.py threads a "was split" flag (input_was_split / was_split) through _split_str, _split_list and _split_str_lines with the intent of short-circuiting the ordered separator-tier loop (TEXT_SPLIT_OPTIONS / MD_SPLIT_OPTIONS) once an earlier tier has resolved the input. As the issue describes, the flag can never become True: every base case returns False, and the recursive cases only OR those False values together. The maintainers' own # pragma: no cover on the break already flagged the branch as never executed. As a result every tier (newline, ., ?!, ;, :, ,, brackets, space, -, and the final hard split) always re-scans every already-resolved chunk, doing up to ~10x more splitting work than the tiered design intends (e.g. 240 _split_str calls instead of 48 on a 40-sentence input in my measurements).

Description

The fix makes the flag report what the loop actually needs to know to stop safely: whether every chunk returned so far fits within max_tokens.

  • _split_str now returns True when its result is fully within budget (a fitting chunk, or a split whose parts all fit). It returns False when a chunk could not be split with the current tier, so weaker tiers keep running. A chunk that trims down to "" is still reported as unresolved, so the next tier drops it exactly as before.
  • _split_list ANDs the per-line flags instead of ORing them, so one fitting line no longer masks another oversized line.
  • _split_str_lines breaks when the flag is True; the # pragma: no cover is removed because the branch is now reachable and covered by the new tests.

Note why the flag must be AND (all-chunks-fit) rather than the issue's suggested "a split happened here" (True on any cutpoint): a strong separator can split most of a text while leaving one oversized chunk that contains no occurrence of that separator. Breaking on "a split happened" would leave that chunk over budget; weaker tiers must keep running while any oversized chunk remains. This matches the correctness concern raised in the issue discussion. The third new test (test_split_text_lines_keeps_splitting_oversized_remainders) locks this behavior in.

Outputs are unchanged by this fix: I verified byte-identical chunk output for 24 corpus cases (all existing unit-test inputs, the issue's repro, mixed-separator inputs with oversized remainders, whitespace-only text, CRLF/CR handling, hard-split cases) before and after the change — the early exit only fires when the remaining tiers are provably no-ops (they only act on chunks that exceed max_tokens). Efficiency on the corpus improved from e.g. 43→7 _split_str + 9→0 _split_list calls on the issue's 30-line repro, and 240→48 on a 40-sentence input.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations (ruff format clean on both touched files; ruff check reports only a pre-existing ISC finding in untouched test lines, present on main with current ruff)
  • All unit tests pass, and I have added new tests where possible
  • I didn't break anyone 😄

Validation

  • New regression tests in python/tests/unit/text/test_text_chunker.py (fail on main, pass with the fix):
    • test_split_str_flags_fully_resolved_input — the flag is True for fully resolved inputs and False when a tier cannot split a chunk;
    • test_split_text_lines_stops_after_fully_resolved_tier — no weaker tier runs once one tier resolved every chunk (on main this records 9 _split_list calls; with the fix, 0);
    • test_split_text_lines_keeps_splitting_oversized_remainders — oversized chunks without the current tier's separators are still split by weaker tiers.
  • pytest tests/unit/text/ — 32 passed (29 existing + 3 new).
  • pytest tests/unit/core_plugins/ — 467 passed, 3 skipped (covers conversation_summary_plugin, the other in-repo consumer of the chunker).
  • 24-case output-identity corpus: byte-identical chunk outputs before/after the fix.

AI assistance

Transparency note: this fix and its tests were prepared with the help of an AI coding agent (branch/commit authored under the DZH166 account, which reviewed and validated the change locally). No generated bulk content is included; the change is limited to the two files in the diff.

@DZH166
DZH166 requested a review from a team as a code owner November 14, 2026 22:18
Copilot AI balanced review requested due to automatic review settings November 14, 2026 22:18
@DZH166
DZH166 deployed to github-app-auth November 14, 2026 22:18 — with GitHub Actions Active
@semantic-kernel-automation semantic-kernel-automation Bot added the python Pull requests for the Python Semantic Kernel label Nov 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block the text-chunker early-termination change.

No open issues 🎉

What changed in this PR

Fixes #14490 by allowing Python’s text chunker to stop trying weaker separator tiers once every chunk fits the token budget, avoiding redundant processing while continuing to split oversized remainders.

Changes:

  • Redefines the split flag to track whether all returned chunks fit.
  • Combines flags with AND and enables the separator loop’s early exit.
  • Adds regression tests for flag behavior, early termination, and oversized remainders.
File Description
python/​tests/​unit/​text/​test_text_chunker.py Adds three regression tests for safe early termination.
python/​semantic_kernel/​text/​text_chunker.py Updates flag propagation and stops processing once all chunks fit.

Review effort: Balanced

This branch was successfully deployed

1 active deployment
github-app-auth — 2e97ae57 Deployed Nov 14, 2026 by DZH166 via add_label #29692
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: text_chunker's was_split flag can never be True, so every separator tier always runs (dead early-exit)

2 participants