fix: merge a short trailing split in EmbeddingBasedDocumentSplitter

[ShousenZHANG]

Related Issues

• fixes EmbeddingBasedDocumentSplitter: a short trailing split is never merged into the previous chunk #12436

Proposed Changes:

_merge_small_splits only merges forward: it keeps a running accumulator and folds the next split into it while the accumulator is below min_length. Whatever is left in that accumulator when the loop ends is appended unconditionally, so a final split shorter than min_length is emitted as its own document — silently breaking the documented promise that "splits below this length will be merged". This is common in practice: a closing line, a signature, a table footer.

The trailing split is now merged into its predecessor instead. The merge is guarded by the same max_length limit that already governs forward merges, so the existing tradeoff is preserved: when a merge would reach max_length the short split is left alone, exactly as test_merge_small_splits_respect_max_length already documents for a blocked forward merge. Because the guard is strictly < max_length, the merged result can never become something _split_large_splits has to re-split.

Both _split_document and _split_document_async go through this helper, so both paths are covered.

How did you test it?

Three unit tests in test_embedding_based_document_splitter.py:

• test_merge_small_splits_merges_short_trailing_split — the regression. Fails on main (returns ["Long enough text ", "Ok."]), passes with this change.
• test_merge_small_splits_keeps_short_trailing_split_when_max_length_blocks — the backward merge must not override max_length.
• test_merge_small_splits_keeps_a_lone_short_split — a single split has nothing to merge into and must be returned as is.

Locally: hatch run test:unit test/components/preprocessors/ → 361 passed; hatch run test:types → no issues in 447 source files; hatch run fmt-check on both changed files → clean.

Notes for the reviewer

The behaviour when a merge is blocked by max_length is deliberately unchanged — min_length stays best-effort in that case, matching the existing forward-merge behaviour rather than introducing a second rule.

One overlap to flag: #12434 also touches this file, but in _find_split_points (~line 376) and its tests, roughly 50 lines away from _merge_small_splits. I don't expect a conflict; happy to rebase if that one lands first.

This PR was fully generated with an AI assistant. I have reviewed the changes and run the relevant tests.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@ShousenZHANG is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

• Fixes EmbeddingBasedDocumentSplitter._merge_small_splits so a final split shorter than min_length is merged into the preceding split when the combined length remains < max_length, aligning behavior with the documented “small splits will be merged” promise (best-effort when max_length blocks).
• Adds targeted unit tests covering the trailing-short-split regression, the max_length guard behavior for backward merges, and the single-split edge case.
• Adds a release note describing the user-visible behavior change.

Changes:

• Update _merge_small_splits to perform a backward merge for a short trailing split (subject to the existing < max_length constraint).
• Add three unit tests to lock in the new behavior and edge cases.
• Add a release note entry for the fix.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
haystack/components/preprocessors/embedding_based_document_splitter.py	Adds backward-merge logic for a short trailing split while preserving the existing < max_length merge guard.
test/components/preprocessors/test_embedding_based_document_splitter.py	Adds regression + edge-case coverage for trailing split merging and max_length blocking.
releasenotes/notes/merge-short-trailing-split-8f5b640fdbc3ccbe.yaml	Documents the fix and its interaction with max_length.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.