Skip to content

fix: PythonCodeSplitter secondary-split pieces missing qualified name context#11875

Open
GovindhKishore wants to merge 4 commits into
deepset-ai:mainfrom
GovindhKishore:fix/python-code-splitter-secondary-split-context
Open

fix: PythonCodeSplitter secondary-split pieces missing qualified name context#11875
GovindhKishore wants to merge 4 commits into
deepset-ai:mainfrom
GovindhKishore:fix/python-code-splitter-secondary-split-context

Conversation

@GovindhKishore

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

PythonCodeSplitter._secondary_split() falls back to a plain line-based split for oversized functions/methods/classes. Only the first resulting piece naturally contains the def/class signature line, so every piece after that has no identifying text at all, which hurts retrieval ranking for those pieces.

Fixed by reusing the qualified_name value already computed in _secondary_split() (currently only used in the logger.warning(...) call) and prepending it as a # qualified_name comment line to every piece after the first.

How did you test it?

Manually verified locally with a reproduction script: created a Document with a single function exceeding the oversized threshold, ran it through PythonCodeSplitter().run(...), and confirmed:

  • Piece 0 still starts with the original def function_name(...): line (unchanged).
  • Pieces 1+ now start with # function_name before the body content (previously had no identifying text at all).

Notes for the reviewer

Open to feedback on the prepended format (comment style, placement) before I finalize with tests and a release note.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/preprocessors
  python_code_splitter.py
Project Total  

This report was generated by python-coverage-comment-action

@GovindhKishore GovindhKishore force-pushed the fix/python-code-splitter-secondary-split-context branch from dc9ac34 to ea9f6cb Compare July 6, 2026 18:40
@vercel

vercel Bot commented Jul 6, 2026

Copy link
Copy Markdown

@GovindhKishore is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@GovindhKishore GovindhKishore marked this pull request as ready for review July 6, 2026 18:51
@GovindhKishore GovindhKishore requested a review from a team as a code owner July 6, 2026 18:52
@GovindhKishore GovindhKishore requested review from anakin87 and removed request for a team July 6, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: PythonCodeSplitter secondary-split pieces lose function/method identity, hurting retrieval ranking

1 participant