Skip to content

fix: deep-copy document metadata in PythonCodeSplitter - #12424

Open
sainikhiljuluri wants to merge 1 commit into
deepset-ai:mainfrom
sainikhiljuluri:fix/python-code-splitter-deepcopy-meta
Open

fix: deep-copy document metadata in PythonCodeSplitter#12424
sainikhiljuluri wants to merge 1 commit into
deepset-ai:mainfrom
sainikhiljuluri:fix/python-code-splitter-deepcopy-meta

Conversation

@sainikhiljuluri

Copy link
Copy Markdown

Problem

PythonCodeSplitter copies the input document's metadata shallowly, so nested values are shared between every chunk and with the caller's document:

doc = Document(content=class_source, meta={"tags": ["code"]})
chunks = PythonCodeSplitter(min_effective_lines=2, max_effective_lines=5).run(documents=[doc])["documents"]

chunks[0].meta["tags"] is chunks[1].meta["tags"]   # True
chunks[0].meta["tags"] is doc.meta["tags"]         # True

chunks[0].meta["tags"].append("shape")
chunks[1].meta["tags"]   # ['code', 'shape']
doc.meta["tags"]         # ['code', 'shape']   <- the caller's document is mutated

Root cause

Two shallow copies:

  • _build_chunk_meta does meta.update({k: v for k, v in parent_doc.meta.items() ...}), so every chunk holds references into the input document's metadata.
  • _secondary_split does meta = dict(base_meta), so the pieces of an oversized unit share the same nested objects with each other.

Fix

Deep-copy at both sites, matching the other splitters.

This is the same defect as #12248, fixed for the Markdown, CSV and hierarchical splitters by #12249. PythonCodeSplitter was not covered there. I checked the remaining splitters — DocumentSplitter, RecursiveDocumentSplitter, CSVDocumentCleaner and DocumentCleaner are already safe, so this should close out the family.

Tests

Two regression tests in TestFileNamePropagation, one per fixed site, written in the same shape as the one #12249 added to test_markdown_header_splitter.py.

Reverting the fix fails both, on the leaked mutation itself:

assert ['code', 'shape'] == ['code']
assert ['code', 'giant'] == ['code']

test/components/preprocessors/ is green: 369 passed, 10 skipped (69 -> 71 in this file). ruff check and ruff format --check clean.

Risk

Behavioural change is limited to chunks no longer aliasing the caller's metadata. Deep-copying runs once per chunk on metadata that was already being copied, so the added cost is proportional to metadata size, not document size.


AI assistance was used to write this change.

PythonCodeSplitter copied the input document's metadata shallowly, so
nested values such as a list under meta["tags"] were shared between
every chunk and with the input document. Editing one chunk's metadata
changed all the others and the caller's document.

This is the same defect fixed for the Markdown, CSV and hierarchical
splitters in deepset-ai#12249; PythonCodeSplitter was not covered there. The
secondary line-based split of oversized units had it too.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sainikhiljuluri
sainikhiljuluri requested a review from a team as a code owner August 21, 2026 06:10
@sainikhiljuluri
sainikhiljuluri requested review from julian-risch and removed request for a team August 21, 2026 06:10
@vercel

vercel Bot commented Aug 21, 2026

Copy link
Copy Markdown

@sainikhiljuluri is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Aug 21, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @sainikhiljuluri, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from julian-risch August 21, 2026 07:31
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Aug 21, 2026
@HaystackBot
HaystackBot marked this pull request as draft August 21, 2026 07:31
@HaystackBot
HaystackBot marked this pull request as ready for review August 21, 2026 08:01
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Aug 21, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @sainikhiljuluri! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants