bug: NlpSentenceChunking.chunk() uses list(set(sens)) which destroys sentence order

[kuishou68]

Bug Description

In crawl4ai/chunking_strategy.py, the NlpSentenceChunking.chunk() method returns list(set(sens)):

def chunk(self, text: str) -> list:
    from nltk.tokenize import sent_tokenize
    sentences = sent_tokenize(text)
    sens = [sent.strip() for sent in sentences]
    return list(set(sens))  # BUG: set() is unordered!

Problem

set() in Python is unordered — converting sentences to a set and back to a list:

1. Destroys sentence order: sentences are returned in arbitrary order, not document order
2. Removes duplicate sentences that may be intentionally present

Fix

Return sens directly: return sens

[SamSi0322]

I'll take this. One-line fix — list(set(sens)) destroys sentence order. PR incoming.

[liamoss1231-ai]

Nice find, thanks for reporting this.

…

On Fri, Apr 10, 2026, 10:19 AM SamSi0322 ***@***.***> wrote: *SamSi0322* left a comment (unclecode/crawl4ai#1909) <#1909 (comment)> I'll take this. One-line fix — list(set(sens)) destroys sentence order. PR incoming. — Reply to this email directly, view it on GitHub <#1909 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CAXZQSIFFRE6PVY6U6JQGKT4VEGHPAVCNFSM6AAAAACXTSYME2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMRUG44DQOBRGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ntohidi]

Thanks for the clear report @kuishou68, and @SamSi0322 and @liamoss1231-ai for jumping on it.

I've pushed a fix on fix/nlp-sentence-chunking-1909 that adopts the approach from #1910 — returning sens directly without dedup. Chunking should split, not filter; if downstream consumers need deduplication they can handle it themselves.

While fixing this, I also found that NlpSentenceChunking.__init__() had a broken re-import (from crawl4ai.le.legacy.model_loader import ...) that shadows the working top-level import. Removed that in the same commit.