fix(scraping): run post-processing on the selected content, not body - #2247
Open
Yasser-Ameur wants to merge 1 commit into
Open
fix(scraping): run post-processing on the selected content, not body#2247Yasser-Ameur wants to merge 1 commit into
Yasser-Ameur wants to merge 1 commit into
Conversation
With css_selector or target_elements set, cleaned_html kept <b> under only_text, emitted inline style, onclick and data-* attributes, empty elements and whole base64 image payloads, and let a <script>, <style>, <noscript>, <meta>, <link>, mermaid SVG or <form> inside the selection survive untouched. _scrap() builds content_element from a deep copy of the selector match, which detaches it from body. Nine passes mutated body after that copy: the only_text unwrap, base64 src truncation, remove_empty_elements_fast, remove_unwanted_attributes_fast, mermaid SVG replacement, style/link/ meta/noscript removal, script removal and remove_forms. All nine fed a tree that was never the one lxml.tostring() serialised into cleaned_html. excluded_tags and excluded_selector run before the copy, which is why they kept working and masked this. Point all nine passes at content_element, which is body itself when no selector is given, so the no-selector path is unchanged. Link, media and table extraction stay on body so whole-page collection there is unaffected.
Yasser-Ameur
marked this pull request as ready for review
September 9, 2026 20:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When
css_selectorortarget_elementsis set, the post-processing passessilently stop running.
only_text=Trueleaves<b>in place, and inlinestyle,onclick,data-*and whole base64data:image payloads reachcleaned_htmland the markdown verbatim. Without a selector the same optionswork, which is what makes it easy to miss.
In
LXMLWebScrapingStrategy._scrap,content_elementis acopy.deepcopyofthe selector match, so it is detached from
body. The passes that follow keptmutating
body, whilecleaned_htmlis serialised fromcontent_element. Nineof them were affected: the
only_textunwrap, base64img srctruncation,empty-element removal, attribute stripping, mermaid SVG replacement,
style/link/meta/noscript removal, script removal and
remove_forms._process_elementand table extraction deliberately stay onbody, becauselink and media collection is page-wide by design and an existing test asserts
that.
List of files changed and why
crawl4ai/content_scraping_strategy.py- point the post-processing passes atthe element that is actually serialised.
tests/test_selector_post_processing.py- new, mirrorstests/test_issue_1484_css_selector.py.How Has This Been Tested?
pytest tests/test_selector_post_processing.pyis 27 passed on this branch and4 failed, 23 passed with the source reverted, for instance
assert '<style>' not in cleanedfinding<style>.job { color: red; }</style>.Four selector shapes are parametrised, with the boundaries: no selector match,
a single
<b>selected,only_text=False,keep_data_attributes=True, thesidebar still excluded, and links still collected page-wide.
The wider sweep is 152 passed, 16 failed, the 16 being the pre-existing
test_pdf_download_limits.pyfailures ondevelop.Checklist: