fix(html): extract <details>/<summary> content instead of discarding it - #4451
fix(html): extract <details>/<summary> content instead of discarding it#4451r0h1tb wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Bug fix in HTML parsing: previously dropped <details>/<summary> content is now extracted, with tests pinned; low-risk mapping change.
Re-trigger cubic
Both tags were mapped to `RemovedBlock`, so `partition_html()` dropped every disclosure widget along with everything inside it. Pages that build their FAQ or documentation out of accordions lost all of their questions and answers, and the caller got a shorter element list rather than an error. `<details>` is an ordinary block container, so it becomes `Flow` like `<div>`. `<summary>` is the widget's heading -- the spec calls it "a summary, caption, or legend for the rest of the contents" -- so it becomes `Heading` and emits a `Title`. That is what the reporter asked for, and it is what makes the fix useful downstream: `chunk_by_title()` opens a new section at each `Title`, so a question stays attached to its own answer instead of blending into its neighbours. `<summary>` was listed twice, under both phrasing and the removed-form group; there is now a single mapping. Resolves Unstructured-IO#3919. Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
7b06863 to
67191e5
Compare
|
High — a leading image or block wrapper makes a
Please add summary-specific traversal that preserves the title role for the summary label across leading images and ordinary block wrappers, while keeping the details body and post-summary tail outside the title and avoiding a duplicate title for a nested heading. The regression should assert exact element types/order and two separate chunks; covering link/emphasis and page metadata on the label would also protect the annotations carried by this path. (authored by codex) |
Problem
<details>and<summary>were both mapped toRemovedBlock, sopartition_html()discarded the whole subtree. On the FAQ page from the issue, every question and answer disappears and the caller gets a shorter element list rather than an error:Fix
<details>is an ordinary block container, so it becomesFlow, like<div>.<summary>becomesHeading. The spec calls it "a summary, caption, or legend for the rest of the contents", the issue asks for the questions as titles, and it is what makes the fix useful downstream —chunk_by_title()opens a new section at eachTitle, so a question stays attached to its own answer instead of blending into the next entry.<summary>was also listed twice, under both phrasing and the removed-form group; there is now one mapping.Scope
Only these two tags.
<dl>,<figure>and the form/nav tags stay inRemovedBlock:<figure>/<figcaption>removal is asserted byDescribeRemovedBlock.it_is_skipped_during_parsing, so it is a deliberate decision, not an oversight.<dl>/<dt>/<dd>also drop their content, andListBlock's docstring already says "maybe a<dl>element at some point". That looks worth doing, but reclassifying it changes the ancestor chain that_category_depthwalks (there is a parametrised case attest_parser.py:298relying on<dl><dd>nesting), so it belongs in its own PR rather than riding along here.Tests
Four tests in
test_unstructured/partition/html/test_partition.py. With the parser change reverted on this branch:Also checked by hand and behaving: unclosed
<details>,<summary>not the first child, two<summary>siblings, empty<summary>,<summary>outside any<details>, attributes on<details>, inline markup inside<summary>,<details>inside a table cell and inside a list item, 20-deep nesting, and a JSON round-trip.Suite before 146 failed / 2531 passed, after 146 failed / 2535 passed — the same 146 IDs, all pre-existing here (pandoc and the ML extras aren't installed locally). 14 modules that cannot be collected without those extras were excluded from both runs.
Note: this bumps to
0.26.4, as does #4449 — whichever lands second needs the version and changelog re-pointed.