Fix HTML blocks nested in indented blocks (#1096) by inchang-ing · Pull Request #1688 · Python-Markdown/markdown

inchang-ing · 2026-11-04T05:52:16Z

Description

Fixes #1096. Supersedes #1666.

A raw HTML block nested in a list item, definition, blockquote, admonition or footnote was not recognized. The HTML block preprocessor only sees the document before it is parsed, where the nesting indentation (or the > of a blockquote) cannot be told apart from an indented code block or a paragraph. The block parser then treated the HTML as Markdown: it parsed Markdown inside it, and split it into paragraphs at every blank line, often producing invalid HTML such as <p><div>.

This adds a RawHtmlBlockProcessor. Once a container has removed its own indentation or markers, a block which starts with a block-level tag, comment, processing instruction, CDATA section or declaration is extracted with the same extractor as the preprocessor (or the md_in_html extractor, when that extension is enabled). The preprocessor is unchanged, so the top level of a document and extensions that work on the document text see exactly what they did before.

Behavior

Nested raw HTML blocks follow the same rules as at the top level, with two deliberate differences, now documented under "Differences" in docs/index.md:

A nested raw HTML block must start a block of its own: after a blank line, right after a list marker or the : of a definition, or on the first line of the nested content. It cannot interrupt a paragraph, so a block-level tag on a later line of a paragraph, including a lazy continuation line, stays part of that paragraph. Without this, a fenced code block nested in a list item, which fenced_code does not recognize there, would have the HTML of its example rendered rather than shown.
A tag which is never closed runs to the end of the nesting block rather than to the end of the document.

As at the top level, Markdown is no longer parsed within such a block, unless md_in_html is enabled and the element has a markdown attribute. The attribute now works in nested content. Its content is parsed as at the top level of the document, so a paragraph in it gets a <p> even in a tight list item.

Inline tags, such as <span>, are unaffected. HTML in an indented code block is still code.

Implementation notes

Containers hand their content to the parser one block at a time, so a raw block with blank lines in it is collected across calls, keyed by its container. A comment, processing instruction, CDATA section or declaration with a blank line in it is held as text until it is terminated. Only each new block is searched for the terminator, so this is linear in the number of blocks.
Two new BlockProcessor methods, which do nothing unless overridden, let a processor release blocks it holds:
- end_blocks(parent) is called when a nested call to parseBlocks is done.
- finish() is called when the outermost call is done.
A held block is kept only while a call still in progress parses into an element that has its container along its chain of last children. Otherwise it is written out before the call returns. An extension that parses into an element of its own and then moves what was parsed (as footnotes does) therefore gets everything before parseChunk returns. Content held for a container that is part of the tree, such as a list item, may be added to it after the call returns. This is documented in the extension API docs, along with the requirement that a BlockParser subclass overriding parseBlocks must call the parent's method.
In the first block of a list item, the lines after the first are not de-indented the way the item's later blocks are, and indented nested-item lines are split off. Raw blocks there are de-indented the same way ListIndentProcessor de-indents later blocks, so the first block and later blocks of an item give the same result.
md_in_html now parses an element's content in a clean parser state, so the state of the list item it is in (a tight item's text has no p) does not carry over.
Text on the same line after a closing tag has its leading whitespace stripped, as a top-level paragraph would. Text on later lines is left alone, as it may be an indented code block.

Changes during review

On top of the original two implementation commits:

Fix content being lost when an extension parses into an element of its own and an unclosed raw block was held for it. Add tests.
Keep collecting raw blocks with blank lines in containers nested in such an element.
Fix quadratic time for a long unterminated comment in a container. Over five seconds for 32,000 paragraphs, now about 0.7 seconds.
Extract a raw block right after a list marker or definition colon, which was previously still parsed as Markdown.
Strip the whitespace before same-line text in tight items.
Merge master.
Move the changelog entry, which the merge had left in the released 3.11.3 section, to Unreleased. Rewrite it under Changed, and add the new hooks under Added.
Document the behavior in docs/index.md, nested md_in_html in its extension page, and the hooks in the extension API docs.

Compatibility notes

Output changes for documents with raw HTML in list items, definitions, blockquotes, admonitions or footnotes. Markdown that used to be parsed inside such HTML, such as a Markdown list inside <details> in a list item, is now left as raw text, as it is at the top level, unless md_in_html is used. The changelog calls this out.
md_in_html is in maintenance mode. The changes to it make it work in nested content and keep the list item's parser state out of the element's content.
Split list items off an indented block of the previous item #1670 is compatible. The code merges cleanly, and together they give correct results for list items following raw HTML without a blank line. Whichever lands second needs the changelog conflict resolved and the expected output of test_comment_unclosed_list_item updated, since Split list items off an indented block of the previous item #1670 intentionally changes that list behavior.

Verification

python -m unittest discover tests passes, including the new tests in test_indented_html_blocks.py. Several of them fail against the code before each fix.
The pymdown-extensions test suite passes (955 tests), and its container extensions (details, tabbed, blocks) render the same as before.
Lint, spell check and link check pass.

AI Assistance Disclosure

No AI tools were used in preparing this PR.
AI tools were used and I have disclosed which ones, and have fully reviewed and verified their output.

AI tools were use to assist with the following:
- Code
- Code Comments
- Documentation
- Changelog Entry
- Commit Message
- The Description Above
- Other: Tests
Claude (Claude Code) was used to review the original change, write the fixes made during review and their tests, and write the documentation, changelog, commit messages and this description. The original implementation commits were prototyped with AI assistance in compiling the cases that should pass, as disclosed in Fix HTML blocks nested in indented blocks #1666.

Checklist

This PR follows the contribution guidelines.
The code follows the Code Style Guide.
The commit message follows the Commit Message Style Guide.
I have added or updated relevant docs, including release notes if applicable which follow the Documentation Style Guide.
I have added or updated relevant tests.
I have not requested, and will not request, an automated AI review for this PR.

The HTML preprocessor can only extract a raw block that starts at the beginning of a line. Within a container (a list item, an admonition, etc.) the block is indented, which cannot be told apart from an indented code block, so it was left as text. The block parser then treated it as inline HTML and split it at any blank lines. Once the container has removed its indentation, a block that starts with a block level tag can only be a raw block. A new `RawHtmlBlockProcessor` extracts it there, using the same extractor as the preprocessor (or the `md_in_html` extractor when that extension is enabled). Containers pass their content on a block at a time, so a raw block with blank lines in it is collected across calls, keyed by the container, until it is closed. `RawHtmlFlushTreeprocessor` finishes anything still open when the document has been parsed. The preprocessor is not changed, so extensions that work on the text of the document before it is parsed see exactly what they saw before, and indented code blocks, paragraph continuation lines and list items render as they always have. Content of a blockquote is left alone to keep its current behavior. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Fixes Python-Markdown#1096

The two new CDATA tests describe the section by name in their docstrings, and checkspelling scans test docstrings, so it failed on that word in both.

inchang-ing · 2026-11-05T08:57:08Z

The spelling check failed on "CDATA" in the two new test docstrings (test_cdata_unclosed, test_cdata_spaced_close); added the word to .spell-dict alongside the other markup terms already listed there (XHTML, CDATA-style acronyms). Pushed as 17029a9.

…tionary The spelling check flagged etree and IndentedRawHtml in a new test docstring. Following the review suggestion, wrap code terms in code spans (which are not spellchecked) in the new comments and docstrings, and drop the CDATA dictionary entry added earlier so the dictionary matches upstream again. Code spans also render correctly in the generated API documentation.

waylan · 2026-11-09T19:03:27Z

As a reminder, this is the replacement for #1666. I am expecting @facelessuser to review this.

facelessuser · 2026-11-09T19:55:18Z

Yep, this is on my list, as soon as I can get to it.

facelessuser · 2026-11-14T15:14:03Z

I'm diving into this today. I suspect there will be some issues that need addressing.

A raw block left open in a container is held until a later call to the parser hands the container more blocks. An extension that parses into an element of its own and then moves what was parsed out of it never hands that element anything more, so whatever was held for it was only written out at the end of the document, into an element nothing refers to, and was lost. At the end of every call to `parseBlocks`, a new `end_blocks` hook lets a processor release what it holds. The raw HTML processor keeps holding a block only for a parent on the last-child path of the element the outermost call parses into, which is where every container that will be handed more blocks is. For any other parent it writes the block out before the call returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An extension that parses into an element of its own and then moves what was parsed lost any raw block left open in it. Cover an unclosed raw block and an unterminated comment, and check that a raw block closed within the same call is still collected across its blank line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A raw block left open in a container was held only when the container was on the last-child path of the element the outermost call to the parser parses into. A container within an element an extension parses into on its own, such as a list item, is not, so each of its blocks was written out as soon as the call for it returned, and a raw block with a blank line in it was split into paragraphs. The parser now keeps the parent of every call in progress. A raw block is held for as long as some call still in progress parses into an element the container is on the last-child path of, and is written out in the container once the outermost such call is done, in the parser state it was held in. `end_blocks` is called only for nested calls; the outermost still ends in `finish`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

facelessuser · 2026-11-14T15:42:10Z

FYI, I found some dropped text cases and some quadratic slowdown in areas that have recently been fixed. I am root-causing those issues and will push changes to this PR directly. I'll also attempt fixes for anything else I find. At the end, I will post any caveats to this approach that I determine to be worth mentioning.

A comment, processing instruction, `CDATA` section or declaration with a blank line in it is held a block at a time until it is ended. For each block, everything held so far was copied, joined and searched for the terminator again, which took quadratic time in the number of blocks: 32 thousand paragraphs after an unterminated comment in a list item took over five seconds. No terminator has a newline in it, so one can not span blocks. Keep the terminator with the held blocks and search only the new block, joining the blocks once the construct is ended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A raw block in the first block of a list item or definition was left to be parsed as it always had been, so Markdown was parsed in it while the same block in any later block of the item was raw HTML. Extract it there too. The first block of a list item is handed on without the indent of its lines after the first removed, unlike the item's later blocks, and with each indented nested item line split off from it. Remove the indent as `ListIndentProcessor` does, and join a split-off line back onto the raw block it is in with a newline rather than a blank line. `md_in_html` parsed the content of an element in the parser state it was in, so in a tight list item or definition, paragraphs in it had no `p`. Parse the content in a state of its own, as it would be at the top of the document. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Text after a raw block on the same line kept the whitespace before it. At the top of the document a paragraph strips that, but in a tight list item or definition the text is added to the item's text as it is, so the whitespace showed up in the output. Strip it from text on the line the raw block ends on. Text on a later line is left alone, as it may be an indented code block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Describe under Differences how a raw HTML block is recognized at the top level of a document and, nested in a list item, definition, blockquote, admonition or footnote, the two ways it differs: it must start a block of its own rather than interrupt a paragraph, and an unclosed tag runs to the end of the nesting block rather than of the document. Move the changelog entry, which the merge of `master` left in the 3.11.3 release, to Unreleased under Changed, and describe what changes: Markdown is no longer parsed in such blocks, `md_in_html` now works in them, and they can not interrupt a paragraph. Add the new `finish` and `end_blocks` block processor methods under Added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Note on the `md_in_html` page that a raw HTML block nested in a list item, definition, blockquote, admonition or footnote may have a `markdown` attribute, and that its content is parsed as at the top level of the document, with an example. Describe in the extension API documentation when `end_blocks` and `finish` are called, what a block processor holding blocks across calls must release in them, and what a block processor which calls the parser can rely on: everything parsed into an element of its own is there when the call returns, but content held for an element at the end of the tree may be added after. List `RawHtmlBlockProcessor` as an example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`BlockParser.parseBlocks` calls the `end_blocks` and `finish` methods of block processors. A subclass which overrides it without calling the parent class's method never does, so blocks a processor holds, such as an unclosed raw HTML block in a list item, are never released and their content is lost. Say so in the extension API documentation and in the docstring of `parseBlocks`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

facelessuser · 2026-11-14T16:34:20Z

Known limitations

These are not addressed by this PR. Some depend on #1670, and some predate it:

Without Split list items off an indented block of the previous item #1670, a list item marker on the line right after a nested raw HTML block, with no blank line, does not start a new item:
- - a\n\n <div>x</div>\n- b gives b as a list nested in the first item, where master absorbed it as paragraph text.
- In a nested list, an indented - c right after the block becomes an indented code block, as it would at the top level.
With Split list items off an indented block of the previous item #1670, both are new items as expected.
A raw HTML block cannot span list items. A tag opened in one item and closed after the next item's marker is held as raw text, including that marker, without Split list items off an indented block of the previous item #1670. With it, the HTML ends up badly nested across the two items.
With Split list items off an indented block of the previous item #1670:
- Under sane_lists, a marker of the other list type right after a nested raw HTML block becomes a nested list rather than text.
- Under def_list, a list marker right after a raw HTML block in a definition stays in the definition, since Split list items off an indented block of the previous item #1670 does not cover definition lists.
When a footnote ends with a raw HTML block, its backlink is added to the <p> the block's placeholder is in, giving <p><div>…</div> <a …></p>. master produces the same structure for HTML at the end of a footnote.
Not caused by this PR: in - item\n <div>x</div>, with the continuation line indented one to three spaces, the HTML block preprocessor extracts the block at the top level and a stray <br /> is inserted in the list item. This happens on master too.

facelessuser · 2026-11-14T16:38:03Z

I've tried to dig into all the areas I could think of, and yes, I used AI because I do not fully trust that I could capture all the possible implications with a simple review. This is a big change.

I've applied a number of fixes and updated documentation, which hopefully clearly identifies what is changing.

As I have said before, this kind of change is a big one. It obviously changes the behavior of tags in indented constructs and things like blockquotes to allow them to behave more as they do at the root level (as much as possible). That is probably all the time I can spend on it this weekend.

waylan · 2026-11-14T21:27:38Z

Wow, that was quite the effort. If I am understanding your correctly, for some of this, we also need #1670. Does that mean that you have also reviewed #1670? I suppose, more importantly, do you consider either to be ready or is there more to do?

facelessuser · 2026-11-14T21:40:11Z

#1670 doesn't have support for definition lists. I mainly applied #1670 to see if it fixed some interesting cases that turned up during my deep dive, and yes, it did. I also wanted to make sure it was compatible with this change, and it was.

To be honest, it's probably easier to get the list changes in first. I need to dive a little deeper there, but currently, I think that PR is incomplete if it doesn't support definition lists.

I may actually get to the #1670 tomorrow, just so I can quantify exactly what needs to be done in that one. It's definitely the safer one to get in once it is ready. Then we can make sure this works well on top of it.

facelessuser · 2026-11-14T21:52:43Z

I see now that in the list feature, the author mentioned it could be addressed in a separate feature. I would like to see both go in before this one. Whether that happens in one PR or two.

facelessuser and others added 3 commits November 3, 2026 17:57

Handle elements under blockquotes and fix comments

09bbb18

Add a changelog entry for indented raw HTML blocks

0532a4e

Fixes Python-Markdown#1096

inchang-ing mentioned this pull request Nov 4, 2026

Fix HTML blocks nested in indented blocks #1666

Closed

13 of 15 tasks

Add CDATA to the spell dictionary

17029a9

The two new CDATA tests describe the section by name in their docstrings, and checkspelling scans test docstrings, so it failed on that word in both.

waylan reviewed Nov 5, 2026

View reviewed changes

Comment thread .spell-dict Resolved

waylan added the needs-review Needs to be reviewed and/or approved. label Nov 9, 2026

waylan requested a review from facelessuser November 9, 2026 19:03

facelessuser and others added 4 commits November 14, 2026 08:29

Merge branch 'master' into pr-1688

26fcedd

facelessuser and others added 7 commits November 14, 2026 08:44

Fix lint

3f6e5d5

Navigation Menu

Repository navigation

Fix HTML blocks nested in indented blocks (#1096) - #1688

Fix HTML blocks nested in indented blocks (#1096)#1688
inchang-ing wants to merge 16 commits into
Python-Markdown:masterfrom
inchang-ing:html-indent-container

inchang-ing commented Nov 4, 2026 •

edited by facelessuser

Loading

Uh oh!

inchang-ing commented Nov 5, 2026

Uh oh!

Uh oh!

waylan commented Nov 9, 2026

Uh oh!

facelessuser commented Nov 9, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

waylan commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

inchang-ing commented Nov 4, 2026 • edited by facelessuser Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Behavior

Implementation notes

Changes during review

Compatibility notes

Verification

AI Assistance Disclosure

Checklist

Uh oh!

inchang-ing commented Nov 5, 2026

Uh oh!

Uh oh!

waylan commented Nov 9, 2026

Uh oh!

facelessuser commented Nov 9, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Known limitations

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

waylan commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

facelessuser commented Nov 14, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

inchang-ing commented Nov 4, 2026 •

edited by facelessuser

Loading