Fix HTML blocks nested in indented blocks (#1096)

docs/changelog.md

@@ -14,6 +14,28 @@ See the [Contributing Guide](contributing.md) for details.
 
 ## Unreleased
 
+### Added
+
+* Add the `finish` and `end_blocks` methods to `BlockProcessor`, which the
+  block parser calls when its outermost call is done and when a nested call is
+  done. A block processor which holds on to blocks until a later call to the
+  parser must release them there. Content held for a nested block, such as an
+  unclosed raw HTML block in a list item, may be added to it after the call to
+  `parseChunk` which parsed it has returned.
+
+### Changed
+
+* Recognize raw HTML blocks nested in a list item, definition, blockquote,
+  admonition or footnote as at the top level of the document, rather than
+  parsing them as Markdown and splitting them into paragraphs at blank lines
+  (#1096). As at the top level, Markdown is no longer parsed within such a
+  block, unless the `md_in_html` extension is enabled and the element has a
+  `markdown` attribute, which is now supported in those places too. Unlike at
+  the top level, a nested raw HTML block must start a block of its own and
+  cannot interrupt a paragraph, and an unclosed tag runs to the end of the
+  nesting block rather than of the document. See
+  [Raw HTML Blocks](index.md#differences) for details.
+
 ### Fixed
 
 * Fix for complex cases in new emphasis handling where text could be unexpectedly

docs/extensions/api.md

@@ -114,6 +114,33 @@ processor.  There are three methods:
 For perspective, Markdown calls `parseDocument` which calls `parseChunk` which calls `parseBlocks` which calls your
 block processor, which, in turn, might call one of these routines.
 
+#### Holding Blocks Across Calls
+
+A block which contains other blocks, such as a list item or an admonition, hands its content to the parser a block at a
+time, in a call for each. A block processor which needs to see a later block before it knows what to do with an
+earlier one may hold on to blocks across those calls, but must release them once no more are coming. The built-in
+processor for raw HTML blocks nested in other blocks does this for a raw HTML block with a blank line in it. Two
+methods of `BlockProcessor`, which do nothing unless overridden, are called for that:
+
+* [`end_blocks`][markdown.blockprocessors.BlockProcessor.end_blocks] is called with `parent` when a nested call to
+  the parser, which parsed blocks into `parent`, is done. The caller may do as it likes with `parent` once the call
+  returns, so anything held which no call still in progress can be counted on to hand more blocks to must be
+  released here.
+* [`finish`][markdown.blockprocessors.BlockProcessor.finish] is called when the outermost call to the parser is done.
+  Anything still held must be released here.
+
+This affects a block processor which calls the parser itself. If it parses into an element of its own, which is not in
+the tree being parsed (to move what is parsed somewhere else, for example), everything parsed is in that element by
+the time the call returns. If it parses into an element at the end of the tree being parsed, which a later call may
+hand more blocks to (as a list item is), what is held for that element may be added to it after the call returns: once
+a later block closes it, or once the outermost call is done.
+
+!!! warning
+
+    The parser calls these methods from [`parseBlocks`][markdown.blockparser.BlockParser.parseBlocks]. A subclass of
+    `BlockParser` which overrides `parseBlocks` must call the parent class's `parseBlocks` to parse the blocks.
+    Otherwise, blocks held by a block processor are never released, and their content is lost from the output.
+
 #### Example
 
 This example calls out important paragraphs by giving them a border.  It looks for a fence line of exclamation points
@@ -198,6 +225,7 @@ Some block processors in the Markdown source tree include:
 | [`HashHeaderProcessor`][markdown.blockprocessors.HashHeaderProcessor]            | built-in  |  Title hashes (`#`), which may split blocks |
 | [`HRProcessor`][markdown.blockprocessors.HRProcessor]                            | built-in  |  Horizontal lines, e.g., `---`              |
 | [`OListProcessor`][markdown.blockprocessors.OListProcessor]                      | built-in  |  Ordered lists; complex and using `state`   |
+| [`RawHtmlBlockProcessor`][markdown.blockprocessors.RawHtmlBlockProcessor]        | built-in  |  Nested raw HTML; holds blocks across calls |
 | [`AdmonitionProcessor`][markdown.extensions.admonition.AdmonitionProcessor]      | extension |  Render each [Admonition][] in a new `div`  |
 | [`FootnoteBlockProcessor`][markdown.extensions.footnotes.FootnoteBlockProcessor] | extension | Find footnote definitions and store for later use. |
 

docs/extensions/md_in_html.md

@@ -170,6 +170,27 @@ behavior. However, if the value of an element's `markdown` attribute is the same
 parent, the the child element's behavior is observed. For example, a `block` element may contain either `block`
 elements or `span` elements as children and each element will be parsed using the specified behavior.
 
+### In Lists and Other Blocks
+
+A raw HTML block nested in a list item, definition, blockquote, admonition or footnote may have a `markdown`
+attribute as well. Its content is parsed as it would be at the top level of the document, whatever block it is nested
+in. For example, a paragraph within it is wrapped in a `<p>` tag even in a tight list item, whose own text is not.
+
+``` md-render
+---
+extensions: [md_in_html]
+---
+* <div markdown="1">This is a *Markdown* paragraph.</div>
+* A tight list item.
+* Another list item.
+
+    <div markdown="1">
+    This is a *Markdown* paragraph too.
+    </div>
+```
+
+See [Raw HTML Blocks](../index.md#differences) for where a raw HTML block is recognized in nested content.
+
 ### Tag Normalization
 
 While the default behavior is for Markdown to not alter raw HTML, as this extension is parsing the content of raw HTML elements, it will do some normalization of the tags of block-level elements. For example:

docs/index.md

@@ -118,6 +118,35 @@ are summarized below:
     foreseeable future. That said, the [Sane List Extension](extensions/sane_lists.md)
     is available to provide a less surprising behavior.
 
+* __Raw HTML Blocks__
+
+    The [syntax rules](https://daringfireball.net/projects/markdown/syntax#html)
+    state that a block-level HTML element must be separated from surrounding
+    content by blank lines, and that its start and end tags should not be
+    indented. At the top level of a document, Python-Markdown is more lenient: a
+    line which starts with a block-level tag, or with a comment, processing
+    instruction, `CDATA` section or declaration, indented by no more than three
+    spaces, starts a raw HTML block, even when it directly follows a line of a
+    paragraph. The block continues, across blank lines, until its tags are closed,
+    and a tag which is never closed runs to the end of the document. Markdown is
+    not parsed within a raw HTML block (the
+    [Markdown in HTML Extension](extensions/md_in_html.md) can change that).
+
+    Nested in a list item, definition, blockquote, admonition or footnote, a raw
+    HTML block follows the same rules once the indentation of the nesting (or the
+    `>` of a blockquote) is removed, with two differences:
+
+    1. A raw HTML block must start a block of its own: after a blank line, right
+       after a list marker or the `:` of a definition, or on the first line of
+       the nested content. A block-level tag on a later line of a paragraph,
+       including a lazy continuation line, remains part of that paragraph. A
+       fenced code block is not recognized in nested content by the
+       [Fenced Code Blocks Extension](extensions/fenced_code_blocks.md), so
+       without this rule, the HTML of a code example in a list item would be
+       rendered rather than shown.
+    2. A tag which is never closed runs to the end of the list item (or other
+       nesting block) rather than to the end of the document.
+
 * __Automatic Links__
 
     Automatic links recognize URL schemes followed by `://`, including schemes

markdown/blockparser.py

@@ -96,6 +96,8 @@ def __init__(self, md: Markdown):
         self.blockprocessors: util.Registry[BlockProcessor] = util.Registry()
         self.state = State()
         self.md = md
+        # The parent of each call to `parseBlocks` in progress, outermost first.
+        self._frames: list[etree.Element] = []
 
     def parseDocument(self, lines: Iterable[str]) -> etree.ElementTree:
         """ Parse a Markdown document into an `ElementTree`.
@@ -113,6 +115,7 @@ def parseDocument(self, lines: Iterable[str]) -> etree.ElementTree:
             An element tree.
         """
         # Create an `ElementTree` from the lines
+        self._frames = []
         self.root = etree.Element(self.md.doc_tag)
         self.parseChunk(self.root, '\n'.join(lines))
         return etree.ElementTree(self.root)
@@ -147,14 +150,33 @@ def parseBlocks(self, parent: etree.Element, blocks: list[str]) -> None:
         additional `BlockProcessors` which call this method to recursively
         parse a nested block.
 
+        Once the blocks are parsed, this calls each processor's
+        [`end_blocks`][markdown.blockprocessors.BlockProcessor.end_blocks] (for a
+        nested call) or [`finish`][markdown.blockprocessors.BlockProcessor.finish]
+        (for the outermost call), so that a processor can release blocks it holds.
+        A subclass which overrides this method must call it to parse the blocks;
+        otherwise, held blocks are never released and their content is lost.
+
         Arguments:
             parent: The parent element.
             blocks: The blocks of text to parse.
 
         """
-        while blocks:
+        self._frames.append(parent)
+        try:
+            while blocks:
+                for processor in self.blockprocessors:
+                    if processor.test(parent, blocks[0]):
+                        if processor.run(parent, blocks) is not False:
+                            # run returns True or None
+                            break
+        finally:
+            self._frames.pop()
+        if self._frames:
+            # The caller may do as it likes with `parent` from here on.
+            for processor in self.blockprocessors:
+                processor.end_blocks(parent)
+        else:
+            # The outermost call is done, so nothing more will be handed to a processor for these parents.
             for processor in self.blockprocessors:
-                if processor.test(parent, blocks[0]):
-                    if processor.run(parent, blocks) is not False:
-                        # run returns True or None
-                        break
+                processor.finish()