diff --git a/docs/changelog.md b/docs/changelog.md index 7926ab4aa..81368a191 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -14,6 +14,28 @@ See the [Contributing Guide](contributing.md) for details. ## Unreleased +### Added + +* Add the `finish` and `end_blocks` methods to `BlockProcessor`, which the + block parser calls when its outermost call is done and when a nested call is + done. A block processor which holds on to blocks until a later call to the + parser must release them there. Content held for a nested block, such as an + unclosed raw HTML block in a list item, may be added to it after the call to + `parseChunk` which parsed it has returned. + +### Changed + +* Recognize raw HTML blocks nested in a list item, definition, blockquote, + admonition or footnote as at the top level of the document, rather than + parsing them as Markdown and splitting them into paragraphs at blank lines + (#1096). As at the top level, Markdown is no longer parsed within such a + block, unless the `md_in_html` extension is enabled and the element has a + `markdown` attribute, which is now supported in those places too. Unlike at + the top level, a nested raw HTML block must start a block of its own and + cannot interrupt a paragraph, and an unclosed tag runs to the end of the + nesting block rather than of the document. See + [Raw HTML Blocks](index.md#differences) for details. + ### Fixed * Fix for complex cases in new emphasis handling where text could be unexpectedly diff --git a/docs/extensions/api.md b/docs/extensions/api.md index 4822bbc2b..3fc2c91a6 100644 --- a/docs/extensions/api.md +++ b/docs/extensions/api.md @@ -114,6 +114,33 @@ processor. There are three methods: For perspective, Markdown calls `parseDocument` which calls `parseChunk` which calls `parseBlocks` which calls your block processor, which, in turn, might call one of these routines. +#### Holding Blocks Across Calls + +A block which contains other blocks, such as a list item or an admonition, hands its content to the parser a block at a +time, in a call for each. A block processor which needs to see a later block before it knows what to do with an +earlier one may hold on to blocks across those calls, but must release them once no more are coming. The built-in +processor for raw HTML blocks nested in other blocks does this for a raw HTML block with a blank line in it. Two +methods of `BlockProcessor`, which do nothing unless overridden, are called for that: + +* [`end_blocks`][markdown.blockprocessors.BlockProcessor.end_blocks] is called with `parent` when a nested call to + the parser, which parsed blocks into `parent`, is done. The caller may do as it likes with `parent` once the call + returns, so anything held which no call still in progress can be counted on to hand more blocks to must be + released here. +* [`finish`][markdown.blockprocessors.BlockProcessor.finish] is called when the outermost call to the parser is done. + Anything still held must be released here. + +This affects a block processor which calls the parser itself. If it parses into an element of its own, which is not in +the tree being parsed (to move what is parsed somewhere else, for example), everything parsed is in that element by +the time the call returns. If it parses into an element at the end of the tree being parsed, which a later call may +hand more blocks to (as a list item is), what is held for that element may be added to it after the call returns: once +a later block closes it, or once the outermost call is done. + +!!! warning + + The parser calls these methods from [`parseBlocks`][markdown.blockparser.BlockParser.parseBlocks]. A subclass of + `BlockParser` which overrides `parseBlocks` must call the parent class's `parseBlocks` to parse the blocks. + Otherwise, blocks held by a block processor are never released, and their content is lost from the output. + #### Example This example calls out important paragraphs by giving them a border. It looks for a fence line of exclamation points @@ -198,6 +225,7 @@ Some block processors in the Markdown source tree include: | [`HashHeaderProcessor`][markdown.blockprocessors.HashHeaderProcessor] | built-in | Title hashes (`#`), which may split blocks | | [`HRProcessor`][markdown.blockprocessors.HRProcessor] | built-in | Horizontal lines, e.g., `---` | | [`OListProcessor`][markdown.blockprocessors.OListProcessor] | built-in | Ordered lists; complex and using `state` | +| [`RawHtmlBlockProcessor`][markdown.blockprocessors.RawHtmlBlockProcessor] | built-in | Nested raw HTML; holds blocks across calls | | [`AdmonitionProcessor`][markdown.extensions.admonition.AdmonitionProcessor] | extension | Render each [Admonition][] in a new `div` | | [`FootnoteBlockProcessor`][markdown.extensions.footnotes.FootnoteBlockProcessor] | extension | Find footnote definitions and store for later use. | diff --git a/docs/extensions/md_in_html.md b/docs/extensions/md_in_html.md index 13a9a88b3..224ab70cd 100644 --- a/docs/extensions/md_in_html.md +++ b/docs/extensions/md_in_html.md @@ -170,6 +170,27 @@ behavior. However, if the value of an element's `markdown` attribute is the same parent, the the child element's behavior is observed. For example, a `block` element may contain either `block` elements or `span` elements as children and each element will be parsed using the specified behavior. +### In Lists and Other Blocks + +A raw HTML block nested in a list item, definition, blockquote, admonition or footnote may have a `markdown` +attribute as well. Its content is parsed as it would be at the top level of the document, whatever block it is nested +in. For example, a paragraph within it is wrapped in a `
` tag even in a tight list item, whose own text is not. + +``` md-render +--- +extensions: [md_in_html] +--- +*
Title
+ +after
+Title
+some text +
some text +
some text +
some text +
after
+ ''' + ) + ) + + def test_paragraph_chain_of_three(self): + """Three blocks in a row. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + some text +some text +
some text +
some text +
text
+<div>a</div><div>b</div><div>c</div> trailing
+ more
+
+ '''
+ )
+ )
+
+ def test_code_chain_multiline(self):
+ """Same, multi-line blocks. Regression: renders as it always has."""
+
+ self.assertMarkdownRenders(
+ self.dedent(
+ R'''
+ text
+
+ text
+<div>
+ a
+ </div><div>
+ b
+ </div> trailing
+ more
+
+ '''
+ )
+ )
+
+ def test_list_lazy_chain(self):
+ """ Regression: renders as it always has."""
+
+ self.assertMarkdownRenders(
+ self.dedent(
+ R'''
+ - item
+ ): nothing may be inserted between the placeholder and the next item. Regression: renders as it + always has. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + - item +
some text +
code
+
+ '''
+ )
+ )
+
+ def test_fn_div_blank_lines(self):
+ """Raw HTML with blank lines in a footnote. Intended change: a raw block in a container."""
+
+ self.assertMarkdownRenders(
+ self.dedent(
+ R'''
+ Text[^1]
+
+ [^1]:
+ Text1
\n' + 'after ↩
\n' + 'Text1
\n' + 'Note text.
\n' + 'after ↩
\n' + 'Text1
\n' + 'Note text.\n' + '
Text1
\n' + 'Note.
\n' + '<div>\n'
+ '\n'
+ 'code\n'
+ '</div>\n'
+ '\n'
+ '\n'
+ 'Text1
\n' + 'x
\n' + 'Text1
\n' + 'para
+after
+definition
+<div>
+
+ code
+ </div>
+
+ para
+x
+para
+Text1
\n' + 'Note
\n' + 'Note
+item
+after
+item
+Note
+++ ''' + ), + extensions=['admonition'] + ) + + def test_blockquote_in_admonition(self): + """A blockquote is a container too, so HTML in one is raw. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + >++Note
++ **x** ++
Note
++++ **x** ++
++ ''' + ) + ) + + def test_blockquote_div_blank_lines(self): + """Blank lines in the raw block do not end it. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >+ **x** ++
++ ''' + ) + ) + + def test_blockquote_div_after_paragraph(self): + """Raw block after a paragraph in a blockquote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > para + > + >+ + **x** + ++
++ ''' + ) + ) + + def test_blockquote_div_then_paragraph(self): + """A paragraph follows the raw block. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >para
++ **x** ++
++ ''' + ) + ) + + def test_blockquote_text_after_close(self): + """ + Text after the end of the raw block starts its own paragraph. Intended change: a raw block in a + container. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + >+ x ++after
+
++ ''' + ) + ) + + def test_blockquote_two_blocks(self): + """Two raw blocks in a blockquote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >x+more
+
++ ''' + ) + ) + + def test_blockquote_chained_blocks(self): + """Raw blocks that start where the last one ended. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >a+b+
++ ''' + ) + ) + + def test_blockquote_lazy_line_in_html(self): + """A lazy line inside the raw block belongs to it. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >a+b+tail
+
++ ''' + ) + ) + + def test_blockquote_lazy_line_after_html(self): + """ + A lazy line after the raw block is a paragraph in the blockquote. Intended change: a raw block in a + container. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + >+ lazy ++
++ ''' + ) + ) + + def test_blockquote_unclosed_html(self): + """ + A raw block that is never closed ends with the blockquote. Intended change: a raw block in a + container. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + >x+lazy
+
++ x + ++ ''' + ), + extensions=['admonition'] + ) + + def test_cdata_unclosed(self): + """A `CDATA` section which is never closed is text. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + +after
+ ''' + ) + ) + + def test_blockquote_nested(self): + """Raw block in a nested blockquote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > >+ > > **x** + > >+ ''' + ), + self.dedent( + ''' +++ ''' + ) + ) + + def test_blockquote_nested_after_text(self): + """Raw block in a blockquote nested after text. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > text + > >+++ **x** +++ > > x + > >+ ''' + ), + self.dedent( + ''' +++ ''' + ) + ) + + def test_blockquote_in_list(self): + """Raw block in a blockquote in a list item. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + - item + + >text
++++ x +++ > **x** + >+ ''' + ), + self.dedent( + ''' ++
+ ''' + ) + ) + + def test_blockquote_on_list_marker(self): + """Raw block in a blockquote on the line of a list marker. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + - >- +
+item
++++ **x** +++ > **x** + >+ ''' + ), + self.dedent( + ''' ++
+ ''' + ) + ) + + def test_blockquote_comment(self): + """A comment in a blockquote is raw. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > + > text + ''' + ), + self.dedent( + ''' +- +
++++ **x** +++ ++ ''' + ) + ) + + def test_blockquote_pre_blank_line(self): + """A blank line in a raw block does not split it. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >text
++ > a + > + > b + >+ ''' + ), + self.dedent( + ''' +++ ''' + ) + ) + + def test_blockquote_script(self): + """The content of a raw block is not escaped. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > + ''' + ), + self.dedent( + ''' ++ a + + b +++ ++ ''' + ) + ) + + def test_blockquote_table(self): + """A table in a blockquote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >+ >
+ ''' + ), + self.dedent( + ''' ++ > x ++ ''' + ) + ) + + def test_blockquote_md_in_html(self): + """A `markdown` attribute works in a blockquote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >+
++ x + > **x** + >+ ''' + ), + self.dedent( + ''' +++ ''' + ), + extensions=['md_in_html'] + ) + + def test_blockquote_md_in_html_block(self): + """A `markdown` attribute with blank lines. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >++x
++ > + > **x** + > + >+ ''' + ), + self.dedent( + ''' +++ ''' + ), + extensions=['md_in_html'] + ) + + def test_blockquote_in_footnote(self): + """Raw block in a blockquote in a footnote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + text[^1] + + [^1]: + >++x
++ > x + >+ ''' + ), + 'text1
\n' + '\n' + '', + extensions=['footnotes'] + ) + + def test_blockquote_paragraph_continuation(self): + """A raw block can not interrupt a paragraph in a blockquote. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > a + >
\n' + '\n' + '
\n' + '- \n' + '
\n' + '\n' + '\n' + '\n' + '\n' + 'x\n' + '\n' + 'x+ ''' + ), + self.dedent( + ''' +++ ''' + ) + ) + + def test_blockquote_inline_tag(self): + """An inline tag in a blockquote is not a raw block. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > x text + ''' + ), + self.dedent( + ''' +a +
x+++ ''' + ) + ) + + def test_blockquote_code_block(self): + """An indented code block in a blockquote is code. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + >x text
++ > x + >+ ''' + ), + self.dedent( + ''' +++ ''' + ) + ) + + def test_blockquote_fenced_code(self): + """HTML in a fenced code block in a blockquote is code. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > ``` + >+<div> + x + </div> ++ > ``` + ''' + ), + self.dedent( + ''' ++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_in_text_line(self): + """A comment that does not start a block is text. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + text more + ''' + ), + self.dedent( + ''' +++ ''' + ), + extensions=['fenced_code'] + ) + + def test_blockquote_list_marker_html(self): + """ + What follows a list marker in a blockquote is parsed as it always has been. Regression: renders as + it always has. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + > -+
<div>x+ ''' + ), + self.dedent( + ''' +++ ''' + ) + ) + + def test_blockquote_invalid_declaration(self): + """ + Something the extractor does not treat as a raw block is left alone. Regression: renders as it + always has. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + > + > text + ''' + ), + self.dedent( + ''' ++
+- +
x++ ''' + ) + ) + + def test_comment_blank_line_admonition(self): + """A comment with a blank line in it is one raw block. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + text + ''' + ), + self.dedent( + ''' +<!invalid> + text
+++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_blank_line_list(self): + """A comment with a blank line in a list item. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + - a + + + text + ''' + ), + self.dedent( + ''' +Note
+ +text
++
+ ''' + ) + ) + + def test_comment_blank_line_definition(self): + """A comment with a blank line in a definition. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + Term + : a + + + text + ''' + ), + self.dedent( + ''' +- +
+a
+ +text
++
+ ''' + ), + extensions=['def_list'] + ) + + def test_comment_blank_line_blockquote(self): + """A comment with a blank line in a blockquote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + > + > text + ''' + ), + self.dedent( + ''' +- Term
+- +
+a
+ +text
++ ++ ''' + ) + ) + + def test_comment_blank_line_blockquote_in_list(self): + """ + A comment with a blank line in a blockquote in a list item. Intended change: a raw block in a + container. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + - a + + > + > text + ''' + ), + self.dedent( + ''' +text
++
+ ''' + ) + ) + + def test_comment_blank_line_footnote(self): + """A comment with a blank line in a footnote. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + x[^1] + + [^1]: + + text + ''' + ), + '- +
+a
++ ++text
+x1
\n' + '\n' + '', + extensions=['footnotes'] + ) + + def test_comment_many_blank_lines(self): + """A comment with several blank lines in it. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + text + ''' + ), + self.dedent( + ''' +
\n' + '\n' + '
\n' + '- \n' + '\n' + '
\n' + 'text ↩
\n' + '++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_blank_line_then_div(self): + """A raw block follows a comment with a blank line. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + +Note
+ +text
+x+ ''' + ), + self.dedent( + ''' +++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_blank_line_two_comments(self): + """Two comments with blank lines in them. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + + + ''' + ), + self.dedent( + ''' +Note
+ +x+++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_blank_line_after_paragraph(self): + """A comment with a blank line after a paragraph. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + para + + + ''' + ), + self.dedent( + ''' +Note
+ + +++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_blank_line_div_after_close(self): + """A raw block starts on the line where the comment ends. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note +Note
+para
+ +x+ ''' + ), + self.dedent( + ''' +++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_blank_line_md_in_html(self): + """ + A `markdown` attribute after a comment with a blank line. Intended change: a raw block in a + container. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + +Note
+ +x++ **x** ++ ''' + ), + self.dedent( + ''' +++ ''' + ), + extensions=['admonition', 'md_in_html'] + ) + + def test_processing_instruction_blank_line(self): + """A processing instruction with a blank line in it. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + ''' + ), + self.dedent( + ''' +Note
+ +++x
+++ ''' + ), + extensions=['admonition'] + ) + + def test_processing_instruction_blank_line_blockquote(self): + """ + A processing instruction with a blank line in a blockquote. Intended change: a raw block in a + container. + """ + + self.assertMarkdownRenders( + self.dedent( + R''' + > c + > + > ?> + ''' + ), + self.dedent( + ''' +Note
+ ++ ++ ''' + ) + ) + + def test_cdata_blank_line(self): + """A `CDATA` section with a blank line in it. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + ''' + ), + self.dedent( + ''' +++ ''' + ), + extensions=['admonition'] + ) + + def test_declaration_blank_line(self): + """A declaration with a blank line in it. Intended change: a raw block in a container.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + ''' + ), + self.dedent( + ''' +Note
+ +++ ''' + ), + extensions=['admonition'] + ) + + def test_comment_unclosed_admonition(self): + """A comment which is never closed is text. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + + + +Note
+ +<!-- b
+text
+++ ''' + ), + extensions=['admonition'] + ) + + def test_processing_instruction_unclosed(self): + """A processing instruction which is never closed is text. Regression: renders as it always has.""" + + self.assertMarkdownRenders( + self.dedent( + R''' + !!! note + +Note
+text <!-- + c
+--> more
+Note
+<?php + c
+text
+Note
+<![CDATA[ + c
+text
+
Note
+<![CDATA[ + c
+] ]> + after
+Note
+<!DOCTYPE + c
+text
+<div>
+
+ code
+ </div>
+
+ Title
+<div>
+ code
+ </div>
+
+ Title
+trailing
+after
+Title
+trailing + more
+after
+Title
+trailing
+after
+Title
+ +trailing
+after
+item
+trailing
+after
+text
+<div>x</div> trailing
+ more code
+
+ '''
+ )
+ )
+
+ def test_trailing_text_in_code_block_in_admonition(self):
+ """Same inside an admonition (8 spaces). Regression: renders as it always has."""
+
+ self.assertMarkdownRenders(
+ self.dedent(
+ R'''
+ !!! note "Title"
+ x
trailing + more code + ''' + ), + self.dedent( + ''' +Title
+<p>x</p> trailing
+ more code
+
+ Outer
+Inner
+trailing
+after
+after
+ ''' + ), + extensions=[SectionExtension()] + ) + + def test_unterminated_comment(self): + self.assertMarkdownRenders( + self.dedent( + ''' + ::: section + after + + para + ''' + ), + self.dedent( + ''' +item
+ +after
+para
+item
+ +para
+d
++ code + more ++ ''' + ), + self.dedent( + ''' +
+ code + more +
x
+x
+para
+two
+x
+