Plugin output is not being rendered

[jessicah]

Describe the bug

context
I've written a port of markdown-it-abbr to python, but when used with MyST-Parser, the replacement tokens are missing, due to an error of WARNING: No render method for: abbr [myst.render].

expectation
In the rendered HTML, I expected the <abbr> HTML tag and content to be output.

E.g. the following markdown:

*[HTML]: HyperText Markup Language
*[W3C]: World Wide Web Consortium

The HTML specification is maintained by the W3C.

should generate:

<p>The <abbr title="HyperText Markup Language'>HTML</abbr> specification is maintained by the <abbr title="World Wide Web Consortium">W3C</abbr>.</p>

bug
Instead, it generates:

<p>The specification is maintained by the .</p>

I added print(nodes) at https://github.com/jessicah/mdit-py-abbr/blob/main/mdit_py_abbr/index.py#L157, and it showed up fine in console:

[Token(type='text', tag='', nesting=0, attrs={}, map=None, level=0, children=None, content='The ', markup='', info='', meta={}, block=False, hidden=False), Token(type='abbr_open', tag='abbr', nesting=1, attrs={'title': 'HyperText Markup Language'}, map=None, level=0, children=None, content='', markup='', info='', meta={}, block=False, hidden=False), Token(type='text', tag='', nesting=0, attrs={}, map=None, level=0, children=None, content='HTML', markup='', info='', meta={}, block=False, hidden=False), Token(type='abbr_close', tag='abbr', nesting=-1, attrs={}, map=None, level=0, children=None, content='', markup='', info='', meta={}, block=False, hidden=False), Token(type='text', tag='', nesting=0, attrs={}, map=None, level=0, children=None, content=' specification is maintained by the ', markup='', info='', meta={}, block=False, hidden=False), Token(type='abbr_open', tag='abbr', nesting=1, attrs={'title': 'World Wide Web Consortium'}, map=None, level=0, children=None, content='', markup='', info='', meta={}, block=False, hidden=False), Token(type='text', tag='', nesting=0, attrs={}, map=None, level=0, children=None, content='W3C', markup='', info='', meta={}, block=False, hidden=False), Token(type='abbr_close', tag='abbr', nesting=-1, attrs={}, map=None, level=0, children=None, content='', markup='', info='', meta={}, block=False, hidden=False), Token(type='text', tag='', nesting=0, attrs={}, map=None, level=0, children=None, content='.', markup='', info='', meta={}, block=False, hidden=False)]

So the token stream itself seems correct. Also, using markdown-it-py directly works correctly.

I did try adding a render method, but it had no effect. E.g.:

md.add_render_rule("abbr", abbr_render)

and

def abbr_render(self, tokens, idx, options, env):
    return f'<abbr title="{tokens[idx].attrGet("title")}">{tokens[idx].content}</abbr>'

Reproduce the bug

I've created an example repo: https://github.com/jessicah/myst-demo

make [all] runs sphinx-build, and make test runs the test file.

List your environment

myst_parser: 0.18.1
sphinx: 5.3.0
docutils: 0.17.1
markdown_it: 1.1.0

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[jessicah]

It seems like perhaps MyST-Parser's DocutilsRenderer is the culprit, using a fixed list of renderers?

See: https://github.com/executablebooks/MyST-Parser/blob/master/myst_parser/mdit_to_docutils/base.py#L102-L106

This might also explain what was going wrong in my WIP patch in #632

[jessicah]

Ohh, I see, it's because the overridden render methods produces docutils AST nodes, correct? And the plugins produce HTML strings, which is why all the currently supported plugins have their equivalent render_xyz methods in the DocutilsRenderer class. Still, an easy way to extend them would be useful... hmm.

[chrisjsewell]

Heya, yep indeed myst-parser intentionally does not expose markdown-it-py as the underlying markdown parser, it would not be maintainable to support such an API.

If you want to have a look at contributing your plugin to https://github.com/executablebooks/mdit-py-plugins, we can look to add it to myst-parser as an extension 😄
(there is already a node type for it, so should not be too difficult https://docutils.sourceforge.io/docs/ref/doctree.html#abbreviation)

[jessicah]

@chrisjsewell ah, yeah, I was just at that docutils link, although not quite sure how to write the render_abbr method in my python code.

I figured I can make a custom renderer by subclassing myst_parser.mdit_to_docutils.sphinx_.SphinxRenderer, and calling create_md_parser(config, CustomRenderer):

class CustomRenderer(SphinxRenderer):
	def render_abbr(self, token: SyntaxTreeNode) -> None:
		print(token)

class CustomParser(MystParser):
	def parse(self, inputstring: str, document: nodes.document) -> None:
		config: MdParserConfig = document.settings.env.myst_config
		# . . .			
		parser = create_md_parser(config, CustomRenderer)
		# . . . 

def setup(app):
	from myst_parser.sphinx_ext.main import setup_sphinx

	setup_sphinx(app, load_parser=False)
	app.add_source_suffix(".md", "markdown")
	app.add_source_parser(CustomParser)

I'm at least getting the print outputing SyntaxTreeNode(abbr) in my test, so it's just figuring out the final piece...

Some tips on what I should be producing in render_abbr would be great, and I can make PRs for both :)

[chrisjsewell]

Yeh no worries, if you get the PR into executablebooks/mdit-py-plugins, I should be able to take care of the myst-parser side (it should be relatively straight forward)

[jessicah]

executablebooks/mdit-py-plugins#63

[jessicah]

@chrisjsewell any chance you can help me out with the myst-parser side as the plugin exists atm? I'd like to be able to start using in my own project ;-)