Fix ReDoS from unclosed HTML tags (#707)

[wolfgang-aura]

Fixes #707. Supersedes #708, which replaced the tokenizer with a hand-written scanner and was closed by its author. This keeps the existing tokenizer and removes the two slow paths instead.

Cause

Two separate slow paths sit behind the issue's input:

• _tag_is_closed counted opening tags with re.findall('<%s(?:.*?)>' % tag_name, text). From every unclosed <p the lazy .*? scans to the end of the line before failing, so a line of '<p m="1"' * 15000 is quadratic.
• _sorta_html_tokenize_re had two ways to split the same attribute text. The optional (?:[^\t<>"'=/]+:)? namespace group overlaps the attribute name class, which already allows :, and the name could begin with whitespace that the preceding \s+ also matches. An unclosed tag such as 'x <p' + ' a:b=1' * 40 or 'x <p' + ' a=1' * 40 backtracks exponentially. Both _hash_html_spans and the tokenizer split use this regex, so the fix covers the spot raised in the review of fix: prevent ReDoS in HTML tokenizer via bounded substring matching (issue #707) #708.

Change

• The attribute name is now [^\s<>"'=/][^<>"'=/]*=, so it starts at the first non-space character and the namespace group is gone. Each attribute has one parse. One degenerate case changes: <p =1>, with two spaces and an empty attribute name, no longer tokenizes as a tag.
• _tag_is_closed counts openers with str.find, looking for > only up to the end of the current line. That gives the same count as the old regex (a fuzz comparison over 400,000 random strings of <p, <pre, >, newlines and text found no difference) and stays linear, including for many short lines followed by one distant >.
• Three cases in test/test_redos.py, one per shape above, and a CHANGES.md line.

Testing

• make testredos: the three new cases each time out after 4s on master and pass with the change; 10 passed.
• make testone (python test.py -- -knownfailure): 288 passed.
• Timing through markdown() with the change: '<p m="1"' * 15000 0.22s, the two attribute shapes 0.18s each, all three over 4s on master.

Only Windows 11 with Python 3.14 was tested. While checking, a separate slow path showed up on master and is left alone here: markdown('<p m="1"\n' * 20000 + '>') takes about 23s both before and after this change, and a profile puts the time in a re.sub call outside both functions changed here.

Prepared with Claude Code assistance.

[Crozzers]

Why limit it to scanning only until the end of the line? What would it do for a snippet like this:

<div
  style='...'
  id='...'
  ...
>
</div>

[Crozzers]

If you wanted to strictly mimic the behaviour of the original regex you could iterate over splitlines instead:

        # find all open tags on each line. Ignore tags split over multiple lines because ...
        open_tag = f'<{tag_name}'
        open_count = 0
        for line in text.splitlines():
            pos = line.find(open_tag)
            while pos != -1:
                end = line.find('>', pos)
                if end == -1:
                    break
                open_count += 1
                pos = line.find(open_tag, end + 1)
        # check if number of open tags == number of close tags
        if open_count != text.count('</%s>' % tag_name):
            return False

It's a bit more readable and alot easier to understand

[wolfgang-aura]

Thanks, that is clearer. The old regex <tag(?:.*?)> has no DOTALL, so . stopped at a newline and a tag split across lines was never counted. I kept that, which is why the first version stopped at the end of the line. For your <div example nothing changes: it was not counted before and is not counted now.

I switched to your splitlines version in 511e95a. Breaking at the first unclosed <tag on a line keeps the scan linear. The ReDoS tests pass.

[nicholasserra]

Thank you!