gh-156505: Speed up difflib.HtmlDiff for lopsided replacements

[felix314159]

difflib._mdiff() used lists as FIFO queues for unmatched lines and removed
items with pop(0). This repeatedly shifted the remaining items, causing
quadratic behavior for lopsided replacement blocks.

Use collections.deque and popleft() for constant-time FIFO removal. This
preserves the existing output while making the affected line-pairing step scale
approximately linearly.

Benchmarks on matched release builds from CPython main:

Case	Before	After	Speedup
_mdiff(), 64,000-to-1	264 ms	61.9 ms	4.26x
HtmlDiff.make_table(), 64,000-to-1	398 ms	206 ms	1.93x
_mdiff(), 128,000-to-1	999 ms	126 ms	7.94x

The complete test_difflib suite passes. A focused regression test covers
lopsided replacements in both directions, and randomized differential testing
confirmed identical output for _mdiff() and HtmlDiff across context and
wrapping modes.

Fixes #156505

• Issue: difflib.HtmlDiff has quadratic line pairing for lopsided replacements #156505

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[felix314159]

I think the skip news label makes sense here as behavior is unchanged

[eendebakpt]

Do you have benchmarks for more mixed scenario's? In particular cases where the changes could cause a regression?

[felix314159]

I benchmarked the parent commit against this PR using 15 paired randomized samples with the same non-debug interpreter. So basically I tested identical inputs, sparse replacements, balanced replacements, and mixed insertion/deletion/replacement workloads. The larger mixed cases ranged from unchanged to like 2% faster. A four-line _mdiff case measured about 1.45% slower, but remember this is only around 0.3 µs per call. The corresponding HtmlDiff case differed by only 0.15%. So IMO there is no meaningful regression outside the lopsided case this PR fixes.

[StanFromIreland]

CC @dg-pb @tim-one

[dg-pb]

Sensible improvement. No new imports. Change makes sense in straight forward manner. No risk for degradation for any cases. Not much to add really.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 LTO + PGO 3.x (tier-1) has failed when building commit 1f25c33.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/864/builds/9792) and take a look at the build logs.
4. Check if the failure is related to this commit (1f25c33) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/864/builds/9792

Summary of the results of the build (if available):

==

Click to see traceback logs

Note: switching to '1f25c333a280561e86cc0af69af307b53ddbaaac'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 1f25c333a280 gh-156505: Speed up difflib.HtmlDiff for lopsided replacements (GH-156506)
Switched to and reset branch 'main'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:3456: clean-retain-profile] Error 1 (ignored)
./Modules/socketmodule.c: In function ‘getsockaddrarg’:
./Modules/socketmodule.c:2664:9: warning: ‘strncpy’ specified bound 64 equals destination size [-Wstringop-truncation]
 2664 |         strncpy((char *)sa->salg_name, name, sizeof(sa->salg_name));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
lto-wrapper: warning: using serial compilation of 6 LTRANS jobs
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
lto-wrapper: warning: using serial compilation of 8 LTRANS jobs
lto-wrapper: warning: using serial compilation of 5 LTRANS jobs
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
lto-wrapper: warning: using serial compilation of 7 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 7 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: warning: using serial compilation of 7 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 128 LTRANS jobs
lto-wrapper: warning: using serial compilation of 128 LTRANS jobs
lto-wrapper: warning: using serial compilation of 128 LTRANS jobs
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1044:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1089:9,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1067:1:
./Modules/expat/xmltok.c:397:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  397 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1042:8: note: destination object ‘buf’ of size 1
 1042 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1044:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1129:9,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1067:1:
./Modules/expat/xmltok.c:397:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  397 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1042:8: note: destination object ‘buf’ of size 1
 1042 |   char buf[1];
      |        ^~~
./Programs/_testembed.c: In function ‘main’:
./Programs/_testembed.c:3115:1: warning: ‘/var/lib/buildbot/worker/cstratak-CentOS9-x86_64/3.x.cstratak-CentOS9-x86_64.lto-pgo/build/Programs/_testembed.gcda’ profile count data file not found [-Wmissing-profile]
 3115 | }
      | ^
Python/ceval.c: In function ‘_Py_BuildMap_StackRefSteal’:
Python/ceval.c:943:11: warning: ‘args_o_temp’ may be used uninitialized [-Wmaybe-uninitialized]
  943 |     res = _PyDict_FromItems(
      |           ^~~~~~~~~~~~~~~~~~
  944 |         args_o, 2,
      |         ~~~~~~~~~~
  945 |         args_o+1, 2,
      |         ~~~~~~~~~~~~
  946 |         half_args
      |         ~~~~~~~~~
  947 |     );
      |     ~      
In file included from Python/ceval.h:11,
                 from Python/ceval.c:3:
./Include/internal/pycore_dict.h:333:23: note: by argument 3 of type ‘struct PyObject * const *’ to ‘_PyDict_FromItems’ declared here
  333 | PyAPI_FUNC(PyObject *)_PyDict_FromItems(
      |                       ^~~~~~~~~~~~~~~~~
In file included from Python/ceval.c:702:
Python/ceval.c:938:52: note: ‘args_o_temp’ declared here
  938 |     STACKREFS_TO_PYOBJECTS(arguments, half_args*2, args_o);
      |                                                    ^~~~~~
Python/ceval.c: In function ‘_PyCallMethodDescriptorFast_StackRef’:
Python/ceval.c:862:11: warning: ‘args_o_temp’ may be used uninitialized [-Wmaybe-uninitialized]
  862 |     res = cfunc(self, (args_o + 1), total_args - 1);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Python/ceval.c:862:11: note: by argument 2 of type ‘struct PyObject * const *’ to ‘struct PyObject * <Tcf0> (struct PyObject *, struct PyObject * const *, Py_ssize_t)’
In file included from Python/ceval.c:702:
Python/ceval.c:856:51: note: ‘args_o_temp’ declared here
  856 |     STACKREFS_TO_PYOBJECTS(arguments, total_args, args_o);
      |                                                   ^~~~~~
Python/ceval.c:862:11: warning: ‘_101’ may be used uninitialized [-Wmaybe-uninitialized]
  862 |     res = cfunc(self, (args_o + 1), total_args - 1);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Python/ceval.c:862:11: note: by argument 2 of type ‘struct PyObject * const *’ to ‘struct PyObject * <Tcf0> (struct PyObject *, struct PyObject * const *, Py_ssize_t)’
In file included from Python/ceval.c:702:
Python/ceval.c:856:51: note: ‘args_o_temp’ declared here
  856 |     STACKREFS_TO_PYOBJECTS(arguments, total_args, args_o);
      |                                                   ^~~~~~
Python/ceval.c: In function ‘_PyCallMethodDescriptorFastWithKeywords_StackRef’:
Python/ceval.c:883:11: warning: ‘args_o_temp’ may be used uninitialized [-Wmaybe-uninitialized]
  883 |     res = cfunc(self, (args_o + 1), total_args-1, NULL);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Python/ceval.c:883:11: note: by argument 2 of type ‘struct PyObject * const *’ to ‘struct PyObject * <Tcf6> (struct PyObject *, struct PyObject * const *, Py_ssize_t, struct PyObject *)’
In file included from Python/ceval.c:702:
Python/ceval.c:877:51: note: ‘args_o_temp’ declared here
  877 |     STACKREFS_TO_PYOBJECTS(arguments, total_args, args_o);
      |                                                   ^~~~~~
Python/ceval.c:883:11: warning: ‘_101’ may be used uninitialized [-Wmaybe-uninitialized]
  883 |     res = cfunc(self, (args_o + 1), total_args-1, NULL);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Python/ceval.c:883:11: note: by argument 2 of type ‘struct PyObject * const *’ to ‘struct PyObject * <Tcf6> (struct PyObject *, struct PyObject * const *, Py_ssize_t, struct PyObject *)’
In file included from Python/ceval.c:702:
Python/ceval.c:877:51: note: ‘args_o_temp’ declared here
  877 |     STACKREFS_TO_PYOBJECTS(arguments, total_args, args_o);
      |                                                   ^~~~~~
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 4 LTRANS jobs
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 2 LTRANS jobs
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1044:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1089:9:
./Modules/expat/xmltok.c:397:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  397 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute.part.0’:
./Modules/expat/xmltok.c:1042:8: note: destination object ‘buf’ of size 1
 1042 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1044:3,
    inlined from ‘parsePseudoAttribute.part.0’ at ./Modules/expat/xmltok.c:1129:9:
./Modules/expat/xmltok.c:397:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  397 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute.part.0’:
./Modules/expat/xmltok.c:1042:8: note: destination object ‘buf’ of size 1
 1042 |   char buf[1];
      |        ^
lto-wrapper: warning: using serial compilation of 81 LTRANS jobs
lto-wrapper: warning: using serial compilation of 86 LTRANS jobs
lto-wrapper: warning: using serial compilation of 85 LTRANS jobs

make: *** [Makefile:2481: buildbottest] Error 3