Skip to content

Commit dde954e

Browse files
committed
gh-109638: Fix excessive backtracking in csv.Sniffer doublequote detection
The doublequote-detection pattern used two greedy character classes that both admitted the quote character, so the engine had to try every way of splitting a run of quotes between them. Excluding the quote from the first class forces the quote that follows it to match at the first one available, which removes the ambiguity while leaving detection intact. The second class must keep admitting the quote, since that is what lets `"a""b"` match -- narrowing both classes, or making the first one atomic while it still admits the quote, silently disables the detection.
1 parent 23b2b40 commit dde954e

3 files changed

Lines changed: 17 additions & 1 deletion

File tree

Lib/csv.py

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -330,8 +330,14 @@ def _guess_quote_and_delimiter(self, data, delimiters):
330330

331331
# if we see an extra quote between delimiters, we've got a
332332
# double quoted format
333+
#
334+
# The first character class must exclude the quote character. If it
335+
# can cross a quote, the engine has to try every way of splitting a
336+
# run of quotes between the two classes, which is pathological on
337+
# quote-heavy input (gh-109638). The second class must keep admitting
338+
# the quote, since that is what lets `"a""b"` match.
333339
dq_regexp = re.compile(
334-
r"((%(delim)s)|^)\W*%(quote)s[^%(delim)s\n]*%(quote)s[^%(delim)s\n]*%(quote)s\W*((%(delim)s)|$)" % \
340+
r"((%(delim)s)|^)\W*%(quote)s[^%(delim)s%(quote)s\n]*%(quote)s[^%(delim)s\n]*%(quote)s\W*((%(delim)s)|$)" % \
335341
{'delim':re.escape(delim), 'quote':quotechar}, re.MULTILINE)
336342

337343

Lib/test/test_csv.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1501,6 +1501,12 @@ def test_delimiters(self):
15011501
self.assertEqual(dialect.delimiter, ',')
15021502
self.assertEqual(dialect.quotechar, '"')
15031503

1504+
def test_sniff_regex_backtracking(self):
1505+
# gh-109638: this artificial sample used to take about ten seconds.
1506+
sniffer = csv.Sniffer()
1507+
sample = '"",' * 60 + '"' * 60 + '0' + '"' * 60 + '0'
1508+
self.assertEqual(sniffer.sniff(sample).delimiter, ',')
1509+
15041510
def test_doublequote(self):
15051511
sniffer = csv.Sniffer()
15061512
dialect = sniffer.sniff(self.header1)
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Fix excessive regular expression backtracking in :class:`csv.Sniffer` on
2+
quote-heavy samples. Detecting a doubled-quote dialect could try every way of
3+
splitting a run of quotes between two character classes, making
4+
:meth:`~csv.Sniffer.sniff` take seconds on samples of a few hundred characters.

0 commit comments

Comments
 (0)