Skip to content

Commit 2f87bf9

Browse files
serhiy-storchakaclaude
serhiy-storchaka
and
claude
committed
gh-154668: Add escaping mode to PyUnicode_FromFormat()
With the "#" flag, the "c", "s", "S", "U" and "V" conversions now escape special and non-printable characters, and with the additional "+" flag -- also all non-ASCII characters. Bytes which cannot be decoded from UTF-8 in the "s" and "V" conversions are escaped as \xNN, valid non-ASCII characters are escaped as \uNNNN or \UNNNNNNNN. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 5062427 commit 2f87bf9

6 files changed

Lines changed: 334 additions & 10 deletions

File tree

‎Doc/c-api/unicode.rst‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -477,6 +477,30 @@ APIs:
477477
| ``-`` | The converted value is left adjusted (overrides the ``0`` |
478478
| | flag if both are given). |
479479
+-------+-------------------------------------------------------------+
480+
| ``#`` | Escape special and non-printable characters for the ``c``, |
481+
| | ``s``, ``S``, ``U`` and ``V`` conversions. |
482+
| | Use a colon as a separator for the ``T`` and ``N`` |
483+
| | conversions. |
484+
+-------+-------------------------------------------------------------+
485+
| ``+`` | Escape also all non-ASCII characters (only in combination |
486+
| | with the ``#`` flag for the ``c``, ``s``, ``S``, ``U`` and |
487+
| | ``V`` conversions). |
488+
+-------+-------------------------------------------------------------+
489+
490+
With the ``#`` flag, the ``c``, ``s``, ``S``, ``U`` and ``V``
491+
conversions escape special and non-printable characters
492+
like :func:`repr` does,
493+
but without adding surrounding quotes:
494+
backslash, tab, carriage return and line feed are escaped as
495+
``\\``, ``\t``, ``\r`` and ``\n`` respectively,
496+
other non-printable characters are escaped as ``\xNN`` if they are
497+
ASCII, and as ``\uNNNN`` or ``\UNNNNNNNN`` otherwise.
498+
With the ``+`` flag, all non-ASCII characters are escaped as well.
499+
For the ``s`` and ``V`` conversions,
500+
bytes which cannot be decoded from UTF-8 are escaped as ``\xNN``,
501+
so they can always be distinguished from valid non-ASCII characters.
502+
The precision is applied to the string before escaping,
503+
the width -- after escaping.
480504
481505
The length modifiers for following integer conversions (``d``, ``i``,
482506
``o``, ``u``, ``x``, or ``X``) specify the type of the argument
@@ -630,6 +654,10 @@ APIs:
630654
.. versionchanged:: 3.13
631655
Support for ``%T``, ``%#T``, ``%N`` and ``%#N`` formats added.
632656
657+
.. versionchanged:: next
658+
Support for flags ``#`` and ``+`` for the ``c``, ``s``, ``S``,
659+
``U`` and ``V`` conversions added.
660+
633661
634662
.. c:function:: PyObject* PyUnicode_FromFormatV(const char *format, va_list vargs)
635663

‎Doc/whatsnew/3.16.rst‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -814,7 +814,12 @@ C API changes
814814
New features
815815
------------
816816

817-
* TODO
817+
* :c:func:`PyUnicode_FromFormat` now supports the escaping mode for
818+
the ``c``, ``s``, ``S``, ``U`` and ``V`` conversions:
819+
with the ``#`` flag, special and non-printable characters in the output
820+
are escaped,
821+
and with the additional ``+`` flag -- also all non-ASCII characters.
822+
(Contributed by Serhiy Storchaka in :gh:`154668`.)
818823

819824
Porting to Python 3.16
820825
----------------------

‎Lib/test/test_capi/test_unicode.py‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -535,6 +535,114 @@ def check_format(expected, format, *args):
535535
check_format("repr= 12",
536536
b'repr=%5.2V', None, b'123')
537537

538+
# test the escaping mode ("#" flag) with %c
539+
check_format('A',
540+
b'%#c', c_int(ord('A')))
541+
check_format('\\n',
542+
b'%#c', c_int(ord('\n')))
543+
check_format('\\\\',
544+
b'%#c', c_int(ord('\\')))
545+
check_format('\\x1b',
546+
b'%#c', c_int(0x1b))
547+
check_format('\\x7f',
548+
b'%#c', c_int(0x7f))
549+
check_format('\\u009f',
550+
b'%#c', c_int(0x9f))
551+
check_format('\xe9',
552+
b'%#c', c_int(0xe9))
553+
check_format('\\u2028',
554+
b'%#c', c_int(0x2028))
555+
check_format('\U0001f600',
556+
b'%#c', c_int(0x1f600))
557+
check_format('\\ud800',
558+
b'%#c', c_int(0xd800))
559+
check_format('\\udc9f',
560+
b'%#c', c_int(0xdc9f))
561+
check_format('\\u00e9',
562+
b'%+#c', c_int(0xe9))
563+
check_format('\\u20ac',
564+
b'%+#c', c_int(0x20ac))
565+
check_format('\\U0001f600',
566+
b'%+#c', c_int(0x1f600))
567+
568+
# test the escaping mode with %s
569+
check_format('a\\tb\\r\\nc\\\\d',
570+
b'%#s', b'a\tb\r\nc\\d')
571+
check_format('quote\'"',
572+
b'%#s', b'quote\'"')
573+
check_format('caf\xe9',
574+
b'%#s', b'caf\xc3\xa9')
575+
check_format('caf\\u00e9',
576+
b'%+#s', b'caf\xc3\xa9')
577+
# valid UTF-8 is escaped as a character, invalid -- as a raw byte
578+
check_format('\\u009f',
579+
b'%#s', b'\xc2\x9f')
580+
check_format('\\x9f',
581+
b'%#s', b'\x9f')
582+
check_format('\\x9f',
583+
b'%+#s', b'\x9f')
584+
check_format('a\\xffb',
585+
b'%#s', b'a\xffb')
586+
check_format('abc\\xc3',
587+
b'%#s', b'abc\xc3')
588+
# the precision is applied before escaping, the width -- after
589+
check_format('ab\\n',
590+
b'%#.3s', b'ab\ncd')
591+
check_format(' ab\\n',
592+
b'%#6.3s', b'ab\ncd')
593+
check_format('ab\\n ',
594+
b'%-#6.3s', b'ab\ncd')
595+
check_format('a\\xff',
596+
b'%#.2s', b'a\xffbc')
597+
check_format('abc',
598+
b'%#.4s', b'abc\xc3\xa9')
599+
600+
# test the escaping mode with %U, %V and %S
601+
check_format('a\\tb',
602+
b'%#U', 'a\tb')
603+
check_format('caf\xe9',
604+
b'%#U', 'caf\xe9')
605+
check_format('caf\\u00e9',
606+
b'%+#U', 'caf\xe9')
607+
check_format('a\\tb',
608+
b'%#S', 'a\tb')
609+
check_format('caf\xe9',
610+
b'%#S', 'caf\xe9')
611+
check_format('caf\\u00e9',
612+
b'%+#S', 'caf\xe9')
613+
check_format(' ab',
614+
b'%#5.2S', 'ab\ncd')
615+
# a lone surrogate in a string argument stays a surrogate
616+
check_format('\\udc9f',
617+
b'%#U', '\udc9f')
618+
check_format('\\udc9f',
619+
b'%+#U', '\udc9f')
620+
check_format(' ab',
621+
b'%#5.2U', 'ab\ncd')
622+
check_format('a\\tb',
623+
b'%#V', 'a\tb', b'ignored')
624+
check_format('\\u009f',
625+
b'%#V', None, b'\xc2\x9f')
626+
check_format('\\x9f',
627+
b'%#V', None, b'\x9f')
628+
629+
# test the escaping mode with %ls and %lV
630+
check_format('a\\tb',
631+
b'%#ls', c_wchar_p('a\tb'))
632+
check_format('caf\xe9',
633+
b'%#ls', c_wchar_p('caf\xe9'))
634+
check_format('caf\\u00e9',
635+
b'%+#ls', c_wchar_p('caf\xe9'))
636+
check_format('a\\tb',
637+
b'%#lV', None, c_wchar_p('a\tb'))
638+
639+
# "+" requires "#" and is only supported for %c, %s, %S, %U and %V
640+
for format in (b'%+c', b'%+s', b'%+S', b'%+U', b'%+d', b'%+#d',
641+
b'%+#R', b'%+#A', b'%+#T', b'%+#p'):
642+
with self.subTest(format=format):
643+
with self.assertRaises(SystemError):
644+
PyUnicode_FromFormat(format, py_object('abc'))
645+
538646
# test integer formats (%i, %d, %u, %o, %x, %X)
539647
check_format('010',
540648
b'%03i', c_int(10))
@@ -2003,6 +2111,11 @@ def test_format(self):
20032111
writer.write_char(ord('.'))
20042112
self.assertEqual(writer.finish(), 'abc 123.')
20052113

2114+
def test_format_escape(self):
2115+
writer = self.create_writer(0)
2116+
self.writer_format(writer, b'%#s %+#s', b'a\tb\xff', b'caf\xc3\xa9')
2117+
self.assertEqual(writer.finish(), 'a\\tb\\xff caf\\u00e9')
2118+
20062119
def test_recover_error(self):
20072120
# test recovering from PyUnicodeWriter_Format() error
20082121
writer = self.create_writer(0)

‎Misc/NEWS.d/next/C_API/2026-07-25-10-40-00.gh-issue-154668.fEsc4p.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
Add the escaping mode to :c:func:`PyUnicode_FromFormat`:
2+
with the ``#`` flag,
3+
the ``c``, ``s``, ``S``, ``U`` and ``V`` conversions now escape special
4+
and non-printable characters,
5+
and with the additional ``+`` flag -- also all non-ASCII characters.

‎Modules/_testlimitedcapi/unicode.c‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1824,6 +1824,29 @@ test_string_from_format(PyObject *self, PyObject *Py_UNUSED(ignored))
18241824
CHECK_FORMAT_2("%1.5V", "None", NULL, "None");
18251825
CHECK_FORMAT_2("%1.5lV", "None", NULL, L"None");
18261826

1827+
// Strings: escaping mode ('#' and '+' flags)
1828+
CHECK_FORMAT_1("%#c", "c", 'c');
1829+
CHECK_FORMAT_1("%#c", "\\n", '\n');
1830+
CHECK_FORMAT_1("%+#c", "\\n", '\n');
1831+
CHECK_FORMAT_1("%#s", "a\\tb", "a\tb");
1832+
CHECK_FORMAT_1("%+#s", "a\\tb", "a\tb");
1833+
CHECK_FORMAT_1("%#ls", "a\\tb", L"a\tb");
1834+
CHECK_FORMAT_1("%#S", "None", Py_None);
1835+
CHECK_FORMAT_1("%+#S", "None", Py_None);
1836+
CHECK_FORMAT_1("%#U", "None", unicode);
1837+
CHECK_FORMAT_2("%#V", "None", unicode, "ignored");
1838+
CHECK_FORMAT_2("%#V", "a\\tb", NULL, "a\tb");
1839+
CHECK_FORMAT_2("%#lV", "a\\tb", NULL, L"a\tb");
1840+
CHECK_FORMAT_1("%#6.3s", " ab\\n", "ab\ncd");
1841+
1842+
// The '+' flag requires the '#' flag and is only supported for
1843+
// the 'c', 's', 'S', 'U' and 'V' conversions
1844+
CHECK_FORMAT_1("%+c", NULL, 'c');
1845+
CHECK_FORMAT_1("%+s", NULL, "abc");
1846+
CHECK_FORMAT_1("%+d", NULL, 1);
1847+
CHECK_FORMAT_1("%+#d", NULL, 1);
1848+
CHECK_FORMAT_1("%+#R", NULL, Py_None);
1849+
18271850
Py_XDECREF(unicode);
18281851
Py_RETURN_NONE;
18291852

0 commit comments

Comments
 (0)