Repository navigation
Description
Description
If a client sends a JSON body with a Content-Type charset that Python doesn't recognise, for example
Content-Type: application/json; charset=not-a-charset
PyramidParser and AIOHTTPParser raise an uncaught LookupError: unknown encoding: not-a-charset. The server then responds with 500 Internal Server Error. The documented behaviour for an unreadable JSON body is a 400 {"json": ["Invalid JSON body."]}.
Any client can trigger this, because the value comes straight from a request header.
Why it happens:
- Pyramid:
PyramidParser._raw_load_jsoncallscore.parse_json(req.body, encoding=req.charset). Insideparse_json,s.decode(encoding)raisesLookupErrorfor an unknown codec.parse_jsononly convertsUnicodeDecodeErrorintoJSONDecodeError, soParser.load_jsonnever catches theLookupError. - aiohttp:
AIOHTTPParser.load_jsoncallsreq.json(). aiohttp'sRequest.text()runsbytes_body.decode(self.charset or "utf-8"), which raises the sameLookupError.load_jsononly catchesJSONDecodeErrorandUnicodeDecodeError.
The other parsers (Flask, Django, Bottle, Falcon) ignore the charset and decode the body as UTF-8, so they return 200. Issue #427 fixed the similar case of a body that isn't valid UTF-8. This is a different input that still gets through.
Reproduction
webargs dev @ 3c7f387, Python 3.15, pyramid 2.1, aiohttp 3.14.5, marshmallow 4.4.0.
import asyncio
import webtest
import webtest_aiohttp
from aiohttp import web
from pyramid.config import Configurator
from webargs import fields
from webargs.aiohttpparser import parser as aio_parser
from webargs.pyramidparser import parser as pyr_parser
ARGS = {"name": fields.Str()}
BODY = b'{"name": "Steve"}'
HEADERS = {"Content-Type": "application/json; charset=not-a-charset"}
def pyramid_view(request):
return pyr_parser.parse(ARGS, request, location="json")
with Configurator() as config:
config.add_route("echo", "/echo")
config.add_view(pyramid_view, route_name="echo", renderer="json")
pyr_app = webtest.TestApp(config.make_wsgi_app())
async def aio_view(request):
return web.json_response(await aio_parser.async_parse(ARGS, request, location="json"))
aio = web.Application()
aio.router.add_post("/echo", aio_view)
aio_app = webtest_aiohttp.TestApp(aio, loop=asyncio.new_event_loop())
for name, app in [("pyramid", pyr_app), ("aiohttp", aio_app)]:
try:
res = app.post("/echo", BODY, headers=HEADERS, expect_errors=True)
print(f"{name}: {res.status} {res.body!r}")
except Exception as exc:
print(f"{name}: UNCAUGHT {type(exc).__name__}: {exc}")Actual
pyramid: UNCAUGHT LookupError: unknown encoding: not-a-charset
aiohttp: 500 Internal Server Error b'500 Internal Server Error\n\nServer got itself in trouble'
Expected
pyramid: 400 Bad Request b'{"json": ["Invalid JSON body."]}'
aiohttp: 400 Bad Request b'{"json": ["Invalid JSON body."]}'
That is, the same handling as other undecodable JSON bodies (see #427).
Activity
sirosen commentedon Nov 16, 2026
Although the submitted PR (#1121) was not acceptable, I think this might be a real issue. I am at least willing to entertain the idea.
It can be handled today by users if they subclass the parser and customize it a bit, so it doesn't strike me as severe.
hemtilva commentedon Nov 16, 2026
Thanks for keeping this open, and sorry about #1121. You're right, catching LookupError around the whole json() call could hide unrelated KeyErrors too. That was careless.
I see two narrower options. One is to check just the charset up front with
codecs.lookup(req.charset)and return the normal 400 only if that fails. The other is to ignore an unknown charset and decode as UTF-8, which is what the Flask/Django/Bottle/Falcon parsers effectively do already. The second seems more consistent to me, but I'm happy either way. If you think it's worth fixing, I'd write it by hand with a test for each parser. If not, no problem.sirosen commentedon Nov 16, 2026
Are you personally using pyramid or aiohttp with webargs? I'm disinclined to rush to make a change if no impacted users are asking for it.
Frankly, you are on very thin ice here. You threw a clearly LLM-generated issue at us -- it's about 6 times longer than it needs to be to adequately describe the issue -- and a generated PR that wasted someone else's time. Based on your most recent reply, I'm not fully confident you understand why your last PR was unacceptable, both from a policy and a quality perspective.