Skip to content

Unknown charset in a JSON request's Content-Type crashes PyramidParser and AIOHTTPParser (500 instead of 400) #1120

Open

Description

@hemtilva

Description

If a client sends a JSON body with a Content-Type charset that Python doesn't recognise, for example

Content-Type: application/json; charset=not-a-charset

PyramidParser and AIOHTTPParser raise an uncaught LookupError: unknown encoding: not-a-charset. The server then responds with 500 Internal Server Error. The documented behaviour for an unreadable JSON body is a 400 {"json": ["Invalid JSON body."]}.

Any client can trigger this, because the value comes straight from a request header.

Why it happens:

  • Pyramid: PyramidParser._raw_load_json calls core.parse_json(req.body, encoding=req.charset). Inside parse_json, s.decode(encoding) raises LookupError for an unknown codec. parse_json only converts UnicodeDecodeError into JSONDecodeError, so Parser.load_json never catches the LookupError.
  • aiohttp: AIOHTTPParser.load_json calls req.json(). aiohttp's Request.text() runs bytes_body.decode(self.charset or "utf-8"), which raises the same LookupError. load_json only catches JSONDecodeError and UnicodeDecodeError.

The other parsers (Flask, Django, Bottle, Falcon) ignore the charset and decode the body as UTF-8, so they return 200. Issue #427 fixed the similar case of a body that isn't valid UTF-8. This is a different input that still gets through.

Reproduction

webargs dev @ 3c7f387, Python 3.15, pyramid 2.1, aiohttp 3.14.5, marshmallow 4.4.0.

import asyncio

import webtest
import webtest_aiohttp
from aiohttp import web
from pyramid.config import Configurator

from webargs import fields
from webargs.aiohttpparser import parser as aio_parser
from webargs.pyramidparser import parser as pyr_parser

ARGS = {"name": fields.Str()}
BODY = b'{"name": "Steve"}'
HEADERS = {"Content-Type": "application/json; charset=not-a-charset"}


def pyramid_view(request):
    return pyr_parser.parse(ARGS, request, location="json")


with Configurator() as config:
    config.add_route("echo", "/echo")
    config.add_view(pyramid_view, route_name="echo", renderer="json")
    pyr_app = webtest.TestApp(config.make_wsgi_app())


async def aio_view(request):
    return web.json_response(await aio_parser.async_parse(ARGS, request, location="json"))


aio = web.Application()
aio.router.add_post("/echo", aio_view)
aio_app = webtest_aiohttp.TestApp(aio, loop=asyncio.new_event_loop())

for name, app in [("pyramid", pyr_app), ("aiohttp", aio_app)]:
    try:
        res = app.post("/echo", BODY, headers=HEADERS, expect_errors=True)
        print(f"{name}: {res.status} {res.body!r}")
    except Exception as exc:
        print(f"{name}: UNCAUGHT {type(exc).__name__}: {exc}")

Actual

pyramid: UNCAUGHT LookupError: unknown encoding: not-a-charset
aiohttp: 500 Internal Server Error b'500 Internal Server Error\n\nServer got itself in trouble'

Expected

pyramid: 400 Bad Request b'{"json": ["Invalid JSON body."]}'
aiohttp: 400 Bad Request b'{"json": ["Invalid JSON body."]}'

That is, the same handling as other undecodable JSON bodies (see #427).

Activity

  1. sirosen commented on Nov 16, 2026

    @sirosen
    Collaborator

    Although the submitted PR (#1121) was not acceptable, I think this might be a real issue. I am at least willing to entertain the idea.
    It can be handled today by users if they subclass the parser and customize it a bit, so it doesn't strike me as severe.

  2. hemtilva commented on Nov 16, 2026

    @hemtilva
    Author

    Thanks for keeping this open, and sorry about #1121. You're right, catching LookupError around the whole json() call could hide unrelated KeyErrors too. That was careless.

    I see two narrower options. One is to check just the charset up front with codecs.lookup(req.charset) and return the normal 400 only if that fails. The other is to ignore an unknown charset and decode as UTF-8, which is what the Flask/Django/Bottle/Falcon parsers effectively do already. The second seems more consistent to me, but I'm happy either way. If you think it's worth fixing, I'd write it by hand with a test for each parser. If not, no problem.

  3. sirosen commented on Nov 16, 2026

    @sirosen
    Collaborator

    Are you personally using pyramid or aiohttp with webargs? I'm disinclined to rush to make a change if no impacted users are asking for it.

    Frankly, you are on very thin ice here. You threw a clearly LLM-generated issue at us -- it's about 6 times longer than it needs to be to adequately describe the issue -- and a generated PR that wasted someone else's time. Based on your most recent reply, I'm not fully confident you understand why your last PR was unacceptable, both from a policy and a quality perspective.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions