Skip to content

fix: an unhandled exception over gRPC is masked instead of sent as UNKNOWN plus its repr - #52

Merged
AlexeyShalaev merged 2 commits into
masterfrom
fix/grpc-unhandled-exception-masking
Sep 7, 2026
Merged

fix: an unhandled exception over gRPC is masked instead of sent as UNKNOWN plus its repr#52
AlexeyShalaev merged 2 commits into
masterfrom
fix/grpc-unhandled-exception-masking

Conversation

@AlexeyShalaev

Copy link
Copy Markdown
Member

Summary

An exception that is not a ServiceError reached a gRPC caller with its own message. Over HTTP the
same exception is a masked 500; over gRPC it fell through to grpc.aio, whose answer for an
exception it was never told about is UNKNOWN with repr() of it. The row from the issue's lab,
before and after, same call over both transports:

    case         HTTP                                gRPC (before)
    unexpected   500 code=internal_error, no detail  UNKNOWN: Unexpected <class 'RuntimeError'>: dividing by the number of retries, which is zero

    case         HTTP                                gRPC (after)
    unexpected   500 code=internal_error, no detail  INTERNAL: internal_error  [x-error-code=internal_error]

The unexpected row is now identical to the ledger row, which is a ServiceError with
public=False. That is the property worth having: a caller cannot tell an error the service hid
from an error it never knew about, and neither carries a sentence nobody wrote. In the service the
lab is modelled on, the exception text held a DSN with a password.

The design

UnhandledErrorInterceptor is the gRPC counterpart of the HTTP stack's UnhandledErrorMiddleware,
and it sits at the same depth of the chain:

UnitScopeInterceptor            ← correlation ids are bound here
  UnhandledErrorInterceptor     ← new: inside the scope, so the log is correlated
    metrics interceptor         ← unchanged: it already classified these as INTERNAL
      your static interceptors
      your interceptors_factory results
        ServiceErrorInterceptor ← unchanged, innermost
          your servicer

It aborts with INTERNAL, detail internal_error and x-error-code: internal_error — byte for
byte what a public=False ServiceError already produced — and logs the exception with its
traceback. grpc.aio.AbortError and grpc.RpcError are re-raised, because a handler that aborted
already chose its status, and asyncio.CancelledError is a BaseException, so a caller walking
away never reaches it.

Installed unconditionally, with no new keyword argument, exactly as UnhandledErrorMiddleware is
over HTTP: not leaking is not a preference. map_service_errors=False keeps meaning what it meant —
it turns off the mapping of the errors you declared — with the consequence, now documented, that an
unmapped ServiceError arrives as a masked INTERNAL rather than as its own message.

What I rejected

Folding the catch-all into ServiceErrorInterceptor. Shortest diff, wrong position.
That interceptor is innermost by design, so it would take every exception before the caller's own
interceptors could map theirs. Starlette dispatches handlers by type, so on HTTP a handler for your
exception type always wins over the Exception catch-all; a separate outer layer is what reproduces
that ordering here. It also covers an exception raised by an interceptor rather than by a servicer.

Installing grpc-server-kit's AsyncExceptionHandlerInterceptor by default. It maps unknown
exceptions to INTERNAL, but it also maps ValueError to INVALID_ARGUMENT, TimeoutError to
DEADLINE_EXCEEDED and FileNotFoundError to NOT_FOUND, where the HTTP side masks all three to
500. That would close the leak and re-open the disagreement somewhere else, and it sends no
x-error-code. Because it composes inside the new layer, using it deliberately still works
exactly as before.

ErrorKind.CONFLICT stays on ALREADY_EXISTS

The issue asked whether it should. It should. ABORTED also renders as 409, but it is the
transaction-conflict code and the gRPC contract tells clients to retry it at a higher level — wrong
advice for "this order was already paid", which will never succeed on a retry.
FAILED_PRECONDITION is spoken for by PRECONDITION_FAILED, and keeping the table injective is
what lets a gateway translate in both directions. A client that must tell a duplicate key from a
state conflict branches on x-error-code, which carries the exact code either way. Changing it
would be a wire change that buys nothing; the reasoning is now written next to the table instead of
nowhere.

Tests

  • tests/unit/test_error_parity.py — new. One property over both transports: a public=False
    ServiceError and an undeclared RuntimeError each produce kind INTERNAL, code
    internal_error and no trace of the secret, over HTTP and over gRPC; a public NOT_FOUND still
    arrives intact on both. Four of its six cases pass on master; the fifth is the never-declared
    gRPC one, which fails.
  • tests/unit/test_grpc.py — the interceptor in isolation: the masked abort, the traceback and the
    method name on the log record, an unmapped ServiceError, AbortError / RpcError /
    CancelledError passing through, and its position in the chain both with and without
    map_service_errors.
  • tests/integration/test_grpc_integration.py — the reporter's call over a real socket: a servicer
    raising RuntimeError, a real client, INTERNAL and internal_error on the wire.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring / internal

A caller that was branching on UNKNOWN to detect a server crash now sees INTERNAL, which is the
correction being made. Nothing else on the wire changes.

Checklist

  • Tests added or updated
  • make check passes locally (ruff + mypy)
  • CHANGELOG.md left alone — Release Please generates it from the Conventional Commit subjects (and from the PR title on a squash merge)
  • Documentation updated (if the public API changed), docs/agents.md included

Related issues

Closes #50

Alex Shalaev added 2 commits September 7, 2026 15:17
…KNOWN plus its repr

ServiceErrorInterceptor claimed ServiceError and everything else fell through to
grpc.aio, whose answer for an exception it was never told about is UNKNOWN with
repr() of it. The HTTP side masks the same exception to a 500 with
code=internal_error and keeps the message in the log, so the promise of the
shared error model held for the errors a service declared and broke for the ones
it did not — and those are the ones whose wording nobody reviewed. In the
service the report came from, that text carried a DSN with a password.

UnhandledErrorInterceptor is the gRPC counterpart of the HTTP stack's
UnhandledErrorMiddleware and sits at the same depth: inside UnitScopeInterceptor
so the traceback is logged with the RPC's correlation ids, and outside the
metrics interceptor and the caller's own, so an exception type you map yourself
still reaches your interceptor first. It aborts with INTERNAL, detail
internal_error and x-error-code: internal_error, which is byte for byte what a
public=False ServiceError already produced, and re-raises AbortError and
RpcError — a status somebody chose deliberately. CancelledError is a
BaseException and never reaches it.

It is installed unconditionally, exactly as its HTTP counterpart is.
map_service_errors=False still turns off only the mapping of declared errors;
an unmapped ServiceError then arrives as a masked INTERNAL rather than as its
own message.

ErrorKind.CONFLICT keeps mapping to ALREADY_EXISTS. ABORTED also renders as 409
but tells clients to retry at a higher level, which a state conflict will not
survive, and FAILED_PRECONDITION is spoken for by PRECONDITION_FAILED; the
reason now sits next to the table instead of nowhere.

Refs #50
…nsports

The errors page said public=False masks at every transport and left the other
half unsaid, so the pages a reader lands on described the gap as if it were the
design. They now carry the masked-by-default table for an undeclared exception,
the note that neither default_exception_handlers=False nor
map_service_errors=False removes that layer, and the reason CONFLICT maps to
ALREADY_EXISTS rather than to ABORTED.

The gRPC adapter page gains the section on anything that is not a ServiceError
and the interceptor chain grows its new row, as do the blueprint's two
renderings of it. The runbook entry for a domain error becoming INTERNAL says
what map_service_errors=False now leaves in place, the checklist gains the line
about catching Exception and returning its str(), and agents.md carries the new
name, the chain and a widened rule 16.

Refs #50
@AlexeyShalaev
AlexeyShalaev merged commit 85218f1 into master Sep 7, 2026
6 checks passed
@AlexeyShalaev
AlexeyShalaev deleted the fix/grpc-unhandled-exception-masking branch September 7, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An unexpected exception reaches a gRPC caller with its message; the HTTP side masks the same error

1 participant