Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 40 additions & 6 deletions docs/adapters/grpc.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,18 +90,43 @@ real code only reaches your logs. The full mapping table is in

Turn it off with `map_service_errors=False`.

## Anything that is not a `ServiceError`

`UnhandledErrorInterceptor` is the last resort, and it is always installed:

```python
raise RuntimeError("dividing by the number of retries, which is zero")
# → the RPC aborts with INTERNAL
# → details "internal_error", trailing metadata "x-error-code: internal_error"
# → the exception and its traceback go to the log, with the RPC's correlation ids
```

Byte for byte what a `public=False` `ServiceError` produces, which is the point: the two are
indistinguishable to a caller. Without it `grpc.aio` answers `UNKNOWN` with `repr()` of the
exception, so the one error class whose wording nobody reviewed is the one that reaches the wire
verbatim.

Three things are left alone, because each already carries a decision:
`grpc.aio.AbortError` and `grpc.RpcError` (a status a handler chose itself) are re-raised, and
`asyncio.CancelledError` never reaches it — a caller that walked away is not a failure to report.

`map_service_errors=False` does not remove it. That flag turns off the mapping of the errors you
declared; with it off a `ServiceError` reaches the client as a masked `INTERNAL` like any other
unhandled exception, never as its own message.

## Interceptor ordering

`grpc.aio` hands control in list order, so the first entry is the **outermost** wrapper. The
entrypoint assembles the chain like this:

```
UnitScopeInterceptor ← outermost: everything below sees a live scope
metrics interceptor ← records the status the client actually receives
your static interceptors
your interceptors_factory results
ServiceErrorInterceptor ← innermost: closest to the servicer
your servicer
UnitScopeInterceptor ← outermost: everything below sees a live scope
UnhandledErrorInterceptor ← the net: inside the scope, so the log is correlated
metrics interceptor ← records the status the client actually receives
your static interceptors
your interceptors_factory results
ServiceErrorInterceptor ← innermost: closest to the servicer
your servicer
```

!!! warning "Why `ServiceErrorInterceptor` is innermost"
Expand All @@ -113,6 +138,15 @@ UnitScopeInterceptor ← outermost: everything below sees a live scope

It still sits *inside* the metrics interceptor, so aborts are recorded with their real status.

!!! note "Why `UnhandledErrorInterceptor` is not"

It is the mirror of the HTTP stack's `UnhandledErrorMiddleware`, and it sits at the same
depth: inside the layer that binds the correlation ids, so the masked abort is logged with
the request id, and outside everything else, so it only ever sees what nobody below it
claimed. An interceptor of yours that maps an exception type of its own still gets it first —
including grpc-server-kit's `AsyncExceptionHandlerInterceptor`, if that is the mapping you
want. Compose exactly as before.

```python
GrpcEntrypoint(
config=config,
Expand Down
21 changes: 17 additions & 4 deletions docs/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,12 +251,17 @@ once per run inside the application scope and appended to `spec.warmers`.
| `ErrorKind` | `INVALID`, `UNAUTHENTICATED`, `FORBIDDEN`, `NOT_FOUND`, `CONFLICT`, `PRECONDITION_FAILED`, `TOO_MANY_REQUESTS`, `DEADLINE_EXCEEDED`, `UNAVAILABLE`, `NOT_IMPLEMENTED`, `INTERNAL` | 400/401/403/404/409/412/429/504/503/501/500 over HTTP |
| `ErrorInfo` | `kind`, `code`, `detail=None`, `params={}`, `public=True`, `status_override=None`, `headers=None`; `.http_status`; `ErrorInfo.from_service_error(exc)` | the normalized view every renderer takes |
| `mask_private_error` | `mask_private_error(info) -> ErrorInfo` | `public=False` collapses to a generic `internal_error` 500 |
| `INTERNAL_ERROR_CODE` | `"internal_error"` | the code every masked error renders as, on both transports |
| `RenderedError` | `status_code`, `body`, `media_type="application/problem+json"`, `headers=None` | |
| `ProblemDetailsRenderer` | `ProblemDetailsRenderer(*, type_base=None)` | RFC 9457; the default renderer |

`HTTP_STATUS_BY_KIND`, `status_title` and `to_json_safe` live in
`HTTP_STATUS_BY_KIND`, `INTERNAL_ERROR_CODE`, `status_title` and `to_json_safe` live in
`servicewright.core.errors` and are not re-exported at the top level.

`ErrorKind.CONFLICT` maps to `ALREADY_EXISTS` and not to `ABORTED`: both mean 409, but `ABORTED`
tells the client to retry at a higher level, which a state conflict will not survive. Both status
tables are injective, so a status maps back to exactly one kind.

### Request context

| Name | Signature | Notes |
Expand Down Expand Up @@ -308,7 +313,7 @@ installed and raise (or degrade) at construction instead.
|---|---|
| `servicewright.adapters.fastapi` | `FastApiEntrypoint`, `FastApiPlugin`, `HttpConfig`, `MiddlewareConfig`, `HealthConfig`, `CORSMiddlewareConfig`, `LoggingMiddlewareConfig`, `GZipMiddlewareConfig`, `CorrelationIdMiddlewareConfig`, `MetricsInstrumentatorConfig`, `UnitScopeDep`, `UnitScopeMiddleware`, `get_unit_scope`, `current_unit_scope`, `setup_default_exception_handlers`, `setup_metrics_instrumentator`, `LivenessResponse`, `ReadinessResponse`, `ProblemDetails`, `XUserId`, `IdempotencyKey`, `AuthorizationHeader`, `XFingerprintHeader`, `OtelBaggageSetter`, `StructlogSetter`, `get_default_context_setters`, `RoutesRegisterer`, `ConfigureApp` |
| `servicewright.adapters.litestar` | `LitestarEntrypoint`, `LitestarPlugin`, `LitestarConfig`, `HealthConfig`, `build_health_routes`, `UnitScopeMiddleware`, `get_unit_scope`, `current_unit_scope`, `RouteRegisterer`, `ConfigureApp` |
| `servicewright.adapters.grpc` | `GrpcEntrypoint`, `GrpcPlugin`, `GrpcConfig`, `ServicerRegisterer`, `InterceptorFactory`, `ServiceErrorInterceptor`, `GRPC_STATUS_BY_KIND`, `ERROR_CODE_TRAILING_METADATA`, `GrpcHealthBridge`, `UnitScopeInterceptor`, `current_unit_scope`, `GrpcServerMetricsRecorder`, `IDEMPOTENCY_KEY_METADATA`, `get_idempotency_key`, `get_client_ip`, `get_user_agent`, `get_client_context` |
| `servicewright.adapters.grpc` | `GrpcEntrypoint`, `GrpcPlugin`, `GrpcConfig`, `ServicerRegisterer`, `InterceptorFactory`, `ServiceErrorInterceptor`, `UnhandledErrorInterceptor`, `GRPC_STATUS_BY_KIND`, `ERROR_CODE_TRAILING_METADATA`, `GrpcHealthBridge`, `UnitScopeInterceptor`, `current_unit_scope`, `GrpcServerMetricsRecorder`, `IDEMPOTENCY_KEY_METADATA`, `get_idempotency_key`, `get_client_ip`, `get_user_agent`, `get_client_context` |
| `servicewright.adapters.apscheduler4` (and `.apscheduler3`) | `SchedulerEntrypoint`, `SchedulerPlugin`, `ScheduledJob`, `ScheduledJobFunc`, `SchedulerJobMetricsRecorder`, `SchedulerError`, `DuplicateScheduleError` |
| `servicewright.adapters.dishka` | `DishkaContainer`, `DishkaScope` |
| `servicewright.adapters.settings` | `BaseServiceSettings`, `LoggingSettings`, `MetricsSettings`, `TracingSettings`, `ErrorTrackingSettings` |
Expand All @@ -333,7 +338,9 @@ Entrypoint constructors, all keyword-only:
context_setters=None, map_service_errors=True, enable_metrics=False,
metrics_prefix=None, kind="grpc", essential=True)`. `GrpcConfig` defaults:
`port=50051`, `grace_period=30.0`, `enable_reflection=False`, `enable_channelz=False`,
`health_service_names=()`, `health_refresh_interval=5.0`.
`health_service_names=()`, `health_refresh_interval=5.0`. Interceptor chain, outermost first:
`UnitScopeInterceptor`, `UnhandledErrorInterceptor`, metrics, yours,
`ServiceErrorInterceptor`.
* `SchedulerEntrypoint(jobs, enable_metrics=False, metrics_prefix=None,
kind="scheduler", essential=True)`; `ScheduledJob(id, func, trigger, args=(), kwargs={},
max_instances=None, misfire_grace_time=None, coalesce=None)`.
Expand Down Expand Up @@ -420,7 +427,13 @@ Each `*Plugin` takes exactly the same arguments as its entrypoint and exposes `.
16. **`ServiceError(public=False)` masks everything at the transport**: the client gets a
generic `internal_error` 500 (or `INTERNAL` over gRPC) and the real code only reaches the
log. A subclass's `code` is derived from its class name, so renaming the class is a wire
change.
change. **An exception that is not a `ServiceError` is masked identically**, by
`UnhandledErrorMiddleware` over HTTP and `UnhandledErrorInterceptor` over gRPC — both
installed unconditionally, neither removed by `default_exception_handlers=False` or
`map_service_errors=False`, which only stop the mapping of the errors you declared. So a
caller cannot tell an error you hid from one you never knew about, and no exception text
reaches the wire. Do not catch-and-return an exception in a handler or servicer to "make the
error nicer": that is the one way to get its message back onto the wire.
17. **One unit scope per request, not two.** If the DI framework's own integration owns the
request scope (dishka's `setup_dishka`), switch servicewright's off —
`MiddlewareConfig(unit_scope=False)` or `LitestarConfig(unit_scope=False)` — otherwise
Expand Down
16 changes: 10 additions & 6 deletions docs/blueprints/grpc-service.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ pip install "servicewright[grpc,dishka,postgres,metrics,observability]"
flowchart LR
C["gRPC clients"] --> EP["GrpcEntrypoint<br/>:50051"]
EP --> I1["UnitScopeInterceptor"]
I1 --> I2["metrics"]
I1 --> I0["UnhandledErrorInterceptor"]
I0 --> I2["metrics"]
I2 --> I3["your interceptors"]
I3 --> I4["ServiceErrorInterceptor"]
I4 --> S["Servicers"]
Expand Down Expand Up @@ -97,15 +98,18 @@ Your interceptors land between the framework's:

```
UnitScopeInterceptor ← scope is live for everything below
metrics ← records the status the client receives
AuthInterceptor ← yours
LoggingInterceptor ← yours
ServiceErrorInterceptor
OrdersServicer
UnhandledErrorInterceptor ← nothing below leaves without a status
metrics ← records the status the client receives
AuthInterceptor ← yours
LoggingInterceptor ← yours
ServiceErrorInterceptor
OrdersServicer
```

That is why a generic "map everything to INTERNAL" interceptor of yours cannot swallow a
deliberate `NOT_FOUND`: the domain error has already become an abort by the time it reaches you.
And it is why anything your interceptors do *not* map still reaches the client as `INTERNAL` with
`x-error-code: internal_error` rather than as `UNKNOWN` plus the exception's own text.

## 4. main

Expand Down
52 changes: 52 additions & 0 deletions docs/concepts/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ servicewright solves that with three replaceable pieces:
```mermaid
flowchart LR
E["raise OrderNotFoundError"] --> N["ErrorInfo<br/>kind · code · detail · params · public"]
X["raise anything else"] --> N
N --> M["mask_private_error()"]
M --> R["HttpErrorRenderer"]
M --> K["ErrorKind → grpc.StatusCode"]
Expand Down Expand Up @@ -66,6 +67,17 @@ except ServiceError as exc:
| `DEADLINE_EXCEEDED` | 504 | `DEADLINE_EXCEEDED` |
| `INTERNAL` | 500 | `INTERNAL` |

Both columns are injective, so a status maps back to exactly one kind — which is what lets a
gateway translate between the two without guessing.

One row is worth stating out loud, because it has a plausible-looking alternative: `CONFLICT`
renders as `ALREADY_EXISTS` and not as `ABORTED`, although both mean 409 to a gRPC gateway.
`ABORTED` is the transaction-conflict code, and the gRPC contract tells clients to retry it at a
higher level — the wrong advice for "this order was already paid", which will never succeed on a
retry. `FAILED_PRECONDITION` is spoken for by `PRECONDITION_FAILED`. A client that has to tell a
duplicate key from a state conflict branches on `x-error-code`, which carries the exact code
either way, rather than on the status.

## What the client sees

Over HTTP, an RFC 9457 problem document with `Content-Type: application/problem+json`:
Expand Down Expand Up @@ -111,6 +123,34 @@ leak through a custom renderer by accident.
public = False
```

## Errors you never declared

An exception that is not a `ServiceError` at all — a `KeyError` off a dict, a `RuntimeError` from
a driver — is masked the same way, and by the same rule, on both transports:

| | HTTP | gRPC |
| --- | --- | --- |
| status | 500 | `INTERNAL` |
| machine code | `"code": "internal_error"` | `x-error-code: internal_error` |
| message | none | `internal_error` |
| the exception itself | logged with its traceback | logged with its traceback |

So a caller cannot tell an error you hid from an error you never knew about, and neither one
carries a sentence you did not write. That matters most for the second kind, because its wording
is whatever a library chose: an exception text is where a DSN with a password, a failing row or a
file path ends up.

Over HTTP this is `UnhandledErrorMiddleware` plus the `Exception` handler; over gRPC it is
`UnhandledErrorInterceptor`. Both are installed unconditionally, and neither one is what
`map_service_errors=False` or `default_exception_handlers=False` turns off — those switch off the
mapping of the errors you *did* declare, and an undeclared exception then reaches the client as a
masked internal error rather than as its own message.

!!! warning "The gRPC default without it"

`grpc.aio` answers an exception it was not told about with `UNKNOWN` and `repr()` of it —
status code and message both chosen by the transport, out of material nobody reviewed.

## Own the wire format

If your product has its own error envelope, implement one renderer and every default handler
Expand Down Expand Up @@ -173,6 +213,18 @@ from servicewright.core.errors import status_title, to_json_safe
| Deadline exceeded | 504, `code="deadline_exceeded"` |
| Anything unhandled | masked 500, logged with the request id — and the response carries that id too |

## The gRPC equivalents

| Situation | Result |
| --- | --- |
| `ServiceError` | its kind's status; masked when `public=False` |
| Anything unhandled | `INTERNAL`, `x-error-code: internal_error`, logged with the correlation ids |
| A handler that called `context.abort()` itself | left alone — the status was already chosen |
| The caller went away (`CancelledError`) | left alone — not an error to report |

The two interceptors doing that sit at different depths of the chain, and
[the adapter page](../adapters/grpc.md#interceptor-ordering) explains why.

Add handlers for your own exception types, or switch the defaults off entirely:

```python
Expand Down
3 changes: 3 additions & 0 deletions docs/operations/checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ to a failure someone has already had.
`to_json_safe`.
- [ ] A masked 500 still carries the request id (it does by default — do not move the
unhandled-error layer).
- [ ] Nothing catches a broad `Exception` in a handler or servicer and returns its `str()` to
the client. The unhandled-error layer masks what reaches it; a message you re-raise
yourself as a public error is yours to have read.

## Observability

Expand Down
4 changes: 4 additions & 0 deletions docs/operations/runbooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,10 @@ Check, in order:
`ServiceErrorInterceptor` is installed innermost precisely so your own interceptors cannot
intercept the domain error first — but code inside the servicer still can.

With the mapper off, `UnhandledErrorInterceptor` stays: an unmapped `ServiceError` then arrives as
a masked `INTERNAL` with `x-error-code: internal_error`, exactly like an exception nobody
declared. The log line next to it carries the real one.

### Validation errors have a different shape than the rest

They should not — every default handler renders through the same renderer. A 422 with a foreign
Expand Down
8 changes: 7 additions & 1 deletion servicewright/adapters/grpc/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,12 @@

from .config import GrpcConfig
from .entrypoint import GrpcEntrypoint, GrpcPlugin, InterceptorFactory, ServicerRegisterer
from .errors import ERROR_CODE_TRAILING_METADATA, GRPC_STATUS_BY_KIND, ServiceErrorInterceptor
from .errors import (
ERROR_CODE_TRAILING_METADATA,
GRPC_STATUS_BY_KIND,
ServiceErrorInterceptor,
UnhandledErrorInterceptor,
)
from .health import GrpcHealthBridge
from .interceptors import UnitScopeInterceptor, current_unit_scope
from .metadata import (
Expand All @@ -28,6 +33,7 @@
"InterceptorFactory",
"ServiceErrorInterceptor",
"ServicerRegisterer",
"UnhandledErrorInterceptor",
"UnitScopeInterceptor",
"current_unit_scope",
"get_client_context",
Expand Down
23 changes: 17 additions & 6 deletions servicewright/adapters/grpc/entrypoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
from ._imports import AsyncMetricsInterceptor, AsyncServer, bind_server_port, create_async_grpc_server, grpc
from .config import DEFAULT_HEALTH_SERVICE_NAME, GrpcConfig
from .context import get_default_context_setters
from .errors import ServiceErrorInterceptor
from .errors import ServiceErrorInterceptor, UnhandledErrorInterceptor
from .health import GrpcHealthBridge
from .interceptors import UnitScopeInterceptor
from .metrics import GrpcServerMetricsRecorder
Expand Down Expand Up @@ -56,10 +56,11 @@ class GrpcEntrypoint(ServerEntrypoint):
config: Self-contained server configuration (NOT read from settings).
servicers: Callback registering servicers on the gRPC server.
interceptors: Static interceptors. They wrap the servicer *inside* the
unit-scope and metrics interceptors but *outside* the service-error
mapper, so a generic exception handler of yours (e.g. the kit's
``AsyncExceptionHandlerInterceptor``) sees an already-mapped
``AbortError`` instead of swallowing the domain error.
unit-scope, unhandled-error and metrics interceptors but *outside*
the service-error mapper, so a generic exception handler of yours
(e.g. the kit's ``AsyncExceptionHandlerInterceptor``) sees an
already-mapped ``AbortError`` instead of swallowing the domain
error, and still gets first refusal on everything else.
interceptors_factory: Optional callback returning extra interceptors,
resolved at ``bind`` time with the :class:`ServiceContext`.
context_setters: Bridges that push the per-RPC context (request id, user
Expand All @@ -69,7 +70,10 @@ class GrpcEntrypoint(ServerEntrypoint):
map_service_errors: Convert raised
:class:`~servicewright.core.errors.ServiceError` into the mapped
``grpc.StatusCode`` abort (non-public errors masked). Default
``True``.
``True``. Turning it off does not uninstall
:class:`~servicewright.adapters.grpc.UnhandledErrorInterceptor`:
a ``ServiceError`` then reaches the client as a masked ``INTERNAL``
like any other unhandled exception, never as its own message.
enable_metrics: Add the RPC metrics interceptor, recording through the
app's configured metrics sink (``ObsConfig(metrics=...)`` + the
matching extra, e.g. servicewright[metrics] for prometheus).
Expand Down Expand Up @@ -201,6 +205,13 @@ async def _collect_interceptors(self, ctx: ServiceContext[Any, Any]) -> list[grp
setters = get_default_context_setters() if self._context_setters is None else self._context_setters
interceptors: list[grpc.aio.ServerInterceptor] = [UnitScopeInterceptor(ctx.container, context_setters=setters)]

# Inside the unit scope (so the masked abort is logged with the RPC's
# correlation ids) but outside everything else: nothing below may leave
# without a status, and grpc.aio's own answer for an exception it was
# not told about is UNKNOWN with repr() of it. Not optional, for the
# same reason UnhandledErrorMiddleware is not optional over HTTP.
interceptors.append(UnhandledErrorInterceptor())

if self._enable_metrics:
recorder = GrpcServerMetricsRecorder(ctx.observability.metrics, prefix=self._metrics_prefix)
interceptors.append(AsyncMetricsInterceptor(recorder, service_name=ctx.service_name))
Expand Down
Loading