Hi gRPC team,
We are looking for clarification on how gRPC-Java manages memory (Direct Buffers vs. JVM Heap) for incoming streaming messages while they wait in the server executor queue.
Service Profile:
Type: Long-lived Bidirectional / Streaming RPCs
Active Streams: ~15,000 concurrent streams
Transport: ~64 HTTP/2 connections from Envoy to each pod
gRPC Threads: 24
Message Size: Raw wire message is ~500 KB; deserialized Java object is ~5 MB
Memory Budget: 4 GB Heap (-Xmx4096m), ~3 GB Direct/Off-heap memory
Flow Control: Netty BDP tuning enabled (8 MB max window per connection)
Questions:
-
Memory location while waiting in Executor Queue: Once Netty deframes a message and submits the task to the gRPC server executor, but before a worker thread picks it up and enters onMessage():
1a. Does this queued message reside off-heap in Netty Direct Buffers (raw bytes), or is it already deserialized into the custom Protobuf object on the JVM Heap?
1b. Assuming 1a is true => At what exact point are the underlying Direct Buffers released back to Netty's allocator,
-
Burst Protection & Direct Memory Sizing (Assuming Off-Heap): Assuming the queued messages reside in Direct Buffers (off-heap, assuming 1a is true) from the above: if there is a sudden surge where ~8,000 streams connect simultaneously and each sends a 500 KB first message, in-flight raw data could reach ~4 GB (8,000 × 500 KB), exceeding our 3 GB Direct Memory limit and risking an off-heap OOM.
2a. Does HTTP/2 connection/stream flow control automatically prevent Direct Memory from growing beyond our 3 GB limit, or could this cause an off-heap OOM?
2b. To protect against this surge, is using a bounded queue on the gRPC executor recommended, or do you recommend any other optimized way for this?
Thanks,
Aditya
Hi gRPC team,
We are looking for clarification on how gRPC-Java manages memory (Direct Buffers vs. JVM Heap) for incoming streaming messages while they wait in the server executor queue.
Service Profile:
Type: Long-lived Bidirectional / Streaming RPCs
Active Streams: ~15,000 concurrent streams
Transport: ~64 HTTP/2 connections from Envoy to each pod
gRPC Threads: 24
Message Size: Raw wire message is ~500 KB; deserialized Java object is ~5 MB
Memory Budget: 4 GB Heap (-Xmx4096m), ~3 GB Direct/Off-heap memory
Flow Control: Netty BDP tuning enabled (8 MB max window per connection)
Questions:
Memory location while waiting in Executor Queue: Once Netty deframes a message and submits the task to the gRPC server executor, but before a worker thread picks it up and enters onMessage():
1a. Does this queued message reside off-heap in Netty Direct Buffers (raw bytes), or is it already deserialized into the custom Protobuf object on the JVM Heap?
1b. Assuming 1a is true => At what exact point are the underlying Direct Buffers released back to Netty's allocator,
Burst Protection & Direct Memory Sizing (Assuming Off-Heap): Assuming the queued messages reside in Direct Buffers (off-heap, assuming 1a is true) from the above: if there is a sudden surge where ~8,000 streams connect simultaneously and each sends a 500 KB first message, in-flight raw data could reach ~4 GB (8,000 × 500 KB), exceeding our 3 GB Direct Memory limit and risking an off-heap OOM.
2a. Does HTTP/2 connection/stream flow control automatically prevent Direct Memory from growing beyond our 3 GB limit, or could this cause an off-heap OOM?
2b. To protect against this surge, is using a bounded queue on the gRPC executor recommended, or do you recommend any other optimized way for this?
Thanks,
Aditya