flow-controlled client streams can be leaked when the server half-closes

[benjaminp]

When the server half-closes, NettyClientStream attempts to cancel the stream if the client has not half-closed:

grpc-java/netty/src/main/java/io/grpc/netty/NettyClientStream.java

Lines 344 to 347 in 4d0d548

	if (endOfStream) {
	if (!isOutboundClosed()) {
	handler.getWriteQueue().enqueue(new CancelClientStreamCommand(this, null), true);
	}

However, AbstractClientStream.isOutboundClosed() reflects whether the application has called AbstractClientStream.halfClose() not whether the end-of-stream has actually been sent. The end-of-stream flag may never be sent if the control flow window is exhausted. This can prevent streams that would normally have terminated from ever dying.

A standalone reproduction of this bug is available at https://github.com/benjaminp/grpc_stream_leak_bug. The reproducer uses a client-streaming RPC. The client writes a large amount of data in a series of messages while the server implementation sleeps to allow the client to fill the control window and then replies with its unary response. Because of the bug, each call leaves the stream open. After making several calls, the client exhausts MAX_CONCURRENT_STREAMS and can no longer make calls.

$ ./gradlew run
> Task :run
client: launching server: leak-demo-client/bin/leak-demo-server
client: server pid=254266 port=38687
client: creating channel
client: call 0: sending 1064960 bytes
client: call 0: OK, server read 16384
client: call 1: sending 1064960 bytes
client: call 1: OK, server read 16384
client: call 2: sending 1064960 bytes
client: call 2: OK, server read 16384
client: call 3: sending 1064960 bytes
client: call 3: failed: Status{code=DEADLINE_EXCEEDED, description=CallOptions deadline exceeded after 4.999898215s. Name resolution delay 0.000000000 seconds. [closed=[], committed=[remote_addr=/127.0.0.1:38687]], cause=null}
client: call 4: sending 1064960 bytes
client: call 4: failed: Status{code=DEADLINE_EXCEEDED, description=CallOptions deadline exceeded after 4.999914485s. Name resolution delay 0.000000000 seconds. [closed=[], committed=[remote_addr=/127.0.0.1:38687]], cause=null}
client: call 5: sending 1064960 bytes
client: call 5: failed: Status{code=DEADLINE_EXCEEDED, description=CallOptions deadline exceeded after 4.999885146s. Name resolution delay 0.000000000 seconds. [closed=[], committed=[remote_addr=/127.0.0.1:38687]], cause=null}
BUG: 3 of 6 calls failed even though every earlier call completed OK; the completed calls' HTTP/2 streams were never closed and exhausted the server's MAX_CONCURRENT_STREAMS=3