CASSANDRA-21191: [CEP-59] Implementation of In-Band Connection Draining (Graceful Disconnect) by AshenScribe · Pull Request #4953 · apache/cassandra

AshenScribe · 2026-07-22T03:37:16Z

AshenScribe

StorageService.drain() was synchronized on the same monitor as drain(boolean). Pre-CEP-59, drain() called drain(false) directly on the same thread, so reentrancy made this safe. Now drain() hands a callback to gracefulDisconnect(...), which invokes it asynchronously (Netty close-listener or scheduler thread) — so drain(false) can run on a different thread than the one blocked in drain()'s await(). Since that thread still holds the monitor while waiting, drain(false) can never acquire it → deadlock whenever a client is still connected at drain time.

Fix: drop synchronized from drain(), use a dedicated ReentrantLock scoped only to drain() (preserves "one drain at a time"), leave drain(boolean)'s own synchronized untouched.

Copilot

Pull request overview

Implements CEP-59 graceful native-protocol disconnection for controlled Cassandra shutdowns.

Changes:

Adds capability advertisement, event registration, and connection draining.
Integrates draining with shutdown operations and metrics.
Adds configuration, documentation, and tests.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`test/unit/org/apache/cassandra/service/StorageServiceTest.java`	Tests graceful-disconnect settings.
`test/unit/org/apache/cassandra/config/DatabaseDescriptorTest.java`	Tests configuration defaults and updates.
`test/unit/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutorTest.java`	Propagates the new drain exception.
`test/distributed/org/apache/cassandra/distributed/test/GracefulDisconnectTest.java`	Adds distributed feature tests.
`src/java/org/apache/cassandra/transport/SimpleClient.java`	Handles graceful-disconnect events.
`src/java/org/apache/cassandra/transport/Server.java`	Tracks subscribers and stops accepting connections.
`src/java/org/apache/cassandra/transport/messages/StartupMessage.java`	Defines the capability key.
`src/java/org/apache/cassandra/transport/messages/OptionsMessage.java`	Advertises feature support.
`src/java/org/apache/cassandra/transport/InitialConnectionHandler.java`	Advertises support before negotiation.
`src/java/org/apache/cassandra/transport/Event.java`	Defines the protocol event.
`src/java/org/apache/cassandra/tools/nodetool/Drain.java`	Handles drain timeout errors.
`src/java/org/apache/cassandra/tools/NodeProbe.java`	Propagates drain timeouts.
`src/java/org/apache/cassandra/service/StorageServiceMBean.java`	Exposes settings and timeout contract.
`src/java/org/apache/cassandra/service/StorageService.java`	Orchestrates graceful shutdown.
`src/java/org/apache/cassandra/service/NativeTransportService.java`	Exposes subscribed channels.
`src/java/org/apache/cassandra/metrics/ClientMetrics.java`	Adds draining metrics.
`src/java/org/apache/cassandra/config/DatabaseDescriptor.java`	Exposes configuration accessors.
`src/java/org/apache/cassandra/config/Config.java`	Defines feature configuration.
`doc/modules/cassandra/pages/managing/operating/graceful_disconnect.adoc`	Documents operation and compatibility.
`conf/cassandra.yaml`	Documents stable configuration defaults.
`conf/cassandra_latest.yaml`	Enables the feature in latest configuration.

Suppressed comments (1)

src/java/org/apache/cassandra/transport/SimpleClient.java:85

Removing this suppression makes Netty Promise fail the IllegalImport check in .build/checkstyle.xml:104. Restore the established import-level suppression.

import io.netty.util.concurrent.Promise;

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

 import java.util.UUID;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.CountDownLatch;


 import java.util.concurrent.BlockingQueue;
 import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.SynchronousQueue; // checkstyle: permit this import
+import java.util.concurrent.SynchronousQueue;


SiyaoIsHiding · 2026-08-14T01:04:24Z

                    supportedOptions.put(StartupMessage.CQL_VERSION, cqlVersions);
                    supportedOptions.put(StartupMessage.COMPRESSION, compressions);
                    supportedOptions.put(StartupMessage.PROTOCOL_VERSIONS, ProtocolVersion.supportedVersions());
+                    supportedOptions.put(StartupMessage.GRACEFUL_DISCONNECT, List.of(String.valueOf(DatabaseDescriptor.getGracefulDisconnectEnabled())));


+            if (!drainComplete.await(DatabaseDescriptor.getGracefulDisconnectGracePeriod(), MILLISECONDS))
+                throw new TimeoutException("Timed out waiting for drain to complete after graceful disconnect");


+                if (connectedChannels.decrementAndGet() == 0)
+                {
+                    timeoutTask.cancel(false);
+                    runOnceAction.run();


+    public boolean graceful_disconnect_enabled = false;
+
+    public volatile DurationSpec.LongMillisecondsBound graceful_disconnect_grace_period = new DurationSpec.LongMillisecondsBound(5000);


+    @Test
+    public void testGracefulDisconnectEnabled()
+    {
+        Assertions.assertThat(StorageService.instance.getGracefulDisconnectEnabled()).isFalse();


+        AtomicBoolean actionStarted = new AtomicBoolean(false);
+        AtomicInteger connectedChannels = new AtomicInteger(channelGroup.size());


+        if (bindChannel != null && bindChannel.isOpen())
+        {
+            logger.info("Stopping native transport acceptor on {}", bindChannel.localAddress());
+            // syncUninterruptibly ensures we wait for the port to actually close
+            bindChannel.close().syncUninterruptibly();


+
+== Solution
+
+Introduce an in-band signal — `GRACEFUL_DISCONNECT` — so that the server can notify clients (that have subscribed) connection before closing it. This gives drivers time to:


SiyaoIsHiding

Preliminary review, yet to dig into the core mechanism.
The most convincing test is to put together a client-server scenario, where it throws an error when graceful disconnect is disabled, and does not throw an error when graceful disconnect is enabled. This is the whole purpose of CEP-59.
But the existing tests are only about configs, whether server sends event or start draining, etc. We need the test to prove the error is gone.
If the existing integration test harness is not enough to put together such scenario, then a manual testing is needed.

SiyaoIsHiding · 2026-08-14T00:45:18Z

            connection.setCompressor(Compressor.LZ4Compressor.instance);
        }
+        if (version.isGreaterOrEqualTo(ProtocolVersion.V5))
+            options.put(StartupMessage.GRACEFUL_DISCONNECT, "GRACEFUL_DISCONNECT");


graceful disconnect is opted in by REGISTER, not in startup message

SiyaoIsHiding · 2026-08-14T01:04:24Z

                    supportedOptions.put(StartupMessage.CQL_VERSION, cqlVersions);
                    supportedOptions.put(StartupMessage.COMPRESSION, compressions);
                    supportedOptions.put(StartupMessage.PROTOCOL_VERSIONS, ProtocolVersion.supportedVersions());
+                    supportedOptions.put(StartupMessage.GRACEFUL_DISCONNECT, List.of(String.valueOf(DatabaseDescriptor.getGracefulDisconnectEnabled())));


SiyaoIsHiding · 2026-08-14T01:26:30Z

+                }
+            });
+            if (!drainComplete.await(DatabaseDescriptor.getGracefulDisconnectGracePeriod(), MILLISECONDS))
+                throw new TimeoutException("Timed out waiting for drain to complete after graceful disconnect");


Why does drain throws a TimeoutException when graceful period passes? it should at most log a warning and proceed IMO

SiyaoIsHiding · 2026-08-14T01:37:26Z

    @VisibleForTesting
    Gauge<Integer> connectedNativeClients;

+    public AtomicInteger connectionsDraining;


We need tests for metrics too

AshenScribe · 2026-08-15T05:34:39Z

@SiyaoIsHiding I need a java driver supporting cep 59. I am unable to get one, so further testing can't be done by me.

SiyaoIsHiding · 2026-08-19T03:00:11Z

@AshenScribe I sent you a working java driver and a working native protocol on Feb 25, I also hopped on an half hour meeting with you around the same time when I showed you how to do necessary end-to-end testing.

AshenScribe · 2026-08-19T03:08:53Z

Hi @SiyaoIsHiding yes I remember that meeting. I'll do so.

As of drivers, since shanzita took over the java driver, I was waiting for her branch of native protocol, as that will be the official code to be merged now, which I received just yesterday.

SiyaoIsHiding · 2026-08-19T04:47:37Z

She didn't have a full server side implementation to test against until last month, either. You two needs each other to do end-to-end testing, so you both will need to test against WIP code, and you should expect changes through out the PR review process.

Shanzita · 2026-08-19T18:34:55Z

 import java.util.UUID;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CopyOnWriteArrayList;
+import org.apache.cassandra.utils.concurrent.CountDownLatch


I think you missed a semicolon at the end of this import.

yup, I also skipped the import style check skip patching it now.

Also wait some moment before reviewing the StorageService class, I am going to change the approach there.

# Conflicts: # src/java/org/apache/cassandra/service/StorageService.java

AshenScribe · 2026-08-20T13:06:13Z

added new logic for draining.
Todo

Write comprehensive test cases for GracefulDisconnectLifecycle, it contains basic ones only
Write comprehensive test cases for GracefulDisconnect feature, it contains basic ones only
Write comprehensive test cases for new metrics
Do manual testing using new java driver and native-protocol and this PR. Current test include draining with graceful_disconnect_enabled set to true and false and used cqlsh to test legacy behaviour.

SiyaoIsHiding · 2026-08-21T07:38:14Z

Here are a list of different ways for a node to disconnect and whether graceful disconnect should kick in.

Way to Disconnect	Graceful disconnect?
nodetool disablebinary	yes
nodetool disablebinary -f ("Use -f to interrupt client requests that have already started")	no
nodetool drain	yes
nodetool stopdaemon	yes
nodetool decommission	yes
nodetool decommission -f	yes
JVM shutdown hook/ SIGTERM/ systemctl stop	yes
ccm node1 stop (SIGTERM)	yes
SIGKILL	no

If everyone agree with the above behavior, this table should go into the CEP and the operating manual.

Assuming we want the above behavior, I'd suggest we add graceful disconnect to NativeTransportService.stop(bool force) when force == false, because all ways above that should gracefully disconnect calls NativeTransportService.stop(false).

Way to Disconnect	Callstack to graceful disconnect
nodetool disablebinary	StorageService.stopNativeTransport(bool force) -> CassandraDaemon.stopNativeTransport(bool force) -> NativeTransportService.stop(bool force)
nodetool disablebinary -f	Same as above, and it calls NativeTransportService.stop(true), so it should skip the graceful disconnect
nodetool drain	StorageService.drain(bool) -> StorageService.shutdownClientServers() -> StorageService.stopNativeTransport(false)
nodetool stopdaemon	CassandraDaemon.stop() -> CassandraDaemon.destroyClientTransports() -> NativeTransportService.stop()
nodetool decommission	StorageService.decommission() -> SingleNodeSequences.decommission(true, force) -> StorageService.shutdownNetowrking() -> StorageService.shutdownClientServers()
nodetool decommission -f	Exactly same above
JVM shutdown hook/ SIGTERM/ systemctl stop	Runtime.getRuntime().addShutdownHook(drainOnShutdown); -> StorageService.drain(true) -> shutdownClientServers()
SIGKILL	not handled

AshenScribe commented Jul 22, 2026

View reviewed changes

SiyaoIsHiding requested a balanced review from Copilot August 14, 2026 00:07

Copilot started reviewing on behalf of SiyaoIsHiding August 14, 2026 00:07 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

SiyaoIsHiding reviewed Aug 14, 2026

View reviewed changes

SiyaoIsHiding changed the title ~~cep 59~~ CASSANDRA-21191: [CEP-59] Implementation of In-Band Connection Draining (Graceful Disconnect) Aug 14, 2026

Shanzita reviewed Aug 19, 2026

View reviewed changes

AshenScribe added 3 commits August 20, 2026 00:08

cep 59

e75ba3c

cep 59

bcefd0d

# Conflicts: # src/java/org/apache/cassandra/service/StorageService.java

creating a new FSM based model

c0641da

AshenScribe force-pushed the feature/cep-59/metrics branch from bdeaf90 to c0641da Compare August 20, 2026 12:59

		if (!drainComplete.await(DatabaseDescriptor.getGracefulDisconnectGracePeriod(), MILLISECONDS))
		throw new TimeoutException("Timed out waiting for drain to complete after graceful disconnect");

		public boolean graceful_disconnect_enabled = false;

		public volatile DurationSpec.LongMillisecondsBound graceful_disconnect_grace_period = new DurationSpec.LongMillisecondsBound(5000);

		AtomicBoolean actionStarted = new AtomicBoolean(false);
		AtomicInteger connectedChannels = new AtomicInteger(channelGroup.size());


		== Solution

		Introduce an in-band signal — `GRACEFUL_DISCONNECT` — so that the server can notify clients (that have subscribed) connection before closing it. This gives drivers time to:

Uh oh!

Conversation

AshenScribe commented Jul 22, 2026

Uh oh!

AshenScribe left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SiyaoIsHiding left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshenScribe commented Aug 15, 2026

Uh oh!

SiyaoIsHiding commented Aug 19, 2026

Uh oh!

AshenScribe commented Aug 19, 2026

Uh oh!

SiyaoIsHiding commented Aug 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AshenScribe commented Aug 20, 2026

Uh oh!

SiyaoIsHiding commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants