feat: add TcpStackAwareSocketChannel for z/OS TCP/IP stack recovery — GH#4776

[balhar-jakub]

Closes #4776

Problem

The z/OS TCP/IP stack can restart (EDC5122I / NetworkRecycledException) while Tomcat is reading from or writing to an accepted client socket. Tomcat must discard the dead connection safely.

Fix

• Refactored isTcpStackRestarted and isRecycledClass from instance methods to package-private static methods on TomcatAcceptFixConfig
• Added the opt-in apiml.tcpStackAwareSocketChannel.enabled configuration property, defaulting to false so non-z/OS deployments are unaffected
• Created ExcludedSocketOps interface for Lombok @Delegate exclusion list
• Created TcpStackAwareSocketChannel inner class:
  • Intercepts read(ByteBuffer), read(ByteBuffer[], int, int), write(ByteBuffer), write(ByteBuffer[], int, int)
  • Detects EDC5122I/NetworkRecycledException, safely closes the dead socket, and re-throws so Tomcat discards the connection
  • Delegates wrapper close and blocking-mode behavior safely to the accepted socket
• Added unit coverage for read/write variants, socket closure, configuration disabled/enabled behavior, and z/OS restart detection

Configuration

This is a z/OS-only mitigation and is disabled by default. Enable it only for affected z/OS deployments after staging validation:

apiml:
  tcpStackAwareSocketChannel:
    enabled: true

Validation

• ./gradlew :apiml-tomcat-common:clean :apiml-tomcat-common:build — BUILD SUCCESSFUL (77 tests)
• ./gradlew clean && ./gradlew build — BUILD SUCCESSFUL

[balhar-jakub]

QA + Security Review — PR #4792 (#4776)

Verdict: APPROVED

Build and Tests

• ./gradlew :apiml-tomcat-common:clean build — BUILD SUCCESSFUL, 76/76 tests pass
• 16 new tests added covering all read/write/safeClose/config scenarios

Pavel's Lens — All 8 Rules Checked

Rule	Status	Notes
1. Config Consistency	OK	apiml.tcpStackAwareSocketChannel.enabled has Spring default :true. Follows existing undocumented pattern (same as server.tomcat.retryRebindTimeoutSecs).
2. Deduplication	OK	isTcpStackRestarted/isRecycledClass properly extracted to static methods. Old instance method delegates. Minor: NETWORK_RECYCLED_EXCEPTION_CLASS constant on line 176 is now dead code — not used after refactor. Suggest removing in follow-up.
3. Null Safety	OK	isTcpStackRestarted checks getMessage() != null, getCause() != null, cause != t (cycle guard).
4. Test Parametrization	OK	Tests are varied enough (read/write, single/scatter, pass/fail, config on/off, integration) to not collapse cleanly into parameterized. 16 distinct test cases with good edge coverage.
5. Security Boundaries	OK	No auth/TLS/CORS/input validation changes. This is z/OS socket infrastructure. Config toggle safely disables. No secrets, no data exposure. safeClose is best-effort with catch-and-ignore.
6. z/OS Awareness	EXCELLENT	Purpose-built for z/OS TCP/IP stack recovery. Handles EDC5122I and com.ibm.net.NetworkRecycledException. Graceful degradation via config toggle.
7. Log Quality	OK	All log.debug with exception context. Messages identify operation type (read/scatter read/write/scatter write). safeClose silently swallows close errors — correct for best-effort.
8. TODO Tracking	OK	No TODO/FIXME/HACK comments found.

Minor Notes (non-blocking)

1. NETWORK_RECYCLED_EXCEPTION_CLASS constant (line 176) is dead code after the static refactor — can be cleaned up in a follow-up.
2. Instance isTcpStackRestarted wrapper (lines 277-279) delegates to static version — may still be referenced by pre-existing tests calling channel.isTcpStackRestarted(). Acceptable.

Security Assessment

• No new auth/authz boundaries introduced
• No input validation changes
• No secrets or credentials in code
• No data exposure via logs (debug level, exception context only)
• MethodHandle usage for implCloseSelectableChannel/implConfigureBlocking follows existing pattern in FixedServerSocketChannel
• @Delegate with ExcludedSocketOps correctly excludes final methods and manually overridden methods

APPROVED — ready for merge.

[balhar-jakub]

DCO check failed — all 3 commits are missing Signed-off-by lines.

Fix: git rebase --signoff v3.x.x && git push --force-with-lease

[sonarqubecloud]

Quality Gate failed

Failed conditions
78.6% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

💡 Need a hand with PR review? Try Gitar by Sonar!

[balhar-jakub]

Multi-Reviewer Review

Verdict: Approve with minor NITs.

Findings:

• ℹ️ NIT — Typo in IMPL_CLOSE_SELECTABGLE_CHANNEL_HANLE (should be HANDLE) — pre-existing, but inherited.
• ℹ️ NIT — Method counts in ExcludedSocketOps comment are slightly off (14/6 split is approximate).
• ℹ️ NIT — close() is excluded but not overridden — defense-in-depth via implCloseSelectableChannel() works.
• ℹ️ NIT — No toString() override.
• ℹ️ NIT — safeClose swallows IOException silently; consider trace-level log.
• ✅ Test coverage is excellent (every read/write variant tested, mockStatic for NetworkRecycledException).

Good:

• Feature flag apiml.tcpStackAwareSocketChannel.enabled defaults to false — safe opt-in.
• @Delegate approach correctly handles 20 excluded methods + delegates the rest.
• isTcpStackRestarted now static (was instance) — cleaner since no instance state needed.
• New tests cover both the existing isTcpStackRestarted and the new TcpStackAwareSocketChannel.

Optional follow-up:

• Update apiml-tomcat-common CHANGELOG / docs to mention the new flag.
• Consider a tenant-level integration test that toggles the flag and verifies behavior.

Required before merge: none. All findings are NIT-level polish.

[richard-salac]

The start.sh scripts are missing support for the new configuration property.

[richard-salac]

When the feature is enabled on z/OS, APIML is not able to accept connections and logs:

2026-08-28 15:29:24.836 <ZWEAGW1:https-jsse-nio-0.0.0.0-10010-Poller:33620144> ZWESVUSR ERROR (o.a.t.u.n.NioEndpoint) Failed to register socket with selector from poller
java.nio.channels.IllegalSelectorException
	at java.base/sun.nio.ch.SelectorImpl.register(SelectorImpl.java:253)
	at java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:236)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:1004)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1066)
	at java.base/java.lang.Thread.run(Thread.java:854)
2026-08-28 15:29:26.401 <ZWEAGW1:https-jsse-nio-0.0.0.0-10010-Poller:33620144> ZWESVUSR ERROR (o.a.t.u.n.NioEndpoint) Failed to register socket with selector from poller
java.nio.channels.IllegalSelectorException
	at java.base/sun.nio.ch.SelectorImpl.register(SelectorImpl.java:253)
	at java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:236)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:1004)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1089)
	at java.base/java.lang.Thread.run(Thread.java:854)

[richard-salac]

When the feature is enabled on z/OS, APIML is not able to accept connections and logs:

2026-08-28 15:29:24.836 <ZWEAGW1:https-jsse-nio-0.0.0.0-10010-Poller:33620144> ZWESVUSR ERROR (o.a.t.u.n.NioEndpoint) Failed to register socket with selector from poller
java.nio.channels.IllegalSelectorException
	at java.base/sun.nio.ch.SelectorImpl.register(SelectorImpl.java:253)
	at java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:236)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:1004)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1066)
	at java.base/java.lang.Thread.run(Thread.java:854)
2026-08-28 15:29:26.401 <ZWEAGW1:https-jsse-nio-0.0.0.0-10010-Poller:33620144> ZWESVUSR ERROR (o.a.t.u.n.NioEndpoint) Failed to register socket with selector from poller
java.nio.channels.IllegalSelectorException
	at java.base/sun.nio.ch.SelectorImpl.register(SelectorImpl.java:253)
	at java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:236)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:1004)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1089)
	at java.base/java.lang.Thread.run(Thread.java:854)

Review notes:

Background: the two z/OS TCP/IP stack-restart failure modes

1. Listening socket can't accept new connections. Solved by FixedServerSocketChannel
   (apiml-tomcat-common/src/main/java/org/zowe/apiml/product/web/TomcatAcceptFixConfig.java:175),
   originally added in a851b8f33 (2022), refined in 8189db715 (2024). Detects EDC5122I /
   com.ibm.net.NetworkRecycledException from accept(), closes and rebinds in a retry loop.
   This part works and was not touched by gh4776.
2. Already-accepted client connections go stale when the stack restarts — their fd refers to a
   dead stack instance. gh4776 tried to fix this with TcpStackAwareSocketChannel
   (TomcatAcceptFixConfig.java:363), a SocketChannel subclass wrapping accepted sockets to
   detect the same errors on read/write, close, and rethrow. This is the broken new part.

Review findings (PR 4792, /code-review)

1. CONFIRMED — critical. TcpStackAwareSocketChannel (line 363) cannot be registered with a
   real java.nio.channels.Selector. Tomcat's NioEndpoint.Poller calls
   sc.register(selector, SelectionKey.OP_READ, socketWrapper) on every accepted channel
   (NioEndpoint.java:705). The JDK's SelectorImpl.register() needs the channel to implement the
   package-private sun.nio.ch.SelChImpl, which only the real SocketChannelImpl implements — not
   this wrapper. Result: every accepted connection fails registration the instant the flag is
   enabled (Failed to register socket with selector from poller, logged and swallowed at
   NioEndpoint.java:706-707, key endpoint.nio.registerFail) — independent of whether a stack
   restart ever happens. Disabling the flag by default (fb7e318d8) just hides this on non-z/OS CI;
   it doesn't fix the intended z/OS use case.
2. CONFIRMED. ExcludedSocketOps (line 333) omits bind, setOption, shutdownInput,
   shutdownOutput. These are declared to return covariant SocketChannel (this), but Lombok's
   @Delegate has no special-casing for fluent returns — it generates
   return this.delegate.bind(...), silently unwrapping to the raw channel on any chained call. The
   sibling FixedServerSocketChannel's Overridden interface already excludes bind() for this
   exact reason; it was missed here.
3. CONFIRMED — minor. Test read_shouldCloseAndRethrowOnNetworkRecycledException
   (TomcatAcceptFixConfigTest.java:410) stubs isRecycledClass(any()) to always return true,
   so it never exercises the real class-name-matching logic, only the downstream close/rethrow
   wiring.

Why FixedServerSocketChannel never hit the same SelChImpl bug

Structurally identical wrapper pattern, but never exercised: the listening socket is set to
blocking mode (NioEndpoint.java:266, serverSock.configureBlocking(true); // mimic APR behavior) and the acceptor thread just calls serverSock.accept() directly
(NioEndpoint.java:535) — never Selector.register(). Selector.register() requires
non-blocking mode, so the SelChImpl cast is never reached for the listening socket. Accepted
client sockets, by contrast, are explicitly set non-blocking
(NioEndpoint.java:488, socket.configureBlocking(false)) specifically so the Poller can
register them for event-driven reads — that registration call is exactly where
TcpStackAwareSocketChannel breaks.

Is it fixable in place? No — conclusion: drop the wrapper, redesign

SelChImpl is package-private JDK-internal; it can't be robustly implemented by an
application-level wrapper (would require sitting in sun.nio.ch and matching vendor/version
internals — OpenJDK vs IBM Semeru/OpenJ9 differ). A real fix would require intercepting reads/
writes at Tomcat's own NioChannel (org.apache.tomcat.util.net.NioChannel, a plain object, not
a SelectableChannel — confirmed via Tomcat 10.1 sources: NioEndpoint.setSocketOptions() at
line 464-482 does channel.reset(socket, newWrapper), and Poller.events() at line 696 does
socketWrapper.getSocket().getIOChannel() to get the raw channel for registration — one field,
NioChannel.sc, used for both I/O and registration). That's invasive and adds Tomcat-internals
coupling on top of existing JDK-internals coupling.

Also: TcpStackAwareSocketChannel.read()/write() (lines 373-423) only log a debug line and call
safeClose(delegate) before rethrowing the same exception — Tomcat's own NIO error handling
already closes sockets on IOException from read/write. So the wrapper's actual value-add is thin
(a debug log line), while its cost (breaks all connections) is total.

Alternative design (recommended)

Don't detect the restart per-connection (unreliable — idle connections never read again, so the
exception never fires; and wrapping breaks registration). Instead, treat the already-working
accept()-side detection as the single trigger to bulk-reap every connection that predates the
restart, using Tomcat's own public API — no subclassing, no reflection beyond what's already used.

• org.apache.tomcat.util.net.AbstractEndpoint.getConnections() — public, returns
  Set<SocketWrapperBase<S>> of all live connections. Used by Tomcat itself the same way for
  pause() (AbstractProtocol.java:1184) and in Nio2Endpoint housekeeping.
• SocketWrapperBase.close() — public, idempotent; closes the channel, removes it from
  connections, decrements the connection counter, releases the handler.
• Closing a SocketChannel (an InterruptibleChannel) from another thread is JDK-guaranteed to
  unblock any thread currently blocked in a read/write on it (AsynchronousCloseException) — so
  this also solves the "stuck thread" case without needing to catch any specific exception.

private void closeStaleConnections() {
    for (SocketWrapperBase<?> wrapper : abstractEndpoint.getConnections()) {
        try {
            wrapper.close();
        } catch (Exception e) {
            log.debug("Failed to close stale connection after TCP/IP stack restart", e);
        }
    }
}

private synchronized void rebind(int stateBefore) throws IOException {
    if (state.compareAndSet(stateBefore, stateBefore + 1)) {
        try {
            closeStaleConnections();   // NEW
            socket.close();
            bindWithWait();
            rebindHandler.run();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        } catch (Exception e) {
            throw new IOException("Cannot rebind the port", e);
        }
    }
}

This lets us delete TcpStackAwareSocketChannel, ExcludedSocketOps, the @Delegate wrapper, and
the tcpStackAwareSocketChannelEnabled flag — removing both correctness findings and ~120 lines of
JDK/Tomcat-internals-coupled code. Trivially unit-testable (mock getConnections() to return mock
SocketWrapperBases, assert close() called).

Known gap: detection still rides on the acceptor thread calling accept() again, so if the stack
dies during a lull with no incoming connections, the sweep is delayed until the next client
connects. Optional follow-up: a low-frequency background probe to trigger the same sweep
independently. Treated as a nice-to-have, not required for the initial fix.

How to verify the SelChImpl finding on real z/OS

No actual TCP/IP stack restart needed — the bug fires on the very first accepted connection while
the flag is enabled.

1. Standalone JDK repro (isolates JDK-vendor behavior, no apiml/Tomcat needed): a small
   self-contained SocketChannel subclass + Selector.open() + register() call, run with the
   exact JDK build used on the z/OS LPAR. Expect it to throw (ClassCastException or
   IllegalSelectorException, exact type is vendor-dependent — this pins it down precisely). Full
   snippet given earlier in the conversation transcript (search for SelChImplRepro).
2. End-to-end on the gateway: set apiml.tcpStackAwareSocketChannel.enabled=true, enable
   logging.level.org.apache.tomcat.util.net.NioEndpoint=DEBUG, send one ordinary request. Expect
   the client to hang/time out and the server log to show
   Failed to register socket with selector from poller (Tomcat's hardcoded message, key
   endpoint.nio.registerFail in org/apache/tomcat/util/net/LocalStrings.properties) with a
   stack trace matching the exception type from step 1.

[richard-salac]

cause == null if t == cause from the t.getClause implementation. The cause != t check is unnecesary

[richard-salac]

is recycled is called only once. Can be internalized.

[richard-salac]

It is useful to know if there are sockets that cannot be freed and why. Debug is rarely turned on in production. Suggestion to add a number of connections that could not be closed. If there is a repeating problem, the user can be asked to turn on debug and replicate.

[richard-salac]

Why was the method moved from FixedServerSocketChannel class? Seems unnecessary as the method is called only from there.

[richard-salac]

Dead code

[richard-salac]

getConnections returns a set. Why conversion to Array is needed?

[richard-salac]

It is a practice we follow to specify times(x) even for 1

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
97.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud