ZOOKEEPER-4946: Bound the join in Login.shutdown() - #2446
Open
IvanKhanas wants to merge 1 commit into
Open
Conversation
Login.shutdown() interrupted the TGT renewal thread once and then joined it without a timeout. Shell swallowed that interrupt by rethrowing InterruptedException as IOException without restoring the interrupt status, so the thread went back to sleeping until the next refresh and the join never returned. SendThread calls this on exit, so ZooKeeper.close() hung with it. Shell now restores the interrupt status, Login tracks the request in a volatile flag instead of relying on a single interrupt, and the join is bounded by zookeeper.kerberos.shutdownTimeoutMs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Login.shutdown()calledt.interrupt()once and thent.join()with notimeout. The interrupt could be lost:
Shell.runCommand()caughtInterruptedExceptionfromprocess.waitFor()and rethrew it as anIOExceptionwithout restoring the interrupt status.Loginread that as afailed
kinit, retried, and went back to sleeping until the next scheduledrefresh, which is around 8 hours for a 10 hour ticket. The join waited all of it.
SendThreadcallsLogin.shutdown()as its last action, andClientCnxn.close()joinsSendThreadwithout a timeout, soZooKeeper.close()could hang.
NIOServerCnxnFactoryandNettyServerCnxnFactorycalllogin.shutdown()during their own shutdown as well.Fix
Shellrestores the interrupt status at both sites where it catchesInterruptedException.Loginrecords the request in a volatile flag instead of relying on a singleinterrupt, and checks it before running
kinitand before enteringreLogin().Login.shutdown()bounds the join withzookeeper.kerberos.shutdownTimeoutMs(5s default) and warns when the thread has to be abandoned. The thread is a
daemon, so abandoning it does not keep the JVM alive.
Tests
ShellTestis new: interrupts a thread insideShell.execCommandand assertsthe interrupt status survives. Runs in under 0.1s.
KerberosTicketRenewalTest.shouldNotBlockForeverWhenRenewalThreadDoesNotExitparks the renewal thread in a call that ignores interrupts and asserts
shutdown()still returns.Both fail without the production change: the first on the interrupt status, the
second by timing out, which is the reported symptom. The three existing tests in
KerberosTicketRenewalTestare unaffected.