Skip to content

ZOOKEEPER-4946: Bound the join in Login.shutdown() - #2446

Open
IvanKhanas wants to merge 1 commit into
apache:masterfrom
IvanKhanas:ZOOKEEPER-4946
Open

ZOOKEEPER-4946: Bound the join in Login.shutdown()#2446
IvanKhanas wants to merge 1 commit into
apache:masterfrom
IvanKhanas:ZOOKEEPER-4946

Conversation

@IvanKhanas

Copy link
Copy Markdown
Contributor

Problem

Login.shutdown() called t.interrupt() once and then t.join() with no
timeout. The interrupt could be lost: Shell.runCommand() caught
InterruptedException from process.waitFor() and rethrew it as an
IOException without restoring the interrupt status. Login read that as a
failed kinit, retried, and went back to sleeping until the next scheduled
refresh, which is around 8 hours for a 10 hour ticket. The join waited all of it.

SendThread calls Login.shutdown() as its last action, and
ClientCnxn.close() joins SendThread without a timeout, so ZooKeeper.close()
could hang. NIOServerCnxnFactory and NettyServerCnxnFactory call
login.shutdown() during their own shutdown as well.

Fix

  • Shell restores the interrupt status at both sites where it catches
    InterruptedException.
  • Login records the request in a volatile flag instead of relying on a single
    interrupt, and checks it before running kinit and before entering
    reLogin().
  • Login.shutdown() bounds the join with zookeeper.kerberos.shutdownTimeoutMs
    (5s default) and warns when the thread has to be abandoned. The thread is a
    daemon, so abandoning it does not keep the JVM alive.

Tests

  • ShellTest is new: interrupts a thread inside Shell.execCommand and asserts
    the interrupt status survives. Runs in under 0.1s.
  • KerberosTicketRenewalTest.shouldNotBlockForeverWhenRenewalThreadDoesNotExit
    parks the renewal thread in a call that ignores interrupts and asserts
    shutdown() still returns.

Both fail without the production change: the first on the interrupt status, the
second by timing out, which is the reported symptom. The three existing tests in
KerberosTicketRenewalTest are unaffected.

Login.shutdown() interrupted the TGT renewal thread once and then joined it
without a timeout. Shell swallowed that interrupt by rethrowing
InterruptedException as IOException without restoring the interrupt status,
so the thread went back to sleeping until the next refresh and the join never
returned. SendThread calls this on exit, so ZooKeeper.close() hung with it.

Shell now restores the interrupt status, Login tracks the request in a
volatile flag instead of relying on a single interrupt, and the join is
bounded by zookeeper.kerberos.shutdownTimeoutMs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant