Password server on redundant VRs binds only the VRRP VIP, so guests never receive their password

[MitchDrage]

problem

SUMMARY
On a redundant VR, the password server is started with both the VRRP VIP and the router's own guest IP, but passwd_server_ip.py binds only the first of the two. Guests derive the password server address from the DHCP server-identifier, which is the router's own guest IP, so their requests to port 8080 are answered with a TCP RST. Passwords accumulate uncollected in /var/cache/cloud/passwords- and password injection silently fails for every guest on every isolated network with redundant routers.

This is not to be confused with #12161, which changes the DNS list for redundant VPC VRs only and does not affect this path.

CAUSE
I believe this was introduced in #10183, which made the VR's own guest IP unconditional in dnsmasq's listen-address (fixing #10182). That deliberately changed the behaviour established by #3477, where redundant VRs listened on the VIP only. dnsmasq now hands out the VR's own IP as the DHCP server-identifier, but the password server is still started as , and binds addresses[0], the VIP, so guests query an address nothing listens on.

In the environment below, the guest network is 192.168.0.0/24, the VRRP VIP (gateway) is 192.168.0.1, and the primary router's own guest IP is 192.168.0.48.

EXPECTED RESULTS

The guest retrieves its password from the address it was given as the DHCP server-identifier, and the entry is removed from the password cache.

ACTUAL RESULTS

The password server listens on the VIP only:

root@r-3050-VM:~# ss -lntp | grep 8080
LISTEN 0 5 192.168.0.1:8080 0.0.0.0:* users:(("python",pid=3372,fd=3))

Passwords are delivered by the management server and never collected:

root@r-3050-VM:~# cat /var/cache/cloud/passwords-192.168.0.1
192.168.0.82=
192.168.0.188=
192.168.0.38=

Metadata on port 80 works for every guest, because cloudstack binds all addresses. Both agents complete a full metadata crawl and only the password request fails. Note the 61 second stall in the cloudbase-init crawl, which is the password request timing out:

192.168.0.82 - - [.../Jul/2026:03:30:36 +0000] "GET /latest/meta-data/instance-id HTTP/1.1" 200 291 "-" "python-requests/2.33.1"
192.168.0.82 - - [.../Jul/2026:03:31:37 +0000] "GET /latest/meta-data/service-offering HTTP/1.1" 200 302 "-" "python-requests/2.33.1"
192.168.0.38 - - [.../Jul/2026:04:55:56 +0000] "GET /latest/meta-data/instance-id HTTP/1.1" 200 291 "-" "Cloud-Init/26.1-0ubuntu1~22.04.1"
192.168.0.38 - - [.../Jul/2026:04:55:56 +0000] "GET /latest/user-data HTTP/1.1" 200 351 "-" "Cloud-Init/26.1-0ubuntu1~22.04.1"

The guest requests port 8080 at the router's own guest IP (192.168.0.48), not at the VIP, and is reset because nothing is bound there:

05:03:16.482340 IP 192.168.0.38.43828 > 192.168.0.48.8080: Flags [S], seq 2121724, length 0
05:03:16.482361 IP 192.168.0.48.8080 > 192.168.0.38.43828: Flags [R.], seq 0, ack 2121725, win 0, length 0
05:03:17.484909 IP 192.168.0.38.43838 > 192.168.0.48.8080: Flags [S], seq 2008568524, length 0
05:03:17.484944 IP 192.168.0.48.8080 > 192.168.0.38.43838: Flags [R.], seq 0, ack 2008568525, win 0, length 0

This is not a firewall problem. The rules are present and counting the SYNs above as accepted:

root@r-3050-VM:~# iptables-save | grep -E 'dport (80|8080)'
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -s 192.168.0.0/24 -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -s 192.168.0.0/24 -i eth0 -p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT

17 204 10640 ACCEPT 6 -- eth0 * 192.168.0.0/24 0.0.0.0/0 tcp dpt:8080 state NEW

Confirmed with a DNAT rule on the primary router, after which every pending password was collected immediately:

iptables -t nat -I PREROUTING -i eth0 -d 192.168.0.48/32 -p tcp --dport 8080
-j DNAT --to-destination 192.168.0.1:8080

versions

ACS: 4.22.1.0
systemvm template: systemvm-kvm-4.22.0-x86_64
Hypervisor: KVM.
Guests tested: Windows Server 2022 (cloudbase-init) and Ubuntu 22.04 (cloud-init 26.1). Same guest templates work correctly on a shared network served by a single, non-redundant VR.

CONFIGURATION
Advanced zone, KVM, isolated guest networks using a network offering with redundant routers enabled. UserData/password service provided by VirtualRouter.

The steps to reproduce the bug

1. Create an isolated network using an offering with redundant VRs enabled and UserData provided by VirtualRouter.
2. Deploy a guest from any template with a password-enabled image (cloudbase-init or cloud-init), or reset the password of an existing guest.
3. Observe the guest never receives its password.

What to do about it?

passwd_server_ip.py: start a listener for every address supplied rather than only addresses[0]. allowAddresses already contains both, and savepassword.sh already POSTs to every local IP, so no other change is needed. Keeping addresses[0] as listeningAddress preserves the existing /var/cache/cloud/passwords- path. This is what I am running locally using custom userdata, verified across save, fetch and acknowledge.

Something like this??
FROM:
(Starting at line 182 of passwd_server_ip.py)

    global listeningAddress
    global allowAddresses
    if len(sys.argv) > 1:
        addresses = sys.argv[1].split(",")
        if len(addresses) > 0:
            listeningAddress = addresses[0]
            allowAddresses.append(addresses[0])
        if len(addresses) > 1:
            allowAddresses.append(addresses[1])

    server_address = (listeningAddress, 8080)
    passwordServer = ServerClass(server_address, HandlerClass)
    passwordServer.allow_reuse_address = True
    sa = passwordServer.socket.getsockname()
    initToken()
    loadPasswordFile()
    syslog.syslog('serve_password running on %s:%s' % (sa[0], sa[1]))
    try:
        passwordServer.serve_forever()

TO:

    global listeningAddress
    global allowAddresses
    addresses = [listeningAddress]
    if len(sys.argv) > 1:
        addresses = [a.strip() for a in sys.argv[1].split(",") if a.strip()]
        listeningAddress = addresses[0] # Need to keep listeningAddress as-is on the first address as it's global. Used in getPasswordFile()
        for addr in addresses:
            if addr not in allowAddresses:
                allowAddresses.append(addr)

    initToken()
    loadPasswordFile()
    servers = []
    for addr in addresses:
        server = ServerClass((addr, 8080), HandlerClass)
        server.allow_reuse_address = True
        servers.append(server)
        syslog.syslog('serve_password running on %s:8080' % addr)
    for server in servers[1:]: # Add each subsequent server as a thread. Primary server starts in the try block.
        t = threading.Thread(target=server.serve_forever)
        t.daemon = True
        t.start()
    try:
        servers[0].serve_forever()

[weizhouapache]

@MitchDrage
can you test #13194 which reverts #10183 ?

[MitchDrage]

Ahhh ok. Didn't find that one when I was looking for an existing issue. Could have saved myself two hours of typing!
That reverses the change that I believe introduced the issue, so it's very likely to solve it.
I have a temp solution in place so I'll have to schedule some time to get this tested. I'll come back to you when I can.