LoadBalancer services in a VPC with the Firewall service get no firewall rule #114

Since apache/cloudstack#12706 (CloudStack 4.23.0.0) a VPC offering can include the Firewall
service for public IPs. The VPC virtual router then drops ingress traffic to a public IP that
has a port forwarding, load balancing or static NAT rule unless a firewall rule allows it (a
FIREWALL_<ip> chain ending in DROP).

applyLoadBalancerRules (cloudstack_loadbalancer.go, main at 363cd03, same in v1.3.0-rc1)
decides per port between a firewall rule and a network ACL item from the services of the
cluster's network:

if isFirewallSupported(network.Service) {
    lb.updateFirewallRule(lbRule.Publicipid, ...)
} else if isNetworkACLSupported(network.Service) {
    lb.updateNetworkACL(..., network.Id)
}

In a VPC the Firewall service belongs to the VPC, not to the tier. A VPC tier reports
NetworkACL but not Firewall, so the CCM only adds an ACL item, and the service port stays
behind the DROP of the VPC firewall. Removal (pruneFirewallRule / pruneNetworkACLRule) uses
the same switch.

Reproduced

CloudStack 4.23.0.0 (KVM), VPC offering with Firewall:VpcVirtualRouter, tier offering with
Dhcp, Dns, Lb, NetworkACL, PortForwarding, SourceNat, StaticNat, UserData, Vpn (no Firewall),
CKS cluster v1.35.4 (1 control, 1 worker), CCM v1.3.0-rc1, nginx behind a LoadBalancer
service on port 80.

Service on the cluster's public IP (spec.loadBalancerIP set to the IP that CKS uses for the
API, so the IP is already configured on the VR):

• The CCM created the load balancer rule (port 80, Active) and a network ACL item for TCP 80
  ("Managed by the CloudStack Kubernetes Provider"), but no firewall rule.
• TCP 80 on that IP timed out from two different clients, while TCP 6443 on the same IP, which
  has a firewall rule, answered.
• After adding a firewall rule for TCP 80 on that IP by hand, the service answered at once
  (HTTP 200 from both clients).
• On kubectl delete svc the CCM removed the load balancer rule and the ACL item. The firewall
  rule stayed, as expected, since the CCM does not know about it; a fix has to remove the rule it
  created, because the IP is shared with the cluster API.

Service on a new public IP (no loadBalancerIP):

• Same rules as above: load balancer rule and ACL item, no firewall rule.
• In addition, the new IP was never configured on the VPC virtual router: no IpAssocVpcCommand
  was sent when the load balancer rule was applied, the IP was missing on the VR's public
  interface, and a firewall rule added by hand did not change that. This part is a CloudStack
  issue (CreatePortForwardingRuleCmd on a VPC tier doesn't trigger ipassoc ( firewall and conserve mode on in vpc offering) cloudstack#14146): once a public IP of a tier has a firewall rule (here the
  cluster IP, which has rules for the API), no further public IP of that tier is added to the
  router. Fix: VPC: associate tier public IPs when a public IP of the tier has firewall rules cloudstack#14681. Until then the CCM change alone is not enough for this case.
  In a VPC without the Firewall service the same kind of service on its own IP works.

Expected

When the cluster network is a VPC tier and the VPC has the Firewall service (the VPC's service
list from listVPCs contains Firewall), create a firewall rule for each service port on the
service's public IP in addition to the ACL item, honouring loadBalancerSourceRanges, and remove
it together with the service. The order in the code is already right (load balancer rule first,
then the firewall rule): on CloudStack versions without the fix for apache/cloudstack#14146 the
opposite order keeps a new public IP off the router.

Related

• CKS: cluster on a VPC tier fails when the VPC has the Firewall service (#12706) cloudstack#14668: the same gap in CKS itself (cluster API and SSH), fix in
  CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service cloudstack#14670.
• CreatePortForwardingRuleCmd on a VPC tier doesn't trigger ipassoc ( firewall and conserve mode on in vpc offering) cloudstack#14146 (and VPC firewall: a firewall rule on a new VPC public IP before any PF/LB rule prevents the IP from being configured on the VR cloudstack#14669): once a public IP of a VPC tier has a
  firewall rule, further public IPs of the tier are not added to or removed from the router; fix
  in VPC: associate tier public IPs when a public IP of the tier has firewall rules cloudstack#14681.
• Not covered here: a tier whose own offering includes the Firewall service takes the firewall
  branch and gets no ACL item; whether that matters depends on the ACL of the tier.