Skip to content

fix(k8s): use the node's configured iptables backend for node prep - #2145

Open
guanchzhou wants to merge 1 commit into
apple:mainfrom
guanchzhou:fix-k8s-node-prep-iptables-backend
Open

fix(k8s): use the node's configured iptables backend for node prep#2145
guanchzhou wants to merge 1 commit into
apple:mainfrom
guanchzhou:fix-k8s-node-prep-iptables-backend

Conversation

@guanchzhou

Copy link
Copy Markdown

Fixes #2120

The node-prep script hardcodes /usr/sbin/iptables-nft for its two TCP MSS clamping rules, but the default node image runs a kernel with no nftables support, so those calls can never succeed and set -e aborts node prep.

Evidence, inside a node left behind by a failed container k8s create

$ ls /proc/net/nf_tables*
ls: cannot access '/proc/net/nf_tables*': No such file or directory

$ /usr/sbin/iptables-nft -t mangle -S
iptables v1.8.11 (nf_tables): Could not fetch rule set generation id: Invalid argument

$ ls -l /usr/sbin/iptables
/usr/sbin/iptables -> /etc/alternatives/iptables
$ /usr/sbin/iptables --version
iptables v1.8.11 (legacy)

The same two rules, unchanged, applied through /usr/sbin/iptables:

$ /usr/sbin/iptables -t mangle -A OUTPUT  -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
$ /usr/sbin/iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
$ /usr/sbin/iptables -t mangle -S | grep TCPMSS
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
-A OUTPUT -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220

Why name iptables rather than a backend

alternatives resolves iptables to whichever backend the image is configured for. On kindest/node that is legacy, which works; on an image whose kernel does carry nftables it selects nft. Naming a specific backend is what makes the script image-dependent, so this is strictly more portable than the line it replaces.

Appending || true would also stop the abort, but it would silently drop the MSS clamping those rules exist to provide.

A note on the confusing error message

Because nodePrepScript runs under set -e and these are its last two lines, the failure aborts prep and what surfaces is the accumulated stdout of the commands that already succeeded:

Error: node prep failed on cs-repro-2120: net.ipv4.ip_forward = 1
registry.k8s.io/pause:3.10.1

That names a sysctl that worked and an image tag that worked, not the command that failed. Not changed here, but surfacing the failing command's stderr would make this class of failure self-diagnosing. Two smaller observations from the same reproduction, also left alone: a create that fails leaves the node running and holding its published port, and --cpus/--memory make no difference (reproduced at 2/4 GiB as well as the reporter's 6/16 GiB).

Verification

  • swift build --product container and swift build --target k8s both succeed.
  • The behavioural claim is verified by running the exact commands in a real failed node, as above: the nft backend errors, iptables applies both rules.
  • Not verified: an end-to-end container k8s create with the patched plugin installed. That needs the built plugin placed into the install root in place of the packaged one, which I did not want to do on this machine. If you would like that run before merging, say so and I will do it.

Environment: macOS 27.0 (Tahoe), Apple silicon, container 1.2.2 from the Homebrew formula, node image docker.io/kindest/node:v1.35.5@sha256:ce977ae… (the plugin default).

Per CONTRIBUTING: I used AI assistance while investigating and drafting, and I can explain and justify both changed lines — the change is a substitution of one shell binary for another, and the reasoning is the nftables-absence evidence above.

The node-prep script hardcodes /usr/sbin/iptables-nft for the two TCP MSS
clamping rules, but the default node image runs a kernel with no nftables
support, so those calls always fail:

  $ ls /proc/net/nf_tables*
  ls: cannot access '/proc/net/nf_tables*': No such file or directory
  $ /usr/sbin/iptables-nft -t mangle -S
  iptables v1.8.11 (nf_tables): Could not fetch rule set generation id: Invalid argument

/usr/sbin/iptables resolves through alternatives to the backend the image is
actually set up for -- legacy in kindest/node -- and the same two rules apply
cleanly through it. On an image whose kernel does carry nftables, alternatives
selects nft, so naming iptables is strictly more portable than naming a
specific backend.

Because nodePrepScript runs under set -e and these are its last two lines, the
failure aborts node prep and surfaces the accumulated stdout of the commands
that already succeeded, which is why the reported error names a sysctl that
worked rather than the command that failed.

Fixes apple#2120
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: container k8s create fails when node preparation invokes iptables-nft

1 participant