Add bridge mode networking

[torarnv]

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Bridged networking can be useful for example for mDNS/Bonjour, LAN-visible services without port forwarding, or simply to have the LAN/router manage the IPs of the containers.

Fixes #489

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[torarnv]

This is an early RFC of the feature. It requires the com.apple.vm.networking entitlement to use https://developer.apple.com/documentation/virtualization/vzbridgednetworkdeviceattachment, which requires a provisioning profile blessed by Apple, otherwise container-runtime-linux crashes on start (unless you locally disable AMFI). So if we land this feature it probably needs some guardrails so it's not enabled for local dev unless explicitly opted in. And the release machinery needs to sign with a blessed profile and signing identity.

[dcantah]

@torarnv Sorry for not getting back on the other issues yet, I have a large backlog.. 😆 But, what you point out is essentially why we haven't taken a stab at this yet. We'd have to figure out how we can go about the entitlement deficiencies and we hadn't explored too deeply.

[torarnv]

Thanks @dcantah! No rush, just wanted to try this out for myself locally, and for now this solves my immediate need 😄 But would be cool to upstream at some point!

For the entitlement, I assume it won't be a problem to get it blessed for the releases 😉 So the challenge we have to solve is setting the GH release pipeline of this repo up with the right certs/profiles, and ensuring local devs do not pull the feature in accidentally. I don't know how the Homebrew formula is going to deal with it, but I guess they'd either need to pull the binary releases from this repo, or live without the bridged network feature.

I'm happy to look further into this if you think there's a path forward for upstreaming this. Let me know if you have other comments/concerns with the code as is. Thanks!

[torarnv]

@dcantah 2aa9d21 should take care of the dev experience for people without a blessed cert in their CODESIGN_OPTS or run with AMFI off.

[jglogan]

@torarnv I'm glad you took a crack at it. As Danny mentioned, we're going to be slammed this month with more basic work, so we won't be in a place where we can merge it soon.

However, this change is an excellent architecture driver. We're in the midst of refactoring the network model to clean up the separation of concerns between client, API server, and network plugin (we're doing this for runtime as well). We need to work out the best way to represent configuration for different network modes, and this needs to work in a scenario where there are additional (builtin or third party) network plugins. Since you have an additional mode here, we can refine this PR and the main together to allow bridged mode to slot in cleanly.

Let's start on that front. Once I have #1421 merged and make a few of the simpler network changes, let's sort out what the config model might like. We can either follow up in here on this, or back in #489.

[torarnv]

Totally understandable @jglogan ! 😅 Happy to use this PR as inspiration/discussion for how to do this properly at some point. Some random things that came to mind when working on this:

• Making Interface.ipv4Address optional in apple/containerization@23c8a66 spread a bit through the codebase. Dunno if there are better ways to model the lack of an up-front configured IP. OTOH, a machine could strictly speaking have only IPv6, so making IPv4 optional is perhaps in line with that?
• I opted for extending container-network-vmnet with a bridged mode, rather than implementing a new network plugin, as a new plugin didn't seem necessary when the work of creating the final BridgedNetworkInterface happens in the Linux runtime plugin anyways, but perhaps is misunderstood the architecture. The vmnet in the existing network plugin is a bit of a misnomer already since NAT uses VZNATNetworkDeviceAttachment and not VZVmnetNetworkDeviceAttachment AFAIU.
• container ls and container inspect doesn't show the resulting IP. Should it? We'd need new RPC to feed back the interfaces, etc if so. Since the DHCP happens via the kernel during boot, these should be available very early.

Let's discuss further when #1421 and related changes land 🤝

[curdbecker]

Please check out also the adoption of the PR on top of the current state of the main branch with added support for helper-based bridging support. This allows to avoid using the entitlement completely during development while also having the option to work directly with bridged networking (because I actually need it right now 😂).

I imagine also that it is quite okay to have a less optimal performance during testing or even in general, e.g. in my current case where I just would simply bridge a container to a VLAN network interface, but Docker for Mac seems to basically obstruct people from using the features offered by the Virtualization Framework - or basically the entire VM (see also here, here and here - if someone is interested - had to take their entire virtualization service apart to set simple settings on the VM :-/). And this is exactly where container is amazingly different.

See #1622

P.S.: @torarnv your idea with having the kernel negotiate the DHCP lease and then also retrieve the nameservers from there without having a dedicated DHCP client in the init container is really amazing. This is quite elegant and simple :)