Run BESS inside a container

[jianfenw]

I wonder if there is a way of running BESS inside a container.
The scenario is as following (1) multiple BESS processes running at a single host (2) a vSwitch (e.g. BESS) that connects all BESS processes with the physical NIC.
Is it feasible to do so?
I am quite curious and really appreciate if you could provide guides (or possible hints) for me to try out this scheme.

[trozet]

Hi @jwangee. Yes it is possible. We already do it in the Nimbess project. The docker images are here:
https://cloud.docker.com/u/nimbess/repository/docker/nimbess/bess

The only caveat is that the kmod needs to be compiled and loaded on the host first.

[jianfenw]

Nice. This looks interesting. Nimbess seems to be a fast-path Kubernetes CNI option.
I could run BESS on my local host. Also, I've compiled and loaded kmod on the host, which I guess is necessary for creating VPort and forwarding packets for containers. I could also run BESS on host and creating VPort which is visible inside a specific container.
Then, I tried to run BESS insdie the Nimbess/bess container, but failed with the following messages. I wonder if you've gone through this.

$ docker run --privileged -it --rm nimbess/bess bash
[root@0684b28bb899 bess]# ./bessctl/bessctl
Type "help" for more information.
Connection to localhost:10514 failed
Perhaps bessd daemon is not running locally? Try "daemon start".
$ daemon start
E1025 18:33:43.111024 38 bessd.cc:356] Failed to launch a daemon process
*** Error: Cannot start BESS daemon

Followup:
I figured it out. We still have to mount the Hugepage host directory to the container.
$ docker run -v /dev/hugepages/:/dev/hugepages/ --privileged -it --rm nimbess/bess bash

[trozet]

Right, unless you use "-m 0" which I think will disable hugepages and bessd will start. Here is the k8s yaml we use to launch bess:

https://github.com/nimbess/nimbess-agent/blob/master/k8s/nimbess.yaml#L55

[jianfenw]

Thanks a lot for confirming this. It helps. I could run BESS as a vSwitch for containers.
One more thing that I am curious (I didn't succeed in doing so in my local machine): is it possible to run multiple such BESS containers together at the same machine (and forwarding packets between VPorts)? (I doubt it because these BESS containers would share the same set of Hugepages.)

[trozet]

You might be able to do it if you have multiple numa nodes and you isolate hugepage mem allocation per BESS instance to a single node (also configure lcores to the right socket). That's something that is missing in BESS right now. Today if you specify a huge page size with -m arg, it will automatically try to allocate that size from all sockets.

[trozet]

I started on:
trozet@6f558f7

It will let you start BESS and allocate hugepages from only 1 socket. However it is crashing when the workers are started. Even when I specify worker core it is still using core 0. Looking into it.

[trozet]

Looks like the worker 0 was being used by Nimbess controller by default and that was causing bessd to crash. I added a check during AddWorker to ensure the core requested is valid for the socket being used. Works for me. Give it a try and see if you can launch multiple bessd containers with hugepages on different sockets:

#946