Use Akka Stream based MesosClient implementation

[chetanmeh]

Current MesosClient is an actor based implementation. It would useful to investigate if we can switch to an stream based implementation. For example we can use Marathon Mesos Client code (Apache License). Some possible benefits

1. Supports proper back pressure
2. Simplify offer processing by making use of various stream operators
3. Uses Scala Protobuf which further simplifies constructing the objects. There is a Gradle Plugin for that which would be useful in our case
4. Provides a Sink for outbound calls which use a single long lived connection
5. Supports Mesos with HA i.e. if there is a redirect on initial subscribe then it would reconnect to new master. However if Mesos Master changes post subscribe then possibly whole MesosClient need to be recreated

[chetanmeh]

@tysonnorris @mcdan @Himavanth Thoughts?

[tysonnorris]

I'm not sure there are significant benefit to making the MesosClient an akka stream based interface.
The existing MesosClientHttpConnection is already stream based, it is only the external interface that is actor based. If you are integrating with a stream system, maybe that makes it simpler, but we are not, for now at least - WDYT? I think this would be more interesting to consider in the "bigger picture" with openwhisk, e.g. if ContainerPool and ContainerProxy were changed to be stream based, and could maybe use marathon mesos client directly in that case, if it works easily and reliably.

Using ScalaProtobuf would be nice in terms of getting better scala friendly protobuf objects, but it comes at the cost of complicating the build process (not sure if marathon mesos client helps there?), so I have mixed feelings about that. The mesos.proto is huge, and I experimented with using this directly in the past, and decided at the time that pre-built java simplicity was worth the cost of having java converters added to the code.

I definitely agree that Mesos HA needs testing, but I'm not sure that is a complicated problem (to redirect and reconnect as needed), compared to switching to a stream based api at the client.