[Suggestion] Experiment that uses OTP 28+ "priority" messages

[lukebakken]

RabbitMQ series

4.2.x

Operating system (distribution) used

Linux

How is RabbitMQ deployed?

Other

What would you like to suggest for a future version of RabbitMQ?

main...lukebakken:rmq-rabbitmq-server:lukebakken/priority-credit-flow-messages

When you run those changes with OTP 28, it'll use priority messages to control the rabbit_reader process. I haven't done extensive testing yet, just some runs with absurdly low watermarks. The idea is that rabbit_reader should stop reading from the TCP socket as soon as it receives an alarm message, rather than having to wade through its mailbox first to get to the conserve_resources message.

I'll report back when I have time to test in an environment with monitoring in place. What I'm hoping to see is less memory usage if a publisher is publishing large messages and an alarm is triggered.

[gomoripeti]

it's great to use the newest features or Erlang/OTP. specifically for rabbit_read did you experience a long message queue? As I think it uses {active, once} so there should only be 1 message from the socket in the mailbox at once (there could be other messages ofc). Maybe the channel process can have a long message queue, but it does not handle conserve_resources but it does handle credit_flow messages

[lukebakken]

Let me look into that, thanks for bringing up {active, once}.

[lukebakken]

Yep you are right on. I didn't think to look into that detail.

What I'd like to accomplish is a smarter credit flow system that increases TCP backpressure more gradually as system/operational load increases in RabbitMQ.

[michaelklishin]

Note that Erlang 28 is not fully supported by RabbitMQ. We can tackle this as a research project, sure but horus need a set of dramatic changes to support Erlang 28's encoding/term format changes.

[lukebakken]

For now, @gomoripeti's answer is correct in that the way that rabbit_reader works doesn't require priority messages. I'll close this discussion and will start a new one if/when I have something useful 😉