The fact that most of the memory used is not associated with any specific messages (some of the "binary links" in your screenshot) suggests that this memory is being used by operating system resources not directly managed by RabbitMQ. My biggest suspect will be open minded.

To test this theory, you can run netstat

and see if you get similar results (if you are using rabbitmq on the default port is 5672):

root@rabbitmqhost:~# netstat -ntpo | grep -E ':5672\>'
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name Timer
tcp6       0      0 xxx.xxx.xxx.xxx:5672    yyy.yyy.yyy.yyy:57656    ESTABLISHED 27872/beam.smp   off (0.00/0/0)
tcp6       0      0 xxx.xxx.xxx.xxx:5672    yyy.yyy.yyy.yyy:49962    ESTABLISHED 27872/beam.smp   off (0.00/0/0)
tcp6       0      0 xxx.xxx.xxx.xxx:5672    yyy.yyy.yyy.yyy:56546    ESTABLISHED 27872/beam.smp   off (0.00/0/0)
tcp6       0      0 xxx.xxx.xxx.xxx:5672    yyy.yyy.yyy.yyy:50726    ESTABLISHED 27872/beam.smp   off (0.00/0/0)
⋮

      

        
        
        
      

    

The interesting part is the last column showing "Timer Off". This indicates that this connection is not using keepalives, which means that they will just hang out where there are resources if the client dies without being able to close them.

There are two ways to avoid this problem:

• TCP keepalives
• application protocol bit

TCP keepalives

They are processed by the kernel. Whenever the connection sees no packets for a certain amount of time, the kernel tries to send some probes to see if there is another side. Since the current Linux defaults (e.g. 4.12) are quite high (7200 seconds + 9 probes every 75 seconds> 2 hours) rabbitmq does not use them by default .

To activate them, you must add it to your rabbitmq.config

:

[
  {rabbit, [
    ⋮
    {tcp_listen_options,
         [
         ⋮
         {keepalive,     true},
         ⋮
         ]
    },
    ⋮
  ]},
  ⋮
].

      

        
        
        
      

    

and possibly lower the timeouts to more reasonable values. Something like this might work (but YMMV of course):

root@rabbitmqhost:~# sysctl net.ipv4.tcp_keepalive_intvl=30
root@rabbitmqhost:~# sysctl net.ipv4.tcp_keepalive_probes=3
root@rabbitmqhost:~# sysctl net.ipv4.tcp_keepalive_time=60

      

        
        
        
      

    

Application Receive Bits

They are handled by the actual messaging protocols (e.g. AMQP, STOMP, MQTT) but require the client to opt out. As each protocol is different, you should check the documentation to configure it in your client applications.

Conclusion

The safest option in terms of preventing dangling resources is TCP keepalives, since you don't have to rely on the behavior of your client applications. However, they are less versatile, and if misconfigured in a high-performance but “explosive” system, it can result in degraded performance as false positives will cause reconnections.

Application eavesdropping is a smaller-scale option if you need to avoid this problem while maintaining system performance, but they require more coordination, in the sense that clients have to choose and choose their own reasonable timeouts. Since you can never be 100% sure that your clients won't die without gracefully closing connections, enabling TCP keepalives as a backup (even with higher timeouts) might also be a good idea.