Erlang garbage collection
I need your help investigating Erlang's memory consumption problem. Typical, isn't it?
We have two different deployment schemes.
- In the first scheme, we run many identical nodes in small virtual machines (in Amazon AWS), one node per machine. Each machine has 4 GB of RAM.
- In another deployment scheme, we run these nodes on large machines with painless coverage (with 64GB of RAM) with many nodes per machine. In this deployment, the nodes are isolated in docker containers (limited to 4GB memory).
I noticed that a heap of processes in docked nodes wipes out 3 times more RAM than heaps in non-blocked nodes with the same load. I suspect garbage collection on non-blocked nodes is more aggressive. Unfortunately I don't have garbage collection statistics, but I would like to get them as soon as possible.
To provide more information, I must say that we are using HiPE R17.1 on Ubuntu 14.04 with a kernel. In both schemes, we run 8 schedulers per node and use the fullsweep_after
default flag .
My blind guess is that Erlang's garbage collection relies (somehow) on /proc/meminfo
(which is not relevant in a dockerized environment) by default. I am not a C-guy and am not familiar with internal emulators, so can someone point me to the places in the Erlang sources that are responsible for garbage collection and some emulator options that I can use to customize this behavior?
source to share
Unfortunately, virtual machines often try to be smarter with memory management than necessary, and this doesn't always fit well with Erlang's memory management model. Erlang tends to allocate and release a large number of small chunks of memory, which is very different from conventional applications, which usually allocate and release a small number of large chunks of memory.
One of these technologies is Transparent Huge Pages (THP), which some operating systems provide by default, and which makes the Erlang nodes running in such virtual machines grow (until they crash).
https://access.redhat.com/solutions/46111
https://www.digitalocean.com/company/blog/transparent-huge-pages-and-alternative-memory-allocators/
https://docs.mongodb.org/manual/tutorial/transparent-huge-pages/
So, make sure THP is turned off - this is the first thing you can check.
Another is trying to tweak the memory settings used when starting the Erlang virtual machine itself, for example see this post:
Result that worked for us:
-MBas aobf -MBlmbcs 512 -MEas aobf -MElmbcs 512
Another theory about memory allocators:
http://www.erlang-factory.com/static/upload/media/139454517145429lukaslarsson.pdf
And a more detailed description of memory flags:
source to share
First of all, you need to know that Erlang's garbage collection is process-based. Each process is GC at its own time and independently of each other. Thus, garbage collection on your system depends only on the data in your processes, and not on the operating system itself.
However, there could be a difference between memory consumption from Eralang's point of view and systems point of view. This is why comparing erlang:memory
to what your system says is always a good idea (it might show you some binary leaks or other memory problems).
If you want to learn a little about Erlang internals, I would recommend these two discussions:
https://www.youtube.com/watch?v=QbzH0L_0pxI
https://www.youtube.com/watch?v=YuPaX11vZyI
And from a little better debugging of your memory management, I could recommend starting at http://ferd.github.io/recon/
source to share