Why does docker see container hitting rss limit?

I am trying to understand why limits decided that a task should be killed and how it does accounting. When my GCE Docker container kills the process, it shows something like:

Task in /404daacfcf6b9e55f71b3d7cac358f0dc921a2d580eed460c2826aea8e43f05e killed as a result of limit of /404daacfcf6b9e55f71b3d7cac358f0dc921a2d580eed460c2826aea8e43f05e 
memory: usage 2097152kB, limit 2097152kB, failcnt 74571 
memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0 
kmem: usage 0kB, limit 18014398509481983kB, failcnt 0 
Memory cgroup stats for /404daacfcf6b9e55f71b3d7cac358f0dc921a2d580eed460c2826aea8e43f05e: cache:368KB rss:2096784KB rss_huge:0KB mapped_file:0KB writeback:0KB inactive_anon:16KB active_anon:2097040KB inactive_file:60KB active_file:36KB unevictable:0KB 
[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name 
[ 4343]     0  4343     5440       65      15        0             0 bash 
[ 4421]     0  4421   265895     6702      77        0             0 npm 
[ 4422]     0  4422    12446     2988      28        0             0 gunicorn 
[ 4557]     0  4557   739241   346035    1048        0             0 gunicorn 
[ 4560]     0  4560     1086       24       8        0             0 sh 
[ 4561]     0  4561     5466      103      15        0             0 bash 
[14594]     0 14594   387558   168790     672        0             0 node 
Memory cgroup out of memory: Kill process 4557 (gunicorn) score 662 or sacrifice child 
Killed process 4557 (gunicorn) total-vm:2956964kB, anon-rss:1384140kB, file-rss:0kB 

      

        
        
        
      

    

Presumably the memory has reached the 2GB usage limit and something is about to die. According to the group's stats, I seem to have 2GB of use in active_anon

and out rss

.

When I look at the process statistics table, I cannot see where the 2 GB is located:

For rss

, I see two main processes 346035 + 168790 = 514MB

? For total_vm

I see three main processes 265895 + 739241 + 387558 = 1.4GB

?

But when he decides to kill the shooting process, he says that he had 3GB Total VM and 1.4GB Anon RSS. I don't see how this follows from the numbers above ...

For most of life, according to top

, the shooting process seems to be buzzing along with 555m RES

and 2131m VIRT

and 22% MEM * 2.5GB box = 550MB

memory usage. (I have not yet been able to timely assess its significance top

at the time of his death ...)

• Can anyone help me figure this out?
• Under what reporting are these amounts 2 GB? (virtual? rss? something else?)
• Is there anything other than top
  
  / ps
  
  I should be using to keep track of how much memory this process is using for docker killing it?