Proxmox high memory usage on one server.

Dec 26, 2018
138
2
23
35
Hello.
We have one server using more memory than it's suppose to do.

I have configured zfs to only use 512MB ram max.
And i have configured the Ceph OSDs to use 1GB each.

But still i am missing

Code:
root@proxmox1:~# top -o %MEM

top - 09:08:09 up 2 days, 12:53,  2 users,  load average: 5,83, 6,61, 6,33
Tasks: 575 total,   1 running, 574 sleeping,   0 stopped,   0 zombie
%Cpu(s): 17,4 us,  1,9 sy,  0,0 ni, 79,6 id,  0,3 wa,  0,0 hi,  0,7 si,  0,0 st
GiB Mem :    125,8 total,     39,6 free,     80,9 used,      5,3 buff/cache
GiB Swap:      0,0 total,      0,0 free,      0,0 used.     43,8 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                             
   5818 root      20   0   49,6g  48,1g   0,0g S 655,0  38,2   9127:09 kvm                                                                                 
   4549 root      20   0    9,4g   8,1g   0,0g S  29,1   6,4   1785:01 kvm                                                                                 
   3315 ceph      20   0    1,7g   0,8g   0,0g S   4,3   0,6 112:14.74 ceph-osd                                                                             
   2967 ceph      20   0    1,5g   0,7g   0,0g S   0,7   0,5  15:32.95 ceph-osd                                                                             
   2735 ceph      20   0    1,0g   0,7g   0,0g S   1,3   0,5  32:24.80 ceph-mon                                                                             
   3311 ceph      20   0    1,7g   0,6g   0,0g S   2,3   0,5  71:00.27 ceph-osd                                                                             
   2968 ceph      20   0    1,5g   0,5g   0,0g S   1,0   0,4  15:55.41 ceph-osd                                                                             
   2738 root      rt   0    0,6g   0,2g   0,0g S   3,6   0,2 102:57.32 corosync                                                                             
   2736 ceph      20   0    0,5g   0,2g   0,0g S   0,0   0,1   3:14.23 ceph-mgr                                                                             
   3395 www-data  20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:05.45 pveproxy                                                                             
1955496 www-data  20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:06.44 pveproxy worker                                                                     
1955497 www-data  20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:06.82 pveproxy worker                                                                     
1955495 www-data  20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:06.96 pveproxy worker                                                                     
   3212 root      20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:13.47 pvedaemon worke                                                                     
   3210 root      20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:14.07 pvedaemon worke                                                                     
   3211 root      20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:11.74 pvedaemon worke                                                                     
   3209 root      20   0    0,3g   0,1g   0,0g S   0,0   0,1   0:03.90 pvedaemon                                                                           
   3349 root      20   0    0,3g   0,1g   0,0g S   0,0   0,1   1:21.01 pve-ha-crm                                                                           
   3434 root      20   0    0,3g   0,1g   0,0g S   0,0   0,1   2:21.12 pve-ha-lrm                                                                           
   2936 root      20   0    0,3g   0,1g   0,0g S   0,0   0,1   6:48.29 pve-firewall                                                                         
   2941 root      20   0    0,3g   0,1g   0,0g S   1,7   0,1  18:00.84 pvestatd                                                                             
   2595 root      20   0    0,7g   0,1g   0,0g S   0,7   0,0  12:42.69 pmxcfs                                                                               
   3435 www-data  20   0    0,1g   0,1g   0,0g S   0,0   0,0   0:03.87 spiceproxy                                                                           
   3804 statd     20   0    0,1g   0,0g   0,0g S   0,0   0,0   0:00.01 rpc.statd                                                                           
1955494 www-data  20   0    0,1g   0,0g   0,0g S   0,0   0,0   0:04.64 spiceproxy work                                                                     
   1455 root      20   0    0,1g   0,0g   0,0g S   0,0   0,0   0:06.79 systemd-journal                                                                     
4099739 root      20   0    0,0g   0,0g   0,0g S   0,0   0,0   0:00.06 systemd                                                                             
      1 root      20   0    0,2g   0,0g   0,0g S   0,0   0,0   0:23.75 systemd                                                                             
   1991 root      20   0    0,0g   0,0g   0,0g S   0,0   0,0   0:00.12 ceph-crash                                                                           
4132723 root      20   0    0,0g   0,0g   0,0g S   0,0   0,0   0:00.02 sshd                                                                                 
4099730 root      20   0    0,0g   0,0g   0,0g S   0,0   0,0   0:00.07 sshd

Code:
root@proxmox1:~# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c 
09:10:46     0     0      0     0    0     0    0     0    0   277M  381M
Code:
root@proxmox1:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         128803       82947       40431          88        5424       44735


A vm using 48GB and one using 8GB. 4 osd's using 1 gb each,
So roughly it should be 48+8+1+1+1+1+0.5(zfs)+2gb(system) =62GB, so i am missing 20GB

This calculation checks out on all other 3 servers.
 
Hi,
it might be the kernel that (wrongly) uses too much memory. There are other users experiencing similar issues, see this thread on the PVE-User mailing list.
 
It was a 2012r2 server, we disabled balooning, seems for work for now.
To free up the space we just migrated the VM to another host, then back again, and the the memory was back to normal :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!