High RAM usage in Proxmox 4.4

eth · Feb 23, 2017

I have 32 gb of memory which I shared in two CT's:
1. 2gb of RAM - Centos 6 for haproxy and nginx.
2. 29 gb of RAM - Centos 7 for Percona Cluster database.

I've noticed high swapping on 29 gb VM in decided to stop it. And then I completely removed it from the system.

When I SSH'ed into the Proxmox host, it showed me that 12 GB of RAM are still used somewhere.

Code:

root@pve:~# free -mh
             total       used       free     shared    buffers     cached
Mem:           31G        12G        18G        91M        36M       421M
-/+ buffers/cache:        12G        19G
Swap:         8.0G       842M       7.2G

It's impossible, because the only virtual machine that is left has a 2GB RAM limit, and uses probably only 700 MB of it.

I ran a bash script to count total RAM usage from all processes and it came to 833 Megabytes:

Code:

ps aux | awk '{sum+=$6} END {print sum / 1024}'
833.605

So I'm wondering, where did other 11 Gigabytes go to? Is there a leak?

Ashley · Feb 23, 2017

Are you using ZFS?

eth · Feb 23, 2017

No, I don't use ZFS. I've attached my ps.

eth · Feb 23, 2017

Dropping caches didn't help.

Code:

root@pve:~# echo 1 > /proc/sys/vm/drop_caches
root@pve:~# free -mh
             total       used       free     shared    buffers     cached
Mem:           31G        12G        19G        91M       828K       158M
-/+ buffers/cache:        12G        19G
Swap:         8.0G       834M       7.2G
root@pve:~# echo 2 > /proc/sys/vm/drop_caches
root@pve:~# free -mh
             total       used       free     shared    buffers     cached
Mem:           31G        12G        19G        91M       1.3M       154M
-/+ buffers/cache:        11G        19G
Swap:         8.0G       834M       7.2G
root@pve:~# echo 3 > /proc/sys/vm/drop_caches
root@pve:~# free -mh
             total       used       free     shared    buffers     cached
Mem:           31G        12G        19G        91M       1.6M       139M
-/+ buffers/cache:        11G        19G
Swap:         8.0G       834M       7.2G

Ashley · Feb 23, 2017

Don't have any ram/tmpfs using alot of storage?

What does df -h show?

eth · Feb 23, 2017

df -h doesn't show anything unusual.

Code:

root@pve:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs           6.3G  8.8M  6.3G   1% /run
/dev/dm-0       583G   41G  518G   8% /
tmpfs            16G   43M   16G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/fuse        30M   20K   30M   1% /etc/pve

eth · Feb 23, 2017

Here is my top:

Code:

root@pve:/run# top -d1
top - 02:15:02 up 3 days, 21:24,  2 users,  load average: 0.24, 0.15, 0.11
Tasks: 378 total,   1 running, 377 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.1 us,  4.2 sy,  0.0 ni, 92.2 id,  0.9 wa,  0.0 hi,  0.6 si,  0.0 st
KiB Mem:  32946120 total, 13166684 used, 19779436 free,    16008 buffers
KiB Swap:  8388604 total,   834664 used,  7553940 free.   368448 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20349 501       20   0  151908  19580  11736 S  16.8  0.1   0:00.17 php
 6573 root      20   0  148060  15088  11420 S   1.0  0.0   0:00.64 php
10922 99        20   0   13100    788    636 S   1.0  0.0   0:45.61 dnsmasq
18747 501       20   0   56052   3328   1824 S   1.0  0.0   0:00.16 nginx
    1 root      20   0   30168   4120   2168 S   0.0  0.0   0:09.49 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.25 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   1:25.46 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   9:13.22 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:05.41 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.48 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.50 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:05.43 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   1:24.74 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      rt   0       0      0      0 S   0.0  0.0   0:00.46 watchdog/2
   17 root      rt   0       0      0      0 S   0.0  0.0   0:05.41 migration/2
   18 root      20   0       0      0      0 S   0.0  0.0   1:19.72 ksoftirqd/2
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
   21 root      rt   0       0      0      0 S   0.0  0.0   0:00.42 watchdog/3
   22 root      rt   0       0      0      0 S   0.0  0.0   0:05.45 migration/3
   23 root      20   0       0      0      0 S   0.0  0.0   1:19.33 ksoftirqd/3
   25 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
   26 root      rt   0       0      0      0 S   0.0  0.0   0:00.70 watchdog/4
   27 root      rt   0       0      0      0 S   0.0  0.0   0:05.51 migration/4
   28 root      20   0       0      0      0 S   0.0  0.0   1:31.40 ksoftirqd/4

eth · Feb 23, 2017

Here is someone having the same issue in Ubuntu and simply went for a kernel upgrade:
http://superuser.com/questions/7931...on-linux-ubuntu-server-not-disk-cache-buffers

Ashley · Feb 23, 2017

What kernel version are you running?

eth · Feb 23, 2017

4.4.35-1 (that comes in the latest install package) but I am now doing a dist-upgrade to 4.4.40-1 through the pve-no-subscription repo.

eth · Feb 23, 2017

Just rebooted into 4.4.40-1 and looks like I'm hitting the same error:

Code:

root@pve:~# free -mh
             total       used       free     shared    buffers     cached
Mem:           31G        30G       896M        52M        93M        16G
-/+ buffers/cache:        13G        17G
Swap:         8.0G       8.2M       8.0G
root@pve:~# ps aux | awk '{sum+=$6} END {print sum / 1024}'
3152.41

Ashley · Feb 23, 2017

What happens if you power down the other VM?

eth · Feb 23, 2017

Not much. Problem still there.

Code:

root@pve:~# free -mh
             total       used       free     shared    buffers     cached
Mem:           31G        31G       228M        52M       114M        17G
-/+ buffers/cache:        14G        17G
Swap:         8.0G        15M       8.0G
root@pve:~# pct shutdown 101
root@pve:~# free -mh
             total       used       free     shared    buffers     cached
Mem:           31G        31G       226M        51M       109M        17G
-/+ buffers/cache:        13G        17G
Swap:         8.0G       120K       8.0G

eth · Mar 21, 2017

I just wanted to follow up on this.
My problem was that I set the innodb_buffer_pool_size too high in my mysql settings which led to extreme swapping.
You should not worry about high RAM usage - it's normal.

gosha · Mar 22, 2017

Whom to believe?

1. free -mh:

2. WEB-GUI:

3. htop:

---
Best regards,
Gosha

fabian · Mar 22, 2017

the only difference is whether buffer and/or cache pages are counted as "free" or as "used", but all three "views" show the same memory situation.

gosha · Mar 22, 2017

Exactly!

And in the first post everything is fine too. The really used memory (not for buffer/cache) is about 1 GB.
And this confirmed by:
ps aux | awk '{sum+=$6} END {print sum / 1024}'

Search

Search

High RAM usage in Proxmox 4.4

eth

Well-Known Member

Ashley

Member

eth

Well-Known Member

Attachments

eth

Well-Known Member

Ashley

Member

eth

Well-Known Member

eth

Well-Known Member

eth

Well-Known Member

Ashley

Member

eth

Well-Known Member

eth

Well-Known Member

Ashley

Member

eth

Well-Known Member

eth

Well-Known Member

gosha

Well-Known Member

fabian

Proxmox Staff Member

gosha

Well-Known Member