Proxmox uses full swap and uses much more memory than allocated for VMs

[fcukinyahoo]

I have 9 KVMs running on Proxmox 3.2 and total of 5G of ram given to 9 KVMs. Proxmox server has 8GB of ram and 7GB of SWAP. For most of the time when i check summary of the server i see that it uses all 7.50Gb of ram and all 7GB of full SWAP. Well that is bad for multiple reasons. I can not run more servers even though I should have another 2GB of ram should be available for me.

So the question is where should I look, what should I fix to have better Ram utilization. I know that Proxmox 3.3 is now available, I read the release notes but I couldn't notice any improvement for Ram.

Would love to hear your suggestions. Thanks in advance.

 

[fcukinyahoo]

any idea on this memory leak?

 

[udo]

any idea on this memory leak?

Hi,
you are sure that you have an memory leak?

Look with "top" and press M to see the memory consumption.

Any additional software on the server?


Udo

 

[fcukinyahoo]

There is no additional software running on the server. Clean installation of 3.2 proxmox. Below is the output of "top". And I am certain that memory allocated by me for those 9 servers are not higher than 5GB. However, as I can see from "top" command they consume more Ram than allocated for them. Top output follows.

top - 22:02:20 up 124 days, 3:00, 1 user, load average: 0.06, 0.08, 0.02
Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.7 us, 6.0 sy, 0.0 ni, 89.9 id, 0.2 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem: 8031900 total, 7973108 used, 58792 free, 125324 buffers
KiB Swap: 7340024 total, 5889256 used, 1450768 free, 218876 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
371746 root 20 0 3065m 1.4g 1460 S 0.7 18.5 688:20.62 kvm
918199 root 20 0 3286m 1.3g 1432 S 0.7 17.4 798:21.21 kvm
641488 root 20 0 2911m 1.2g 1524 S 1.3 16.1 832:01.82 kvm
1001075 root 20 0 1715m 826m 1556 S 5.7 10.5 141:12.46 kvm
759636 root 20 0 1820m 678m 1428 S 0.0 8.7 84:37.50 kvm
568707 root 20 0 1055m 494m 1468 S 0.0 6.3 35:04.55 kvm
380461 root 20 0 1134m 427m 1428 S 0.0 5.5 39:08.18 kvm
247561 root 20 0 706m 332m 1460 S 0.3 4.2 23:04.91 kvm
247491 root 20 0 692m 330m 1544 S 0.0 4.2 7:17.34 kvm
396213 root 20 0 280m 29m 6220 S 0.0 0.4 0:04.40 pvedaemon worke
396280 root 20 0 281m 28m 6176 S 0.0 0.4 0:03.61 pvedaemon worke
396148 root 20 0 280m 27m 6052 S 0.0 0.4 0:04.26 pvedaemon worke
396406 www-data 20 0 280m 25m 4676 S 0.0 0.3 0:05.19 pveproxy worker
396616 www-data 20 0 278m 23m 4460 S 0.7 0.3 0:04.36 pveproxy worker
396898 www-data 20 0 278m 22m 4136 S 0.7 0.3 0:03.15 pveproxy worker
2183 root 20 0 331m 22m 21m S 0.0 0.3 34:16.73 pmxcfs
2404 root 20 0 190m 19m 3624 S 0.0 0.2 495:53.82 pvestatd
397684 root 20 0 71244 3720 2900 S 0.0 0.0 0:00.03 sshd
1898 root 10 -10 5452 3404 2240 S 0.0 0.0 9:04.53 iscsid
397690 root 20 0 20124 2936 1676 S 0.0 0.0 0:00.02 bash
2023 root 20 0 11188 2712 600 S 0.0 0.0 12:28.41 ksmtuned
2018 root 20 0 122m 2608 872 S 0.0 0.0 0:03.44 rsyslogd
397759 postfix 20 0 39896 2376 1836 S 0.0 0.0 0:00.00 pickup
2409 www-data 20 0 269m 2324 856 S 0.0 0.0 2:05.03 pveproxy
2380 root 20 0 260m 2112 864 S 0.0 0.0 1:55.17 pvedaemon
2433 www-data 20 0 269m 1756 736 S 0.0 0.0 6:18.62 spiceproxy work
397921 root 20 0 23244 1648 1168 R 0.0 0.0 0:00.03 top

 

[udo]

Hi,
how much memory do you have configured for you VMs?

grep memory /etc/pve/qemu-server/*.conf

And what kind of VM use much more RAM? I had one win2003-server with memory issues - the 32bit-vm has 4gb assigned but over 40 used. Stop/start solved the problem.

Udo

 

[fcukinyahoo]

Here is the give ram amount to each kvm. I have several of them so i dont think it is only one server's fault. None of them run at 100%, three of them usually average at 90 - 95 percent, and others are usually around %70. After restarting the whole server, the problem doesnt happen for weeks or months but little by little swap space is getting full and server's physical Ram start being filled with something else. Then of course kvms start shutting down. Interesting thing is when it shutdown a kvm couple weeks ago, it released 2gb from swap and another 512m from physical. Here is the output.

# grep memory /etc/pve/qemu-server/*.conf
/etc/pve/qemu-server/100.conf:memory: 256
/etc/pve/qemu-server/101.conf:memory: 1024
/etc/pve/qemu-server/102.conf:memory: 256
/etc/pve/qemu-server/103.conf:memory: 256
/etc/pve/qemu-server/104.conf:memory: 512
/etc/pve/qemu-server/105.conf:memory: 1024
/etc/pve/qemu-server/106.conf:memory: 1024
/etc/pve/qemu-server/107.conf:memory: 512

 

[sumsum]

Good morning
I discovered as well some swapiness of servers which we upgraded to the latest proxmox release. Even if enough RAM is available. We have still some Servers with Proxmox 3.1 running, having a uptime for about one year. On those servers no swap is used at all.
Never the less. I found a script a while ago which show you which process is using how much swap space. Cant remember where i found this script. Here's the code :

#!/bin/bash
# Get current swap usage for all running processes
# Erik Ljungstrom 27/05/2011
SUM=0
OVERALL=0
for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do
PID=`echo $DIR | cut -d / -f 3`
PROGNAME=`ps -p $PID -o comm --no-headers`
for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
do
let SUM=$SUM+$SWAP
done
echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
let OVERALL=$OVERALL+$SUM
SUM=0

done
echo "Overall swap used: $OVERALL"

This script will can be executed with the following command :

./getswap.sh | sort -n -k 5

The Result looks something like this :

PID=4781 - Swap used: 424 - (smartd )
PID=4088 - Swap used: 448 - (rpc.statd )
PID=4441 - Swap used: 488 - (nmbd )
PID=4755 - Swap used: 488 - (sshd )
PID=774528 - Swap used: 492 - (udevd )
PID=793382 - Swap used: 492 - (udevd )
PID=561 - Swap used: 520 - (udevd )
PID=4450 - Swap used: 532 - (smbd )
PID=4474 - Swap used: 764 - (smbd )
PID=4765 - Swap used: 800 - (pmxcfs )
PID=4753 - Swap used: 1840 - (snmpd )
PID=5539 - Swap used: 3556 - (pve-firewall )
PID=320301 - Swap used: 4024 - (pvedaemon worke )
PID=759686 - Swap used: 4164 - (pvedaemon worke )
PID=760484 - Swap used: 4392 - (pvedaemon worke )
PID=168247 - Swap used: 5428 - (pvedaemon )

Hope this snipped will help some of you.
have a funky day
tom

 

[fcukinyahoo]

This is excellent Tom, thanks a lot. I m sure it will be helpful to many of us experiencing similar problem. However, I have already upgraded the server to the latest 3.3 and I have not experienced the same issue since the upgrade. It has been 4 days since upgrade and memory usage and allocation proportions seems correct. Also, only 138kb swap is being used which also makes sense. I will keep watching the server if same issue happens again then will post the output of the script here. Thanks.