Proxmox uses full swap and uses much more memory than allocated for VMs

Discussion in 'Proxmox VE: Installation and configuration' started by fcukinyahoo, Oct 29, 2014.

  1. fcukinyahoo

    fcukinyahoo New Member

    Joined:
    Nov 29, 2012
    Messages:
    27
    Likes Received:
    0
    I have 9 KVMs running on Proxmox 3.2 and total of 5G of ram given to 9 KVMs. Proxmox server has 8GB of ram and 7GB of SWAP. For most of the time when i check summary of the server i see that it uses all 7.50Gb of ram and all 7GB of full SWAP. Well that is bad for multiple reasons. I can not run more servers even though I should have another 2GB of ram should be available for me.

    So the question is where should I look, what should I fix to have better Ram utilization. I know that Proxmox 3.3 is now available, I read the release notes but I couldn't notice any improvement for Ram.

    Would love to hear your suggestions. Thanks in advance.
     
  2. fcukinyahoo

    fcukinyahoo New Member

    Joined:
    Nov 29, 2012
    Messages:
    27
    Likes Received:
    0
    any idea on this memory leak?
     
  3. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,835
    Likes Received:
    159
    Hi,
    you are sure that you have an memory leak?

    Look with "top" and press M to see the memory consumption.

    Any additional software on the server?


    Udo
     
  4. fcukinyahoo

    fcukinyahoo New Member

    Joined:
    Nov 29, 2012
    Messages:
    27
    Likes Received:
    0
    There is no additional software running on the server. Clean installation of 3.2 proxmox. Below is the output of "top". And I am certain that memory allocated by me for those 9 servers are not higher than 5GB. However, as I can see from "top" command they consume more Ram than allocated for them. Top output follows.

    top - 22:02:20 up 124 days, 3:00, 1 user, load average: 0.06, 0.08, 0.02
    Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie
    %Cpu(s): 3.7 us, 6.0 sy, 0.0 ni, 89.9 id, 0.2 wa, 0.0 hi, 0.2 si, 0.0 st
    KiB Mem: 8031900 total, 7973108 used, 58792 free, 125324 buffers
    KiB Swap: 7340024 total, 5889256 used, 1450768 free, 218876 cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    371746 root 20 0 3065m 1.4g 1460 S 0.7 18.5 688:20.62 kvm
    918199 root 20 0 3286m 1.3g 1432 S 0.7 17.4 798:21.21 kvm
    641488 root 20 0 2911m 1.2g 1524 S 1.3 16.1 832:01.82 kvm
    1001075 root 20 0 1715m 826m 1556 S 5.7 10.5 141:12.46 kvm
    759636 root 20 0 1820m 678m 1428 S 0.0 8.7 84:37.50 kvm
    568707 root 20 0 1055m 494m 1468 S 0.0 6.3 35:04.55 kvm
    380461 root 20 0 1134m 427m 1428 S 0.0 5.5 39:08.18 kvm
    247561 root 20 0 706m 332m 1460 S 0.3 4.2 23:04.91 kvm
    247491 root 20 0 692m 330m 1544 S 0.0 4.2 7:17.34 kvm
    396213 root 20 0 280m 29m 6220 S 0.0 0.4 0:04.40 pvedaemon worke
    396280 root 20 0 281m 28m 6176 S 0.0 0.4 0:03.61 pvedaemon worke
    396148 root 20 0 280m 27m 6052 S 0.0 0.4 0:04.26 pvedaemon worke
    396406 www-data 20 0 280m 25m 4676 S 0.0 0.3 0:05.19 pveproxy worker
    396616 www-data 20 0 278m 23m 4460 S 0.7 0.3 0:04.36 pveproxy worker
    396898 www-data 20 0 278m 22m 4136 S 0.7 0.3 0:03.15 pveproxy worker
    2183 root 20 0 331m 22m 21m S 0.0 0.3 34:16.73 pmxcfs
    2404 root 20 0 190m 19m 3624 S 0.0 0.2 495:53.82 pvestatd
    397684 root 20 0 71244 3720 2900 S 0.0 0.0 0:00.03 sshd
    1898 root 10 -10 5452 3404 2240 S 0.0 0.0 9:04.53 iscsid
    397690 root 20 0 20124 2936 1676 S 0.0 0.0 0:00.02 bash
    2023 root 20 0 11188 2712 600 S 0.0 0.0 12:28.41 ksmtuned
    2018 root 20 0 122m 2608 872 S 0.0 0.0 0:03.44 rsyslogd
    397759 postfix 20 0 39896 2376 1836 S 0.0 0.0 0:00.00 pickup
    2409 www-data 20 0 269m 2324 856 S 0.0 0.0 2:05.03 pveproxy
    2380 root 20 0 260m 2112 864 S 0.0 0.0 1:55.17 pvedaemon
    2433 www-data 20 0 269m 1756 736 S 0.0 0.0 6:18.62 spiceproxy work
    397921 root 20 0 23244 1648 1168 R 0.0 0.0 0:00.03 top
     
  5. udo

    udo Well-Known Member
    Proxmox Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,835
    Likes Received:
    159
    Hi,
    how much memory do you have configured for you VMs?
    Code:
    grep memory /etc/pve/qemu-server/*.conf
    
    And what kind of VM use much more RAM? I had one win2003-server with memory issues - the 32bit-vm has 4gb assigned but over 40 used. Stop/start solved the problem.

    Udo
     
  6. fcukinyahoo

    fcukinyahoo New Member

    Joined:
    Nov 29, 2012
    Messages:
    27
    Likes Received:
    0
    Here is the give ram amount to each kvm. I have several of them so i dont think it is only one server's fault. None of them run at 100%, three of them usually average at 90 - 95 percent, and others are usually around %70. After restarting the whole server, the problem doesnt happen for weeks or months but little by little swap space is getting full and server's physical Ram start being filled with something else. Then of course kvms start shutting down. Interesting thing is when it shutdown a kvm couple weeks ago, it released 2gb from swap and another 512m from physical. Here is the output.

    # grep memory /etc/pve/qemu-server/*.conf
    /etc/pve/qemu-server/100.conf:memory: 256
    /etc/pve/qemu-server/101.conf:memory: 1024
    /etc/pve/qemu-server/102.conf:memory: 256
    /etc/pve/qemu-server/103.conf:memory: 256
    /etc/pve/qemu-server/104.conf:memory: 512
    /etc/pve/qemu-server/105.conf:memory: 1024
    /etc/pve/qemu-server/106.conf:memory: 1024
    /etc/pve/qemu-server/107.conf:memory: 512
     
  7. sumsum

    sumsum Member
    Proxmox Subscriber

    Joined:
    Oct 26, 2009
    Messages:
    157
    Likes Received:
    2
    Good morning
    I discovered as well some swapiness of servers which we upgraded to the latest proxmox release. Even if enough RAM is available. We have still some Servers with Proxmox 3.1 running, having a uptime for about one year. On those servers no swap is used at all.
    Never the less. I found a script a while ago which show you which process is using how much swap space. Cant remember where i found this script. Here's the code :
    Code:
    #!/bin/bash
    # Get current swap usage for all running processes
    # Erik Ljungstrom 27/05/2011
    SUM=0
    OVERALL=0
    for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` ; do
    PID=`echo $DIR | cut -d / -f 3`
    PROGNAME=`ps -p $PID -o comm --no-headers`
    for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
    do
    let SUM=$SUM+$SWAP
    done
    echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
    let OVERALL=$OVERALL+$SUM
    SUM=0
    
    done
    echo "Overall swap used: $OVERALL"
    This script will can be executed with the following command :

    Code:
    ./getswap.sh | sort -n -k 5
    The Result looks something like this :
    Code:
    PID=4781 - Swap used: 424 - (smartd )
    PID=4088 - Swap used: 448 - (rpc.statd )
    PID=4441 - Swap used: 488 - (nmbd )
    PID=4755 - Swap used: 488 - (sshd )
    PID=774528 - Swap used: 492 - (udevd )
    PID=793382 - Swap used: 492 - (udevd )
    PID=561 - Swap used: 520 - (udevd )
    PID=4450 - Swap used: 532 - (smbd )
    PID=4474 - Swap used: 764 - (smbd )
    PID=4765 - Swap used: 800 - (pmxcfs )
    PID=4753 - Swap used: 1840 - (snmpd )
    PID=5539 - Swap used: 3556 - (pve-firewall )
    PID=320301 - Swap used: 4024 - (pvedaemon worke )
    PID=759686 - Swap used: 4164 - (pvedaemon worke )
    PID=760484 - Swap used: 4392 - (pvedaemon worke )
    PID=168247 - Swap used: 5428 - (pvedaemon )


    Hope this snipped will help some of you.
    have a funky day
    tom
     
    chrone likes this.
  8. fcukinyahoo

    fcukinyahoo New Member

    Joined:
    Nov 29, 2012
    Messages:
    27
    Likes Received:
    0
    This is excellent Tom, thanks a lot. I m sure it will be helpful to many of us experiencing similar problem. However, I have already upgraded the server to the latest 3.3 and I have not experienced the same issue since the upgrade. It has been 4 days since upgrade and memory usage and allocation proportions seems correct. Also, only 138kb swap is being used which also makes sense. I will keep watching the server if same issue happens again then will post the output of the script here. Thanks.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice