KVM - Swap Question

[BitRausch]

Hi @all,

For a couple of days I am observing a certain behavior which I would like to share an get some help on it from you to figure out what is causing it:

I am running 3 KVM's (1x Linux, 1x SBS 2008, 1 Windows 7). When I start the system my swapfile starts with 64MB in use. After a week or so it reaches 1 GB. After a a weekly Backup the Swap-Size increases to 1.7 GB. The problem I have is that the responsiveness of my Windows machines starts to decrease. Everything is still running but it looks to me that the the VM's using swap memory instead of RAM. When I shutdown the Windows VM's and restart again swapfile starts at 64 Mb again and the VM's are working fast again.
Even if I only run the Linux VM and the SBS 2008 VM swapsize starts to increase and eventually responsivness of the SBS Machine goes down.

After doing some research I changed the vm.swapiness from 60 to 40 which actually reduced the swap increase a little but it does not stop.

So my question is, since I am not very linux savvy, where can I see in more detail what process/program/vm is causing the swapfile to increase?
What can I do to stop swapfile to increase?
If this is a normal behavior - how can I optimize the speed/responsiveness of the Windows VM's?

Any help or suggestions are welcome.
If you need more information please let me know.

Kind regards
B

System:
CPU: Intel Q9400
RAM 8 GB (shows at 7.8 GB)
Swap partition: 7 GB
Raid 10 - 1 TB

Memory allocated to KVM's:
Linux - 1024 MB
SBS 2008 - 5120
Windows 7 - 1024

Backup
Method: Snapshot, compressed

Fileformat of VM's : raw

pveversion -v

pve-manager: 1.5-8 (pve-manager/1.5/4674)
running kernel: 2.6.32-1-pve
proxmox-ve-2.6.32: 1.5-4
pve-kernel-2.6.24-8-pve: 2.6.24-16
pve-kernel-2.6.32-1-pve: 2.6.32-4
qemu-server: 1.1-11
pve-firmware: 1.0-3
libpve-storage-perl: 1.0-10
vncterm: 0.9-2
vzctl: 3.0.23-1pve8
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.11.1-2
ksm-control-daemon: 1.0-3

pveperf

CPU BOGOMIPS:      21333.13
REGEX/SECOND:      793061
HD SIZE:           49.22 GB (/dev/mapper/pve-root)
BUFFERED READS:    188.98 MB/sec
AVERAGE SEEK TIME: 7.94 ms
FSYNCS/SECOND:     2836.63
DNS EXT:           1085.73 ms
DNS INT:           1047.83 ms (local)

current free (after I restarted it on Saturday/Sunday)

             total       used       free     shared    buffers     cached
Mem:       8179160    5687548    2491612          0      65148      12904
-/+ buffers/cache:    5609496    2569664
Swap:      7340024     556604    6783420

This is how it looks as of now:

 

[BitRausch]

Hi @All,

The increasing Swapsize is starting to realy impact VM performance.
After monitoring it for 5 days swapsize increased to over 2 GB even so memory usage is at only half the available memory.
I was wondering if I am the only one who has this problem?
Can that be related to KSM?


current free:

                                  total       used       free     shared    buffers     cached
Mem:                       8179160    8106436      72724          0       9312    4065688
-/+ buffers/cache:     4031436    4147724
Swap:                      7340024    2454696    4885328

I have attached a screenshot as of today.

Any help on this would be great.

Kind regards

 

[l.mierzwa]

If You got some 64MB swap usage right after boot up then You are low on ram, that's all. Buy more or decrease memory alocation for Your virtual machines. You got 1024+5120+1024=7148MB out of 8GB allocated to virtual machines (don't forget that virtual machine can eat a little more that the the memory size allocated to it) and You got proxmox host machine disk cache fighting with Your virtual machines. You could try to lower swappines on Your proxmox host (echo 0 > /proc/sys/vm/swappiness) but I don't expect it to help much. Maybe settings cache=none on virtual machine disks would also help as promox host would not cache writes.

 

[BitRausch]

Hi I.mierzwa,

Thank you very much for your reply. Unfortunately the board is already maxed out with ram. I changed swappiness from default 60 to 40 which, as you already mentioned, did help a little but not much. I did not try to set it lower so. Regarding your last suggestion: Where can I find more information about the cache parameter? I am currently using the RAW format. I would like to switch to VIRTIO but I am not sure if this is stable with SBS 2008 and Windows 7 VM's. Would I need the cache for VIRTIO ? But I will also reduce the allocated RAM for the VM's. It is just a little confusing for me since the same VM's worked on the 2.6.24 Kernel and I did not had this issue...

Kind regards
Kind regards

 

[l.mierzwa]

Regarding your last suggestion: Where can I find more information about the cache parameter?

run "man qm" (how to set it in proxmox) and "man kvm" (technical info), it should help if You got a lot of writes on Your virtual machines.

I am currently using the RAW format. I would like to switch to VIRTIO but I am not sure if this is stable with SBS 2008 and Windows 7 VM's. Would I need the cache for VIRTIO ? But I will also reduce the allocated RAM for the VM's. It is just a little confusing for me since the same VM's worked on the 2.6.24 Kernel and I did not had this issue...

Do You have any kind of host proxmox host performance monitoring like collectd or cacti? It would help You to see how memory and swap usage changes over time, maybe virtual machine is not doing much so proxmox kernel pushes it's data to swap as they are rarely used, then when access Your virtual machine it needs to fetch it's memory from swap. Virtual machine memory can also be swapped out during backup as it is I/O intensive and it quickly fills caches with data. Monitoring would anwser many of this questions.

 

[BitRausch]

Thx again!

Do You have any kind of host proxmox host performance monitoring like collectd or cacti?

Thats exactly what I was looking for. I will look into the monitoring. Have not used collectd or cacti before.

It would help You to see how memory and swap usage changes over time, maybe virtual machine is not doing much so proxmox kernel pushes it's data to swap as they are rarely used, then when access Your virtual machine it needs to fetch it's memory from swap.

That was my assumption too. The strange thing is the Web GUI shows me only half of the memory is used (see pic. in previous post but when I look at the free command it shows me a lot of ram used for cached in use).

In any case I will setup a better monitoring just to get more details as you suggested.

Kind regards

 

[dietmar]

The strange thing is the Web GUI shows me only half of the memory is used (see pic. in previous post but when I look at the free command it shows me a lot of ram used for cached in use).

That is not strange. 'Cached' memory is only used as cache (and can be used for other things immediately) - so you can count that as 'free' memory.