KVM machines very slow /unreachable during vmtar backup

michaeljk

Active Member
Oct 7, 2009
58
2
28
We have configured 9 KVM images on a i7 server (2.67GHz, 12 GB RAM, 1,5TB Hardware RAID1: 3ware Inc 9650SE SATA-II RAID PCIe), they are running very good. Unfortunatly, the virtual machines are nearly unreachable if we try to make snapshot-backups. We are using a nfs-mount over a gigabit network-link, so bandwith is no problem. An example for a small KVM-image:

Nov 20 01:13:34 INFO: Total bytes written: 8826415616 (31.17 MiB/s)
Nov 20 01:13:40 INFO: archive file size: 8.22GB

When the backup starts, the load on the KVM-image increases and goes up to 21-25. The local system status on the proxmox-webpage displays IO delays of 30 to 50%, so I guess there must be a problem with the speed of the local filesystem. We've tested the backup on the local disk (instead of the nfs-share) too, but with the same results. The KVM becomes unresponsive during the backup and works normal after that. File-compression is disabled, we use raw-images as harddisks for the KVM's.

Any ideas how we can improve this?
 

michaeljk

Active Member
Oct 7, 2009
58
2
28
pveperf:

CPU BOGOMIPS: 42765.40
REGEX/SECOND: 963899
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 99.99 MB/sec
AVERAGE SEEK TIME: 14.11 ms
FSYNCS/SECOND: 642.22
DNS EXT: 166.88 ms
DNS INT: 7.78 ms
 

michaeljk

Active Member
Oct 7, 2009
58
2
28
Code:
tw_cli /c4/p0 show all      
/c4/p0 Status = OK
/c4/p0 Model = ST31000528AS
/c4/p0 Firmware Version = CC35
/c4/p0 Spindle Speed = 7200 RPM
/c4/p0 Link Speed Supported = 1.5 Gbps and 3.0 Gbps
/c4/p0 Link Speed = 3.0 Gbps

Unfortunatly, the system wasn't completely idle at this time, perhaps I can make a fresh test some time later. The raw KVM-Images have different sizes from 10 GB up to 70 GB.
 

Elbandi

Member
Apr 4, 2011
18
4
23
Same issue here. The hw is a HP DL160 G6 with 3 SAS 15K disk in raid5. SW: base debian lenny with drbd+lvm. The backup is mounted as a nfs share.

CPU BOGOMIPS: 72535.20
REGEX/SECOND: 858073
HD SIZE: 11.00 GB (/dev/cciss/c0d0p2)
BUFFERED READS: 187.79 MB/sec
AVERAGE SEEK TIME: 3.85 ms
FSYNCS/SECOND: 3047.92
DNS EXT: 56.68 ms
DNS INT: 17.06 ms

Load goes up to 40-70. and the memory usage too: http://forum.proxmox.com/attachment.php?attachmentid=459&d=1301932706&stc=1
 

Attachments

  • memory-day.png
    memory-day.png
    29.5 KB · Views: 47

Elbandi

Member
Apr 4, 2011
18
4
23
How many VMs do you run (OpenVZ or KVM)?
We use only KVM. ~10 guest / node. (2 hardy, others are lucid).

Hmm, we updated the system to 1.8, and now the load are only 10-20 when backuping. (normal load is 1-2)
 

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,884
429
103
Austria
www.proxmox.com
Your backup target is on NFS? If so, maybe the NFS server or the network is too slow - try to make a backup to local disk to verify that.
 

Elbandi

Member
Apr 4, 2011
18
4
23
Your backup target is on NFS? If so, maybe the NFS server or the network is too slow - try to make a backup to local disk to verify that.
1G interface both side, tested:
# time dd if=/dev/zero of=/mnt/pve/backup2/test.img bs=1M count=2k
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 22.613 s, 95.0 MB/s

real 0m22.616s
user 0m0.000s
sys 0m2.740s
 

Elbandi

Member
Apr 4, 2011
18
4
23
Hi,
you measure mem cache... use with dd the flag "conv=fdatasync" to get real values.


Udo

# time dd if=/dev/zero of=/mnt/pve/backup2/test.img bs=1M count=2k conv=fdatasync
Code:
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 22.8933 s, 93.8 MB/s

real    0m22.907s
user    0m0.000s
sys     0m1.960s

# time dd if=/dev/zero of=/mnt/pve/backup2/test.img bs=1k count=2M conv=fdatasync
Code:
2097152+0 records in
2097152+0 records out
2147483648 bytes (2.1 GB) copied, 26.8243 s, 80.1 MB/s

real    0m27.703s
user    0m0.320s
sys     0m5.860s

nfs client:
Code:
ii  libnfsidmap2                        0.20-1                     An nfs idmapping library
ii  nfs-common                          1:1.1.2-6lenny2            NFS support files common to client and server

nfs server:
Code:
ii  libnfsidmap2                              0.23-2                            An nfs idmapping library
ii  unfs3                                     0.9.22+dfsg-2                     User-space NFSv3 Server

I try a test backup to local disk.
 

obrienmd

Member
Oct 14, 2009
109
0
16
OK, after a little more investigation - it seems that the local disk is maxed out at 100-120MB/s write and 100% activity (via iostat), while the array the lvm volume is on is at a similar read speed with 30-50% activity (it's much faster than the backup target disk). My question is - why would this make our VMs so slow? The array is only 30-50% active, and only the backup disk is really being maxed-out, speed wise.
 

obrienmd

Member
Oct 14, 2009
109
0
16
I've been watching our enviro (2.6.32-6 newest, core i7-870 (4x cores w/ HT, 8x 'cores' total) and 16GB DDR3, Adaptec HW raid w/ BBU) live tonight, here's some more info:
-System load is low and i/o performance (tested w/ basic hdparm) is good on guest VM being backed up, and guests not being backed up
-System load on host is fairly high (hovers around 3/3/3) on the VM host, but htop shows plenty of cpu idle (not sure how HT plays into this)
-Any service on any system 'reacts' significantly more slowly than it should (timeouts of served web pages, slow sql queries, etc.)
 

obrienmd

Member
Oct 14, 2009
109
0
16
More: getting lots of these on guests while backup is running (whether or not on the current guest):
[698496.068018] BUG: soft lockup - CPU#3 stuck for 67s! [zabbix_agentd:1092]
[698535.653750] BUG: soft lockup - CPU#0 stuck for 138s! [mysqld:2176]
 

obrienmd

Member
Oct 14, 2009
109
0
16
Also possibly useful: during backup, host is seeing 30-40k context switches per second. Will report back when backup is complete to note what it 'settles down' to.
 

obrienmd

Member
Oct 14, 2009
109
0
16
Context switches are 30-35k per second when backup has completed, so not much difference there.
 

gkovacs

Well-Known Member
Dec 22, 2008
507
48
48
Budapest, Hungary
You should try to change the IO scheduler to deadline or noop, and test for a couple of days.

The default CFQ scheduler is performing very badly with HW RAID (especially noticeable during backups) because it has no idea of the block device layout, so it reorders requests in a suboptimal way. At the same time the RAID controller tries to do the same, but it has no idea of the kernel queue, so basically two realtime IO governors are working against each other. The deadline and noop schedulers are much simpler, so they basically let the RAID controller do the majority of the work, causing much less harm.

It's possible on the fly with one command (reboot will clear this):

Code:
cat >/sys/block/sda/queue/scheduler
deadline
After that a quick check should confirm you are running the selected scheduler:
Code:
cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq
 
Last edited:

atran

Member
Dec 9, 2010
37
0
6
I also have this issue!

My volumes that contains the VM's already using deadline (btw vzdump uses ionice priority: 7 which would not work with Deadline scheduler).
Also during backup the net traffic used was minimal (256KB/s)(i used "bwm-ng" to measure my NIC: "bwm-ng -I eth0 -T avg -A 10 -d").
other older Proxmox (1.7/1.8) hosts don't have this problem.

the problem host:

pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-6-pve
proxmox-ve-2.6.32: 1.9-50
pve-kernel-2.6.32-4-pve: 2.6.32-33
pve-kernel-2.6.32-6-pve: 2.6.32-50
qemu-server: 1.1-32
pve-firmware: 1.0-14
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-1
ksm-control-daemon: 1.0-6

the backup took so long (almost 2 days) that i eventually disable the NAS so that backup would failed (is there any better way to cancel a running backup?)

here the log:

114: Nov 26 23:53:16 INFO: Starting Backup of VM 114 (qemu)
114: Nov 26 23:53:16 INFO: running
114: Nov 26 23:53:16 INFO: status = running
114: Nov 26 23:53:17 INFO: backup mode: snapshot
114: Nov 26 23:53:17 INFO: ionice priority: 7
114: Nov 26 23:53:17 INFO: Logical volume "vzsnap-kvm-host3-0" created
114: Nov 26 23:53:17 INFO: creating archive '/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.tgz'
114: Nov 26 23:53:17 INFO: adding '/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.tmp/qemu-server.conf' to archive ('qemu-server.conf')
114: Nov 26 23:53:17 INFO: adding '/dev/pve/vzsnap-kvm-host3-0' to archive ('vm-disk-virtio0.raw')
114: Nov 28 19:10:02 INFO: gzip: stdout: Input/output error
114: Nov 28 19:10:02 INFO: received signal - terminate process
114: Nov 28 19:10:04 INFO: Logical volume "vzsnap-kvm-host3-0" successfully removed
114: Nov 28 20:35:05 ERROR: Backup of VM 114 failed - command '/usr/lib/qemu-server/vmtar '/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.tmp/qemu-server.conf' 'qemu-server.conf' '/dev/pve/vzsnap-kvm-host3-0' 'vm-disk-virtio0.raw'|gzip - >/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.dat' failed with exit code 1

hopefully someone tracked down the cause of this IO problem!?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!