KVM machines very slow /unreachable during vmtar backup

michaeljk · Nov 20, 2009

We have configured 9 KVM images on a i7 server (2.67GHz, 12 GB RAM, 1,5TB Hardware RAID1: 3ware Inc 9650SE SATA-II RAID PCIe), they are running very good. Unfortunatly, the virtual machines are nearly unreachable if we try to make snapshot-backups. We are using a nfs-mount over a gigabit network-link, so bandwith is no problem. An example for a small KVM-image:

Nov 20 01:13:34 INFO: Total bytes written: 8826415616 (31.17 MiB/s)
Nov 20 01:13:40 INFO: archive file size: 8.22GB

When the backup starts, the load on the KVM-image increases and goes up to 21-25. The local system status on the proxmox-webpage displays IO delays of 30 to 50%, so I guess there must be a problem with the speed of the local filesystem. We've tested the backup on the local disk (instead of the nfs-share) too, but with the same results. The KVM becomes unresponsive during the backup and works normal after that. File-compression is disabled, we use raw-images as harddisks for the KVM's.

Any ideas how we can improve this?

dietmar · Nov 20, 2009

michaeljk said:
Any ideas how we can improve this?

Use a fast local disk setup, for example RAID 10 on fast disks.

What's your current disk speed? Please run 'pveperf' when there is no load on the server:

# pveperf

michaeljk · Nov 23, 2009

pveperf:

CPU BOGOMIPS: 42765.40
REGEX/SECOND: 963899
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 99.99 MB/sec
AVERAGE SEEK TIME: 14.11 ms
FSYNCS/SECOND: 642.22
DNS EXT: 166.88 ms
DNS INT: 7.78 ms

dietmar · Nov 23, 2009

michaeljk said:
BUFFERED READS: 99.99 MB/sec
AVERAGE SEEK TIME: 14.11 ms

Well, you HD is incredible slow - what kind of HD is that?

Oh - I assume its a normal SATA disk - should be good enough.

How large is the VM disk?

michaeljk · Nov 23, 2009

Code:

tw_cli /c4/p0 show all      
/c4/p0 Status = OK
/c4/p0 Model = ST31000528AS
/c4/p0 Firmware Version = CC35
/c4/p0 Spindle Speed = 7200 RPM
/c4/p0 Link Speed Supported = 1.5 Gbps and 3.0 Gbps
/c4/p0 Link Speed = 3.0 Gbps

Unfortunatly, the system wasn't completely idle at this time, perhaps I can make a fresh test some time later. The raw KVM-Images have different sizes from 10 GB up to 70 GB.

Elbandi · Apr 4, 2011

Same issue here. The hw is a HP DL160 G6 with 3 SAS 15K disk in raid5. SW: base debian lenny with drbd+lvm. The backup is mounted as a nfs share.

CPU BOGOMIPS: 72535.20
REGEX/SECOND: 858073
HD SIZE: 11.00 GB (/dev/cciss/c0d0p2)
BUFFERED READS: 187.79 MB/sec
AVERAGE SEEK TIME: 3.85 ms
FSYNCS/SECOND: 3047.92
DNS EXT: 56.68 ms
DNS INT: 17.06 ms

Load goes up to 40-70. and the memory usage too: http://forum.proxmox.com/attachment.php?attachmentid=459&d=1301932706&stc=1

dietmar · Apr 5, 2011

Elbandi said:
Load goes up to 40-70. http://forum.proxmox.com/attachment.php?attachmentid=459&d=1301932706&stc=1

How many VMs do you run (OpenVZ or KVM)?

Elbandi · Apr 5, 2011

dietmar said:
How many VMs do you run (OpenVZ or KVM)?

We use only KVM. ~10 guest / node. (2 hardy, others are lucid).

Hmm, we updated the system to 1.8, and now the load are only 10-20 when backuping. (normal load is 1-2)

dietmar · Apr 5, 2011

Your backup target is on NFS? If so, maybe the NFS server or the network is too slow - try to make a backup to local disk to verify that.

Elbandi · Apr 5, 2011

dietmar said:
Your backup target is on NFS? If so, maybe the NFS server or the network is too slow - try to make a backup to local disk to verify that.

1G interface both side, tested:

# time dd if=/dev/zero of=/mnt/pve/backup2/test.img bs=1M count=2k
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 22.613 s, 95.0 MB/s

real 0m22.616s
user 0m0.000s
sys 0m2.740s

udo · Apr 5, 2011

Elbandi said:
1G interface both side, tested:

Hi,
you measure mem cache... use with dd the flag "conv=fdatasync" to get real values.

Udo

Elbandi · Apr 5, 2011

udo said:
Hi,
you measure mem cache... use with dd the flag "conv=fdatasync" to get real values.

Udo

# time dd if=/dev/zero of=/mnt/pve/backup2/test.img bs=1M count=2k conv=fdatasync

Code:

2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 22.8933 s, 93.8 MB/s

real    0m22.907s
user    0m0.000s
sys     0m1.960s

# time dd if=/dev/zero of=/mnt/pve/backup2/test.img bs=1k count=2M conv=fdatasync

Code:

2097152+0 records in
2097152+0 records out
2147483648 bytes (2.1 GB) copied, 26.8243 s, 80.1 MB/s

real    0m27.703s
user    0m0.320s
sys     0m5.860s

nfs client:

Code:

ii  libnfsidmap2                        0.20-1                     An nfs idmapping library
ii  nfs-common                          1:1.1.2-6lenny2            NFS support files common to client and server

nfs server:

Code:

ii  libnfsidmap2                              0.23-2                            An nfs idmapping library
ii  unfs3                                     0.9.22+dfsg-2                     User-space NFSv3 Server

I try a test backup to local disk.

obrienmd · Nov 17, 2011

We are having this issue as well, but backup to a local disk.

obrienmd · Nov 28, 2011

OK, after a little more investigation - it seems that the local disk is maxed out at 100-120MB/s write and 100% activity (via iostat), while the array the lvm volume is on is at a similar read speed with 30-50% activity (it's much faster than the backup target disk). My question is - why would this make our VMs so slow? The array is only 30-50% active, and only the backup disk is really being maxed-out, speed wise.

obrienmd · Nov 28, 2011

I've been watching our enviro (2.6.32-6 newest, core i7-870 (4x cores w/ HT, 8x 'cores' total) and 16GB DDR3, Adaptec HW raid w/ BBU) live tonight, here's some more info:
-System load is low and i/o performance (tested w/ basic hdparm) is good on guest VM being backed up, and guests not being backed up
-System load on host is fairly high (hovers around 3/3/3) on the VM host, but htop shows plenty of cpu idle (not sure how HT plays into this)
-Any service on any system 'reacts' significantly more slowly than it should (timeouts of served web pages, slow sql queries, etc.)

obrienmd · Nov 28, 2011

More: getting lots of these on guests while backup is running (whether or not on the current guest):
[698496.068018] BUG: soft lockup - CPU#3 stuck for 67s! [zabbix_agentd:1092]
[698535.653750] BUG: soft lockup - CPU#0 stuck for 138s! [mysqld:2176]

obrienmd · Nov 28, 2011

Also possibly useful: during backup, host is seeing 30-40k context switches per second. Will report back when backup is complete to note what it 'settles down' to.

obrienmd · Nov 28, 2011

Context switches are 30-35k per second when backup has completed, so not much difference there.

gkovacs · Nov 28, 2011

You should try to change the IO scheduler to deadline or noop, and test for a couple of days.

The default CFQ scheduler is performing very badly with HW RAID (especially noticeable during backups) because it has no idea of the block device layout, so it reorders requests in a suboptimal way. At the same time the RAID controller tries to do the same, but it has no idea of the kernel queue, so basically two realtime IO governors are working against each other. The deadline and noop schedulers are much simpler, so they basically let the RAID controller do the majority of the work, causing much less harm.

It's possible on the fly with one command (reboot will clear this):

Code:

cat >/sys/block/sda/queue/scheduler
deadline

After that a quick check should confirm you are running the selected scheduler:

Code:

cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq

atran · Nov 28, 2011

I also have this issue!

My volumes that contains the VM's already using deadline (btw vzdump uses ionice priority: 7 which would not work with Deadline scheduler).
Also during backup the net traffic used was minimal (256KB/s)(i used "bwm-ng" to measure my NIC: "bwm-ng -I eth0 -T avg -A 10 -d").
other older Proxmox (1.7/1.8) hosts don't have this problem.

the problem host:

pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-6-pve
proxmox-ve-2.6.32: 1.9-50
pve-kernel-2.6.32-4-pve: 2.6.32-33
pve-kernel-2.6.32-6-pve: 2.6.32-50
qemu-server: 1.1-32
pve-firmware: 1.0-14
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-1
ksm-control-daemon: 1.0-6

the backup took so long (almost 2 days) that i eventually disable the NAS so that backup would failed (is there any better way to cancel a running backup?)

here the log:

114: Nov 26 23:53:16 INFO: Starting Backup of VM 114 (qemu)
114: Nov 26 23:53:16 INFO: running
114: Nov 26 23:53:16 INFO: status = running
114: Nov 26 23:53:17 INFO: backup mode: snapshot
114: Nov 26 23:53:17 INFO: ionice priority: 7
114: Nov 26 23:53:17 INFO: Logical volume "vzsnap-kvm-host3-0" created
114: Nov 26 23:53:17 INFO: creating archive '/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.tgz'
114: Nov 26 23:53:17 INFO: adding '/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.tmp/qemu-server.conf' to archive ('qemu-server.conf')
114: Nov 26 23:53:17 INFO: adding '/dev/pve/vzsnap-kvm-host3-0' to archive ('vm-disk-virtio0.raw')
114: Nov 28 19:10:02 INFO: gzip: stdout: Input/output error
114: Nov 28 19:10:02 INFO: received signal - terminate process
114: Nov 28 19:10:04 INFO: Logical volume "vzsnap-kvm-host3-0" successfully removed
114: Nov 28 20:35:05 ERROR: Backup of VM 114 failed - command '/usr/lib/qemu-server/vmtar '/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.tmp/qemu-server.conf' 'qemu-server.conf' '/dev/pve/vzsnap-kvm-host3-0' 'vm-disk-virtio0.raw'|gzip - >/mnt/pve/LiveNAS-NFS/vzdump-qemu-114-2011_11_26-23_53_16.dat' failed with exit code 1

hopefully someone tracked down the cause of this IO problem!?

KVM machines very slow /unreachable during vmtar backup

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Renowned Member

Attachments

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Distinguished Member

Renowned Member

Member

Member

Member

Member

Member

Member

Renowned Member

Member