Search results

Looks like there is a regression of some sort, perhaps between LVM drivers?

Come to think of it, we never had this problem w/ 2.6.35 either. Someone w/ 2.6.35: Can you post the output of lvs --version? For 2.6.32-6-53, it's: LVM version: 2.02.39 (2008-06-27) Library version: 1.02.27 (2008-06-25) Driver version: 4.20.6 Apparently, this is a pretty new driver...

We have tested w/ cfq, noop actually improves the situation over cfq, so ionice on the source at least doesn't seem to be causing the issue.

Unlike the other user (atran), I don't have enough data / test machines to perfectly correlate to a specific version. All of our machines showing major load during backup are running newest pvestest kernel 2.6.32 built Nov 24.

Tom - Do you know what changes were made in the script? Or is it perhaps a kernel changed that caused this? Many of us love lvm snapshot backups, but the performance regression has been huge.

Or, perhaps, do any of the Proxmox folks have an idea of what may have changed between these versions to cause load issues?

VERY interesting result from last night: On a hunch, rather than moving bwlimit down to 10000, I changed our size parameter (the maximum snapshot diff size) from 32768 to 8192. We had increased the size parameter a while ago because we were hitting the max of default on some VMs that used to be...

Also, if 10K improves it, the situation will still be quite poor, as backups will take a very, very long time. If scaling down bwlimit further helps, should we expect a faster backup target disk to allow us to bring bwlimit back up?

proxmox-ve-2.6.32: 1.9-53 I changed /sys/block/sda/queue/scheduler to noop, which seemed to help system performance a bit in general. Elevator for other drives is cfq, as they are single SATA (boot/pve and backup) drives. I will change backup speed to 10k and check resulting system load...

Here you go: CPU BOGOMIPS: 44756.82 REGEX/SECOND: 862539 HD SIZE: 98.43 GB (/dev/mapper/array-test) BUFFERED READS: 464.61 MB/sec AVERAGE SEEK TIME: 9.18 ms FSYNCS/SECOND: 3595.76 DNS EXT: 48.39 ms DNS INT: 41.49 ms (praece.com)

This is interesting: http://www.nikhef.nl/~dennisvd/lvmcrap.html

Our VMs are on lvm, on Areca 1222 w/ raid cache enabled and BBU, 8x 1TB SAS drives. It's quite fast. Is there a way to run pveperf on a physical volume? We have no mounted file systems from this storage on PVE host.

pveperf on PVE root (on SATA SSD): CPU BOGOMIPS: 44756.82 REGEX/SECOND: 772976 HD SIZE: 7.14 GB (/dev/mapper/pve-root) BUFFERED READS: 141.48 MB/sec AVERAGE SEEK TIME: 0.25 ms FSYNCS/SECOND: 149.38 DNS EXT: 53.63 ms DNS INT: 42.97 ms (praece.com)...

OK, last night (w/ 25000 bwlimit) was much better - machines were generally reachable during entire backup. However, system load was still extremely high at points during backup (with corresponding high load on guest VMs - backup starts at 23:00): VZdump logs: https://gist.github.com/1417896...

Interesting. I see i/o on devices much higher than that throughout the process using iostat -xk 1... At any rate, I'll give the bwlimit of 25000 a shot and post back here.

I see that. However, I'm not sure it's actually affecting anything for these kvm/lvm snapshot images, I have it set to 65536 and it didn't seem to limit i/o on any device... Regardless, I have made the change to 25000 and will report back after tonight's backup.

What does the 'limit' represent? I assumed it was KB/s. I have set it to 25000 and will report back after tonight's backup attempt.

pve-manager: 1.9-26 (pve-manager/1.9/6567) running kernel: 2.6.32-6-pve proxmox-ve-2.6.32: 1.9-53 pve-kernel-2.6.24-9-pve: 2.6.24-18 pve-kernel-2.6.24-8-pve: 2.6.24-16 pve-kernel-2.6.32-6-pve: 2.6.32-53 qemu-server: 1.1-32 pve-firmware: 1.0-14 libpve-storage-perl: 1.0-19 vncterm: 0.9-2 vzctl...

These are all kvm vms - You can see that I have the 'bandwidth limit' in vzdump at 65536 KB/s in vzdump.conf, but I'm not sure what that actually does, as i/o rates on the source and backup drives go _far_ above that during backup.

More info - The issue started around midnight (00:00) tonight, right in the middle of the backup for VM 107. Here are the logs for backups of VM 107 and 108: