It would really help if we have a reproducible test case. Maybe you can try to write a script to generate
many thousands of files in a folder, so that the bug triggers after running that script in a standard debian template?
[CRITICAL] Huge IO load causes freezing during backups
- Thread starter gkovacs
- Start date
It would really help if we have a reproducible test case. Maybe you can try to write a script to generate
many thousands of files in a folder, so that the bug triggers after running that script in a standard debian template?
The new kernel -108 did not help in my case, see also the following thread (which I will update as well) http://forum.proxmox.com/threads/9097-vzdump-using-lvm-snapshot-kills-the-box?p=78898#post78898
This particular server is a Dell R815 - See below for storage specs. PERC H700, SAS - Proxmox 1.9 worked fine, we did a clean upgrade from 1.9 to 3.0.x (complete system wipe and reinstall)
I attempted a manual backup of one OpenVZ container last night, which failed locking the entire system up. I have tried OpenVZ containers and KVM virtual machines, there is no rhyme or reason to this problem. Log output below and version info. Looks like it completes one backup, the files are there (on a brand new NFS backup system I turned up yesterday) and then it just craps out. Kernel logs are visible here: http://pastebin.com/UuiJ9Yz0
We have the same issue on a few brand new Dell R620s with the Perc H310 controller. Brand new clean installs of Prox 3.0.x - with only OpenVZ containers and local storage for backups. I have tried ext3 / ext4 and NFS mounts for vzdump backups, with the same results. I have also tried both deadline and CFQ schedulers as recommended.
I am happy to keep testing stuff out
Hope this helps.
Joe Jenkins
root@xx5:/var/log# vzdump 107 -storage ProxBackupsNFS
INFO: starting new backup job: vzdump 107 --storage ProxBackupsNFS
INFO: Starting Backup of VM 107 (openvz)
INFO: CTID 107 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: trying to remove stale snapshot '/dev/xx5/vzsnap-xx5-0'
INFO: umount: /mnt/vzsnap0: not mounted
ERROR: command 'umount /mnt/vzsnap0' failed: exit code 1
INFO: Logical volume "vzsnap-xx5-0" successfully removed
INFO: creating lvm snapshot of /dev/mapper/xx5-vz ('/dev/xx5/vzsnap-xx5-0')
INFO: Logical volume "vzsnap-xx5-0" created
INFO: creating archive '/mnt/pve/ProxBackupsNFS/dump/vzdump-openvz-107-2013_07_31-21_34_17.tar'
INFO: Total bytes written: 3482634240 (3.3GiB, 5.9MiB/s)
INFO: archive file size: 3.24GB
^^ at this point backup hangs, load skyrockets, server requires reboot
top - 21:59:15 up 22:11, 3 users, load average: 195.39, 147.68, 79.54
Tasks: 1732 total, 1 running, 1731 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 0.3 sy, 0.0 ni, 75.0 id, 22.7 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65954312 total, 42607184 used, 23347128 free, 784576 buffers
KiB Swap: 35651576 total, 0 used, 35651576 free, 27281784 cached
Write failed: Broken pipe
root@xx5:/mnt/pve/ProxBackupsNFS/dump# ls -la
total 12
drwxr-xr-x 3 root root 4096 Jul 31 21:45 .
drwxr-xr-x 4 root root 4096 Jul 31 15:26 ..
-rw-r--r-- 1 root root 3482634240 Jul 31 21:45 vzdump-openvz-107-2013_07_31-21_34_17.tar
drwxr-xr-x 2 root root 4096 Jul 31 21:34 vzdump-openvz-107-2013_07_31-21_34_17.tmp
root@xx5:/mnt/pve/ProxBackupsNFS/dump#
YY6G3
1
Assembly, Cable, PERC7, 2.5, R810, V2
0P77F
0
DISK PROGRAM..., FLASH..., FW6, S-FF
NU209
1
Battery, Primary, 7WHR, 1C Lithium Ion, PERCI
TCVRR
2
Power Supply, 1100W, Redundant, LITEON
XXFVX
1
Assembly, Card, PERCH700-INT, 512M, Serial Attached Scsi
G176J
1
Assembly, Carrier, Hard Drive SAS-SATAU, 2.5, V2
7260R
0
PREPARATION MATERIAL..., DEVIATION..., SERVICE CHARGE..., INCRS #2
X160K
1
HARD DRIVE..., 146G, SAS6, 10K, 2.5, SGT-FF
This particular server is a Dell R815 - See below for storage specs. PERC H700, SAS - Proxmox 1.9 worked fine, we did a clean upgrade from 1.9 to 3.0.x (complete system wipe and reinstall)
I attempted a manual backup of one OpenVZ container last night, which failed locking the entire system up. I have tried OpenVZ containers and KVM virtual machines, there is no rhyme or reason to this problem. Log output below and version info. Looks like it completes one backup, the files are there (on a brand new NFS backup system I turned up yesterday) and then it just craps out. Kernel logs are visible here: http://pastebin.com/UuiJ9Yz0
We have the same issue on a few brand new Dell R620s with the Perc H310 controller. Brand new clean installs of Prox 3.0.x - with only OpenVZ containers and local storage for backups. I have tried ext3 / ext4 and NFS mounts for vzdump backups, with the same results. I have also tried both deadline and CFQ schedulers as recommended.
I am happy to keep testing stuff out
Hope this helps.Joe Jenkins
root@xx5:/var/log# vzdump 107 -storage ProxBackupsNFS
INFO: starting new backup job: vzdump 107 --storage ProxBackupsNFS
INFO: Starting Backup of VM 107 (openvz)
INFO: CTID 107 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: trying to remove stale snapshot '/dev/xx5/vzsnap-xx5-0'
INFO: umount: /mnt/vzsnap0: not mounted
ERROR: command 'umount /mnt/vzsnap0' failed: exit code 1
INFO: Logical volume "vzsnap-xx5-0" successfully removed
INFO: creating lvm snapshot of /dev/mapper/xx5-vz ('/dev/xx5/vzsnap-xx5-0')
INFO: Logical volume "vzsnap-xx5-0" created
INFO: creating archive '/mnt/pve/ProxBackupsNFS/dump/vzdump-openvz-107-2013_07_31-21_34_17.tar'
INFO: Total bytes written: 3482634240 (3.3GiB, 5.9MiB/s)
INFO: archive file size: 3.24GB
^^ at this point backup hangs, load skyrockets, server requires reboot
top - 21:59:15 up 22:11, 3 users, load average: 195.39, 147.68, 79.54
Tasks: 1732 total, 1 running, 1731 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 0.3 sy, 0.0 ni, 75.0 id, 22.7 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65954312 total, 42607184 used, 23347128 free, 784576 buffers
KiB Swap: 35651576 total, 0 used, 35651576 free, 27281784 cached
Write failed: Broken pipe
root@xx5:/mnt/pve/ProxBackupsNFS/dump# ls -la
total 12
drwxr-xr-x 3 root root 4096 Jul 31 21:45 .
drwxr-xr-x 4 root root 4096 Jul 31 15:26 ..
-rw-r--r-- 1 root root 3482634240 Jul 31 21:45 vzdump-openvz-107-2013_07_31-21_34_17.tar
drwxr-xr-x 2 root root 4096 Jul 31 21:34 vzdump-openvz-107-2013_07_31-21_34_17.tmp
root@xx5:/mnt/pve/ProxBackupsNFS/dump#
YY6G3
1
Assembly, Cable, PERC7, 2.5, R810, V2
0P77F
0
DISK PROGRAM..., FLASH..., FW6, S-FF
NU209
1
Battery, Primary, 7WHR, 1C Lithium Ion, PERCI
TCVRR
2
Power Supply, 1100W, Redundant, LITEON
XXFVX
1
Assembly, Card, PERCH700-INT, 512M, Serial Attached Scsi
G176J
1
Assembly, Carrier, Hard Drive SAS-SATAU, 2.5, V2
7260R
0
PREPARATION MATERIAL..., DEVIATION..., SERVICE CHARGE..., INCRS #2
X160K
1
HARD DRIVE..., 146G, SAS6, 10K, 2.5, SGT-FF
On all of my servers around June 23rd we started seeing higher load when performing backups.
Nearly all of our VMs are KVM.
I posted a graph of load here: http://forum.proxmox.com/threads/9097-vzdump-using-lvm-snapshot-kills-the-box?p=78751#post78751
Most of the short spikes are backups, after June 23rd the load spikes higher when doing backups.
I believe that June 23rd is when we updated to wheezy
Nearly all of our VMs are KVM.
I posted a graph of load here: http://forum.proxmox.com/threads/9097-vzdump-using-lvm-snapshot-kills-the-box?p=78751#post78751
Most of the short spikes are backups, after June 23rd the load spikes higher when doing backups.
I believe that June 23rd is when we updated to wheezy
No, I am using ext4 on top of LUKS encrypted SATA disk that is connected to the motherboard using AHCI.
We have been using this same setup since Proxmox 1.5 and have seen a significant increase in load after upgrading to 3.0.
NFS you say?
Wonder if this bug is in our kernel, this references what the 3.0 release kernel was based on:
https://access.redhat.com/site/solutions/318163
Then I found this: http://www.marshut.com/wnwt/io-performance-regressions-with-kernel-2-6-32-358.html
And this: http://gtowey.blogspot.com/2013/02/serious-xfs-performance-regression-in.html
Seems like RedHat has been regressing IO performance over and over lately.
Current Proxmox kernel is based on: vzkernel-2.6.32-042stab078.28.src.rpm
That is based on RedHat 2.6.32-358.6.1.el6
It seems like the Proxmox 3.0 release kernel had IO issues and so does the current kernel we are using.
Unfortunately our kernel is based on the latest RedHat release kernel 2.6.32-358 so there are no newer kernels that would have fixes for these IO issues.
Now that we know why we have these issues maybe our combined efforts can track down a solution.
Wonder if this bug is in our kernel, this references what the 3.0 release kernel was based on:
https://access.redhat.com/site/solutions/318163
Then I found this: http://www.marshut.com/wnwt/io-performance-regressions-with-kernel-2-6-32-358.html
And this: http://gtowey.blogspot.com/2013/02/serious-xfs-performance-regression-in.html
Seems like RedHat has been regressing IO performance over and over lately.
Current Proxmox kernel is based on: vzkernel-2.6.32-042stab078.28.src.rpm
That is based on RedHat 2.6.32-358.6.1.el6
It seems like the Proxmox 3.0 release kernel had IO issues and so does the current kernel we are using.
Unfortunately our kernel is based on the latest RedHat release kernel 2.6.32-358 so there are no newer kernels that would have fixes for these IO issues.
Now that we know why we have these issues maybe our combined efforts can track down a solution.
Seems to be a NFSv4 problem only? PVE uses NFSv3 unless users manually change this in the storage.cfg file.
I have also noticed performance degration on NFS but this degration has first materialized in latest kernel releases.
I also run into this problem, regardless whether the backup is done using NFSv3 or v4, just tried that. Is there any other way we can help the support team find the bug?
According to e100's references it seems that NFS is hit by a bug in locking so maybe if NFS is configured to not use locking by adding this option in storage.cfg: nolock or
to keep locks enabled but ensure locks are only used local: local_lock=all
to keep locks enabled but ensure locks are only used local: local_lock=all
I have lsi card, but I don't use it for vm or backup storage, only proxmox os.
All my vms and backups are on netapp storages through nfsv4. (so through intel e1000 cards)
here the nfs mount options:
options rw,noatime,nodiratime,noacl,vers=4,rsize=65536,wsize=65536,hard,proto=tcp
I could be interesting to see your settings
I would like also notice promox users, that the new backup implementation for kvm vms (not openvz), use some kind of live block migration without snapshot, so if you have a lot of writes on a lot of new blocks in your vms disks, theses writes must be writen live on your vms disk and in parallel on your backup storage. (Only once by block)
That mean that if you have a very slow nfs storage vs vm storage, not enough fast to handle new vms writes (slow disk, remote backup with big latencies,...), you could have some slowdown in your vm. (But it depend of the vm write workload, and speed difference between backup storage and vm storage)
Last edited:
I am using the default nfs3 settings:
rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.5.9,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.5.9
But it seems the bug is not just limited to nfs as storage since others reported the same problem with iscsi and smb. Does anyone have more information about his problem that we could use to pinpoint a direction where we can look at?
rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.5.9,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.5.9
But it seems the bug is not just limited to nfs as storage since others reported the same problem with iscsi and smb. Does anyone have more information about his problem that we could use to pinpoint a direction where we can look at?
Guys,
I had the same issue over the weekend. Our vms were all unresponsive. After logging into the host, It was unresponsive. ps was hanging so I couldnt get a full list of processes but I did manage to get the following:
The load average and zombie processes are worrying. Most of the crond processes had become zombies so I couldnt kill them. Had to reset the server.
The system was doing its backup at the time it happened. Server has only openvz containers running (15 in total) but my backup does NOT use NFS so this is on a pure ext3 local filesystem
Output of backup
I had the same issue over the weekend. Our vms were all unresponsive. After logging into the host, It was unresponsive. ps was hanging so I couldnt get a full list of processes but I did manage to get the following:
Code:
top - 10:44:37 up 35 days, 21:01, 5 users, load average: [B]688.71, 698.57, 808.51
Tasks: 1665 total, 1 running, 1137 sleeping, 0 stopped, 527 zombie[/B]
%Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 25.0 id, 74.8 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65380704 total, 52229376 used, 13151328 free, 741200 buffers
KiB Swap: 131071992 total, 1391552 used, 129680440 free, 36075552 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17 root 20 0 0 0 0 S 8 0.0 184:55.47 ksoftirqd/3
4 root 20 0 0 0 0 S 2 0.0 55:59.18 ksoftirqd/0
9 root 20 0 0 0 0 S 1 0.0 46:13.48 ksoftirqd/1
919101 root 20 0 25740 2896 1176 R 1 0.0 0:00.30 top
2688 root 20 0 388m 30m 28m S 0 0.0 117:36.13 pmxcfs
4843 27 20 0 2224m 781m 4888 S 0 1.2 71:07.05 mysqld
83258 27 20 0 2353m 998m 4000 S 0 1.6 79:29.20 mysqld
906024 root 20 0 189m 36m 4792 S 0 0.1 1:23.31 vzdump
912986 www-data 20 0 217m 50m 3612 S 0 0.1 1:49.62 pveproxy worker
1 root 20 0 10604 828 688 S 0 0.0 0:28.97 init
2 root 20 0 0 0 0 S 0 0.0 0:00.06 kthreadd
3 root rt 0 0 0 0 S 0 0.0 0:02.74 migration/0
5 root rt 0 0 0 0 S 0 0.0 0:00.00 migration/0
6 root rt 0 0 0 0 S 0 0.0 0:05.42 watchdog/0
7 root rt 0 0 0 0 S 0 0.0 0:06.86 migration/1
8 root rt 0 0 0 0 S 0 0.0 0:00.00 migration/1
10 root rt 0 0 0 0 S 0 0.0 0:04.15 watchdog/1
11 root rt 0 0 0 0 S 0 0.0 0:01.28 migration/2The load average and zombie processes are worrying. Most of the crond processes had become zombies so I couldnt kill them. Had to reset the server.
The system was doing its backup at the time it happened. Server has only openvz containers running (15 in total) but my backup does NOT use NFS so this is on a pure ext3 local filesystem
Output of backup
Code:
INFO: starting new backup job: vzdump --quiet 1 --mailto XXXX --mode snapshot --compress lzo --storage backup --all 1
INFO: Starting Backup of VM 100 (openvz)
INFO: CTID 100 exist unmounted down
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/backup/dump/vzdump-openvz-100-2013_08_03-00_00_02.tar.lzo'
INFO: Total bytes written: 1480663040 (1.4GiB, 26MiB/s)
INFO: archive file size: 710MB
INFO: Finished Backup of VM 100 (00:00:56)
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/vg1-vz ('/dev/vg1/vzsnap-proxmox-0')
INFO: Logical volume "vzsnap-proxmox-0" created
INFO: creating archive '/backup/dump/vzdump-openvz-101-2013_08_03-00_00_58.tar.lzo'
INFO: Total bytes written: 7490283520 (7.0GiB, 26MiB/s)
INFO: archive file size: 5.18GB
Code:
pve-manager: 3.0-23 (pve-manager/3.0/957f0862)
running kernel: 2.6.32-20-pve
proxmox-ve-2.6.32: 3.0-100
pve-kernel-2.6.32-20-pve: 2.6.32-100
lvm2: 2.02.95-pve3
clvm: 2.02.95-pve3
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-4
qemu-server: 3.0-20
pve-firmware: 1.0-22
libpve-common-perl: 3.0-4
libpve-access-control: 3.0-4
libpve-storage-perl: 3.0-8
vncterm: 1.1-4
vzctl: 4.0-1pve3
vzprocps: not correctly installed
vzquota: 3.1-2
pve-qemu-kvm: 1.4-13
ksm-control-daemon: not correctly installedWhat is your backup storage config ? (os,number of disk, raid type,...)
which network card do you use for your proxmox host ?
Guys, stop hijacking this thread with speculation about nfs and device drivers. This has nothing to do with network, and also not a lot to do with device drivers.
This is a reclusive kernel issue. It is present on all kinds of hardware (sata, scsi, adaptec/lsi/intel/hp raid, etc.), different filesystems (ext3 and ext4) and kernel versions.
It is somehow linked to IO load, and it is more likely to appear on OpenVZ VE's that have many thousands of files, but it has also been reported on KVM machines.
This is a reclusive kernel issue. It is present on all kinds of hardware (sata, scsi, adaptec/lsi/intel/hp raid, etc.), different filesystems (ext3 and ext4) and kernel versions.
It is somehow linked to IO load, and it is more likely to appear on OpenVZ VE's that have many thousands of files, but it has also been reported on KVM machines.
Last edited: