[CRITICAL] Huge IO load causes freezing during backups

dietmar

Proxmox Staff Member
Staff member
Apr 28, 2005
16,491
315
83
Austria
www.proxmox.com
Problem seems to be connected to a single VE having thousands of files in a folder.
It would really help if we have a reproducible test case. Maybe you can try to write a script to generate
many thousands of files in a folder, so that the bug triggers after running that script in a standard debian template?
 

subversion

New Member
Jan 23, 2013
21
0
1
The new kernel -108 did not help in my case, see also the following thread (which I will update as well) http://forum.proxmox.com/threads/9097-vzdump-using-lvm-snapshot-kills-the-box?p=78898#post78898

This particular server is a Dell R815 - See below for storage specs. PERC H700, SAS - Proxmox 1.9 worked fine, we did a clean upgrade from 1.9 to 3.0.x (complete system wipe and reinstall)

I attempted a manual backup of one OpenVZ container last night, which failed locking the entire system up. I have tried OpenVZ containers and KVM virtual machines, there is no rhyme or reason to this problem. Log output below and version info. Looks like it completes one backup, the files are there (on a brand new NFS backup system I turned up yesterday) and then it just craps out. Kernel logs are visible here: http://pastebin.com/UuiJ9Yz0

We have the same issue on a few brand new Dell R620s with the Perc H310 controller. Brand new clean installs of Prox 3.0.x - with only OpenVZ containers and local storage for backups. I have tried ext3 / ext4 and NFS mounts for vzdump backups, with the same results. I have also tried both deadline and CFQ schedulers as recommended.

I am happy to keep testing stuff out :) Hope this helps.

Joe Jenkins

root@xx5:/var/log# vzdump 107 -storage ProxBackupsNFS
INFO: starting new backup job: vzdump 107 --storage ProxBackupsNFS
INFO: Starting Backup of VM 107 (openvz)
INFO: CTID 107 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: trying to remove stale snapshot '/dev/xx5/vzsnap-xx5-0'
INFO: umount: /mnt/vzsnap0: not mounted
ERROR: command 'umount /mnt/vzsnap0' failed: exit code 1
INFO: Logical volume "vzsnap-xx5-0" successfully removed
INFO: creating lvm snapshot of /dev/mapper/xx5-vz ('/dev/xx5/vzsnap-xx5-0')
INFO: Logical volume "vzsnap-xx5-0" created
INFO: creating archive '/mnt/pve/ProxBackupsNFS/dump/vzdump-openvz-107-2013_07_31-21_34_17.tar'
INFO: Total bytes written: 3482634240 (3.3GiB, 5.9MiB/s)
INFO: archive file size: 3.24GB


^^ at this point backup hangs, load skyrockets, server requires reboot


top - 21:59:15 up 22:11, 3 users, load average: 195.39, 147.68, 79.54
Tasks: 1732 total, 1 running, 1731 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 0.3 sy, 0.0 ni, 75.0 id, 22.7 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65954312 total, 42607184 used, 23347128 free, 784576 buffers
KiB Swap: 35651576 total, 0 used, 35651576 free, 27281784 cached
Write failed: Broken pipe


root@xx5:/mnt/pve/ProxBackupsNFS/dump# ls -la
total 12
drwxr-xr-x 3 root root 4096 Jul 31 21:45 .
drwxr-xr-x 4 root root 4096 Jul 31 15:26 ..
-rw-r--r-- 1 root root 3482634240 Jul 31 21:45 vzdump-openvz-107-2013_07_31-21_34_17.tar
drwxr-xr-x 2 root root 4096 Jul 31 21:34 vzdump-openvz-107-2013_07_31-21_34_17.tmp
root@xx5:/mnt/pve/ProxBackupsNFS/dump#




YY6G3
1
Assembly, Cable, PERC7, 2.5, R810, V2

0P77F
0
DISK PROGRAM..., FLASH..., FW6, S-FF

NU209
1
Battery, Primary, 7WHR, 1C Lithium Ion, PERCI

TCVRR
2
Power Supply, 1100W, Redundant, LITEON

XXFVX
1
Assembly, Card, PERCH700-INT, 512M, Serial Attached Scsi

G176J
1
Assembly, Carrier, Hard Drive SAS-SATAU, 2.5, V2

7260R
0
PREPARATION MATERIAL..., DEVIATION..., SERVICE CHARGE..., INCRS #2

X160K
1
HARD DRIVE..., 146G, SAS6, 10K, 2.5, SGT-FF
 

e100

Active Member
Nov 6, 2010
1,235
24
38
Columbus, Ohio
ulbuilder.wordpress.com

Voleatech

New Member
Mar 22, 2013
25
0
1
I was just curious if it was related to NFS. I have the same problem but I am using a Dell T420 with a Perc 710 and NFS as storage for backups.
I never had an issue (used it since 1.9) until the 3.0 update.
 

e100

Active Member
Nov 6, 2010
1,235
24
38
Columbus, Ohio
ulbuilder.wordpress.com
NFS you say?
Wonder if this bug is in our kernel, this references what the 3.0 release kernel was based on:
https://access.redhat.com/site/solutions/318163

Then I found this: http://www.marshut.com/wnwt/io-performance-regressions-with-kernel-2-6-32-358.html
And this: http://gtowey.blogspot.com/2013/02/serious-xfs-performance-regression-in.html

Seems like RedHat has been regressing IO performance over and over lately.

Current Proxmox kernel is based on: vzkernel-2.6.32-042stab078.28.src.rpm
That is based on RedHat 2.6.32-358.6.1.el6

It seems like the Proxmox 3.0 release kernel had IO issues and so does the current kernel we are using.
Unfortunately our kernel is based on the latest RedHat release kernel 2.6.32-358 so there are no newer kernels that would have fixes for these IO issues.

Now that we know why we have these issues maybe our combined efforts can track down a solution.
 

mir

Well-Known Member
Apr 14, 2012
3,483
97
48
Copenhagen, Denmark
NFS you say?
Wonder if this bug is in our kernel, this references what the 3.0 release kernel was based on:
https://access.redhat.com/site/solutions/318163
Seems to be a NFSv4 problem only? PVE uses NFSv3 unless users manually change this in the storage.cfg file.

I have also noticed performance degration on NFS but this degration has first materialized in latest kernel releases.
 

spirit

Well-Known Member
Apr 2, 2010
3,440
147
63
www.odiso.com
Seems to be a NFSv4 problem only? PVE uses NFSv3 unless users manually change this in the storage.cfg file.


I have also noticed performance degration on NFS but this degration has first materialized in latest kernel releases.
Don't have any nfs problem with last kernel, but I'm using nfsv4 on a big netapp storage. (I'm using intel e1000 nics bonded with lacp)
 

Nico Haase

Member
Feb 27, 2013
31
1
8
Darmstadt, Germany
I also run into this problem, regardless whether the backup is done using NFSv3 or v4, just tried that. Is there any other way we can help the support team find the bug?
 

mir

Well-Known Member
Apr 14, 2012
3,483
97
48
Copenhagen, Denmark
According to e100's references it seems that NFS is hit by a bug in locking so maybe if NFS is configured to not use locking by adding this option in storage.cfg: nolock or
to keep locks enabled but ensure locks are only used local: local_lock=all
 

spirit

Well-Known Member
Apr 2, 2010
3,440
147
63
www.odiso.com
I am using NFS3. Spirit do you have a Dell Raid Card or LSI Raid Card too?
I have lsi card, but I don't use it for vm or backup storage, only proxmox os.
All my vms and backups are on netapp storages through nfsv4. (so through intel e1000 cards)

here the nfs mount options:

options rw,noatime,nodiratime,noacl,vers=4,rsize=65536,wsize=65536,hard,proto=tcp

I could be interesting to see your settings



I would like also notice promox users, that the new backup implementation for kvm vms (not openvz), use some kind of live block migration without snapshot, so if you have a lot of writes on a lot of new blocks in your vms disks, theses writes must be writen live on your vms disk and in parallel on your backup storage. (Only once by block)

That mean that if you have a very slow nfs storage vs vm storage, not enough fast to handle new vms writes (slow disk, remote backup with big latencies,...), you could have some slowdown in your vm. (But it depend of the vm write workload, and speed difference between backup storage and vm storage)
 
Last edited:

Voleatech

New Member
Mar 22, 2013
25
0
1
I am using the default nfs3 settings:
rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.5.9,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.5.9

But it seems the bug is not just limited to nfs as storage since others reported the same problem with iscsi and smb. Does anyone have more information about his problem that we could use to pinpoint a direction where we can look at?
 

lynchie

New Member
Aug 6, 2013
5
0
1
Guys,

I had the same issue over the weekend. Our vms were all unresponsive. After logging into the host, It was unresponsive. ps was hanging so I couldnt get a full list of processes but I did manage to get the following:

Code:
top - 10:44:37 up 35 days, 21:01,  5 users,  load average: [B]688.71, 698.57, 808.51
Tasks: 1665 total,   1 running, 1137 sleeping,   0 stopped, 527 zombie[/B]
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 25.0 id, 74.8 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  65380704 total, 52229376 used, 13151328 free,   741200 buffers
KiB Swap: 131071992 total,  1391552 used, 129680440 free, 36075552 cached

    PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
     17 root      20   0     0    0    0 S     8  0.0 184:55.47 ksoftirqd/3
      4 root      20   0     0    0    0 S     2  0.0  55:59.18 ksoftirqd/0
      9 root      20   0     0    0    0 S     1  0.0  46:13.48 ksoftirqd/1
 919101 root      20   0 25740 2896 1176 R     1  0.0   0:00.30 top
   2688 root      20   0  388m  30m  28m S     0  0.0 117:36.13 pmxcfs
   4843 27        20   0 2224m 781m 4888 S     0  1.2  71:07.05 mysqld
  83258 27        20   0 2353m 998m 4000 S     0  1.6  79:29.20 mysqld
 906024 root      20   0  189m  36m 4792 S     0  0.1   1:23.31 vzdump
 912986 www-data  20   0  217m  50m 3612 S     0  0.1   1:49.62 pveproxy worker
      1 root      20   0 10604  828  688 S     0  0.0   0:28.97 init
      2 root      20   0     0    0    0 S     0  0.0   0:00.06 kthreadd
      3 root      rt   0     0    0    0 S     0  0.0   0:02.74 migration/0
      5 root      rt   0     0    0    0 S     0  0.0   0:00.00 migration/0
      6 root      rt   0     0    0    0 S     0  0.0   0:05.42 watchdog/0
      7 root      rt   0     0    0    0 S     0  0.0   0:06.86 migration/1
      8 root      rt   0     0    0    0 S     0  0.0   0:00.00 migration/1
     10 root      rt   0     0    0    0 S     0  0.0   0:04.15 watchdog/1
     11 root      rt   0     0    0    0 S     0  0.0   0:01.28 migration/2
The load average and zombie processes are worrying. Most of the crond processes had become zombies so I couldnt kill them. Had to reset the server.

The system was doing its backup at the time it happened. Server has only openvz containers running (15 in total) but my backup does NOT use NFS so this is on a pure ext3 local filesystem

Output of backup

Code:
INFO: starting new backup job: vzdump --quiet 1 --mailto XXXX --mode snapshot --compress lzo --storage backup --all 1
INFO: Starting Backup of VM 100 (openvz)
INFO: CTID 100 exist unmounted down
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/backup/dump/vzdump-openvz-100-2013_08_03-00_00_02.tar.lzo'
INFO: Total bytes written: 1480663040 (1.4GiB, 26MiB/s)
INFO: archive file size: 710MB
INFO: Finished Backup of VM 100 (00:00:56)
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/vg1-vz ('/dev/vg1/vzsnap-proxmox-0')
INFO: Logical volume "vzsnap-proxmox-0" created
INFO: creating archive '/backup/dump/vzdump-openvz-101-2013_08_03-00_00_58.tar.lzo'
INFO: Total bytes written: 7490283520 (7.0GiB, 26MiB/s)
INFO: archive file size: 5.18GB
Code:
pve-manager: 3.0-23 (pve-manager/3.0/957f0862)
running kernel: 2.6.32-20-pve
proxmox-ve-2.6.32: 3.0-100
pve-kernel-2.6.32-20-pve: 2.6.32-100
lvm2: 2.02.95-pve3
clvm: 2.02.95-pve3
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-4
qemu-server: 3.0-20
pve-firmware: 1.0-22
libpve-common-perl: 3.0-4
libpve-access-control: 3.0-4
libpve-storage-perl: 3.0-8
vncterm: 1.1-4
vzctl: 4.0-1pve3
vzprocps: not correctly installed
vzquota: 3.1-2
pve-qemu-kvm: 1.4-13
ksm-control-daemon: not correctly installed
 

spirit

Well-Known Member
Apr 2, 2010
3,440
147
63
www.odiso.com
I am using the default nfs3 settings:
rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.5.9,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=192.168.5.9

But it seems the bug is not just limited to nfs as storage since others reported the same problem with iscsi and smb. Does anyone have more information about his problem that we could use to pinpoint a direction where we can look at?
What is your backup storage config ? (os,number of disk, raid type,...)

which network card do you use for your proxmox host ?
 

gkovacs

Active Member
Dec 22, 2008
503
45
28
Budapest, Hungary
Guys, stop hijacking this thread with speculation about nfs and device drivers. This has nothing to do with network, and also not a lot to do with device drivers.

This is a reclusive kernel issue. It is present on all kinds of hardware (sata, scsi, adaptec/lsi/intel/hp raid, etc.), different filesystems (ext3 and ext4) and kernel versions.
It is somehow linked to IO load, and it is more likely to appear on OpenVZ VE's that have many thousands of files, but it has also been reported on KVM machines.
 
Last edited:

naja7host

Member
May 15, 2009
78
0
6
Guys,

I had the same issue over the weekend. Our vms were all unresponsive. After logging into the host, It was unresponsive. ps was hanging so I couldnt get a full list of processes but I did manage to get the following:

Code:
top - 10:44:37 up 35 days, 21:01,  5 users,  load average: [B]688.71, 698.57, 808.51
Tasks: 1665 total,   1 running, 1137 sleeping,   0 stopped, 527 zombie[/B]
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 25.0 id, 74.8 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  65380704 total, 52229376 used, 13151328 free,   741200 buffers
KiB Swap: 131071992 total,  1391552 used, 129680440 free, 36075552 cached

    PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
     17 root      20   0     0    0    0 S     8  0.0 184:55.47 ksoftirqd/3
      4 root      20   0     0    0    0 S     2  0.0  55:59.18 ksoftirqd/0
      9 root      20   0     0    0    0 S     1  0.0  46:13.48 ksoftirqd/1
 919101 root      20   0 25740 2896 1176 R     1  0.0   0:00.30 top
   2688 root      20   0  388m  30m  28m S     0  0.0 117:36.13 pmxcfs
   4843 27        20   0 2224m 781m 4888 S     0  1.2  71:07.05 mysqld
  83258 27        20   0 2353m 998m 4000 S     0  1.6  79:29.20 mysqld
 906024 root      20   0  189m  36m 4792 S     0  0.1   1:23.31 vzdump
 912986 www-data  20   0  217m  50m 3612 S     0  0.1   1:49.62 pveproxy worker
      1 root      20   0 10604  828  688 S     0  0.0   0:28.97 init
      2 root      20   0     0    0    0 S     0  0.0   0:00.06 kthreadd
      3 root      rt   0     0    0    0 S     0  0.0   0:02.74 migration/0
      5 root      rt   0     0    0    0 S     0  0.0   0:00.00 migration/0
      6 root      rt   0     0    0    0 S     0  0.0   0:05.42 watchdog/0
      7 root      rt   0     0    0    0 S     0  0.0   0:06.86 migration/1
      8 root      rt   0     0    0    0 S     0  0.0   0:00.00 migration/1
     10 root      rt   0     0    0    0 S     0  0.0   0:04.15 watchdog/1
     11 root      rt   0     0    0    0 S     0  0.0   0:01.28 migration/2
The load average and zombie processes are worrying. Most of the crond processes had become zombies so I couldnt kill them. Had to reset the server.

The system was doing its backup at the time it happened. Server has only openvz containers running (15 in total) but my backup does NOT use NFS so this is on a pure ext3 local filesystem

Output of backup

Code:
INFO: starting new backup job: vzdump --quiet 1 --mailto XXXX --mode snapshot --compress lzo --storage backup --all 1
INFO: Starting Backup of VM 100 (openvz)
INFO: CTID 100 exist unmounted down
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/backup/dump/vzdump-openvz-100-2013_08_03-00_00_02.tar.lzo'
INFO: Total bytes written: 1480663040 (1.4GiB, 26MiB/s)
INFO: archive file size: 710MB
INFO: Finished Backup of VM 100 (00:00:56)
INFO: Starting Backup of VM 101 (openvz)
INFO: CTID 101 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating lvm snapshot of /dev/mapper/vg1-vz ('/dev/vg1/vzsnap-proxmox-0')
INFO: Logical volume "vzsnap-proxmox-0" created
INFO: creating archive '/backup/dump/vzdump-openvz-101-2013_08_03-00_00_58.tar.lzo'
INFO: Total bytes written: 7490283520 (7.0GiB, 26MiB/s)
INFO: archive file size: 5.18GB
Code:
pve-manager: 3.0-23 (pve-manager/3.0/957f0862)
running kernel: 2.6.32-20-pve
proxmox-ve-2.6.32: 3.0-100
pve-kernel-2.6.32-20-pve: 2.6.32-100
lvm2: 2.02.95-pve3
clvm: 2.02.95-pve3
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-4
qemu-server: 3.0-20
pve-firmware: 1.0-22
libpve-common-perl: 3.0-4
libpve-access-control: 3.0-4
libpve-storage-perl: 3.0-8
vncterm: 1.1-4
vzctl: 4.0-1pve3
vzprocps: not correctly installed
vzquota: 3.1-2
pve-qemu-kvm: 1.4-13
ksm-control-daemon: not correctly installed
i have the same probleme two days ago ..

i use NFS as storage for backup .
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!