i/o problem
- Thread starter udi
- Start date
And the /var/log/messages:
Code:
Jan 26 14:57:02 sint kernel: EXT3-fs: barriers disabled
Jan 26 14:57:02 sint kernel: kjournald starting. Commit interval 5 seconds
Jan 26 14:57:02 sint kernel: EXT3-fs (dm-5): using internal journal
Jan 26 14:57:02 sint kernel: EXT3-fs (dm-5): 7 orphan inodes deleted
Jan 26 14:57:02 sint kernel: EXT3-fs (dm-5): recovery complete
Jan 26 14:57:02 sint kernel: EXT3-fs (dm-5): mounted filesystem with ordered data mode
Jan 26 15:04:03 sint kernel: syslogd D ffff88036314e300 0 4834 4109 102 0x00020004
Jan 26 15:04:03 sint kernel: ffff880363febcc8 0000000000000086 ffff880363febc68 ffff88066b987198
Jan 26 15:04:03 sint kernel: 0000000000000001 ffff880363febd58 ffff880363febdd0 0000000000000282
Jan 26 15:04:03 sint kernel: ffff88036314e8c8 ffff880363febfd8 000000000000f788 ffff88036314e8c8
Jan 26 15:04:03 sint kernel: Call Trace:
Jan 26 15:04:03 sint kernel: [<ffffffff8109d3c9>] ? ktime_get_ts+0xa9/0xe0
Jan 26 15:04:03 sint kernel: [<ffffffff8111dcb0>] ? sync_page+0x0/0x50
Jan 26 15:04:03 sint kernel: [<ffffffff814e80d3>] io_schedule+0x73/0xc0
Jan 26 15:04:03 sint kernel: [<ffffffff8111dced>] sync_page+0x3d/0x50
Jan 26 15:04:03 sint kernel: [<ffffffff814e894f>] __wait_on_bit+0x5f/0x90
Jan 26 15:04:03 sint kernel: [<ffffffff8111dea3>] wait_on_page_bit+0x73/0x80
Jan 26 15:04:03 sint kernel: [<ffffffff81092a50>] ? wake_bit_function+0x0/0x40
Jan 26 15:04:03 sint kernel: [<ffffffff81135765>] ? pagevec_lookup_tag+0x25/0x40
Jan 26 15:04:03 sint kernel: [<ffffffff8111e48b>] wait_on_page_writeback_range+0xfb/0x190
Jan 26 15:04:03 sint kernel: [<ffffffff811347a4>] ? generic_writepages+0x24/0x30
Jan 26 15:04:03 sint kernel: [<ffffffff811347e5>] ? do_writepages+0x35/0x40
Jan 26 15:04:03 sint kernel: [<ffffffff8111e5db>] ? __filemap_fdatawrite_range+0x5b/0x60
Jan 26 15:04:03 sint kernel: [<ffffffff8111e658>] filemap_write_and_wait_range+0x78/0x90
Jan 26 15:04:03 sint kernel: [<ffffffff811b7f7a>] vfs_fsync_range+0xba/0x190
Jan 26 15:04:03 sint kernel: [<ffffffff811b80bd>] vfs_fsync+0x1d/0x20
Jan 26 15:04:03 sint kernel: [<ffffffff811b8120>] do_fsync+0x60/0xa0
Jan 26 15:04:03 sint kernel: [<ffffffff811b8190>] sys_fsync+0x10/0x20
Jan 26 15:04:03 sint kernel: [<ffffffff810478d0>] sysenter_dispatch+0x7/0x2eI've run into a very similar problem during stress-testing a W2k8 guest. If a backup is done while running iozone in the guest, the backup task hangs when it tries to remove the snapshot volume with "lvremove". When idle, the same guest can be backed up without any problems. In my setup the problem is reproducible.
I think this is related to the following Debian problem report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=659762
The logical volume of the VM is left suspended, so that all processes accessing it are blocked in "D" state. This can be checked with "dmsetup info -c". If a "s" appears in the Stat field, the volume is still suspended.
Manual recovery is possible with "dmsetup resume <logical volume name>". Both, backup task and guest OS then continue their work.
I think this is related to the following Debian problem report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=659762
The logical volume of the VM is left suspended, so that all processes accessing it are blocked in "D" state. This can be checked with "dmsetup info -c". If a "s" appears in the Stat field, the volume is still suspended.
Manual recovery is possible with "dmsetup resume <logical volume name>". Both, backup task and guest OS then continue their work.
This can happen if the snapshot runs out of space.
When the snapshot is created it has a specific size, default is 1GB.
As data is written to the volume, a Copy on Write (CoW) happens from the volume into the snapshot.
If the snapshot is too small, you run out of space for the CoWs and the snapshot becomes unusable.
You can increase the size of the snapshot by editing /etc/vzdump.conf
With this setting the VM can write 15GB before the snapshot fails to function:
Also, make sure you have enough free space on your LVM volume for whatever size you pick or it will fail to create the snapshot.
When the snapshot is created it has a specific size, default is 1GB.
As data is written to the volume, a Copy on Write (CoW) happens from the volume into the snapshot.
If the snapshot is too small, you run out of space for the CoWs and the snapshot becomes unusable.
You can increase the size of the snapshot by editing /etc/vzdump.conf
With this setting the VM can write 15GB before the snapshot fails to function:
Code:
size: 15000Also, make sure you have enough free space on your LVM volume for whatever size you pick or it will fail to create the snapshot.
I already had increased my snapshot volume size to 12 GiB before. Now I've dumped the LV stats each second, until it hangs. The last output from lvdisplay is:
The volume group is 2 TB and has 1.8 TB free, so free space should not be an issue.
Code:
# open 1
LV Size 40.00 GiB
Current LE 10240
COW-table size 12.00 GiB
COW-table LE 3072
Allocated to snapshot 28.76%
Snapshot chunk size 4.00 KiBThe volume group is 2 TB and has 1.8 TB free, so free space should not be an issue.
Sorry I thought it will.
Yes everything is updated and aptitude update; aptitude full-upgrade doesn't show any updates.
Code:
root@timo:~# pveversion -v
pve-manager: 2.0-38 (pve-manager/2.0/af81df02)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 2.0-60
pve-kernel-2.6.32-6-pve: 2.6.32-55
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.88-2pve1
clvm: 2.02.88-2pve1
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-1
pve-cluster: 1.0-23
qemu-server: 2.0-25
pve-firmware: 1.0-15
libpve-common-perl: 1.0-17
libpve-access-control: 1.0-17
libpve-storage-perl: 2.0-12
vncterm: 1.0-2
vzctl: 3.0.30-2pve1
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-5
ksm-control-daemon: 1.1-1
Code:
root@timo:~# uname -a
Linux timo 2.6.32-7-pve #1 SMP Thu Feb 16 09:00:32 CET 2012 x86_64 GNU/Linux
root@timo:~# cat /proc/version
Linux version 2.6.32-7-pve (root@maui) (gcc version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Thu Feb 16 09:00:32 CET 2012was this system installed using a beta proxmox cd? If so then then there may not be enough space on the lvm volume group .
what is the output of:
vgs
this is ours on a rc-1 system:
what is the output of:
vgs
this is ours on a rc-1 system:
Code:
vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 1.82t 16.00g@bread-baker:
Yes but I was one of the first relaises:
As you can see it is only 4GB.
If we reinstall the node with a new ISO rc1 will the free size will be larger then 4GB? We see on our 1.9 systems that 4GB is small for busy containers.
@dietmar:
Yes but I was one of the first relaises:
Code:
root@timo:~# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 299.50g 4.00g
vmdisks 1 4 0 wz--n- 999.99g 517.98gAs you can see it is only 4GB.
If we reinstall the node with a new ISO rc1 will the free size will be larger then 4GB? We see on our 1.9 systems that 4GB is small for busy containers.
@dietmar:
Code:
root@timo:~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
data pve -wi-ao 197.50g
root pve -wi-ao 75.00g
swap pve -wi-ao 23.00g
vm-200-disk-1 vmdisks -wi--- 150.00g
vm-201-disk-1 vmdisks -wi--- 150.01g
vm-202-disk-1 vmdisks -wi-a- 150.00g
vm-203-disk-1 vmdisks -wi--- 32.00gfor us re install on proxmox beta systems was the easiest way to increase free volume group space.
Currently, vzdump is using only 1GB snapshot space by default (size parameter, see 'man vzdump'). Please try to use 'lvs' during the backup to see if the snapshot runs out of space.
We currently have a size of 4000
Code:
root@timo:~# tail /etc/vzdump.conf
#bwlimit: KBPS
#ionize: PRI
#lockwait: MINUTES
#stopwait: MINUTES
size: 4000
maxfiles: 2
#size: MB
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLISTWhat is the current default freespace?
We increased that from 4GB to max 16GB in 2.0RC1 (depends how large the disk is, max 1/8 of disk size).
I have had very similar issues on my boxes - and have not found a conclusive fix.
In my case I have assumed the issue seems to be the lsi megaraid controllers.
The io timout issues seem to have many bug reports on redhat relating to the 120 second timeout. Changing the disk scheduler to noop helps.. but in short. it still sucks!
Simply running a dd if=/dev/zero of=out.img... can cause it to get hung up on io waits.
My current raid controllers have no ram cache on board. So am changing them all to lsi megaraids with 500MB cache and BBU. Hopefully it will solve the problem.
in short. a fix would be wonderfull!
Currently.. just doing a vsdump backup hangs the system!
Rob
In my case I have assumed the issue seems to be the lsi megaraid controllers.
The io timout issues seem to have many bug reports on redhat relating to the 120 second timeout. Changing the disk scheduler to noop helps.. but in short. it still sucks!
Simply running a dd if=/dev/zero of=out.img... can cause it to get hung up on io waits.
My current raid controllers have no ram cache on board. So am changing them all to lsi megaraids with 500MB cache and BBU. Hopefully it will solve the problem.
in short. a fix would be wonderfull!
Currently.. just doing a vsdump backup hangs the system!
Rob