KVM IO delays

[dad311]

Im running the RC1 on a small home server. Today I tried to create a Centos KVM. Disk IO delay went from 0% > 98% and then the GUI became unresponsive. I had to stop the new KVM.

Known issue? Anything I can do about this issue? Things to try? Never had this problem on Proxmox 1.X. All my file systems are ext3 or ext4.

Thanks!


root@proxmox1:~# pveversion -v
pve-manager: 2.0-38 (pve-manager/2.0/af81df02)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 2.0-60
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.88-2pve1
clvm: 2.02.88-2pve1
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-1
pve-cluster: 1.0-23
qemu-server: 2.0-25
pve-firmware: 1.0-15
libpve-common-perl: 1.0-17
libpve-access-control: 1.0-17
libpve-storage-perl: 2.0-12
vncterm: 1.0-2
vzctl: 3.0.30-2pve1
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-5
ksm-control-daemon: 1.1-1



root@proxmox1:~# uname -a
Linux proxmox1 2.6.32-7-pve #1 SMP Thu Feb 16 09:00:32 CET 2012 x86_64 GNU/Linux

 

[sorajate]

i've fact the same problem while running kvm. the IO delay went very high and cause whole system not responding.

 

[tom]

pls run pveperf and post results.

also give details about your storage hardware.

 

[dad311]

Here is a snapshot with a IO delay of about 25%. The IO delay hit 85% but the system became unresponsive and I was not able to a prompt.


root@proxmox1:~# pveperf
CPU BOGOMIPS: 16877.58
REGEX/SECOND: 810340
HD SIZE: 57.09 GB (/dev/mapper/pve-root)
BUFFERED READS: 11.72 MB/sec
AVERAGE SEEK TIME: 19.40 ms
FSYNCS/SECOND: 349.26
DNS EXT: 74.28 ms
DNS INT: 90.35 ms


root@proxmox1:~# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 59863164 881448 55940848 2% /
tmpfs 2021932 0 2021932 0% /lib/init/rw
udev 2012320 212 2012108 1% /dev
tmpfs 2021932 54024 1967908 3% /dev/shm
/dev/mapper/pve-data 159289044 29303000 129986044 19% /var/lib/vz
/dev/sda1 506724 34170 446391 8% /boot
/dev/fuse 30720 28 30692 1% /etc/pve
/dev/sdc1 672865584 69983452 568702500 11% /home1
/dev/sdb1 1922858352 1014964824 810217928 56% /home2
/dev/sdc2 768317536 54898212 674391000 8% /home3

 

[tom]

again, also give details about your storage hardware.

 

[dad311]

No sure what details your looking for. On my small home sever, I have 4 SATA disks(1 system disk and 4 storage disks). There are no USB or network disks attached.

 

[tom]

raid controller/storage controller, disk type (vendor, model number).

and details about your raid settings, level, cache, etc.

 

[djsami]

Here is a snapshot with a IO delay of about 25%. The IO delay hit 85% but the system became unresponsive and I was not able to a prompt.


root@proxmox1:~# pveperf
CPU BOGOMIPS: 16877.58
REGEX/SECOND: 810340
HD SIZE: 57.09 GB (/dev/mapper/pve-root)
BUFFERED READS: 11.72 MB/sec
AVERAGE SEEK TIME: 19.40 ms
FSYNCS/SECOND: 349.26
DNS EXT: 74.28 ms
DNS INT: 90.35 ms


root@proxmox1:~# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 59863164 881448 55940848 2% /
tmpfs 2021932 0 2021932 0% /lib/init/rw
udev 2012320 212 2012108 1% /dev
tmpfs 2021932 54024 1967908 3% /dev/shm
/dev/mapper/pve-data 159289044 29303000 129986044 19% /var/lib/vz
/dev/sda1 506724 34170 446391 8% /boot
/dev/fuse 30720 28 30692 1% /etc/pve
/dev/sdc1 672865584 69983452 568702500 11% /home1
/dev/sdb1 1922858352 1014964824 810217928 56% /home2
/dev/sdc2 768317536 54898212 674391000 8% /home3

How do I rate it in me?

root@servertr:~# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 99083868 2412896 91637808 3% /
tmpfs 8208492 0 8208492 0% /lib/init/rw
udev 8198880 232 8198648 1% /dev
tmpfs 8208492 18792 8189700 1% /dev/shm
/dev/mapper/pve-data 2173801932 231962088 1941839844 11% /var/lib/vz
/dev/sdc2 505764 34170 445482 8% /boot
/dev/fuse 30720 16 30704 1% /etc/pve


root@servertr:~# pveperf
CPU BOGOMIPS: 40425.95
REGEX/SECOND: 917278
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 67.91 MB/sec
AVERAGE SEEK TIME: 17.29 ms
FSYNCS/SECOND: 12.33
DNS EXT: 160.58 ms
DNS INT: 97.67 ms (sunucuservis.com.tr)
root@servertr:~#

 

[dad311]

No raid. Proxmox is running on a plain Dell T105 with 2 gig of memory and 9 CTs. Avererage CPU usage is <10% and average IO delay is 2%.

Tried to get the hdparms, but no luck installing hdparm, see below. All my disks are low-end newegg specials, which are perfect for home use and never caused any issues with 1.x proxmox.

Sorry for the lack of info, but thats all I have.

root@proxmox1:~# hdparm -I /dev/sda
-bash: hdparm: command not found
root@proxmox1:~# apt-get install hdparm
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
apmd
The following NEW packages will be installed:
hdparm
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 103 kB of archives.
After this operation, 311 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
hdparm
Install these packages without verification [y/N]? y
Err http://ftp.us.debian.org/debian/ squeeze/main hdparm amd64 9.32-1
404 Not Found [IP: 199.6.12.70 80]
Failed to fetch http://ftp.us.debian.org/debian/dists/squeeze/main/binary-amd64/hdparm_9.32-1_amd64.deb 404 Not Found [IP: 199.6.12.70 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

 

[gmckeown]

Run "apt-get update" first, then run apt-get -f install to fix the install issues.You can tune a few settings to get a little better performance. Try setting the fs readahead using blockdev --setra 4096 /dev/sd?.You cannot expect high performance from slower sata disks with no raid controller. Software raid is better than no raid, but will never be as fast as a raid controller with write-back cache.Just to give you an idea how slow your disks are, here is my pveperf on a single 36gb 15,000 rpm SAS disk (compared to you SATA disks, the numbers speak for themselves.)pveperfCPU BOGOMIPS: 23939.73REGEX/SECOND: 1068291HD SIZE: 16.73 GB (/dev/mapper/pve-root)BUFFERED READS: 139.90 MB/secAVERAGE SEEK TIME: 5.43 msFSYNCS/SECOND: 1393.15DNS EXT: 105.05 msDNS INT: 90.44 msWith read ahead tweak:#blockdev --setra 8192 /dev/mapper/pve-root # pveperfCPU BOGOMIPS: 23939.73REGEX/SECOND: 1090277HD SIZE: 16.73 GB (/dev/mapper/pve-root)BUFFERED READS: 166.81 MB/secAVERAGE SEEK TIME: 4.84 msFSYNCS/SECOND: 1395.08DNS EXT: 104.63 msDNS INT: 87.81 ms