IO delay probleme and slow hard disk speed need help

[johnfaiz]

i have dedicated server 128 giga ram processor xeon 2670 hard disk 2 TB sata3 no raid controller and my all vms conf is :
bootdisk: virtio0
cores: 4
cpu: host
cpuunits: 8
ide2: local:iso/winoriginal.iso,media=cdrom
memory: 4096
name: desha3
net0: virtio=7E:A6:24:6A:58:E7,bridge=vmbr1
ostype: win7
sockets: 1
tablet: 0
vga: std
virtio0: local:100/vm-100-disk-1.raw,format=raw,cache=writeback,,size=150G

i have realy high IO delay and my pvperf (all vms running) output:
:~# pveperf
CPU BOGOMIPS: 83202.88
REGEX/SECOND: 730482
HD SIZE: 1828.96 GB (/dev/sda2)
BUFFERED READS: 0.79 MB/sec
AVERAGE SEEK TIME: 503.41 ms
FSYNCS/SECOND: 0.21
DNS EXT: 23.59 ms
DNS INT: 2.30 ms (ovh.net)


i hope any one can help me why this delay happen ? an buffered reads so slow on running vms but if all vms off it will be 170 Mega/sec

 

[wahmed]

Your Avg Seek time of 503.41ms is killing you.

Here is pveperf seek time on a node with 60 GB SSD:
BUFFERED READS 164.50 MB/S
AVG. SEEK TIME : 0.20 MS

Here is seek time on a node with an ancient 40 GB IDE HDD which is already 7 years old.
BUFFERED READS 11.27 MB/S
AVG. SEEK TIME : 13.46 MS

If possible may be do another installation on another hdd and see if you get the same seek time. Or put the a VM on a shared storage and see if the I/O delay goes down.

 

[johnfaiz]

Your Avg Seek time of 503.41ms is killing you.

Here is pveperf seek time on a node with 60 GB SSD:
BUFFERED READS 164.50 MB/S
AVG. SEEK TIME : 0.20 MS

Here is seek time on a node with an ancient 40 GB IDE HDD which is already 7 years old.
BUFFERED READS 11.27 MB/S
AVG. SEEK TIME : 13.46 MS

If possible may be do another installation on another hdd and see if you get the same seek time. Or put the a VM on a shared storage and see if the I/O delay goes down.

thanks but there is any way to make IO delay low ?

 

[wahmed]

Your I/O is very unusually high. I dont think it is even a software issue. Is it a dedicated Proxmox node or you have lots of other thing going on the same machine which could result high i/o?

 

[johnfaiz]

there is a command ? sorry im just new on proxmox thanks

 

[wahmed]

You could install htop and see if you can find any clue there. Htop is a tiny program which shows somewhat details of what is going on within your machine. Look for High CPU/RAM usage, load, tasks etc.

#apt-get install htop
#htop

 

[johnfaiz]

You could install htop and see if you can find any clue there. Htop is a tiny program which shows somewhat details of what is going on within your machine. Look for High CPU/RAM usage, load, tasks etc.

#apt-get install htop
#htop

okay thanks iwill post result

 

[johnfaiz]

 

[udo]

Hi,
your VMs simply make much more IO than the sata-disks can support. This is the reason of the high load and the bad IO-Values.

You need an better IO-Subsystem (or less VMs with heavy IO). If you have an good raid-controller (fast, with BBU secured cache), you can start with 4 SATA disks in RAID-10, better 4 (or 6) SAS disks (15k 4ms) in raid-10.

Udo

 

[johnfaiz]

Hi,
your VMs simply make much more IO than the sata-disks can support. This is the reason of the high load and the bad IO-Values.

You need an better IO-Subsystem (or less VMs with heavy IO). If you have an good raid-controller (fast, with BBU secured cache), you can start with 4 SATA disks in RAID-10, better 4 (or 6) SAS disks (15k 4ms) in raid-10.

Udo

what if i try to limit hard disk speed for each vm ?

 

[wahmed]

Looking at your Screenshot of htop, it makes much more sense why you have such high I/O delay.

As Udo pointed out above, a good raid controller is your friend here. Without knowing much about the nature and number of running VMs it is somewhat hard to pin point the exact cause. But if you are running I/O intensive VMs like database server, you need faster disks. period. If it is mission critical setup and got budget, few SSDs would work wonder. If it is a test setup how about putting 1 SSD and put few VMs on it to see how I/O performs.