Slow Disk reads inside virtual os

A

Arthur Stephens

Member
Apr 19, 2016
47
1
8
66
Running Centos 5.8 Linux 2.6.32-642.1.1.el6.x86_64 on x86_64 on Proxmox pve 4.1-1
This is from the console of the vm...

[root@my ~]# hdparm -Tt /dev/mapper/vg_my-lv_root
Timing cached reads: 10316 MB in 2.00 seconds = 5162.32 MB/sec
Timing buffered disk reads: 38 MB in 3.11 seconds = 12.20 MB/sec

This is from the console of pve 4.1-1

root@PteraCluster:~# hdparm -Tt /dev/mapper/pve-data
Timing cached reads: 16106 MB in 2.00 seconds = 8059.04 MB/sec
Timing buffered disk reads: 2208 MB in 3.00 seconds = 735.76 MB/sec

Can anyone explain to me how to get the vm disk reads working faster than 12.20 MB/sec?
 
From inside the vm
[root@my ~]# fio fio.cfg
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.0.13
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m] [100.0% done] [2220K/532K/0K /s] [583 /148 /0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=10645: Fri Jun 3 09:03:23 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3275.3MB, bw=1518.3KB/s, iops=340 , runt=2209009msec
slat (usec): min=4 , max=152768 , avg=58.28, stdev=1047.91
clat (usec): min=72 , max=1343.5K, avg=150271.65, stdev=148380.42
lat (usec): min=502 , max=1343.5K, avg=150331.45, stdev=148380.53
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 13], 10.00th=[ 21], 20.00th=[ 39],
| 30.00th=[ 58], 40.00th=[ 77], 50.00th=[ 98], 60.00th=[ 126],
| 70.00th=[ 172], 80.00th=[ 243], 90.00th=[ 363], 95.00th=[ 474],
| 99.00th=[ 660], 99.50th=[ 734], 99.90th=[ 873], 99.95th=[ 930],
| 99.99th=[ 1057]
bw (KB/s) : min= 217, max= 8368, per=100.00%, avg=1520.09, stdev=754.37
write: io=840493KB, bw=389615 B/s, iops=85 , runt=2209009msec
slat (usec): min=6 , max=151573 , avg=108.97, stdev=1917.73
clat (usec): min=573 , max=1324.9K, avg=150019.88, stdev=148973.81
lat (usec): min=593 , max=1324.9K, avg=150130.74, stdev=148990.47
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 12], 10.00th=[ 21], 20.00th=[ 38],
| 30.00th=[ 57], 40.00th=[ 76], 50.00th=[ 97], 60.00th=[ 126],
| 70.00th=[ 172], 80.00th=[ 243], 90.00th=[ 363], 95.00th=[ 474],
| 99.00th=[ 668], 99.50th=[ 734], 99.90th=[ 873], 99.95th=[ 930],
| 99.99th=[ 1057]
bw (KB/s) : min= 64, max= 2050, per=100.00%, avg=380.74, stdev=201.88
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.03%
lat (msec) : 2=0.28%, 4=0.59%, 10=2.98%, 20=5.73%, 50=16.66%
lat (msec) : 100=24.87%, 250=29.65%, 500=15.11%, 750=3.68%, 1000=0.38%
lat (msec) : 2000=0.02%
cpu : usr=4.39%, sys=12.98%, ctx=894462, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=752048/w=188554/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=3275.3MB, aggrb=1518KB/s, minb=1518KB/s, maxb=1518KB/s, mint=2209009msec, maxt=2209009msec
WRITE: io=840492KB, aggrb=380KB/s, minb=380KB/s, maxb=380KB/s, mint=2209009msec, maxt=2209009msec

Disk stats (read/write):
dm-0: ios=752111/228999, merge=0/0, ticks=112464208/31828324, in_queue=144301990, util=100.00%, aggrios=732209/198383, aggrmerge=19964/31865, aggrticks=109766234/29348063, aggrin_queue=139113564, aggrutil=100.00%
sda: ios=732209/198383, merge=19964/31865, ticks=109766234/29348063, in_queue=139113564, util=100.00%

From inside PVE
root@PteraCluster:~# fio fio.cfg
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [21384KB/5101KB/0KB /s] [4924/1182/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=48835: Fri Jun 3 09:18:44 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=23894KB/s, iops=3920, runt=140315msec
slat (usec): min=3, max=67468, avg=16.25, stdev=91.20
clat (usec): min=29, max=726968, avg=16242.41, stdev=21368.50
lat (usec): min=34, max=726981, avg=16259.09, stdev=21368.70
clat percentiles (usec):
| 1.00th=[ 62], 5.00th=[ 1240], 10.00th=[ 1944], 20.00th=[ 3184],
| 30.00th=[ 4512], 40.00th=[ 6304], 50.00th=[ 8768], 60.00th=[12224],
| 70.00th=[17024], 80.00th=[24448], 90.00th=[39168], 95.00th=[56064],
| 99.00th=[101888], 99.50th=[124416], 99.90th=[193536], 99.95th=[214016],
| 99.99th=[301056]
bw (KB /s): min=17020, max=50497, per=100.00%, avg=23912.19, stdev=6307.62
write: io=841688KB, bw=5998.6KB/s, iops=980, runt=140315msec
slat (usec): min=4, max=39869, avg=19.37, stdev=107.67
clat (usec): min=21, max=783299, avg=213.30, stdev=5197.09
lat (usec): min=37, max=783315, avg=233.09, stdev=5198.21
clat percentiles (usec):
| 1.00th=[ 42], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 52],
| 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 57], 60.00th=[ 61],
| 70.00th=[ 67], 80.00th=[ 74], 90.00th=[ 91], 95.00th=[ 112],
| 99.00th=[ 498], 99.50th=[ 644], 99.90th=[39168], 99.95th=[76288],
| 99.99th=[250880]
bw (KB /s): min= 4011, max=12818, per=100.00%, avg=6004.73, stdev=1656.37
lat (usec) : 50=2.72%, 100=16.77%, 250=2.17%, 500=0.60%, 750=0.57%
lat (usec) : 1000=0.42%
lat (msec) : 2=5.12%, 4=12.73%, 10=22.23%, 20=16.24%, 50=15.30%
lat (msec) : 100=4.27%, 250=0.84%, 500=0.02%, 750=0.01%, 1000=0.01%
cpu : usr=3.88%, sys=11.51%, ctx=571467, majf=0, minf=26877
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=23893KB/s, minb=23893KB/s, maxb=23893KB/s, mint=140315msec, maxt=140315msec
WRITE: io=841688KB, aggrb=5998KB/s, minb=5998KB/s, maxb=5998KB/s, mint=140315msec, maxt=140315msec

Disk stats (read/write):
dm-0: ios=550156/137856, merge=0/0, ticks=8920000/28816, in_queue=8952340, util=100.00%, aggrios=550281/138922, aggrmerge=0/176, aggrticks=8972672/37028, aggrin_queue=9009944, aggrutil=100.00%
sda: ios=550281/138922, merge=0/176, ticks=8972672/37028, in_queue=9009944, util=100.00%
 
This speeds are not that fast. Are you using SCSI with virtio controller or virtio as as Storage? What cache mode do you use?
 
I assume you are asking where the storage of the vm is. It is not on another storage device but is stored on the pve machine itself and the cache mode of the vm is set to Default None. The drive configuration is RAID10 on RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 05)
 
Good to know, but I asked for the virtualized storage adapter. You can normally choose IDE, SATA, SCSI and VIRTIO. With SCSI you have then the option to use different virtualized storage controllers, including a virtio controller. You should use either VIRTIO as storage type or SCSI with virtio controller.

Could you please provide the simulated storage?
 
OK I think I understand your question now. I was not aware of those options when I started this system up.
So I have a Hard Disk (ide0) as the drive for the virtual. And no way to change it either. :(
 
That explains a lot. IDE is dead slow.

Remove the disk, this frees the disk but it is still visible. Then double click on it which should open a dialog in which you can change to VIRTIO. Next is the tab on the right, where you have to change to boot order. Afterwards it should boot up. Depending on the intelligence of the running linux, it could work out of the box, but that's not sure.
 
OK so I should stop the running vm.
select the Hard Disk on the Hardware tab of the vm.
Click remove on the menu bar for that tab.
The Hard Disk will still be in the list but with no params?
I double click on that?
 
Yes. If you're unsure please create a new VM and try for yourself to familiarize you with the process. Do also a backup if you like.
 
Tried it out on a Centos 2.6 vm - after switching to VIRTIO the console showed no boot device available.
Had to switch back to IDE. :-(
 
iometer: (groupid=0, jobs=1): err= 0: pid=2188: Mon Jun 6 17:24:06 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3282.5MB, bw=15560KB/s, iops=3473 , runt=216019msec
slat (usec): min=4 , max=169221 , avg=33.91, stdev=272.93
clat (usec): min=33 , max=1541.7K, avg=18114.68, stdev=40564.83
lat (usec): min=43 , max=1541.7K, avg=18149.55, stdev=40565.45
clat percentiles (usec):
| 1.00th=[ 386], 5.00th=[ 1752], 10.00th=[ 2320], 20.00th=[ 3472],
| 30.00th=[ 4704], 40.00th=[ 6240], 50.00th=[ 8512], 60.00th=[11584],
| 70.00th=[15808], 80.00th=[22400], 90.00th=[36096], 95.00th=[55040],
| 99.00th=[175104], 99.50th=[280576], 99.90th=[552960], 99.95th=[675840],
| 99.99th=[995328]
bw (KB/s) : min= 7732, max=26738, per=100.00%, avg=15569.69, stdev=2409.81
write: io=833133KB, bw=3856.8KB/s, iops=865 , runt=216019msec
slat (usec): min=6 , max=112090 , avg=46.36, stdev=365.47
clat (usec): min=70 , max=634513 , avg=1030.20, stdev=5919.84
lat (usec): min=98 , max=634573 , avg=1077.63, stdev=5931.72
clat percentiles (usec):
| 1.00th=[ 155], 5.00th=[ 189], 10.00th=[ 203], 20.00th=[ 215],
| 30.00th=[ 227], 40.00th=[ 241], 50.00th=[ 262], 60.00th=[ 282],
| 70.00th=[ 310], 80.00th=[ 354], 90.00th=[ 524], 95.00th=[ 1432],
| 99.00th=[18560], 99.50th=[42752], 99.90th=[86528], 99.95th=[95744],
| 99.99th=[126464]
bw (KB/s) : min= 1646, max= 7090, per=100.00%, avg=3858.77, stdev=648.95
lat (usec) : 50=0.01%, 100=0.05%, 250=9.34%, 500=9.42%, 750=0.78%
lat (usec) : 1000=0.48%
lat (msec) : 2=4.77%, 4=13.94%, 10=25.16%, 20=17.37%, 50=13.88%
lat (msec) : 100=3.24%, 250=1.10%, 500=0.37%, 750=0.08%, 1000=0.02%
lat (msec) : 2000=0.01%
cpu : usr=4.78%, sys=20.94%, ctx=547379, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=750343/w=187049/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
READ: io=3282.5MB, aggrb=15559KB/s, minb=15559KB/s, maxb=15559KB/s, mint=216019msec, maxt=216019msec
WRITE: io=833133KB, aggrb=3856KB/s, minb=3856KB/s, maxb=3856KB/s, mint=216019msec, maxt=216019msec

Disk stats (read/write):
dm-0: ios=750060/190825, merge=0/0, ticks=13532229/998198, in_queue=14531661, util=100.00%, aggrios=749747/188238, aggrmerge=628/2783, aggrticks=13524756/612656, aggrin_queue=14137007, aggrutil=100.00%
vda: ios=749747/188238, merge=628/2783, ticks=13524756/612656, in_queue=14137007, util=100.00%
 
OK answer is yes :) but a little more work to install the virtio network interface from inside the vm.
system-config-network to add the new interface and configure it with the ip address of the old interface.
Up and running :)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom