VirtIO vs SCSI

[tuxis]

Some VMs get huge disk I/O performance increase, simply by switching the disk bus/device to VirtIO Block.

Results with SCSI:

[root@acrux testjes]# dd if=/dev/zero of=testfile bs=1M count=1000 oflag=dsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 93.3164 s, 11.2 MB/s
[root@acrux testjes]# dd if=/dev/zero of=testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 20.9574 s, 51.2 MB/s

Results with VirtIO block:

[root@acrux testjes]# dd if=/dev/zero of=testfile bs=1M count=1000 oflag=dsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 42.9133 s, 24.4 MB/s
[root@acrux testjes]# dd if=/dev/zero of=testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 5.74577 s, 187 MB/s

Can this huge performance increase be expected by using VirtIO? It seems odd that such a seemingly small change has such a large impact

Background information: Ceph cluster with 3 nodes. SCSI controller used is VirtIO SCSI (in both tests).

 

[wolfgang]

Hi,

generally, a test with dd is not really meaningful.
Use fio instead.

What I can tell is that the scsi virtio is better maintained and virtio-blk is the older one.

 

[tuxis]

Hi,

generally, a test with dd is not really meaningful.
Use fio instead.

What I can tell is that the scsi virtio is better maintained and virtio-blk is the older one.

I used `fio` afterwards to test the best-performing setups according to `dd`. VirtIO SCSI controller with SCSI bus for disk with SSD emulation and writeback cache is a winner. Actually, stuff like IO thread with VirtIO SCSI single controller makes things perform worse. I was also surprised to see that SSD emulation makes a significant difference..

https://pve.proxmox.com/wiki/Performance_Tweaks states: "Use virtIO for disk and network for best performance."

Do they mean VirtIO controller or VirtIO bus?

 

[lilszi]

I used `fio` afterwards to test the best-performing setups according to `dd`. VirtIO SCSI controller with SCSI bus for disk with SSD emulation and writeback cache is a winner. Actually, stuff like IO thread with VirtIO SCSI single controller makes things perform worse. I was also surprised to see that SSD emulation makes a significant difference..

https://pve.proxmox.com/wiki/Performance_Tweaks states: "Use virtIO for disk and network for best performance."

Do they mean VirtIO controller or VirtIO bus?

I just signed up to say thank you for this post, I was having performance issues in the VM's on 6g SSD's pulling only 50MB/s
With your Recommendations, it's now up to 150MB/s

 

[mir]

and writeback cache is a winner.

This is dangerous since writes are written to memory and first latter persisted to storage.

 

[David Herselman]

Yes and no... Writes are buffered but flushes are honoured so it would be transactionally safe. Only 'unsafe' ignores flushes...

 

[mir]

Writes are buffered but flushes are honoured so it would be transactionally safe

This is entirely up to the host to deside. write-back cache is equivalent to a hardware raid controller.

 

[virtRoo]

This is dangerous since writes are written to memory and first latter persisted to storage.

And may also affect how live migration works as well.

 

[Proxygen]

Wouldn't SATA be faster than any SCSI combination? I don't need more than 6 devices ever. It seems to be the official recommendation for performance.

https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_hard_disk

Which brings me to (forgive me if this warrants another thread): is it possible to convert a VM's disk? Is this still a valid solution?

 

[gb00s]

... stuff like IO thread with VirtIO SCSI single controller makes things perform worse.

On a single bench maybe, but on a real-life system with several i/o-intensive workloads (VM's and databases), why i/o-threading should be a disadvantage if i/o-threading give you several uncorrelated i/o-venues? Without i/o-threading everything stays in line and has to wait until it is to be handled.

I'm curious now ...

 

[jdancer]

According to this post https://forum.proxmox.com/threads/virtio-scsi-vs-virtio-scsi-single.28426 virtio uses a single controller per disk just like for virtio scsi single.

As to which one is "faster", no idea.

 

[Proxygen]

From what I gathered after a lot of research, including some benchmarks

Real physical SATA is the fastest. But this is an emulated environment, so the fastest option is VirtioSCSI.
Within the device, chose writeback cache (the safe option, not the unsafe one)

 

[christian.g]

I know this is an old thread but i'm currently investigating IOPs on our Ceph cluster.

Test VM Debian 10 (Kernel 4.19)
Test Suite fio with 4k randrw test
Every test was repeated 3 times

First tests to raw rdb block devices from within the VM to identify the best BUS (SCSI vs Virtio) with no cache and writeback cache

BUS	Cache	IOThread	SSD Emulation	Discard	IOPS Read (k)	IOPS Write (k)	BW Read (MB/s)	BW Write (MB/s)
SCSI	None	0	0	0	14 k	6 k	24 MB/s	24 MB/s
SCSI	Writeback	1	1	1	40 k	17 k	164 MB/s	71 MB/s
SCSI	Writeback	0	1	1	40 k	17 k	165 MB/s	71 MB/s
SCSI	Writeback	0	0	0	40 k	17 k	165 MB/s	71 MB/s
SCSI	Writeback	0	0	1	39 k	17 k	163 MB/s	70 MB/s
Virtio	None	0	0	0	14 k	6 k	54 MB/s	22 MB/s
Virtio	Writeback	1	0	1	46 k	20 k	187 MB/s	79 MB/s
Virtio	Writeback	0	0	1	45 k	19 k	185 MB/s	79 MB/s
Virtio	Writeback	0	0	0	45 k	19 k	185 MB/s	79 MB/s

I dropped "None" Cache combinations as the first test showed that writeback is definitely the winner.

So far IOThread didn't slow down anything.

Next i did tests on a mounted ext4 filesystem (not rootfs) and suprinsingly IOThread indeed did slow down things.

BUS	Cache	IOThread	SSD Emulation	Discard	IOPS Read (k)	IOPS Write (k)	BW Read (MB/s)	BW Write (MB/s)
Virtio	Writeback	1	0	1	36 k	16 k	148 MB/s	64 MB/s
Virtio	Writeback	0	0	1	41 k	18 k	169 MB/s	72 MB/s
Virtio	Writeback	0	0	0	45 k	19 k	183 MB/s	78 MB/s
Virtio	Writeback	1	0	0	38 k	16 k	157 MB/s	67 MB/s

I was also surprised that activating discard did slow down things too.

It looks like Virtio with writeback cache and all other options disabled provides the best performance for ext4.
Other filesystems may behave differently.

But is using writeback cache really safe when using live migration? What are the possible problems?

Update:
Doing the raw device tests on Debian 11 (Kernel 5.10) did increase overall performance by 10-15%

BUS	Cache	IOThread	SSD Emulation	Discard	IOPS Read (k)	IOPS Write (k)	BW Read (MB/s)	BW Write (MB/s)
Virtio	Writeback	0	0	0	56 k	24 k	230 MB/s	99 MB/s

Doing the filesystem tests on Debian 11 (Kernel 5.10) did decrease overall performance by 10-15% which is very frustrating.

BUS	Cache	IOThread	SSD Emulation	Discard	IOPS Read (k)	IOPS Write (k)	BW Read (MB/s)	BW Write (MB/s)
Virtio	Writeback	0	0	0	38 k	17 k	160 MB/s	68 MB/s

 

[guletz]

It looks like Virtio with writeback cache and all other options disabled provides the best performance for ext4.
Other filesystems may behave differently.

But is using writeback cache really safe when using live migration? What are the possible problems?

Hi,

Write-back cache is not so safe, because the vDisk will cache in RAM all this data(if buffer will be sufficient large) and will tell to the applications ... OK, I write the data on disk, go on .... When buffers will be full or at X seconds(OS dependent, for ex. Windows use 60 sec. if I remember, and around 5-10 sec. for linux) will be flushed on the vDisk. The real danger is that betheen 2 consecutive data flushes , if anything bad will be happend(like OS crash, host crash, power lose, and so on) then you lost this buffered data(who reside in RAM) .

Live Migration of any VM WITH qemu-guest active and operational, will be ok(live migration will start to flush any buffered data on vDisk, and after that will start to move the vDisk). Without qemu-guest ... yes, you can lose your data if during the proccess(if the VM or the node will crash).

Also note that discard option disabled, is not so good as you think, because the backend storage of the vDisk, will not be able to delete any unused blocks who are deleted by the OS VM filesystem.

Good luck / Bafta !

 

[christian.g]

Write-back cache is not so safe, because the vDisk will cache in RAM all this data(if buffer will be sufficient large) and will tell to the applications ... OK, I write the data on disk, go on .... When buffers will be full or at X seconds(OS dependent, for ex. Windows use 60 sec. if I remember, and around 5-10 sec. for linux) will be flushed on the vDisk. The real danger is that betheen 2 consecutive data flushes , if anything bad will be happend(like OS crash, host crash, power lose, and so on) then you lost this buffered data(who reside in RAM) .

Sure, that's what writeback does.

Live Migration of any VM WITH qemu-guest active and operational, will be ok(live migration will start to flush any buffered data on vDisk, and after that will start to move the vDisk). Without qemu-guest ... yes, you can lose your data if during the proccess(if the VM or the node will crash).

Ok, so the usual writeback drawbacks. Thanks.

Also note that discard option disabled, is not so good as you think, because the backend storage of the vDisk, will not be able to delete any unused blocks who are deleted by the OS VM filesystem.

Yes, but this only applies to flash storage. I just wanted to point out that discard impacts performance. In my case i need discard but this does not automatically applies to others. I've tested on flash storage, would be interesting to see if the "discard" options also impacts performance on spinning disks.

 

[guletz]

Yes, but this only applies to flash storage

Not true!

 

[christian.g]

Not true!

True but i would appreciate if instead of just saying "Not true" you would provide usable information.

I wrongly interpreted the Discard Option! It is about thin provisioning and not about flash storage.

"Disk images in Proxmox are sparse regardless of the image type, meaning the disk image grows slowly as more data gets stored in it. Over time, data gets created and deleted within the filesystem of the disk image. But in a sparse disk image, even after data is deleted, it never reclaims the free space. The VM may report the correct available storage space but Proxmox storage will show higher storage usage. The Discard option allows the node to reclaim the free space that does not have any data. This is equivalent to the TRIM option that was introduced in SSD drives. Before this option can be used, we have to ensure that the VM uses the VirtIO SCSI controller. We can set the SCSI Controller Type under virtual machine's Options tab: ..."
https://www.oreilly.com/library/vie...05/03431488-8696-41e3-92e2-a60482b6e4e9.xhtml

Or
https://gist.github.com/hostberg/86bfaa81e50cc0666f1745e1897c0a56

 

[LnxBil]

Keep also in mind that any host-side caching settings (everything besides NONE) will need additional RAM on the host/hypervisor and often people complain about PVE using too much memory. That's why. It can also be the case that you cache on multiple layers, e.g. with ZFS, NONE is the default and should also be the default if you don't want to cache twice (only on the host, there may also be a cache inside of your guest OS)

 

[christian.g]

All of your points are valid but in the end the Cluster has to be fast enough to fit the needs and with Cache "None" it's not.
I generally favor safety over performance but in this case the numbers are far to bad and users have to do their work without going to have a coffee until the applications are done with the work.

 

[VictorSTS]

Please, would you mind to share the whole fio command you are using to get those numbers?
Thanks!