Very low VM READ/Write performance

[subafr]

Hello, I encounter a problem while testing proxmox for our futur IT Structure.

I have vanilla promox 6 install on a 2x Xeon Silver 4214 with 196 Gb RAM and 2TB NVME.

On the host if I benchmark i have this ( READ ):
root@node01:~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1019MiB/s][r=261k IOPS][eta 00m:00s]

I create a simple VM with this settings:
root@node01:~# qm config 100
agent: 1
bootdisk: scsi0
cores: 24
ide2: local:iso/CentOS-7-x86_64-DVD-2003.iso,media=cdrom
memory: 32000
name: basesyscentos
net0: virtio=A2:A3:F2A:80:77,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsi0: local:100/vm-100-disk-0.qcow2,backup=0,cache=none,iothread=1,size=256G
scsihw: virtio-scsi-pci
smbios1: uuid=cf374ea9-6325-4359-9a7f-591fed4ab535
sockets: 2
vmgenid: 5ac023e9-7b1f-4a6a-b389-369831859671

If I benchmark this vm:
[administrateur@ip98 ~]$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=245MiB/s,w=0KiB/s][r=62.7k,w=0 IOPS][eta 00m:00s]

We got 70% slower performance from 1020mb/s to 245mb/s on the same physical disk...

I have try all cache type:
None
DirectSync
Writethrough
Writeback

But I have always this bad performance.

For the write performance its the same:
Directly on the host:
root@node01:~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
testio: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=626MiB/s][w=160k IOPS][eta 00m:00s]

On the VM:
[administrateur@ip98 ~]$ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
testio: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=241MiB/s][r=0,w=61.6k IOPS][eta 00m:00s]

What can I do to have a better performance like a native i/o ?

Thanks a lot for your help

 

[t.lamprecht]

Hi,

ide2: local:iso/CentOS-7-x86_64-DVD-2003.iso,media=cdrom

So you're using CentOS 7 in the VM? Which kernel version is booted (uname -a)?

You could try a VM using a Distro with a more modern Kernel, like a recent Ubuntu 20.04, Fedora 32 - the virtio-scsi driver got some improvements since the default 3.10 Kernel from CentOS 7, released one 30. June 2013.

You could also try uprading to a newer Kernel in the CentOS 7 VM, if you prefer using that.
Or just use CentOS 8 which ships with the 4.18 kernel.

It could be something else, but worth a try.

 

[subafr]

Thanks for your quick response:
For the benchmark i have this kernel:
[root@ip97 ~]# uname -a
Linux ip97.ip-92-222-105.eu 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


I have try your idea to upgrade the kernel in 5.x:
[root@ip97 ~]# uname -a
Linux ip97.ip-92-222-105.eu 5.7.2-1.el7.elrepo.x86_64 #1 SMP Tue Jun 9 15:37:42 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

The performance is a little better but so far away from the native:
[root@ip97 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=387MiB/s,w=0KiB/s][r=99.1k,w=0 IOPS][eta 00m:00s]

Thanks again for your help

 

[guletz]

Hi,

Inside of this VM, do you mount the / with fstrim and/or without noatime? Also be sure when you do your test disable cron service(in VM and in PMX)

Good luck /Bafta !

 

[t.lamprecht]

The IO scheduler inside the VM may also have something to say, you could check the current one for a /dev/sda drive like:
cat /sys/block/sda/queue/scheduler

If it is using "cfq" you could try using another one, like deadline:
echo deadline > /sys/block/sda/queue/scheduler
Or "noop":
echo noop > /sys/block/sda/queue/scheduler

As those provide often the better performance for NVMe drives.

 

[t.lamprecht]

Oh, and one important thing. Be sure to select the "SSD emulation" checkbox, the VM OS will do optimizations depending on that (cfq for example):

 

[subafr]

Hello all ! Thanks for all your reply !

"Inside of this VM, do you mount the / with fstrim and/or without noatime? Also be sure when you do your test disable cron service(in VM and in PMX)"

This is my host fstab:
root@node01:/var/lib/vz/template/iso# cat /etc/fstab
UUID="daa8e5cc-1d85-4c45-ba90-f86536d76bd8" / ext4 defaults 0 0
UUID="8fca5424-2da8-4a72-935d-d08bb95eff4c" /var/lib/vz ext4 defaults 0 0

And this is my VM fstab:
/dev/mapper/centos-root / xfs defaults 0 0
UUID=6a77d775-e443-48c3-a3a9-bcdf6ffa8871 /boot xfs defaults 0 0
/dev/mapper/centos-home /home xfs defaults 0 0

I don't understand this part:
"Also be sure when you do your test disable cron service(in VM and in PMX)"
You think cron will slow down the VM ?
I repeat in the host I dont have any performance problem:
Jobs: 1 (f=1): [r(1)][100.0%][r=1019MiB/s][r=261k IOPS][eta 00m:00s]


For the scheduler part, by default its:
[root@ip98 ~]# more /sys/block/sda/queue/scheduler
[mq-deadline] kyber bfq none

Performance with this one ( with SSD emulation checked ):
[root@ip98 ~]# more /sys/block/sda/queue/scheduler
[mq-deadline] kyber bfq none
[root@ip98 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
testio: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=366MiB/s,w=0KiB/s][r=93.6k,w=0 IOPS][eta 00m:00s]

Performance none in scheduler:
[root@ip98 ~]# echo none > /sys/block/sda/queue/scheduler
[root@ip98 ~]# more /sys/block/sda/queue/scheduler
[none] mq-deadline kyber bfq
[root@ip98 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=310MiB/s,w=0KiB/s][r=79.4k,w=0 IOPS][eta 00m:00s]

The last two performance test is with "SSD emulation enabled"

Thanks for your help, I try now to install a new VM with SSD Emulation enabled before install and with Centos 8.

 

[subafr]

With Centos 8 kernel 4.18:
[root@ip97 ~]# uname -a
Linux ip97.ip-92-222-105.eu 4.18.0-193.el8.x86_64 #1 SMP Fri May 8 10:59:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

VM settings:
root@node01:/var/lib/vz/template/iso# qm config 101
bootdisk: scsi0
cores: 24
ide2: local:iso/CentOS-8.2.2004-x86_64-dvd1.iso,media=cdrom
memory: 64000
name: testperf
net0: virtio=1A:4718:38:99,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsi0: local:101/vm-101-disk-0.qcow2,backup=0,size=256G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=7a5ce34c-963d-41ab-b37e-4bd078e66786
sockets: 2
vmgenid: 382c8f95-5c69-4f18-9087-5a6ff7e080d8

Same poor performance:
[root@ip97 ~]# uname -a
Linux ip97.ip-92-222-105.eu 4.18.0-193.el8.x86_64 #1 SMP Fri May 8 10:59:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@ip97 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=291MiB/s,w=0KiB/s][r=74.4k,w=0 IOPS][eta 00m:00s]

On the host:
root@node01:~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1005MiB/s][r=257k IOPS][eta 00m:00s]

This is really annoying

 

[t.lamprecht]

scsi0: local:100/vm-100-disk-0.qcow2,backup=0,cache=none,iothread=1,size=256G
scsihw: virtio-scsi-pci

Oh, something I just noticed now: If you want to use IOThreads then you should select the "virtio-scsi-single" as SCSI controller type, may not matter much with just one disk but as it's a quick change it's worth a try too.

Other things to try:
* using raw instead of qcow2
* using the local-lvm storage which should been created by default instead of local (file vs. block based storage)

Something more hackery could be trying to tell the VM that this is actually a NVME
NVMe can currently not yet be choosen as disk bus in PVE webinterface, so for this you would do the following:

qm set 101 --args '-drive file=/var/lib/vz/images/101/vm-101-disk-0.qcow2,if=none,id=D22 -device nvme,drive=D22,serial=1234,bootindex=101' --delete scsi0

To go back again:

qm set 101 --delete args --scsi0 local:101/vm-101-disk-0.qcow2,backup=0,size=256G,ssd=1

 

[subafr]

Hello:
Oh, something I just noticed now: If you want to use IOThreads then you should select the "virtio-scsi-single" as SCSI controller type, may not matter much with just one disk but as it's a quick change it's worth a try too.

I have checked IOThreads only to see if its better with, but no its the same, by the way I have try Virtio single but same result.

Other things to try:
* using raw instead of qcow2
* using the local-lvm storage which should been created by default instead of local (file vs. block based storage)

I have recreate a VM with centos 7 and kernel 5.x in a raw mode:
root@node01:/var/lib/vz/template/iso# qm config 103
agent: 1
bootdisk: scsi0
cores: 24
ide2: local:iso/CentOS-7-x86_64-DVD-2003.iso,media=cdrom
memory: 64000
name: test03
net0: virtio=4E:9D:E6:1B:98:3D,bridge=vmbr1,firewall=1
numa: 0
ostype: l26
scsi0: local:103/vm-103-disk-0.raw,backup=0,size=256G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=95f2fde7-faa6-421c-a7b2-5735f39adcdc
sockets: 2
vmgenid: 096a16c2-78a5-483f-b98e-37d01a65f154

Same result:
[root@ip99 ~]# uname -a
Linux ip99.ip-92-222-105.eu 5.7.2-1.el7.elrepo.x86_64 #1 SMP Tue Jun 9 15:37:42 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@ip99 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=373MiB/s,w=0KiB/s][r=95.5k,w=0 IOPS][eta 00m:00s]

If I enable none scheduler, its a little better but always 60% slower than native:
[root@ip99 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=410MiB/s,w=0KiB/s][r=105k,w=0 IOPS][eta 00m:00s]

* using the local-lvm storage which should been created by default instead of local (file vs. block based storage)
In my installation i dont have local-lvm but only local.
By the way at the start, i have created a ceph cluster ( nvme dedicated disk on each node ), but when i have see the poor performance, i have decided to try in "local" mode check if the performance is terrible too. And when i see the poor performance, i have created this post

qm set 101 --args '-drive file=/var/lib/vz/images/101/vm-101-disk-0.qcow2,if=none,id=D22 -device nvme,drive=D22,serial=1234,bootindex=101' --delete scsi0

If I do this, vm 101 don"t boot anymore.

Thanks again for your tips

 

[subafr]

If I create a LXC container, i'am more near native performance:
[root@testperf5 ~]# uname -a
Linux testperf5 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15 May 2020 15:06:08 +0200) x86_64 x86_64 x86_64 GNU/Linux
[root@testperf5 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
clock setaffinity failed: Invalid argument
clock setaffinity failed: Invalid argument
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=772MiB/s,w=0KiB/s][r=198k,w=0 IOPS][eta 00m:00s]

Something is wrong with kvm :-(

 

[guletz]

Hi,

Could you use instead of " --bs=4k" let say --bs=16k ?

 

[subafr]

With 16k, on host:
root@node01:~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=16k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1)
testio: (groupid=0, jobs=1): err= 0: pid=382890: Wed Jun 17 10:15:41 2020
read: IOPS=216k, BW=3382MiB/s (3547MB/s)(4096MiB/1211msec)

On VM ( raw disk ):
[root@ip99 ~]# uname -a
Linux ip99.ip-92-222-105.eu 5.7.2-1.el7.elrepo.x86_64 #1 SMP Tue Jun 9 15:37:42 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@ip99 ~]# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=16k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1)
testio: (groupid=0, jobs=1): err= 0: pid=1738: Wed Jun 17 12:16:43 2020
read: IOPS=93.1k, BW=1455MiB/s (1525MB/s)(4096MiB/2816msec)

I loose half I/O.

 

[guletz]

Hi,

So with a different block size, the problem is the same(50% less performance for IOPs and BW). The last thing, could you show the output for:

smartctl -a /dev/your-nvme

 

[subafr]

root@node01:~# smartctl -a /dev/nvme1n1p4
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.41-1-pve] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: SAMSUNG MZQLB1T9HAJR-00007
Serial Number: S439NA0MB10806
Firmware Version: EDA5202Q
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 1,920,383,410,176 [1.92 TB]
Unallocated NVM Capacity: 0
Controller ID: 4
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,920,383,410,176 [1.92 TB]
Namespace 1 Utilization: 118,669,811,712 [118 GB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Wed Jun 17 11:50:24 2020 UTC
Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x000f): Security Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size: 512 Pages
Warning Comp. Temp. Threshold: 87 Celsius
Critical Comp. Temp. Threshold: 88 Celsius
Namespace 1 Features (0x02): NA_Fields

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 10.60W - - 0 0 0 0 0 0

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
1 - 4096 0 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 42 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 0%
Data Units Read: 105,454,607 [53.9 TB]
Data Units Written: 46,186,870 [23.6 TB]
Host Read Commands: 643,488,916
Host Write Commands: 385,252,061
Controller Busy Time: 698
Power Cycles: 23
Power On Hours: 3,230
Unsafe Shutdowns: 12
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 42 Celsius
Temperature Sensor 2: 46 Celsius
Temperature Sensor 3: 52 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

Its very boring that virtualization always had IO bottleneck
I have thinked KVM should address this issue but that i see not a all.

Virtio SCSI is the best thing ? ( more native things ) May be it I can try other scsi controlleur type ?
I have try SATA mode, but its worst ... ( like 100mb/s )

 

[subafr]

I have try VMware scsi and vmdk disk image ... get 21mb/sec in read.

I really dont understant why is not straightforward to have decent performance ?
Is someone have a near native performance ? I'am curious to see other fio benchmark with NVME.

 

[subafr]

this is really weird, because with hdparm I dont see a lot of difference between host and VM.

HOST:
root@node02:/var/lib/vz# hdparm -Tt /dev/mapper/pve-data

/dev/mapper/pve-data:
Timing cached reads: 17920 MB in 2.00 seconds = 8970.25 MB/sec
Timing buffered disk reads: 5662 MB in 3.00 seconds = 1886.86 MB/sec

VM:
[root@ip100 ~]# hdparm -Tt /dev/sda
/dev/sda:
Timing cached reads: 16930 MB in 1.99 seconds = 8525.02 MB/sec
Timing buffered disk reads: 9114 MB in 3.00 seconds = 3037.15 MB/sec

DD style benchmark on host:
Write test
root@node02:/var/lib/vz# sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.630894 s, 1.7 GB/s

Cached Read:
root@node02:/var/lib/vz# dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.140753 s, 7.6 GB/s

Uncached Read:
root@node02:/var/lib/vz# /sbin/sysctl -w vm.drop_caches=3
vm.drop_caches = 3
root@node02:/var/lib/vz# dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.488127 s, 2.2 GB/s


DD style benchmark on VM:
Write test
[root@ip100 ~]# sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 enregistrements lus
1024+0 enregistrements écrits
1073741824 octets (1,1 GB) copiés, 0,589614 s, 1,8 GB/s

Cached read:
[root@ip100 ~]# dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 enregistrements lus
1024+0 enregistrements écrits
1073741824 octets (1,1 GB) copiés, 0,155419 s, 6,9 GB/s

Uncached read:
[root@ip100 ~]# /sbin/sysctl -w vm.drop_caches=3
vm.drop_caches = 3
[root@ip100 ~]# dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 enregistrements lus
1024+0 enregistrements écrits
1073741824 octets (1,1 GB) copiés, 0,264406 s, 4,1 GB/s

I dont why i have so much difference with FIO benchmark
I will try to setup an Oracle database and benchmark it to know our real performance.

thanks

 

[t.lamprecht]

I dont why i have so much difference with FIO benchmark

FIO is normally pretty near the worst case raw performance without any, or much less, optimizations.

* nvme controller:

If I do this, vm 101 don"t boot anymore.

You may need to select the boot manually, maybe also using "q35" machine type and maybe even use an OVMF bios, not sure from top of my head.

IMO, exposing NVMe, which is normally accessed in parallel with many queues over PCIe, as a SCSI or SATA device to the guest will make the guest OS handle it like such a device, and do not use many queues and other NVMe features. VirtIO driver could possibly expose that but it seems that they currently don't. That's why I suggested trying that controller type in the first place.