slow NVME disk performance on ZFS

[millosz222]

Hi,

I have with performance DISK in ZFS on VM in proxmox it looks nice but in VM i observe decrease performance

I have 3 disk NVME Corsair MP510 1,8TB all of them is in ZFS RAIDZ

When I did fio benchmark from proxmox in /dev/zvol/RAID I have really nice value

root@px01:/dev/zvol/RAID# fio --name=seqwrite --filename=seqwrite.fio --refill_buffers --rw=write --loops=3 --ioengine=libaio --bs=1m --size=5G --runtime=60 --group_reporting
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=2011MiB/s][w=2011 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=849808: Sun Jul 18 21:37:04 2021
write: IOPS=1939, BW=1939MiB/s (2034MB/s)(15.0GiB/7920msec); 0 zone resets
slat (usec): min=273, max=453, avg=303.07, stdev=27.67
clat (nsec): min=990, max=10317, avg=1143.23, stdev=176.85
lat (usec): min=274, max=460, avg=304.37, stdev=27.71
clat percentiles (nsec):
|  1.00th=[ 1020],  5.00th=[ 1032], 10.00th=[ 1048], 20.00th=[ 1048],
| 30.00th=[ 1064], 40.00th=[ 1080], 50.00th=[ 1112], 60.00th=[ 1128],
| 70.00th=[ 1160], 80.00th=[ 1208], 90.00th=[ 1272], 95.00th=[ 1368],
| 99.00th=[ 1592], 99.50th=[ 1704], 99.90th=[ 2352], 99.95th=[ 2928],
| 99.99th=[ 7776]
bw (  MiB/s): min= 1814, max= 2040, per=99.91%, avg=1937.60, stdev=89.59, samples=15
iops        : min= 1814, max= 2040, avg=1937.60, stdev=89.59, samples=15
lat (nsec)   : 1000=0.01%
lat (usec)   : 2=99.80%, 4=0.14%, 10=0.03%, 20=0.01%
cpu          : usr=42.75%, sys=57.24%, ctx=21, majf=0, minf=14
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,15360,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=1



Run status group 0 (all jobs):
  WRITE: bw=1939MiB/s (2034MB/s), 1939MiB/s-1939MiB/s (2034MB/s-2034MB/s), io=15.0GiB (16.1GB), run=7920-7920msec

but when I did it from VM (centos 8.3) i have problem with performance

[root@localhost ~]# fio --name=seqwrite --filename=seqwrite.fio --refill_buffers --rw=write --loops=3 --ioengine=libaio --bs=1m --size=5G --runtime=60 --group_reporting
seqwrite: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.19
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=1110MiB/s][w=1109 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=3280: Sun Jul 18 15:39:47 2021
write: IOPS=719, BW=720MiB/s (755MB/s)(15.0GiB/21346msec); 0 zone resets
slat (usec): min=342, max=12585, avg=728.40, stdev=578.64
clat (usec): min=2, max=1113, avg= 3.14, stdev= 9.27
lat (usec): min=345, max=12595, avg=732.24, stdev=579.06
clat percentiles (usec):
|  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
| 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
| 70.00th=[    4], 80.00th=[    4], 90.00th=[    4], 95.00th=[    4],
| 99.00th=[    6], 99.50th=[   11], 99.90th=[   18], 99.95th=[   20],
| 99.99th=[  265]
bw (  KiB/s): min=49053, max=1655568, per=100.00%, avg=992474.52, stdev=436697.03, samples=31
iops        : min=   47, max= 1616, avg=968.77, stdev=426.49, samples=31
lat (usec)   : 4=96.55%, 10=2.88%, 20=0.53%, 50=0.03%, 500=0.01%
lat (msec)   : 2=0.01%
cpu          : usr=16.18%, sys=61.19%, ctx=1097, majf=0, minf=13
IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,15360,0,0 short=0,0,0,0 dropped=0,0,0,0
latency   : target=0, window=0, percentile=100.00%, depth=1


Run status group 0 (all jobs):
WRITE: bw=720MiB/s (755MB/s), 720MiB/s-720MiB/s (755MB/s-755MB/s), io=15.0GiB (16.1GB), run=21346-21346msec


Disk stats (read/write):
dm-0: ios=0/7069, merge=0/0, ticks=0/101322, in_queue=101322, util=47.50%, aggrios=0/13676, aggrmerge=0/0, aggrticks=0/175770, aggrin_queue=175769, aggrutil=51.85%
sda: ios=0/13676, merge=0/0, ticks=0/175770, in_queue=175769, util=51.85%

I have proxmox 7.0.8
and the configuration of VM it looks like below

agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 8
efidisk0: RAID:vm-102-disk-0,size=1M
ide2: local:iso/CentOS-Stream-8-x86_64-20210706-boot.iso,media=cdrom
machine: q35
memory: 24048
name: centos.raid
net0: virtio=B6:F2:FC:8D:D5:97,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: RAID:vm-102-disk-1,size=120G
scsihw: virtio-scsi-pci
smbios1: uuid=1e296c70-e03c-4748-a3ae-3a56b08200e1
sockets: 2
vga: virtio
vmgenid: 9d132687-a8de-4fb5-b7b9-79432b9c7e5f

Do you have any idea how i can get full performance on my NVME disk?

 

[ph0x]

I would consider this excellent performance values for inside a VM.

 

[millosz222]

are you sure that it is good performace?

why there is difference?

 

[Dunuin]

You are writing to a virtualized disk. Virtio is paravirtualized, but your CPU still needs to convert everything between the guest and host. So you always get some overhead. Also your hosts uses fast NVMe as protocol but your guest is using the way slower SCSI protocol. Here you loose performance again.
If you want the full performance you need to PCI passthrough the drives to a single VM. so the VM can directly and physically access the drives without the virtualization layer in between.