Hello
There is a:
I was going to build it for a home lab, for virtualization.
I installed Proxmox 8.2.2 on it, in general everything is fine, but a couple of days ago I noticed that the read/write speed when measured from the hypervisor and from a virtual machine (centos, vitio drivers are present) differs by ~30 times. I tried everything that is advised on the proxmox forum: manipulating cache settings, types of virtual controllers, etc. - the result practically does not change.
I assumed that cheap NVME without a DRAM buffer is to blame. I bought a disk with DRAM - the indicators are absolutely the same.
command for testing:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75
I conducted the test on both an empty disk and one filled to 85% - the results were the same:
The question is: is such a difference in speeds in the hypervisor and in the virtual machine normal or not?
If not, where can I dig? Can any BIOS settings affect the situation?
P.S. The investigation was prompted by the need to install a cluster system, so the installation kept crashing at random stages. After several attempts, it became clear that the problem was in the SSD speed.
There is a:
huananzhi x99-F8D-pus
2x E5-2697v4
256GB DDR4 ECC Samsung
2x NVME xishuo 4x 3.0 1TB
2x E5-2697v4
256GB DDR4 ECC Samsung
2x NVME xishuo 4x 3.0 1TB
I was going to build it for a home lab, for virtualization.
I installed Proxmox 8.2.2 on it, in general everything is fine, but a couple of days ago I noticed that the read/write speed when measured from the hypervisor and from a virtual machine (centos, vitio drivers are present) differs by ~30 times. I tried everything that is advised on the proxmox forum: manipulating cache settings, types of virtual controllers, etc. - the result practically does not change.
I assumed that cheap NVME without a DRAM buffer is to blame. I bought a disk with DRAM - the indicators are absolutely the same.
command for testing:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75
I conducted the test on both an empty disk and one filled to 85% - the results were the same:
fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.25
Starting 1 process
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=1133MiB/s,w=379MiB/s][r=290k,w=97.0k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=4635: Fri Aug 9 16:02:38 2024
read: IOPS=290k, BW=1131MiB/s (1186MB/s)(6141MiB/5428msec)
bw ( MiB/s): min= 1123, max= 1135, per=99.84%, avg=1129.59, stdev= 4.39, samples=10
iops : min=287540, max=290682, avg=289175.60, stdev=1123.97, samples=10
write: IOPS=96.7k, BW=378MiB/s (396MB/s)(2051MiB/5428msec); 0 zone resets
bw ( KiB/s): min=384592, max=390632, per=99.88%, avg=386426.40, stdev=2273.09, samples=10
iops : min=96148, max=97658, avg=96606.60, stdev=568.27, samples=10
cpu : usr=33.52%, sys=66.48%, ctx=12, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1131MiB/s (1186MB/s), 1131MiB/s-1131MiB/s (1186MB/s-1186MB/s), io=6141MiB (6440MB), run=5428-5428msec
WRITE: bw=378MiB/s (396MB/s), 378MiB/s-378MiB/s (396MB/s-396MB/s), io=2051MiB (2150MB), run=5428-5428msec
fio-3.25
Starting 1 process
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=1133MiB/s,w=379MiB/s][r=290k,w=97.0k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=4635: Fri Aug 9 16:02:38 2024
read: IOPS=290k, BW=1131MiB/s (1186MB/s)(6141MiB/5428msec)
bw ( MiB/s): min= 1123, max= 1135, per=99.84%, avg=1129.59, stdev= 4.39, samples=10
iops : min=287540, max=290682, avg=289175.60, stdev=1123.97, samples=10
write: IOPS=96.7k, BW=378MiB/s (396MB/s)(2051MiB/5428msec); 0 zone resets
bw ( KiB/s): min=384592, max=390632, per=99.88%, avg=386426.40, stdev=2273.09, samples=10
iops : min=96148, max=97658, avg=96606.60, stdev=568.27, samples=10
cpu : usr=33.52%, sys=66.48%, ctx=12, majf=0, minf=10
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1131MiB/s (1186MB/s), 1131MiB/s-1131MiB/s (1186MB/s-1186MB/s), io=6141MiB (6440MB), run=5428-5428msec
WRITE: bw=378MiB/s (396MB/s), 378MiB/s-378MiB/s (396MB/s-396MB/s), io=2051MiB (2150MB), run=5428-5428msec
Starting 1 process
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=35.1MiB/s,w=11.9MiB/s][r=8991,w=3036 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=111002: Fri Aug 9 16:58:51 2024
read: IOPS=8973, BW=35.1MiB/s (36.8MB/s)(6141MiB/175196msec)
bw ( KiB/s): min=18256, max=40910, per=99.97%, avg=35883.94, stdev=4331.34, samples=350
iops : min= 4564, max=10227, avg=8970.96, stdev=1082.83, samples=350
write: IOPS=2996, BW=11.7MiB/s (12.3MB/s)(2051MiB/175196msec)
bw ( KiB/s): min= 6440, max=14008, per=99.97%, avg=11982.88, stdev=1442.82, samples=350
iops : min= 1610, max= 3502, avg=2995.69, stdev=360.70, samples=350
cpu : usr=3.26%, sys=66.62%, ctx=496196, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=35.1MiB/s (36.8MB/s), 35.1MiB/s-35.1MiB/s (36.8MB/s-36.8MB/s), io=6141MiB (6440MB), run=175196-175196msec
WRITE: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=2051MiB (2150MB), run=175196-175196msec
Disk stats (read/write):
sda: ios=1571546/528954, merge=29/4917, ticks=1015984/258375, in_queue=1274674, util=100.00%
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=35.1MiB/s,w=11.9MiB/s][r=8991,w=3036 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=111002: Fri Aug 9 16:58:51 2024
read: IOPS=8973, BW=35.1MiB/s (36.8MB/s)(6141MiB/175196msec)
bw ( KiB/s): min=18256, max=40910, per=99.97%, avg=35883.94, stdev=4331.34, samples=350
iops : min= 4564, max=10227, avg=8970.96, stdev=1082.83, samples=350
write: IOPS=2996, BW=11.7MiB/s (12.3MB/s)(2051MiB/175196msec)
bw ( KiB/s): min= 6440, max=14008, per=99.97%, avg=11982.88, stdev=1442.82, samples=350
iops : min= 1610, max= 3502, avg=2995.69, stdev=360.70, samples=350
cpu : usr=3.26%, sys=66.62%, ctx=496196, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=35.1MiB/s (36.8MB/s), 35.1MiB/s-35.1MiB/s (36.8MB/s-36.8MB/s), io=6141MiB (6440MB), run=175196-175196msec
WRITE: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=2051MiB (2150MB), run=175196-175196msec
Disk stats (read/write):
sda: ios=1571546/528954, merge=29/4917, ticks=1015984/258375, in_queue=1274674, util=100.00%
The question is: is such a difference in speeds in the hypervisor and in the virtual machine normal or not?
If not, where can I dig? Can any BIOS settings affect the situation?
P.S. The investigation was prompted by the need to install a cluster system, so the installation kept crashing at random stages. After several attempts, it became clear that the problem was in the SSD speed.