Slow (or unexpected) NVme speed even with ext4

V

VDNKH

Renowned Member
Aug 8, 2016
57
2
73
29
Hi, it's me again.
Dear forum friends,

Today I'm here with a question that has me thinking and that I can't seem to make sense of.

I have a server with very good hardware specifications. In fact, I posted about it a while ago to see if I should install ZFS in RAID 10 to host my VMs.

These are the specs:

Dual Intel Xeon Gold 6554S - 2x36c/2x72t - 2.2GHz/3GHz
1.5TB DDR5 ECC 4800MHz
2 x 960GB Micron 7450 Datacenter Class NVMe SSDs (for the OS)
18 x 3.84TB Micron 7500 NVMe SSDs

I used two 3.84TB Micron 7500 SSDs formatted with ext4 to run a test, transferring a backup file from a VM from one NVMe drive to another, and the speed varied between 1GB/s and 1.8GB/s. However, these NVMe drives claim to reach up to 6800MB/s, which is a big difference.

Within a Windows virtual machine, copying a file from one virtual drive to another, I get practically the same speed. Even with a RAID1 or RAID10 ZFS, but I never reach 6800MB/s even with ext4; they all seem to be limited to that speed.

The server isn't slow at all; it backs up VMs very quickly, but I've noticed that it doesn't exceed that write speed, never going over 2GB/s.

If you have any advice, I'd really appreciate it!

I want to at least see if I can improve the speed with ext4 and then move on to ZFS.

im using proxmox 8.4
 
I'd recommend you do these tests with fio instead. Note that you often only get the advertised speeds with higher iodepths.
 
Last edited:
  • Like
Reactions: VDNKH
Impact said:
I'd recommend you do these tests with fio instead. Note that you oftne only get the advertised speeds with higher iodepths.
Click to expand...
I did a sequential write test, quite heavy and the speeds are quite high, but I'm worried about the small/random ones.
 
VDNKH said:
I did a sequential write test, quite heavy and the speeds are quite high, but I'm worried about the small/random ones.
Click to expand...
I used this on my ZFS RAID10:

fio --name=multi_write_benchmark --filename=test_write --rw=write --bs=1M --iodepth=32 --numjobs=6 --time_based --runtime=60s --group_reporting --eta-newline=5s --direct=1 --ioengine=libaio --size=40G

and the result is:

Run status group 0 (all jobs):
WRITE: bw=21.4GiB/s (22.9GB/s), 21.4GiB/s-21.4GiB/s (22.9GB/s-22.9GB/s), io=1281GiB (1376GB), run=60002-60002msec

a little crazy
 
Instead with this one:

fio --name=vm_sql_file_test --filename=ttest_file.img --rw=randrw --bs=8k --iodepth=32 --numjobs=4 --time_based --runtime=300s --group_reporting --eta-newline=5s --direct=1 --ioen
gine=libaio --fsync=1 --size=10G

I get this resul:

Run status group 0 (all jobs):
READ: bw=187MiB/s (196MB/s), 187MiB/s-187MiB/s (196MB/s-196MB/s), io=54.7GiB (58.7GB), run=300001-300001msec
WRITE: bw=186MiB/s (196MB/s), 186MiB/s-186MiB/s (196MB/s-196MB/s), io=54.6GiB (58.7GB), run=300001-300001msec
 
You must log in or register to reply here.
Top Bottom