hdd performance

B

big_blue

Member
Jan 31, 2023
10
0
6
Hi. I am using Proxmox since about 2 years and it is working without problems. Now I want to optimize the performance because of some issues and have found some very strange behavior.

So I bought very similar hardware to compare and tune the system to improve the performance.

This is my setup:
AMD Ryzen 7 5700G
64 GB RAM / 128 GB RAM
2x Kingston 2TB nvme

The installation is on the NVME and is a mirror ZFS filesystem. The "productive" system is running on Proxmox 8, the new system is running on the latest 9.

To get a value for the maximum of HDD speed I run a fio test on the Proxmox server with the following command:
fio --name=write-test --size=1G --filename=fio-testfile --bs=128k --rw=write --direct=1 --numjobs=1 --time_based --runtime=30 --group_reporting
Click to expand...
This command show about 960MiB/s, which is a very bad value by the way. I also have done this test on the "prod" environment and there the value is about 4295MiB/s.
Does someone of you have a similar issue? Is there something wrong with my setup?

I started from scratch with the new setup and will follow some best practices. Please let me know if there is something wrong.

cheers
bb
 
I am using the 2x Kingston 2TB nvme, with a mirror zfs. No additional optimization. I used the installer for the ZFS setup.
 
How to get the information you need. This is the nvme I am using: KINGSTON SFYRD2000G
and this is the output of spool status:
zpool status
pool: rpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-eui.00000000000000000026b768607e7405-part3 ONLINE 0 0 0
nvme-eui.00000000000000000026b76868b01915-part3 ONLINE 0 0 0
Click to expand...
 
Ok, thanks, 2x KINGSTON SFYRD2000G on witch Mainboard?

* Do you check the PCIe Speed and the avaible PCIe lines?

This is a zfs server and mybe you don't tune it in a proper way.

I Run your fio command on a zfs pool with 2x Crucial MX500 (SATA III) on a intel core i5-12400 desktop pc, zfs-2.2.2 and got:
Code: 
fio --name=write-test --size=1G --filename=fio-testfile --bs=128k --rw=write --direct=1 --numjobs=1 --time_based --runtime=30 --group_reporting
write-test: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1
fio-3.36
Starting 1 process
write-test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]                        
write-test: (groupid=0, jobs=1): err= 0: pid=6877: Sun Dec 28 11:57:05 2025
  write: IOPS=3789, BW=474MiB/s (497MB/s)(13.9GiB/30084msec); 0 zone resets
    clat (usec): min=8, max=1568.4k, avg=262.15, stdev=16367.20
     lat (usec): min=9, max=1568.4k, avg=263.45, stdev=16367.20
    clat percentiles (usec):
     |  1.00th=[     10],  5.00th=[     10], 10.00th=[     11],
     | 20.00th=[     13], 30.00th=[     15], 40.00th=[     18],
     | 50.00th=[     26], 60.00th=[     46], 70.00th=[     59],
     | 80.00th=[     64], 90.00th=[    117], 95.00th=[    139],
     | 99.00th=[    174], 99.50th=[    186], 99.90th=[    269],
     | 99.95th=[    310], 99.99th=[1384121]
   bw (  KiB/s): min=19200, max=2223616, per=100.00%, avg=810086.17, stdev=596222.71, samples=36
   iops        : min=  150, max=17372, avg=6328.78, stdev=4658.01, samples=36
  lat (usec)   : 10=5.97%, 20=38.10%, 50=20.16%, 100=25.13%, 250=10.51%
  lat (usec)   : 500=0.11%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.86%, sys=9.96%, ctx=25282, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,113997,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=13.9GiB (14.9GB), run=30084-30084msec
this is what i get und its ok.
 
Last edited:
Hi, the board is a ASROCK, X570 Phantom Gaming 4. link
Yes I checked the speed:
LnkSta: Speed 8GT/s (downgraded), Width x4 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkSta: Speed 16GT/s, Width x4 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

It looks like the first NVME is downgraded to PCIe3.

I am a little bit frustrated because of the same system run with the exactly same components expect ram size with about 4000mb/s speed.
 
Thank you.

Your AMD Ryzen 7 5700G CPU combined with ASROCK, X570 Phantom Gaming 4 has "only" PCIe 3.0 16+4+4 CPU Lanes.
So your PCIe 4.0/3.0 x16 slot get only PCIe 3.0 x16 Lanes max

# https://www.techpowerup.com/cpu-specs/ryzen-7-5700g.c2472

Please see top left AM4 X570 Chipset:
# https://tpucdn.com/cpu-specs/images/connectivity/amd-am4.png

Please see too: ASROCK X570 Phantom Gaming 4:
# https://pg.asrock.com/MB/AMD/X570 Phantom Gaming 4/index.asp#Specification
  • AMD Ryzen series CPUs (Cezanne, Renoir, Pinnacle Ridge)
    - 1 x PCI Express 3.0 x16 Slot (x16 (PCIE1))*
  • Storage
    1 x Hyper M.2 Socket (M2_1), supports M Key type 2230/2242/2260/2280/22110 M.2 PCI Express module up to Gen3x4 (32 Gb/s) (with Cezanne, Renoir, Pinnacle Ridge and Picasso)*
 
Last edited:
  • Like
Reactions: Kingneutron
> It looks like the first NVME is downgraded to PCIe3

You will likely get better performance by doing a zpool detach on the 2nd nvme, and re-using it for data / LXC/VM vdisks.

PCIe3 is OK for proxmox boot/root, but you want better I/O for your virtual environment. And make sure you have backups.

Also, PROTIP - if you're doing zfs mirroring, buy 2 different make/model of SSD. You kind of want one to die first (think EVO + Pro), not have both die around the same time due to identical wear patterns.
 
@news: Thank you very much for your support. All this time, I thought I was getting full speed because I'm using a 5000 series CPU.

Just for fun, I ran another test. I installed Proxmox on a SATA SSD, formatted the individual SSDs with XFS, and repeated the FIO test. Both hard drives show exactly the same performance at around 2200 Mb/s. As a layman, I would have expected a difference, right?

Long story short, I want to upgrade my hardware. Due to the current hardware prices, I am now looking for a combination with DDR4 RAM that will give me full speed. I have currently chosen the 5800X CPU. That should do the trick, right?
Now I just need a motherboard that can accommodate all the necessary components.
- 10 GBit Ethernet (PCIe X8)
- TV card (PCIe X1)
- Graphics card (system) only for installation and to connect a monitor
- Graphics card (PCIe X16) will be integrated into a VM

Does anyone have a recommendation for a motherboard that can accommodate these components?
 
Please think of:
You need on long term PCIe Lanes; PCI-Express: Gen 4, 128 Lanes (CPU only)
Please read the Mainboard Manual.

Add: a german youtube cannel:
# https://youtu.be/sMtADQyDtBg?si=_IXoKG0tWXEMX_wR

A German hardware compare site with this parts and even faster and bigger CPU: AMD Ryzen Threadripper PRO 5955WX, 16C/32T:
# https://geizhals.de/wishlists/4860102
 
Last edited:
You must log in or register to reply here.
Top Bottom