hdd performance

B

big_blue

Member
Jan 31, 2023
12
0
6
Hi. I am using Proxmox since about 2 years and it is working without problems. Now I want to optimize the performance because of some issues and have found some very strange behavior.

So I bought very similar hardware to compare and tune the system to improve the performance.

This is my setup:
AMD Ryzen 7 5700G
64 GB RAM / 128 GB RAM
2x Kingston 2TB nvme

The installation is on the NVME and is a mirror ZFS filesystem. The "productive" system is running on Proxmox 8, the new system is running on the latest 9.

To get a value for the maximum of HDD speed I run a fio test on the Proxmox server with the following command:
fio --name=write-test --size=1G --filename=fio-testfile --bs=128k --rw=write --direct=1 --numjobs=1 --time_based --runtime=30 --group_reporting
Click to expand...
This command show about 960MiB/s, which is a very bad value by the way. I also have done this test on the "prod" environment and there the value is about 4295MiB/s.
Does someone of you have a similar issue? Is there something wrong with my setup?

I started from scratch with the new setup and will follow some best practices. Please let me know if there is something wrong.

cheers
bb
 
I am using the 2x Kingston 2TB nvme, with a mirror zfs. No additional optimization. I used the installer for the ZFS setup.
 
How to get the information you need. This is the nvme I am using: KINGSTON SFYRD2000G
and this is the output of spool status:
zpool status
pool: rpool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-eui.00000000000000000026b768607e7405-part3 ONLINE 0 0 0
nvme-eui.00000000000000000026b76868b01915-part3 ONLINE 0 0 0
Click to expand...
 
Ok, thanks, 2x KINGSTON SFYRD2000G on witch Mainboard?

* Do you check the PCIe Speed and the avaible PCIe lines?

This is a zfs server and mybe you don't tune it in a proper way.

I Run your fio command on a zfs pool with 2x Crucial MX500 (SATA III) on a intel core i5-12400 desktop pc, zfs-2.2.2 and got:
Code: 
fio --name=write-test --size=1G --filename=fio-testfile --bs=128k --rw=write --direct=1 --numjobs=1 --time_based --runtime=30 --group_reporting
write-test: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1
fio-3.36
Starting 1 process
write-test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [W(1)][100.0%][eta 00m:00s]                        
write-test: (groupid=0, jobs=1): err= 0: pid=6877: Sun Dec 28 11:57:05 2025
  write: IOPS=3789, BW=474MiB/s (497MB/s)(13.9GiB/30084msec); 0 zone resets
    clat (usec): min=8, max=1568.4k, avg=262.15, stdev=16367.20
     lat (usec): min=9, max=1568.4k, avg=263.45, stdev=16367.20
    clat percentiles (usec):
     |  1.00th=[     10],  5.00th=[     10], 10.00th=[     11],
     | 20.00th=[     13], 30.00th=[     15], 40.00th=[     18],
     | 50.00th=[     26], 60.00th=[     46], 70.00th=[     59],
     | 80.00th=[     64], 90.00th=[    117], 95.00th=[    139],
     | 99.00th=[    174], 99.50th=[    186], 99.90th=[    269],
     | 99.95th=[    310], 99.99th=[1384121]
   bw (  KiB/s): min=19200, max=2223616, per=100.00%, avg=810086.17, stdev=596222.71, samples=36
   iops        : min=  150, max=17372, avg=6328.78, stdev=4658.01, samples=36
  lat (usec)   : 10=5.97%, 20=38.10%, 50=20.16%, 100=25.13%, 250=10.51%
  lat (usec)   : 500=0.11%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 2000=0.01%
  cpu          : usr=0.86%, sys=9.96%, ctx=25282, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,113997,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=13.9GiB (14.9GB), run=30084-30084msec
this is what i get und its ok.
 
Last edited:
Hi, the board is a ASROCK, X570 Phantom Gaming 4. link
Yes I checked the speed:
LnkSta: Speed 8GT/s (downgraded), Width x4 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkSta: Speed 16GT/s, Width x4 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

It looks like the first NVME is downgraded to PCIe3.

I am a little bit frustrated because of the same system run with the exactly same components expect ram size with about 4000mb/s speed.
 
Thank you.

Your AMD Ryzen 7 5700G CPU combined with ASROCK, X570 Phantom Gaming 4 has "only" PCIe 3.0 16+4+4 CPU Lanes.
So your PCIe 4.0/3.0 x16 slot get only PCIe 3.0 x16 Lanes max

# https://www.techpowerup.com/cpu-specs/ryzen-7-5700g.c2472

Please see top left AM4 X570 Chipset:
# https://tpucdn.com/cpu-specs/images/connectivity/amd-am4.png

Please see too: ASROCK X570 Phantom Gaming 4:
# https://pg.asrock.com/MB/AMD/X570 Phantom Gaming 4/index.asp#Specification
  • AMD Ryzen series CPUs (Cezanne, Renoir, Pinnacle Ridge)
    - 1 x PCI Express 3.0 x16 Slot (x16 (PCIE1))*
  • Storage
    1 x Hyper M.2 Socket (M2_1), supports M Key type 2230/2242/2260/2280/22110 M.2 PCI Express module up to Gen3x4 (32 Gb/s) (with Cezanne, Renoir, Pinnacle Ridge and Picasso)*
 
Last edited:
  • Like
Reactions: Kingneutron
> It looks like the first NVME is downgraded to PCIe3

You will likely get better performance by doing a zpool detach on the 2nd nvme, and re-using it for data / LXC/VM vdisks.

PCIe3 is OK for proxmox boot/root, but you want better I/O for your virtual environment. And make sure you have backups.

Also, PROTIP - if you're doing zfs mirroring, buy 2 different make/model of SSD. You kind of want one to die first (think EVO + Pro), not have both die around the same time due to identical wear patterns.
 
  • Like
Reactions: Johannes S
@news: Thank you very much for your support. All this time, I thought I was getting full speed because I'm using a 5000 series CPU.

Just for fun, I ran another test. I installed Proxmox on a SATA SSD, formatted the individual SSDs with XFS, and repeated the FIO test. Both hard drives show exactly the same performance at around 2200 Mb/s. As a layman, I would have expected a difference, right?

Long story short, I want to upgrade my hardware. Due to the current hardware prices, I am now looking for a combination with DDR4 RAM that will give me full speed. I have currently chosen the 5800X CPU. That should do the trick, right?
Now I just need a motherboard that can accommodate all the necessary components.
- 10 GBit Ethernet (PCIe X8)
- TV card (PCIe X1)
- Graphics card (system) only for installation and to connect a monitor
- Graphics card (PCIe X16) will be integrated into a VM

Does anyone have a recommendation for a motherboard that can accommodate these components?
 
Please think of:
You need on long term PCIe Lanes; PCI-Express: Gen 4, 128 Lanes (CPU only)
Please read the Mainboard Manual.

Add: a german youtube cannel:
# https://youtu.be/sMtADQyDtBg?si=_IXoKG0tWXEMX_wR

A German hardware compare site with this parts and even faster and bigger CPU: AMD Ryzen Threadripper PRO 5955WX, 16C/32T:
# https://geizhals.de/wishlists/4860102
 
Last edited:
I now have the new system up and running, and the NVMe SSDs are now recognized correctly. When I format the SSDs individually with XFS, I get around 3000 MB/s with the FIO test. I also get these values with an mdadm Raid1 and XFS. However, when I create a Mirror ZFS, I'm back to 400 MB/s.

Does anyone have any ideas?
 
Yes, but which image you want to have? I created a default installation, wipe everything from the SSD. Create a ZFS with the GUI and did the performance test.
I have seen a bad performance, so I created a madam raid and formatted the raid with xfs, now the performance is good.
 
big_blue said:
I have seen a bad performance, so I created a madam raid and formatted the raid with xfs, now the performance is good.
Click to expand...
First: if it fulfills your requirements then it is absolutely fine.

But: you are exchanging performance for other features. For me these are important and you have dropped them: technically cheap snapshots, transparent compression, guaranteed integrity - including self-healing and bitrot detection. And replication and ... .

ZFS does a lot to store data in a safe way. This requires several write actions per single request. Of course that's slower than classic systems.

At the end everybody is free to choose the tool which seems to fit - and that is great! :-)


Edit: see also https://forum.proxmox.com/threads/f...y-a-few-disks-should-i-use-zfs-at-all.160037/
 
Last edited:
  • Like
Reactions: Johannes S and news
You must log in or register to reply here.
Top Bottom