Bifurcated PCIe NVMe Adapter with Raid0 and low performance

P4in1

New Member
Sep 23, 2024
1
0
1
Hi @ all,

I'm fairly new to Proxmox and Linux in general and i am trying to upgrade my server with some fancy hardware.
I installed the Asus Hyper with four NVMe cards on my Super Micro X12 Mainboard. I am using 4x4x4x4 bifurcation for this setup and created a RAID0 and added it as a storage for my ZFS. All NVMes were found, working as expected and then i thought i can give it a try and see how fast i get.

My assumption: 4x of those NVMes with a RAID0 setup should get me somewhere over 20GB/s for normal read and write operations.

My "Problem" : I only get 4.5 - 5 GB/s Read and Write speed which i think is underwhelming considering that a single NVMe of my setup alone should score higher than the whole RAID.

Is there some kind of setting i am missing, a switch not toggled or something else? It works, but not how i thought it should and i see videos of people getting this setup running like i wanted it, but i don't know how to "fix" this.

After reading the compatibility chart of the ASUS Hyper and seeing that it might not be compatible i bought this on from Delock, but to no avail.

My System:
MainboardSuper Micro X12DPL-i6
CPUIntel Xeon Silver 4314
RAM256 GB Micron Reg ECC
GPU3090 TI Founders Edition with GPU passthrough
ZFS Pool 12x 1TB NVMe Mirror direct on Board for the System
ZFS Pool 25x 20TB HDD RAIDZ with 1 for parity
ZFS Pool 31x 20TB HDD for network storage
ZFS Pool 44x NVMe via Bifurcated PCIe Adapter

Additional Information:

Proxmox Version: 7.3-4
The GPU is inserted into CPU1 Slot 2 and blocks Slot 3 completely (it's way bigger than i expected it to be)
The NVMe Adapter is inserted into CPU Slot 4

I would appreciate every bit of help to get this running and integrate me into the Linux space. I am still learning and trying to understand how this works in detail.
If any information would be required on how to find the solution, I will find it (but i might need some information on "how to" , because i consider myself a noob in Linux)

Thanks in advance.

Have a good Night or Day :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!