Before any other test I read again your benchmark to optimize bios settings.
Copy/paste error from putty to notepad.
Using psync results are a little lower (less then 5%)
No cache effect with 600 seconds, bandwidth and iops are stable from start to end of fio test.
I noticed growing bandwidth...
I'm building a similar cluster but with this hardware:
-6x Dell PowerEdge R6515 with AMD EPYC 7402P 24-Core Processor
-1 boss card with 2 m.2 240gb Raid1 for OS per host
-4 disks Kingston DC1000M U.2 7,68Tb per host
-8x Kingston KTD-PE432/64GRam 3200Mhz
-6 10Gbps sfp+ per hosts (1 Broadcom Adv...
I solved my usb problems and headache first with:
virtualhere https://www.virtualhere.com/ (cheaper and stable)
I never found usb dongle or peripherals not working on them.
If you choose those...
I've a 3 nodes cluster with 60 pfsense and 60 vlan with other vm.
Every pfsense has 1 lan (virtio) 1 opt (virtio) and 1 wan (e1000 to internet). This configuration is needed cause of backup run on opt network.
Still everyday one or two pfsense lost connection to every network (lan,wan,opt)...
As subject, I think to have mtu parameters on Gui can help a lot of users to solve performance problems with network storage.
I've separated networks with differents mtu to SAN (7980), backup storage (9000), internet (1500).
I solved all my latency/bottleneck problems with:
-4 10gb nic on san
-4 10gb nic on every proxmox (1 bond with 2 nic for communication with san and 1 bond with 2 nic for cluster/vm networking/backup)
-2 1gb nic on every proxmox (1 bond for traffic to internet)
We use 2 10gb switch and 2 1gb...