Proposed Hardware Design – Proxmox VE Compatibility & Shortfalls

[Trevor Meyer]

Hi all,
We’re in the planning phase for a new Proxmox VE cluster and would appreciate feedback on hardware compatibility, known issues, and any shortfalls or design concerns before purchasing.

Planned Usage:
Proxmox over iSCSI + multipath
VM disks on shared LUNs
No Ceph planned initially

Planned Environment (Phase 1)
3-Node Proxmox VE cluster
Shared iSCSI storage vs Ceph/LINSTOR (Maybe Ceph/LISTOR for Phase One and iSCSI for Production workload [Phase 2])
Focus on stability and enterprise‑grade supportability

3x Dell PowerEdge R660xs
Dual Intel Xeon Silver 4514Y (2.0GHz, 16C/32T per CPU)
128GB DDR5 5600MT/s (dual-rank)
Broadcom 57414 dual-port 10/25GbE SFP28, OCP NIC 3.0
Boot planned via BOSS or mirrored SSDs (TBC)

Shared Storage (Considderring iSCSI as we require at least 45TB Storage between all nodes)
Dell PowerVault ME5224 - Dual controller
25Gb iSCSI, 8 ports total
16 × 3.84TB SSD, SAS 24Gbps, 1WPD
RAID configuration likely RAID-10 or RAID-5 (TBC)

Networking
Switching
2 × Netgear VSM4320C-100NES
25G SFP28 access ports
100G QSFP28 uplinks

Best practices for:
x- LACP vs standalone links for storage traffic
x- Jumbo frames end-to-end (any Proxmox gotchas)?
x- Anyone running ME5 iSCSI over Netgear 25G successfully?
x- Any strong reasons to avoid shared iSCSI vs local ZFS for this size cluster?
x- HA behavior during storage controller or path failover?

Questions / Concerns:
x- General experience with ME5xx series + Proxmox?
x- iSCSI multipath stability under load (ALUA, failover times)?
x- Any tuning recommendations (queue depth, IO scheduler, cache mode)?
x- Snapshot / backup impact on ME5 performance?
x- Any known driver or firmware issues with:
x- Intel Xeon Silver 4514Y on Proxmox 9.x (kernel 6.x)
x- Broadcom 57414 OCP NIC (bnxt_en driver stability, SR-IOV, VLANs)?

x- Anything you’d change or watch out for with this design?

Summary
Before committing to purchase, we’d really value:

x- Confirmation this hardware is well supported on Proxmox VE 8.x/9.x
x- Known pitfalls with Broadcom NICs, ME5 storage, or 25Gb iSCSI
x- Real world lessons learned from similar deployments

Thanks in advance for any feedback or recommendations.

 

[LnxBil]

Dual Intel Xeon Silver 4514Y (2.0GHz, 16C/32T per CPU)

Why such low speeds with not so many cores? I would optimize with better CPUs. More cores (>40) or higher frequencies.

Broadcom 57414 dual-port 10/25GbE SFP28, OCP NIC 3.0

We buy always two different NICs to leverage firmware/driver problems and single NIC failure.

Snapshot / backup impact on ME5 performance?

Yes, it will impact the performance, just search the forum about it. Best to not use snapshots over LVM.

RAID configuration likely RAID-10 or RAID-5 (TBC)

Have you looked at ADAPT?

LACP vs standalone links for storage traffic

If you switch can handle that, sure. LACP and then VLAN would work with QoS involved.

Any strong reasons to avoid shared iSCSI vs local ZFS for this size cluster?

I love ZFS, yet I would not want it in a cluster, use proper (dedicated or distributed) shared storage, ZFS is not shared storage.

Confirmation this hardware is well supported on Proxmox VE 8.x/9.x

Dell in general works, yet it may be hard to find someone with the exact specs.

Known pitfalls with Broadcom NICs, ME5 storage, or 25Gb iSCSI

We had problems with 25 Gb Dell-branded GBICs for Broadcom and Intel cards in the last 2 years. Intel GBICs worked and with Broadcom, we had a problem with negotiating the speed with a cisco switch, yet that was a known problem.