Planning network infrastructure for virtualization and future expansion

[rosiecharles]

Hi everyone,

I'm working on learning more about virtualization and network infrastructure planning.

For those running Proxmox in production, how do you design your network to support future growth? Do you separate storage traffic, management, and VM traffic from the beginning, or keep things simple until expansion is needed?

I'd also be interested in how you choose between 2.5GbE and 10GbE, especially for environments that may grow over time.

I'd love to hear about your real-world experience and any lessons you've learned.

Thanks!

 

[bbgeek17]

Hi @rosiecharles,

25GbE is really the starting point for modern greenfield business deployments. If you are planning for future growth, starting with 100GbE infrastructure and using breakout cables to provide 25GbE connectivity may be a good first step.

Whether you should separate storage, management, and VM traffic depends largely on your compliance requirements, performance expectations, operational model, and SLA.

In general, it is a good idea to maintain out-of-band management for hypervisors, storage, networking equipment, and IPMI/BMC interfaces using isolated management switches.

From a data-plane perspective, a redundant pair of 100GbE switches should comfortably handle both front-end and back-end traffic. You can then logically separate traffic classes using VLANs as appropriate. Including management traffic on the same switching fabric is acceptable in a pinch.

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[gunterwa]

You need a high-level design beforehand which maps your your materials in hands to your requirements; Then a low-level design to put your ideas into a spreadsheet... LAG is a must for connections between PVE nodes and uplink switches, in case you have multiple uplink switch MLAG will be the best choic.

For links...

Corosync network -- N x 1G
Storage network -- N x 10G (Ceph is good but costly)
Service network -- N x 100G

SDN is necessary for a multi-nodes PVE cluster, SDN can help you to configure all nodes in a cluster with the same networking scheme easily which I'd had to configure nodes one by one on PVE 7.x ... Same network configuration is important when you migrating VM between nodes.

PVE firewall,IPSet embeded, not perfect but workable.

OOB network is really really important, both router, switch and server's iPMI aggregated in management Vlan will be the last resort when you suffered from box down. Two years ago, one node of a 4 nodes PVE cluster DOWN. Google results show it's a BIOS bug, OOB network saved me... uprgade BIOS SW via iPMI fixed the issue.

 

[bbgeek17]

Corosync network -- N x 1G
Storage network -- N x 10G (Ceph is good but costly)
Service network -- N x 100G

The hardware availability may depend on the region, but as a general approach - if one thinks they need 100G for "service" (user/app access?), having a 10G for backing storage may introduce an unnecessary bottleneck from day one.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox