ESXi->Proxmox, 1G->10G Planning

[Uturn]

Hi,

i want to migrate from ESXi (Essentials) to Proxmox and while there i'm thinking about switching to 10G for the main cluster.

Reasons:
- replication/migration for Proxmox goes faster (only needed for 2-4 VMs right now)
- i use CARP for failover alot, so there are machines running on both servers all the time doing replication via application
- i have a build cluster for compiling sources out of 4 VMs splitted on server1 and server2, so i want to use resources (CPU,mem)
from both servers all the time

A lot of software side configuration stuff i can/will find out myself, but regarding buying hardware i'd like your opinions.
All in all, i plan to use my 2x EPYC systems in HA + i'll reconfigure a third machine so it also runs ProxmoxVE, on top
one quorum VM, as well as a Proxmox Backup Server VM.

So when it comes to 10G i'm thinking about two scenarios.

1) cheaper, works i guess
Plug in one Intel dualport x710 + (10G DAC from fs) in Server1 and Server2 and directly connect both servers over both ports.
So all VMs, as well as Proxmox have 10G for interconnection, the uplink to the 24-port 1G switch will be done by already
existing Intel dualport I350T2 1G cards (RJ45) via LACP for balancing. As all remaining systems have 1G only, this is - right now-
no problem regarding speed i guess

2) more expensive, better, cleaner
I buy a new switch with SFP+ ports, plug in one Intel dualport x710 + (10G DAC from fs) in every server and go to the
SFP+ ports on the switch. All other systems are connected single/via LACP and 1G RJ45. The advantage is, that the uplink
to the switches is also via 10G SFP+ and i need no other cards + with an additional x710 i can also connect the third Proxmox
Backup Server to the switch

General Networking:
I'll have different VLANs the VMs run on, CARP is for failover for machines that replicate permanently on application layer and
plan to use Promox replication only for machines where CARP makes no/less sense. A VM on both server1/server2 acts as firewall
and connects all VMs as needed (VLAN, Routing Domains). This way, also the routing happens with 10G between the cluster and
i don't need physical stuff.

General Hardware:
Are there better, cheaper 10G Nics that work flawlessly with Proxmox? There are a few labled from Inter-Tech, Digitus that also
have Intel chipset and cost sometimes 1/3. Perhaps also Mellanox/Marvell....

Thanks for feedback.

 

[Uturn]

Is anybody having an opinion on this? Also partly answers are fine.

Thanks in advance

 

[Ramalama]

Hi, as Proxmox is based on a relatively new linux kernel, you can basically use any nic.
Probably all of them will work.

But i wouldn't feel quite comfortable with some noname nics with no controller, just an Asic.
I wouldn't even ever buy realtek or even worse Marvel AQtion adapters.

Tbh an intel X550 or the newer X710 are pretty cheap, pretty nice adapters that even support SR-IOV. They would be my preferred choice.
The only downside is, that they are hard to get as genuine on the market, most are china fakes, that work either, but getting hotter and don't reach the full 10g speed.
They max out at around 4/5gbs. But at least they have no bugs.

If you go for mellanox (nvidia), then you wouldn't find fakes at least, but those are expensive compared to intel.
Another good choice is broadcom, but expensive either.

To the second question...
You could connect your Servers back to back, for the replication and migration part....
But i would prefer tbh to buy the switch and use at least those 10g nics for everything else either, since 1g is pretty limited these days

At least the switch allows you to connect more 10g things later on

Cheers

Edit:
https://youtu.be/YdgrHda4sW0
You can get from AliExpress pretty cheap 10g manageable switches

 

[Uturn]

Tbh an intel X550 or the newer X710 are pretty cheap, pretty nice adapters that even support SR-IOV.

Yes, i'll go for these then.

But i would prefer tbh to buy the switch and use at least those 10g

Ok. Thought about the same and will go that way. Sounds more reliable and future proof in case the network grows.

Thanks very much

 

[Uturn]

I'm just thinking, as most switches have only 4xSFP+ ports.

Would the following work for three Proxmox machines:
1x Master Proxmox
1x Backup Proxmox
1x Backup Server

All servers get 2-port Intel X710 cards. With one port i directly connect "Master <----> Backup", so all VMs and Proxmox itself can communicate over 10G. With the remaining port i go into the switch 10G SFP+ as uplink and dedicate this uplink to a VM which acts as firewall/router. This way, i have still two switch ports left for connecting the "Backup Server" machine via 10G as well.