Networking and Clusters

N

N0_Klu3

Well-Known Member
Mar 8, 2020
37
11
48
40
Hi all,

I'm about to build a new cluster with dual 2.5gb nics.

I want one or two LXC's on my main default network (unifi) and the rest on VLAN.
I can do this by leaving the vm/lxc lan on default, and then just using the VLAN tag for all containers I want elsewhere.

But my main question is, what should I do with my cluster network.
Do I install Proxmox host using this? I'm going to put it on its own VLAN.

As in during the install do I use the network I want as my cluster as the host IP too?
And then use the other NIC purely for VM/LXC, or do I roll my Proxmox into this and leave the cluster NIC purely just for cluster traffic?
 
Corosync should be set on both NICs as it can have multiple links. It prefers low latency but uses little bandwidth. How busy will the VMs be in terms of network traffic?
 
SteveITS said:
Corosync should be set on both NICs as it can have multiple links. It prefers low latency but uses little bandwidth. How busy will the VMs be in terms of network traffic?
Click to expand...
Probably not very.

Couple of Windows VM's and about 10 LXC's.

I was going to use Ceph, as my storage across the 3 nodes, on the mgmt LAN.
 
UdoB said:
Make sure to understand the limitations of so few Ceph nodes (and possibly only very few OSDs per node).

Personally I came to the conclusion that I wanted at least five! -- https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/

Disclaimer: I am not using Ceph currently...
Click to expand...
Yeah I've been testing for a year on a smaller cluster.
It's been rock solid so far.

I just want to figure out the best option in terms of 2 NIC utilization and which way to install Proxmox.
 
  • Like
Reactions: UdoB
N0_Klu3 said:
can you help me decide if I put the Proxmox network on the Ceph network or the VM/LXC network?
Click to expand...
What is the "Proxmox network" for you?

My own Homelab is also based on two physical NICs per node. I know this is bad - I would love to have four or more. But this is "just" a Homelab...

Nevertheless there is no fixed solution to structure networks; the following ist only one of many examples:
  • NIC1: untagged = Storage = Ceph + NFS + Samba + PBS
  • NIC2: tagged only = all other networks are trunked and I carve out some VLANs like dmz,lan,wlan,media,isolated,guestwlan,iot,server,admin, ...
    each one is equivalent to a separate bridge on all nodes; most of these bridges do NOT have an IP address on the nodes
  • corosync: both physical NICs are used to establish one ring each
I just define those bridges the classical way in etc/network/interfaces. If I would just now start evaluating PVE I would possibly consider to use the SDN capabilities. But I will definitely not switch...

Of course I have some switches to distribute the networks to my hardware devices, untagged. This is a must!

Some networks are interconnected through a dedicated router with an extensive list of positive rules - while the policy is "deny". Currently this is just a a VM with some magic scripts ("shorewall"). My test-installation of OpnSense does work already, including redundancy via CARP, but migration is too... cumbersome.

Everything around my networks lacks redundancy for now, but... "Homelab".

Again: there's a zillion possibly approaches...
 
  • Like
Reactions: N0_Klu3
UdoB said:
What is the "Proxmox network" for you?

My own Homelab is also based on two physical NICs per node. I know this is bad - I would love to have four or more. But this is "just" a Homelab...

Nevertheless there is no fixed solution to structure networks; the following ist only one of many examples:
  • NIC1: untagged = Storage = Ceph + NFS + Samba + PBS
  • NIC2: tagged only = all other networks are trunked and I carve out some VLANs like dmz,lan,wlan,media,isolated,guestwlan,iot,server,admin, ...
    each one is equivalent to a separate bridge on all nodes; most of these bridges do NOT have an IP address on the nodes
  • corosync: both physical NICs are used to establish one ring each
I just define those bridges the classical way in etc/network/interfaces. If I would just now start evaluating PVE I would possibly consider to use the SDN capabilities. But I will definitely not switch...

Of course I have some switches to distribute the networks to my hardware devices, untagged. This is a must!

Some networks are interconnected through a dedicated router with an extensive list of positive rules - while the policy is "deny". Currently this is just a a VM with some magic scripts ("shorewall"). My test-installation of OpnSense does work already, including redundancy via CARP, but migration is too... cumbersome.

Everything around my networks lacks redundancy for now, but... "Homelab".

Again: there's a zillion possibly approaches...
Click to expand...
Was just thinking about about bonding the 2x 2.5gb for a 5 gb connection for everything?
Ceph, Proxmox, and VM/LXC?

Would having extra bandwidth be better than a dedicated slower NIC?
 
N0_Klu3 said:
Was just thinking about about bonding the 2x 2.5gb for a 5 gb connection for everything?
Click to expand...

When you bond it the result is a single transport. At least corosync wants to have multiple independent physical transports.

To separate storage, admin, poisoned iot, etc you need to have several networks. The VLAN approach is already a not optimal compromise as it shares the hardware.

Note that "speed" is irrelevant to me (while I have only 1 GBit/s on the front). Whatsovever: your mileage may vary --> choose your poison.
 
  • Like
Reactions: N0_Klu3
The Ceph "public network" is where your VMs talk to Ceph. It can be the "Proxmox LAN" however you define that. The Ceph "private network" is used for replication, rebuilding, etc. You can separate the two.

I would put corosync (the Proxmox "cluster network" for lack of a better term) on both.
 
SteveITS said:
The Ceph "public network" is where your VMs talk to Ceph. It can be the "Proxmox LAN" however you define that. The Ceph "private network" is used for replication, rebuilding, etc. You can separate the two.

I would put corosync (the Proxmox "cluster network" for lack of a better term) on both.
Click to expand...
Yeah I plan to do this.
This is the option when creating the cluster to select both NICs right?

And then when you join a cluster to add both NICs of the other nodes?
 
You must log in or register to reply here.
Top Bottom