How to add second LAN to existing PVE Cluster?

[bedibe]

Hi there,

we are using PVE since nearly one year at work and I must say, I'm very, very happy with this Software. Never thought that managing a bunch of VMs could be so easy. Before that we tried to use OpenStack, but it's way to complicated and overkill for our needs.

Currently we have five nodes in our cluster, each with one 1GE-NIC for Uplink and one 1GE-NIC for local Networking. Local Networking is for PVE-Sync as well as for distributed sotorage (NFS) and internal communication between vms. I know, this is no good and currently we are seeing many problems that are caused by sharing the same network for cluster, storage and complete internal traffic.

Now we have decided to enlarge this cluster by one dedicated storage node (no PVE, just exports per NFS and iSCSI) and an additional 10GE network. Each node will get a 10GE-NIC and then be able to use all three NIC. The goal is to have no more timeouts between the hosts, to be able to deploy new VMs really fast including cloning images and most of all, to be creating snapshots and backups without bringing the whole cluster down. This could also be accomplished by running all these tasks on the storage node itself (cloning, backup etc), but having the possibility to do some quick management task via PVE WebUI is very nice and most of all it's the only way to do it for our developers, which have no acces to storage backends itself.

Currently we are using one internal private network 10.10.0.0/16 for all and everything. Each node has an address in the first subnet 10.10.0.0/24 (like 10.10.0.10, 10.10.0.20, ...) and all VMs are using IPs from a higher subnet like 10.10.8.0/24 or 10.10.10.0/24 except for one VM, which has the IP 10.10.0.100. This is our central gateway, run by IPFire and every single packet of one of the other VMs has to pass this VM.

The current bridge looks like this:

# internal traffic 1GE
iface eth1 inet manual
iface vmbr1 inet static
    address 10.10.0.10 # cluster address
   netmask 255.255.0.0 # the complete network
   bridge_ports eth1
   bridge_stp off
   bridge_fd 0
   bridge_vlan_aware yes

This is the point, where I need your help. As far as I see, we have two possibilities:

A) The 10GE interfaces use a different network (like 10.11.0.0/24) and will be storage-only

B) We try to separate the cluster communication from all the rest and dedicate the 1GE-Network to cluster communication and the 10GE Network to storage AND internal traffic

Solution A would be easy to setup .... but me, I'd really prefer the latter one, as it clearly separates the VM-related traffic to the high bandwidth-network and PVE should never ever be unresponsive again. But I'm not sure how to do it. Would it be possible to configure the interfaces like this?

# internal traffic 1GE
iface eth1 inet manual
iface vmbr1 inet static
    address 10.10.0.10 # cluster address
# breaking this interface down to route a /26 ranging from .1 to .62
# this way the gateway with 10.10.0.100 would be routed via vmbr2
# at least I hope so ....
    netmask 255.255.255.192
    bridge_ports eth1
    bridge_stp off
    bridge_fd 0
    bridge_vlan_aware yes

# internal traffic 10GE
iface eth2 inet manual
iface vmbr2 inet static
    address 10.10.1.10 # some new address
    netmask 255.255.0.0 # the complete network?!?
    bridge_ports eth2
    bridge_stp off
    bridge_fd 0
    bridge_vlan_aware yes

I'm not that an experienced networker and I think that this setup might work, but I'm not sure. Currently I cannot test it, as the 10GE network isn't installed yet. And I'd really like to know in advance how it would work, as we need to start merging the images to the new storage as soon as the network is up and running.

Somebody has some better idea? Did I miss something? Am I completely numb and there is a obvious solution?


Our Software:
PVE 5.2 running on Debian 9.4

Thanks for reading and for answering!!
Ben

 

[bedibe]

OK, I guess I'll try to put a separate network on the 10GE NIC. My tests weren't that sucsessfull with pulling a small subnet out of that /16 network and routing it over the 10GE NIC. What I did was to route the 10.10.1.0/24 subnet over the 10GE NICs but then I couldn't reach the virtual hosts no more, which are running on 10.10.10.0/24. Maybe just some stupid misconfiguration but it's pretty hard to test a new routing in a live environment with active VMs.

If I find the time I'll do some tests this weekend with a network emulator or probably with virualbox. I'd be so happy if I could get that configuration running having a small subnet 10.10.0.0/26 routed over the 1GE network and the rest routed over the 10GE network.

Please could somebody give at least some feedback wether this makes sense at all? Routing the internal proxmox traffic over a separate 1 GE Network while VM-traffic AND storage runs on the same 10GE network?