How to connect VMs on different Proxmox hosts

[leom]

Hello community

I have a cluster with 2 Proxmox nodes, which are hosting some VMs on their own. The two hosts are in the same network (172.21.251.0/24), and connect to the switch using LACP (802.3ad). The hosts also have a network bridge (vmbr0) connected to a private network (10.9.16.0/24). Both of the hosts have an IP address in the private network (10.9.16.1 and 10.9.16.101). The VMs can ping the two hosts (172.21.251.0/24) as well as the external network (e.g. 8.8.8.8).

What I want to achieve is to connect VMs in the two hosts to a private network 10.9.16.0/24, which is currently not possible.

/etc/network/interfaces of host1 (/etc/network/interfaces of host2 is similar, just the IP addresses of bond0 and vmbr0 are different.)

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto enp175s0f1
iface enp175s0f1 inet manual

auto enp175s0f0
iface enp175s0f0 inet manual

auto bond0
iface bond0 inet static
        address 172.21.251.124/24
        gateway 172.21.251.254
        bond-slaves enp175s0f0 enp175s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 10.9.16.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -t nat -A POSTROUTING -s '10.9.16.0/24' -o bond0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.9.16.0/24' -o bond0 -j MASQUERADE

Do you have any suggestions on how to do it? Thanks a lot!

 

[mvs]

Do you have any suggestions on how to do it? Thanks a lot!

You need to connect vmbr0 bridges of your nodes on Layer 2 together (bridge-ports line in vmbr0 config). You may use a NIC (eno1 or eno2) and a cable, a bond (eno1 + eno2) and two cables, a vlan over existing bond0, etc.

Config for a vlan with id=100 over bond0 would look like following

auto bond0.100
iface bond0.100 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.9.16.1/24
        bridge-ports bond0.100
        bridge-stp off
        bridge-fd 0

 

[leom]

You need to connect vmbr0 bridges of your nodes on Layer 2 together (bridge-ports line in vmbr0 config). You may use a NIC (eno1 or eno2) and a cable, a bond (eno1 + eno2) and two cables, a vlan over existing bond0, etc.

Config for a vlan with id=100 over bond0 would look like following

Hi mvs, thanks for your suggestions. I tried the vlan over bond0 method and modified the interface files as suggested. But the VMs cannot ping each other yet. Actually the 2 proxmox hosts cannot ping to each other's internal IP addresses yet.

# ping -c 3 10.9.16.101
PING 10.9.16.101 (10.9.16.101) 56(84) bytes of data.
From 10.9.16.1 icmp_seq=1 Destination Host Unreachable
From 10.9.16.1 icmp_seq=2 Destination Host Unreachable
From 10.9.16.1 icmp_seq=3 Destination Host Unreachable

Below is the network information of one of the hosts, if it helps a little.

 

[mvs]

Please check configuration of your switch. You may need to allow vlan100 (tagged) on bond group.

 

[bobmc]

I don't think this is going to be easy to achieve

Q: Is the requirement to connect only the two VM's (VM1 and VM2) or do you need to allow other VM's to communicate?
Q: Can the VM to VM traffic be port based?

 

[leom]

Please check configuration of your switch. You may need to allow vlan100 (tagged) on bond group.

The hosts are sitting in a sever room and I don't have any access to the switches. Maybe I need to talk with the staff there.

 

[leom]

Hi Bobmc,

Actually I am trying to set up a small Slurm cluster on top of these hosts.

I don't think this is going to be easy to achieve

Q: Is the requirement to connect only the two VM's (VM1 and VM2) or do you need to allow other VM's to communicate?

There will be several VMs in each hosts (not just 1) and they are required to communicate with each other.

Q: Can the VM to VM traffic be port based?

Does it mean if there communication are limited to only several ports. I think yes in this case.

 

[bobmc]

The major issue is that your setup has the VM's NAT'ed behind the bond network. I can only see that working with a router-vm being available or with more ip-tables foo than I'm capable of.

I'd be inclined to try it like this



that way the vm's will be able to directly connect to one another. If the VM's need to be able to connect to the rest of the network then you should be able to make that work with iptables rules. Bear in mind though, that devices outside of the 10.9.16.0 subnet will still not be able to initiate direct connections to the VM's