Proxmox Servers on Private Network (OVH Vrack) setup - VM's Not Connecting

[asifejaz]

I have 2 Proxmox Servers on Private Network (OVH Vrack) setup as below.​

Both host servers are communicating with each other, but VM's on both hosts are communicating across servers but within same servers they are working fine. what could be the problem?

​

IP Scheme for Proxmox Hosts and VMs​

1. Proxmox Hosts (Private IPs):
   • Server 1: 192.168.0.2/24
   • Server 2: 192.168.0.6/24
   • Both hosts will be in the same 192.168.0.0/24 subnet, allowing them to communicate directly.
2. VM Network Subnet (Shared across both servers):Assign all VMs on both servers IP addresses from the same subnet. You can use a separate subnet like 10.10.10.0/24 for VM communication.
   For instance:
   • Server 1 VMs:
     • VM1: 10.10.10.10/24
     • VM2: 10.10.10.11/24
     • VM3: 10.10.10.12/24
   • Server 2 VMs:
     • VM1: 10.10.10.20/24
     • VM2: 10.10.10.21/24
     • VM3: 10.10.10.22/24

 

[shanreich]

Can you post your network configuration?

cat /etc/network/interfaces

 

[asifejaz]

Can you post your network configuration?

cat /etc/network/interfaces

Server 1

auto lo
iface lo inet loopback
iface ens3f0np0 inet manual

auto enp129s0f0np0
iface enp129s0f0np0 inet static
address 192.168.0.2/24

iface enp129s0f1np1 inet manual
iface ens3f1np1 inet manual
iface enx16b347375be3 inet manual

auto vmbr0
iface vmbr0 inet static
address 162.19.xx.xx/32
gateway 100.64.xx.xx
bridge-ports ens3f0np0
bridge-stp off
bridge-fd 0
hwaddress A0:88:C2:66:AE:48

auto vmbr1
iface vmbr1 inet static
address 10.10.10.1/24
bridge-ports none
bridge-stp off
bridge-fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2210 -j DNAT --to 10.10.10.10:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2210 -j DNAT --to 10.10.10.10:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2220 -j DNAT --to 10.10.10.20:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2220 -j DNAT --to 10.10.10.20:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2230 -j DNAT --to 10.10.10.30:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2230 -j DNAT --to 10.10.10.30:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2240 -j DNAT --to 10.10.10.40:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2240 -j DNAT --to 10.10.10.40:22
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp129s0f0np0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp129s0f0np0 -j MASQUERADE
post-down iptables -t nat -D PREROUTING -i enp129s0f0np0 -p tcp --dport 2210 -j DNAT --to 10.10.10.10:22
post-up iptables -t nat -A PREROUTING -i enp129s0f0np0 -p tcp --dport 2210 -j DNAT --to 10.10.10.10:22
post-down iptables -t nat -D PREROUTING -i enp129s0f0np0 -p tcp --dport 2220 -j DNAT --to 10.10.10.20:22
post-up iptables -t nat -A PREROUTING -i enp129s0f0np0 -p tcp --dport 2220 -j DNAT --to 10.10.10.20:22
post-down iptables -t nat -D PREROUTING -i enp129s0f0np0 -p tcp --dport 2230 -j DNAT --to 10.10.10.30:22
post-up iptables -t nat -A PREROUTING -i enp129s0f0np0 -p tcp --dport 2230 -j DNAT --to 10.10.10.30:22
post-down iptables -t nat -D PREROUTING -i enp129s0f0np0 -p tcp --dport 2240 -j DNAT --to 10.10.10.40:22
post-up iptables -t nat -A PREROUTING -i enp129s0f0np0 -p tcp --dport 2240 -j DNAT --to 10.10.10.40:22

Server 2

auto lo
iface lo inet loopback

iface enp66s0f0 inet manual

auto enp66s0f1
iface enp66s0f1 inet static
address 192.168.0.6/24

iface enx369f794667b2 inet manual

auto vmbr0
iface vmbr0 inet static
address 162.19.xx.xx/24
gateway 162.19.xx.xx
bridge-ports enp66s0f0
bridge-stp off
bridge-fd 0
hwaddress A8:A1:59:C0:EF:13

auto vmbr1
iface vmbr1 inet static
address 10.10.10.2/24
bridge-ports none
bridge-stp off
bridge-fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2250 -j DNAT --to 10.10.10.50:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2250 -j DNAT --to 10.10.10.50:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2260 -j DNAT --to 10.10.10.60:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2260 -j DNAT --to 10.10.10.60:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2270 -j DNAT --to 10.10.10.70:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2270 -j DNAT --to 10.10.10.70:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 2280 -j DNAT --to 10.10.10.80:22
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 2280 -j DNAT --to 10.10.10.80:22

 

[shanreich]

The issue here is probably that you have 10.10.10.0/24 configured on both hosts and therefore a route gets created that routes all traffic in that subnet to vmbr1. Since you have no bridge_port there is no way for the traffic to leave the host and therefore you cannot reach the VMs on the other host.

 

[asifejaz]

The issue here is probably that you have 10.10.10.0/24 configured on both hosts and therefore a route gets created that routes all traffic in that subnet to vmbr1. Since you have no bridge_port there is no way for the traffic to leave the host and therefore you cannot reach the VMs on the other host.

how Can i fix that?

 

[shanreich]

The best way is probably to use VXLAN for vmbr1 if you cannot physically hook them up