Networking problem

[Rex de Koning]

Hi,

I'm new to Proxmox and trying to setup a new environment. For now just using my homelab, but the same should be build in colocation environment. I think i'm doing something wrong with routing, but I cannot put my finger on it, perhaps someone can point me in the correct direction.

I have two hosts in a clusters. Proxmox1 and Proxmox2. Both of those hosts have a VM on it.
The VM's have internet access but cannot reach each other.

Setup en pings that fail/succeed:

( The vmbr1's on both machines have the same ip so that when I VM migrates the gateway stays the same ). This is a working setup. The VM's need to connect to each other via vmbr2.

PVEVERSION -V:
proxmox-ve: 5.1-26 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-26
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: not correctly installed
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1


Proxmox1

auto enp0s3
iface enp0s3 inet static
address 192.168.1.53
netmask 255.255.255.0
gateway 192.168.1.41
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up echo 1 > /proc/sys/net/ipv4/conf/enp0s3/proxy_arp


iface enp0s8 inet manual

auto vmbr1
iface vmbr1 inet static
address 192.168.10.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up iptables -t nat -A POSTROUTING -s '192.168.10.0/24' -o enp0s3 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.10.0/24' -o enp0s3 -j MASQUERADE

auto vmbr2
iface vmbr2 inet static
address 192.168.20.1
netmask 255.255.255.0
bridge_ports enp0s8
bridge_stp off
bridge_fd 0

Proxmox2

auto enp0s3
iface enp0s3 inet static
address 192.168.1.52
netmask 255.255.255.0
gateway 192.168.1.41
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up echo 1 > /proc/sys/net/ipv4/conf/enp0s3/proxy_arp


iface enp0s8 inet manual

auto vmbr1
iface vmbr1 inet static
address 192.168.10.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
post-up iptables -t nat -A POSTROUTING -s '192.168.10.0/24' -o enp0s3 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.10.0/24' -o enp0s3 -j MASQUERADE

auto vmbr2
iface vmbr2 inet static
address 192.168.20.2
netmask 255.255.255.0
bridge_ports enp0s8
bridge_stp off
bridge_fd 0

VM1 on Proxmox1

auto ens18
iface ens18 inet static
address 192.168.10.3
netmask 255.255.255.0
network 192.168.18.0
broadcast 192.168.10.255
gateway 192.168.10.1
dns-nameservers 8.8.8.8

auto ens19
iface ens19 static
address 192.168.20.3
netmask 255.255.255.0

VM2 on Proxmox2
auto ens18
iface ens18 inet static
address 192.168.10.4
netmask 255.255.255.0
network 192.168.18.0
broadcast 192.168.10.255
gateway 192.168.10.1
dns-nameservers 8.8.8.8

auto ens19
iface ens19 static
address 192.168.20.4
netmask 255.255.255.0

Proxmox1 can ping:
192.168.1.53 ( proxmox1-enp0s3 )
192.168.1.52 ( proxmox2-enp0s3 )
192.168.10.1 ( proxmox1-vmbr1 )
192.168.10.3 ( VM1-ens18 )
192.168.20.1 ( proxmox1-vmbr2 )
192.168.20.2 ( proxmox2-vmbr2 )
192.168.20.3 ( VM1-ens19 )
8.8.8.8 ( 'outside' )

Proxmox1 cannot ping:
192.168.10.4 ( VM2-ens18 )
192.168.20.4 ( VM2-ens19 )

Proxmox2 can ping:
192.168.1.53 ( proxmox1-enp0s3 )
192.168.1.52 ( proxmox2-enp0s3 )
192.168.10.1 ( proxmox2-vmbr1 )
192.168.20.1 ( proxmox1-vmbr2 )
192.168.20.2 ( proxmox2-vmbr2 )
192.168.10.4 ( VM2-ens18 )
192.168.20.4 ( VM2-ens19 )
8.8.8.8 ( 'outside' )

Proxmox2 cannot ping:
192.168.10.3 ( VM1-ens18 )
192.168.20.3 ( VM1-ens19 )

VM1 on proxmox1 can ping:
192.168.1.53 ( proxmox1-enp0s3 )
192.168.1.52 ( proxmox2-enp0s3 )
192.168.10.1 ( proxmox1-vmbr1 )
192.168.20.1 ( proxmox1-vmbr2 )
192.168.10.3 ( VM1-ens18 )
192.168.20.3 ( VM1-ens19 )
8.8.8.8 ( 'outside' )

VM1 on proxmox1 cannot ping:
192.168.20.2 ( proxmox2-vmbr2 )
192.168.10.4 ( VM2-ens18 )
192.168.20.4 ( VM2-ens19 )

VM2 on proxmox2 can ping:
192.168.1.53 ( proxmox1-enp0s3 )
192.168.1.52 ( proxmox2-enp0s3 )
192.168.10.1 ( proxmox2-vmbr1 )
192.168.20.2 ( proxmox2-vmbr2 )
192.168.10.4 ( VM2-ens18 )
192.168.20.4 ( VM2-ens19 )
8.8.8.8 ( 'outside' )

VM1 on proxmox1 cannot ping:
192.168.20.1 ( proxmox1-vmbr2 )
192.168.10.3 ( VM1-ens18 )
192.168.20.3 ( VM1-ens19 )

Kind regards,

Rex de Koning