Windows VM network issue. Routing.

[corroded]

Hi all,

I have the three nodes proxmox cluster with the public IPs for each node and the public IP block for VMs.

In my current configuration I have very strange issue with networking on Ubuntu/Windows VMs.

I can access internet from ubuntu VM. But I can't do the same from windows VM.

yy.yy.yy.yy - public IP
xx.xx.xx.136/29 - public IP block
zz.zz.zz.zz - local IP

eth2 - interface for public IP to access nodes.
eth3 - interface for public IP block and for local network between nodes.


Public IP block is distributed over all three nodes so I can use IP from block on each node.


There is Ubuntu network config:

net0: virtio=22:22:22:22:22:22,bridge=vmbr2

auto ens18
iface ens18 inet static
    address XX.XX.XX.137
    netmask 255.255.255.255
    broadcast XX.XX.XX.137
    dns-nameserver 8.8.4.4 8.8.8.8
    post-up route add XX.XX.XX.142 dev ens18
    post-up route add default gw XX.XX.XX.142
    post-down route del default gw XX.XX.XX.142
    post-down route del XX.XX.XX.142 ens18

Windows VM network config:

net0: virtio=11:11:11:11:11:11,bridge=vmbr2

IP: XX.XX.XX.140
NETMASK:255.255.255.255
GW: XX.XX.XX.142
DNS: 8.8.8.8 8.8.4.4

Node info:

root@px1:~# pveversion -v
proxmox-ve: 4.4-88 (running kernel: 4.4.62-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.62-1-pve: 4.4.62-88
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-50
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-95
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-100
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
ceph: 10.2.7-1~bpo80+1

Node network settings:

root@px1:~# cat /etc/network/interfaces

auto lo
iface lo inet loopback
iface eth0 inet manual
iface eth1 inet manual
iface eth2 inet manual
iface eth3 inet manual

auto vmbr1
iface vmbr1 inet manual
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0
    post-up /etc/pve/kvm-networking.sh

auto vmbr0
iface vmbr0 inet static
    address  YY.YY.YY.89
    netmask  255.255.255.0
    gateway  YY.YY.YY.254
    broadcast  YY.YY.YY.255
    bridge_ports eth2
    bridge_stp off
    bridge_fd 0
    network YY.YY.YY.0
  
auto vmbr2
iface vmbr2 inet static
    address  ZZ.ZZ.ZZ.10
    netmask  255.240.0.0
    bridge_ports eth3
    bridge_stp off
    bridge_fd 0

net.ipv4.ip_forward = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.vmbr2.proxy_arp = 1

Node routing table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         YY.YY.YY.254   0.0.0.0         UG    0      0        0 vmbr0
YY.YY.YY.0     0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
ZZ.ZZ.ZZ.0      0.0.0.0         255.240.0.0     U     0      0        0 vmbr2

In windows VM I can see my public IP block gw in the arp table.
If I do "arp -d XX.XX.XX.142" I can ping some external resources for a while. I think actually only once after clearing the arp table on windows VM. Windows virtio driver is up to date.



At the same time ubuntu VM works like a charm.

But if I add route to XX.XX.XX.142 via vmbr2 on node than internet starting to work on both VMs.

Is all of this possible and how? What I miss? What is the difference between Ubuntu and Windows VMs in this case?

Help me please. This made me crazy already.

Thx

ps: English isn't my first language, so please excuse any mistakes.

 

[corroded]

Some additional information.

I found that Proxmox sends 2 ARP reply to VM. See attached.

EE:EE:EE:EE:EE:EE - Proxmox host eth3 mac

root@px1:~# tcpdump -i vmbr2 -vv arp host XX.XX.XX.137
tcpdump: listening on vmbr2, link-type EN10MB (Ethernet), capture size 262144 bytes
16:29:06.297986 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has ip142.ip-XX-XX-XX.eu tell ip137.ip-XX-XX-XX.eu, length 28
16:29:06.298450 ARP, Ethernet (len 6), IPv4 (len 4), Reply ip142.ip-XX-XX-XX.eu is-at VV:VV:VV:VV:VV:c5 (oui Unknown), length 46
16:29:06.348503 ARP, Ethernet (len 6), IPv4 (len 4), Reply ip142.ip-XX-XX-XX.eu is-at EE:EE:EE:EE:EE:EE (oui Unknown), length 28
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

root@px1:~# tcpdump -i eth3 -vv arp host XX.XX.XX.137
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 262144 bytes
16:29:06.298002 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has ip142.ip-XX-XX-XX.eu tell ip137.ip-XX-XX-XX.eu, length 28
16:29:06.298450 ARP, Ethernet (len 6), IPv4 (len 4), Reply ip142.ip-XX-XX-XX.eu is-at VV:VV:VV:VV:VV:c5 (oui Unknown), length 46
^C
2 packets captured
3 packets received by filter
0 packets dropped by kernel

So, after this I compared arp tables on VMs.
I saw that windows VM use the second reply and add EE:EE:EE:EE:EE:EE to arp table.

arp -a
Interface: XX.XX.XX.137
XX.XX.XX.142  EE:EE:EE:EE:EE:EE dynamic

But ubuntu VM use the first reply and add VV:VV:VV:VV:VV:c5 to arp table.

arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
XX.XX.XX.142           ether   VV:VV:VV:VV:VV:c5   C                     ens18

Is it normal behavior for proxmox? As I understood in case of two ARP answers by standard the second one is need to be used. Like windows do. But there is no internet access on windows VM in that case.