[SOLVED] KVM Guest - Network Issues

[ssaman]

Hello,

We have set up an proxmox cluster with 5 nodes by Hetzner.

After we installed the first VM we had our first issue that we could not ping all Servers.

The strange point is that we can ping some IPs and some not.

for Example. We can ping:
8.8.8.8
213.133.98.98
we can ping our gateway
we can ping our host syste (first node)

and what we cannot:
212.121.132.78
cannot ping other nodes where the Guestsystem is not installed
most of our servers
localnetwork (company)


when we try to "ping google.de" it does not work.
It can resolv the ip, but we dont geht any packets.

dig and nslookup works

Here is our Network setting on the first node:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
#vmbr0

auto eth1
iface eth1 inet manual
#vmbr1

auto eth2
iface eth2 inet static
        address  10.200.0.1
        netmask  255.255.255.128
#Netgear (cluster communication network)

iface eth3 inet manual

auto eth4
iface eth4 inet static
        address  10.200.1.1
        netmask  255.255.255.128
#Zyxel (ceph and data migration network)

iface eth5 inet manual

auto vmbr0
iface vmbr0 inet static
        address  xxx.xxx.44.178
        netmask  255.255.255.128
        gateway  94.130.80.129
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet static
        address  xxx.xxx.55.129
        netmask  255.255.255.224
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0
#Netgear (public subnet)

and bridge setting:

root@c5-node1:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr0           8000.1866da880f1a       no              eth0
vmbr1           8000.1866da880f1b       no              eth1

 

[Alwin]

Have you seen the docs from Hetzner, even though made up to PVE4.4 the networking part should be still valid.
https://wiki.hetzner.de/index.php/Proxmox_VE/en

 

[snachtsheim]

Hi Alwin, I'am a colleague of Serhat.

We already installed three Proxmox 3 Clusters (2-node setup with DRBD) with Hetzner servers and always used this subnet/vmbr setup routed through the host server. With Proxmox 4.4 it is the first time we run into those network issues with a guest VM.

Sometimes ping works, sometimes not and sometimes services like DNS and SSH are working, and after seconds the don't. So it is really hard to reproduce...

We already tried:
rp_filter setting (0, 1, 2)
ethtool -K interface tx off
bridge_fd = 1
gateway ip assigned to ethX or vmbrX

no change/success at all :-(

best regards, Stephan

 

[Alwin]

If it is a routed setup, where are you routes in the network block?

 

[snachtsheim]

thanks for your feedback

route -n from host system:

root@c5-node1:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         94.xxx.xxx.129   0.0.0.0         UG    0      0        0 vmbr0
10.200.0.0      0.0.0.0         255.255.255.128 U     0      0        0 eth2
10.200.1.0      0.0.0.0         255.255.255.128 U     0      0        0 eth4
78.xxx.xxx.128   0.0.0.0         255.255.255.224 U     0      0        0 vmbr1
94.xxx.xxx.128   0.0.0.0         255.255.255.128 U     0      0        0 vmbr0

route -n from guest system:

snachtsheim@performance-test:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         78.xxx.xxx.129   0.0.0.0         UG    0      0        0 ens18
78.xxx.xxx.128   0.0.0.0         255.255.255.224 U     0      0        0 ens18

 

[Alwin]

The systctl config like eg. proxy_arp is also set?

 

[ssaman]

proxy_arp was missing.
We enabled it, but still no improvement.

We have a standard Proxmox 4.4 iso installation shouln't the default settings work?

# sysctl -p /etc/sysctl.conf
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.proxy_arp = 1

 

[Alwin]

Hetzner has their specialized network setup and is not like our install defaults, so this has to be asked Hetzner. But from what I can see, the Hetzner Config is using pointtopoint configurations and vmbr0 without bride_ports connected. https://wiki.hetzner.de/index.php/Proxmox_VE/en

 

[snachtsheim]

Hi Alwin, thats correct but only relevant for communication directly between servers in the public provider subnet.

We resolved the problem: It was a orphaned entry in the routing tables for this subnet. We talked to our hostnig provider and asked to check for this, because that was the last thing we could imagine. And luckily that was the problem. So everything solved now

thanks for your help anyways. I really appreciate it.