No Internet on all VMs after update

painerp · Jan 30, 2020

Yesterday after I updated and rebooted my Server all of my 3 VMs weren't able to connect back to the Internet.
I use a routed setup basically exactly like this https://community.hetzner.com/tutorials/install-and-configure-proxmox_ve/de?title=Proxmox_VE

After some digging I figured out that the Gateway (proxmox host) is still pingable from inside of the VMs.
Pinging the VMs from proxmox failed and also "ping 1.1.1.1 -I vmbr0" fails, Im not sure if this should work though.

Tcpdump on proxmox showed some incoming connections from the VMs but I couldn't get anything special from that.
So i figured it has to be a problem with my proxmox networking but I have no idea what is happening since I haven't changed anything and it still looks correct to me.

Proxmox Network:

Code:

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet manual
        address  [ProxmoxIP]
        netmask  255.255.255.255
        gateway  [Gateway of ProxmoxIP]
        pointopoint [Gateway of ProxmoxIP]

auto vmbr0
iface vmbr0 inet static
        address  [ProxmoxIP]
        netmask  255.255.255.255
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        up ip route add [ExtraIP1]/32 dev vmbr0
        up ip route add [ExtraIP2]/32 dev vmbr0

#intern
auto vmbr1
iface vmbr1 inet static
        address  10.10.10.1
        netmask  24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

VM1 Network:

Thanks for your time!

102020 · Jan 30, 2020

Confirmed same issue here, Anything internal is not working. I'm running a mesh network with OVS for my layer3. No web access while my vmbr0 was configured for ovs, had to switch it back to a standard bridge, but still my internal network is borked.

oguz · Jan 31, 2020

hi,

do you use ifupdown2 by chance? some users are reporting problems[0] with that.

[0]: https://forum.proxmox.com/threads/ifdownup2-breaks-network-on-nodes.64463/

102020 · Jan 31, 2020

oguz said:
hi,

do you use ifupdown2 by chance? some users are reporting problems[0] with that.

[0]: https://forum.proxmox.com/threads/ifdownup2-breaks-network-on-nodes.64463/

Yes indeed using ifupdown2. Will have a look and report back. Thought it may have been something with the OVS update and/or kernel. My temporary fix has been to add auto vmbr1, which drops off if I make any gui changes.

spirit · Feb 4, 2020

102020 said:
Confirmed same issue here, Anything internal is not working. I'm running a mesh network with OVS for my layer3. No web access while my vmbr0 was configured for ovs, had to switch it back to a standard bridge, but still my internal network is borked.

do you use ifupdown2 2.0 ? (dpkg -l|grep ifupdown2?)
if yes, they are a regression with ovs, it should be fixed with
http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/ifupdown2_2.0.1-1+pve4_all.deb

Miki84 · Feb 22, 2020

Same Problem here, with 2.0.1-1+pve4.
I went back to ifupdown2=1.2.8-1+pve4 and all VM are working fine....

spirit · Feb 22, 2020

Miki84 said:
Same Problem here, with 2.0.1-1+pve4.
I went back to ifupdown2=1.2.8-1+pve4 and all VM are working fine....

can you send your /etc/network/interfaces ? (do you use openvswitch ? because they are a known bug if last ifupdown2)

Miki84 · Feb 23, 2020

Sure.

I dont use openvswitch.

Code:

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp4s0
iface enp4s0 inet static
    address  [HostIP]
    netmask  255.255.255.224
    gateway  [Gateway]
    pointopoint [Gateway]
    up route add -net 78.46.X.X netmask 255.255.255.224 gw [Gateway] dev enp4s0


iface enp4s0 inet6 static
    address  [IPv6]
    netmask  128
    gateway  fe80::1
    up sysctl -p

auto vmbr0
iface vmbr0 inet static
    address   [HostIP]
    netmask  255.255.255.255
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    bridge_maxwait 0
    up ip route add [Additional IP for VM] dev vmbr0

spirit · Feb 23, 2020

@Miki84

just curious, why do you call "sysctl -p" in up ?
could you send the result of "ifup -a -d" ?

I wonder if it could be forwarding be disabled by ifupdown2.
can you try to add "ip-forward on' on ipv4 interfaces, and "ip6-forward on" on ipv6 interfaces ?

Miki84 · Feb 23, 2020

I installed Proxmox via the Hetzner installer.
the sysctl -p call comes from the installer.
no idea why

The result of ifup -a -d with the working
ifupdown2 1.2.8-1+pve4 version?

Or with the 2.0 version that didnt work?

spirit · Feb 24, 2020

Miki84 said:
I installed Proxmox via the Hetzner installer.
the sysctl -p call comes from the installer.
no idea why

ok no problem. I didn't known that hetzner was setting up default configuration.

[QUOTE
The result of ifup -a -d with the working
ifupdown2 1.2.8-1+pve4 version?

Or with the 2.0 version that didnt work?
[/QUOTE]

maybe both version if you can.

I have found a bug with pointopoint in both versions. (mainly, when you ifdown the interface, it's not removing the ip address)
but maybe with 2.0 it's worste than before.

I have make a patch and build a fixed deb:
http://odisoweb1.odiso.net/ifupdown2_2.0.1-1+pve4_all.deb
if you can test, it could be great

(I don't have hetzner server to test)

I have send a pull request to ifupdown2 github too:
https://github.com/CumulusNetworks/ifupdown2/pull/149/

Miki84 · Feb 24, 2020

Hi,

here are the results of ifup -a -d.

I

I have make a patch and build a fixed deb:
http://odisoweb1.odiso.net/ifupdown2_2.0.1-1+pve4_all.deb
if you can test, it could be great (I don't have hetzner server to test)

The patch dint work.

But what worked was to add the linte "ip-forward on' on the interfaces.

spirit · Feb 24, 2020

Ok thanks for testing ! I il look for the forwarding issue

spirit · Feb 24, 2020

can't reproduce the forwarding issue :/

can you look in /etc/sysctl.conf or /etc/sysctl.d/*.conf if you have something special setup with "*forwarding*"

spirit · Feb 24, 2020

and maybe the result of "sysctl -p" (as it's launched in up script)

spirit · Feb 24, 2020

Ok, I have found the forward bug. It was a bug that I had fixed in 1.2 but patch seem to be forget during 2.0 dev.
I'll make a patch.
thanks for reporting this !

proxminent · Mar 13, 2020

Same Problem
apt-get install ifenslave
( this removed ifupdown2 )
and then works again ...

dpkg -l|grep ifupdown2
rc ifupdown2 2.0.1-1+pve4 all Network Interface Management tool similar to ifupdown

spirit · Mar 13, 2020

proxminent said:
Same Problem
apt-get install ifenslave
( this removed ifupdown2 )
and then works again ...

should be fixed in ifupdown2_2.0.1-1+pve7_all.deb , already available in no-subscription repo.

what is your version ?

proxminent · Mar 13, 2020

spirit said:
should be fixed in ifupdown2_2.0.1-1+pve7_all.deb , already available in no-subscription repo.

what is your version ?

ifupdown2 2.0.1-1+pve4

spirit · Mar 13, 2020

proxminent said:
ifupdown2 2.0.1-1+pve4

simply "apt update" && "apt dist-upgrade" to update to last version

No Internet on all VMs after update

Member

Active Member

Proxmox Retired Staff

Active Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Distinguished Member

Member

Attachments

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Well-Known Member

Distinguished Member

Well-Known Member

Distinguished Member