Reloading ifupdown2 breaks internet connectivity within containers/VMs... until toggling ip forward

[MrPowerGamerBR]

This is a very strange issue that I'm having on some of my dedicated servers.

Every time I reload the network interfaces with ifupdown2 installed (Example: When applying changes to the SDN network) all traffic in the VMs and containers break.

However there is a fix to this...

echo 0 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/ip_forward

Yup, toggling ip forward to off and then on fixes the issue. You can not just do "echo 1 > /proc/sys/net/ipv4/ip_forward", you need to disable it and then enable it, then the network is fixed.

What can cause this issue? I mean, it is a annoying issue but at least the fix is very easy.

But what is very strange is that this issue is *not* affecting all of my dedis, only two of my four dedicated servers.

Something worth nothing: All of them were installed via OVH/SoYouStart templates, *however* the two that are being affected by this issue used the Proxmox 6 template, while the other two that weren't affected were installed via the Proxmox 5 template and then upgraded to Proxmox 6.

 

[spirit]

Hi, which kind of plugin zone do you used in sdn ?

if sysctl
net.ipv4.conf.all.forwarding = 1

it should work out of the box

if sysctl
net.ipv4.conf.all.forwarding = 0,

then it's possible to enable ip-forward, interface by interface
you can add "ip-forward 1" in your interfaces where your want routing.

sdn code already doing it in /etc/network/interfaces/sdn


could you check with "sysctl -a|grep forwarding" , if value is changing on interfaces before/after reload.


can you send output of "ifreload -a -d" && /etc/network/interfaces + /etc/network/interfaces.d/sdn ?

 

[MrPowerGamerBR]

if sysctl
net.ipv4.conf.all.forwarding = 1

it should work out of the box

if sysctl
net.ipv4.conf.all.forwarding = 0,

then it's possible to enable ip-forward, interface by interface
you can add "ip-forward 1" in your interfaces where your want routing.

sdn code already doing it in /etc/network/interfaces/sdn


could you check with "sysctl -a|grep forwarding" , if value is changing on interfaces before/after reload.


can you send output of "ifreload -a -d" && /etc/network/interfaces + /etc/network/interfaces.d/sdn ?

VXLAN, however I think that this issue is not related to the SDN plugin, if I recall correctly this was happening even without any SDN network. (heck, even containers that do not use any SDN network (uses only the local network, "vmbr1" in this case, is breaking). When I installed "ifupdown" again (which uninstalled "ifupdown2", the issue was fixed.

In "/etc/sysctl.conf" the "net.ipv4.conf.all.forwarding" is set to 0
In "/etc/ufw/sysctl.conf" the "net.ipv4.conf.all.forwarding" is set to 1

(I already tried setting both to 1, but the issue persisted.)

/etc/network/interfaces

auto lo
iface lo inet loopback

iface enp1s0f0 inet manual

iface enp1s0f1 inet manual

iface enp3s0f0u14u2c2 inet manual

auto vmbr0
iface vmbr0 inet dhcp
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0
        mtu 1450

auto vmbr1
iface vmbr1 inet static
        address 10.0.0.1/8
        bridge-ports none
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

/etc/network/interfaces.d/sdn

#version:62

auto vxlan_vxnet1
iface vxlan_vxnet1
        vxlan-id 10000
        vxlan_remoteip IP1
        vxlan_remoteip IP2
        vxlan_remoteip IP3
        mtu 1450

auto vxnet1
iface vxnet1
        bridge_ports vxlan_vxnet1
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 1450

Executing "ifreload -a -d" causes the connection to break until I toggle the IPv4 Forwarding again.

"sysctl -a|grep forwarding" before using "ifreload -a -d": https://gist.github.com/MrPowerGamerBR/245a01df8ef93870499832fa10b937cd
"sysctl -a|grep forwarding" after using "ifreload -a -d": https://gist.github.com/MrPowerGamerBR/53dbc64214fc3b416f09ed729a29cfe5
"ifreload -a d" output: https://gist.github.com/MrPowerGamerBR/4ad48331a9d98177c01c55331444b6f0

 

[spirit]

@MrPowerGamerBR

Hi,

I don't understand why you need forwarding here ? do you have vms on vmbr1 or vmbr0 with vmbr0/1 is as gateway for theses vms?

I'm seeing ifreload disabling forwarding on vmbr0
info: writing '0 ' to file /proc/sys/net/ipv4/conf/vmbr0/forwarding

does "echo 1 > /proc/sys/net/ipv4/conf/vmbr0/forwarding" is fixing your problem ?

if yes:

I think this is because it's in dhcp mode (seem to be a bug). it should happen only on manual interfaces (without ip address).
for static interface, the forwarsind value is not changed.

for now, you can add "ip-forward 1" to vmbr0 to avoid this.

and I will submit a patch to ifupdown2 devs

 

[MrPowerGamerBR]

@MrPowerGamerBR

Hi,

I don't understand why you need forwarding here ? do you have vms on vmbr1 or vmbr0 with vmbr0/1 is as gateway for theses vms?

I'm seeing ifreload disabling forwarding on vmbr0
info: writing '0 ' to file /proc/sys/net/ipv4/conf/vmbr0/forwarding

does "echo 1 > /proc/sys/net/ipv4/conf/vmbr0/forwarding" is fixing your problem ?

if yes:

I think this is because it's in dhcp mode (seem to be a bug). it should happen only on manual interfaces (without ip address).
for static interface, the forwarsind value is not changed.

for now, you can add "ip-forward 1" to vmbr0 to avoid this.

and I will submit a patch to ifupdown2 devs

Only vmbr1 and vxlan1 are used for VMs/containers. vmbr1 traffic is forwarded to vmbr0 with a iptables rule.

I may be wrong (not really too good with networking haha) but the vmbr0 is used to ask OVH/SYS servers to get the machine IP, if you remove the vmbr0 interface, you are not able to access the dedicated server via the static IP anymore. (trust me, that already happened to me... whoops)

Also, this "vmbr0" was created by default by OVH/SYS when setting up the Proxmox 6 template, and now that you told me that... it seems that my non-affected dedis (that were using Proxmox 5 and then upgraded to Proxmox 6) actually has a static interface!

auto vmbr0
iface vmbr0 inet static
        address MachineIpHereFromSoYouStart
        gateway GatewayIpHereFromSoYouStart
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
        mtu 1500

And yes, "echo 1 > /proc/sys/net/ipv4/conf/vmbr0/forwarding" fixes the issue!

So yeah, you were right! Having a dhcp interface breaks forwarding!!

 

[spirit]

Hi, the bug has been fixed upstream in ifupdown2,
a new release should be available soon.

 

[jammin]

Thank you @MrPowerGamerBR for making this ticket. I spent many hours trying to figure out why it wasn't working consistantly until I found this thread.

I'm also using OVH with prebuild and pve6

@spirit doesn't mention which version this was fixed upstream, but this issue still exists in Debian as of today latest version is 3.0.0-1+pve4~bpo10