Fresh VyOS routing setup - consistent drop of packets (100%), but only some applications

[s_sasha]

Hi guys,

We need some help please.

In short - we've setup vyos routing on our pve cluster.

The problem is as soon as vyos vm get's an IP address and starts peering all the other guests on the SAME host with firewall ON (on the nic), SOME applications (curl and rsync - there might be more, but we've tested only those) loose connectivity and timeout, mtr, traceroute and ping continue to work.

We also notice this in the logs (7915 is the vyos vm):

Nov 29 21:50:29 host-five kernel: [90415512.115001] vmbr0: port 11(tap7915i5) entered disabled state
Nov 29 21:50:29 host-five systemd-udevd[2112363]: Using default interface naming scheme 'v240'.
Nov 29 21:50:29 host-five systemd-udevd[2112363]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 29 21:50:29 host-five systemd-udevd[2112363]: Could not generate persistent MAC address for fwbr7915i5: No such file or directory
Nov 29 21:50:29 host-five systemd-udevd[2112371]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 29 21:50:29 host-five systemd-udevd[2112371]: Using default interface naming scheme 'v240'.
Nov 29 21:50:29 host-five systemd-udevd[2112371]: Could not generate persistent MAC address for fwpr7915p5: No such file or directory
Nov 29 21:50:29 host-five systemd-udevd[2112372]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 29 21:50:29 host-five systemd-udevd[2112372]: Using default interface naming scheme 'v240'.
Nov 29 21:50:29 host-five systemd-udevd[2112372]: Could not generate persistent MAC address for fwln7915i5: No such file or directory
Nov 29 21:50:29 host-five kernel: [90415512.228807] fwbr7915i5: port 1(fwln7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.228811] fwbr7915i5: port 1(fwln7915i5) entered disabled state
Nov 29 21:50:29 host-five kernel: [90415512.228922] device fwln7915i5 entered promiscuous mode
Nov 29 21:50:29 host-five kernel: [90415512.228983] fwbr7915i5: port 1(fwln7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.228985] fwbr7915i5: port 1(fwln7915i5) entered forwarding state
Nov 29 21:50:29 host-five kernel: [90415512.236104] vmbr0: port 11(fwpr7915p5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.236107] vmbr0: port 11(fwpr7915p5) entered disabled state
Nov 29 21:50:29 host-five kernel: [90415512.236203] device fwpr7915p5 entered promiscuous mode
Nov 29 21:50:29 host-five kernel: [90415512.236991] vmbr0: port 11(fwpr7915p5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.236994] vmbr0: port 11(fwpr7915p5) entered forwarding state
Nov 29 21:50:29 host-five kernel: [90415512.262228] fwbr7915i5: port 2(tap7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.262231] fwbr7915i5: port 2(tap7915i5) entered disabled state
Nov 29 21:50:29 host-five kernel: [90415512.262356] fwbr7915i5: port 2(tap7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.262358] fwbr7915i5: port 2(tap7915i5) entered forwarding state

Our version (although we don't think it's version specific)

proxmox-ve: 6.4-1 (running kernel: 5.3.13-1-pve)
pve-manager: 6.4-15 (running version: 6.4-15/af7986e6)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.195-1-pve: 5.4.195-1
pve-kernel-5.4.189-2-pve: 5.4.189-2
pve-kernel-5.4.178-1-pve: 5.4.178-1
pve-kernel-5.4.174-2-pve: 5.4.174-2
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-4.15: 5.4-8
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph: 12.2.13-pve1
ceph-fuse: 12.2.13-pve1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve4~bpo10
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.14-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve

Any feedback and/or resolution will be highly appreciated.

 

[spirit]

sound like fragmentation packet && mtu problem. (som protocol, ssh,https for example, use bit "dont fragment" in ip packet, and don't allow fragmented packed). proxmox firewall wil also drop fragmented packets.

 

[s_sasha]

Hi @spirit, thanks for the response, please also note that this manifests only on the hosts with linux bridge, hosts with ovs bridging there's no issues ..