Fresh VyOS routing setup - consistent drop of packets (100%), but only some applications

Jan 8, 2021
13
0
6
35
Hi guys,

We need some help please.

In short - we've setup vyos routing on our pve cluster.

The problem is as soon as vyos vm get's an IP address and starts peering all the other guests on the SAME host with firewall ON (on the nic), SOME applications (curl and rsync - there might be more, but we've tested only those) loose connectivity and timeout, mtr, traceroute and ping continue to work.

We also notice this in the logs (7915 is the vyos vm):

Bash:
Nov 29 21:50:29 host-five kernel: [90415512.115001] vmbr0: port 11(tap7915i5) entered disabled state
Nov 29 21:50:29 host-five systemd-udevd[2112363]: Using default interface naming scheme 'v240'.
Nov 29 21:50:29 host-five systemd-udevd[2112363]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 29 21:50:29 host-five systemd-udevd[2112363]: Could not generate persistent MAC address for fwbr7915i5: No such file or directory
Nov 29 21:50:29 host-five systemd-udevd[2112371]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 29 21:50:29 host-five systemd-udevd[2112371]: Using default interface naming scheme 'v240'.
Nov 29 21:50:29 host-five systemd-udevd[2112371]: Could not generate persistent MAC address for fwpr7915p5: No such file or directory
Nov 29 21:50:29 host-five systemd-udevd[2112372]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Nov 29 21:50:29 host-five systemd-udevd[2112372]: Using default interface naming scheme 'v240'.
Nov 29 21:50:29 host-five systemd-udevd[2112372]: Could not generate persistent MAC address for fwln7915i5: No such file or directory
Nov 29 21:50:29 host-five kernel: [90415512.228807] fwbr7915i5: port 1(fwln7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.228811] fwbr7915i5: port 1(fwln7915i5) entered disabled state
Nov 29 21:50:29 host-five kernel: [90415512.228922] device fwln7915i5 entered promiscuous mode
Nov 29 21:50:29 host-five kernel: [90415512.228983] fwbr7915i5: port 1(fwln7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.228985] fwbr7915i5: port 1(fwln7915i5) entered forwarding state
Nov 29 21:50:29 host-five kernel: [90415512.236104] vmbr0: port 11(fwpr7915p5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.236107] vmbr0: port 11(fwpr7915p5) entered disabled state
Nov 29 21:50:29 host-five kernel: [90415512.236203] device fwpr7915p5 entered promiscuous mode
Nov 29 21:50:29 host-five kernel: [90415512.236991] vmbr0: port 11(fwpr7915p5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.236994] vmbr0: port 11(fwpr7915p5) entered forwarding state
Nov 29 21:50:29 host-five kernel: [90415512.262228] fwbr7915i5: port 2(tap7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.262231] fwbr7915i5: port 2(tap7915i5) entered disabled state
Nov 29 21:50:29 host-five kernel: [90415512.262356] fwbr7915i5: port 2(tap7915i5) entered blocking state
Nov 29 21:50:29 host-five kernel: [90415512.262358] fwbr7915i5: port 2(tap7915i5) entered forwarding state

Our version (although we don't think it's version specific)

Bash:
proxmox-ve: 6.4-1 (running kernel: 5.3.13-1-pve)
pve-manager: 6.4-15 (running version: 6.4-15/af7986e6)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.195-1-pve: 5.4.195-1
pve-kernel-5.4.189-2-pve: 5.4.189-2
pve-kernel-5.4.178-1-pve: 5.4.178-1
pve-kernel-5.4.174-2-pve: 5.4.174-2
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-4.15: 5.4-8
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph: 12.2.13-pve1
ceph-fuse: 12.2.13-pve1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve4~bpo10
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.14-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve

Any feedback and/or resolution will be highly appreciated.
 
Last edited:
sound like fragmentation packet && mtu problem. (som protocol, ssh,https for example, use bit "dont fragment" in ip packet, and don't allow fragmented packed). proxmox firewall wil also drop fragmented packets.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!