Hello everyone. I have the following problem. The network part in the cluster works on Linux Bridge using VLAN. The problem is that one virtual machine receives traffic that is not intended for it. It feels like the Bridge works like a regular network hub, not a network switch, and sends all traffic to all network participants. If my theory is correct, then what could be the cause of this and how can I get rid of it? I would not really like to switch to Openvswitch since everything already works.
Traffic leaking through Linux Bridge
- Thread starter plastilin
- Start date
Hi!
Can you send me the configuration of the VM as well as the FDB of the bridge?
Would it be possible for you to send me a tcpdump of some of this traffic from BOTH the bridge as well as the tap interface?
Can you send me the configuration of the VM as well as the FDB of the bridge?
Code:
qm config <vmid>
bridge fdb show <bridge> > fdb.txtWould it be possible for you to send me a tcpdump of some of this traffic from BOTH the bridge as well as the tap interface?
Code:
tcpdump -i <bridge> -w bridge.pcap
tcpdump -i <tapVM> -w tap.pcap.pcap files are very large to download. How can I transfer them to you?
Code:
agent: 1
balloon: 0
boot: order=virtio0
cores: 1
cpu: host
machine: q35
memory: 2048
meta: creation-qemu=8.1.5,ctime=1732618128
name: 09542
net0: virtio=BC:24:11:C3:9F7,bridge=vmbr0,tag=4048
numa: 1
ostype: l26
parent: auto-daily-241126230422
scsihw: virtio-scsi-single
smbios1: uuid=1bbd1d96-d423-4d63-a47b-eb69d5b256f5
sockets: 2
virtio0: ceph-rbd:vm-1037-disk-0,size=128M
vmgenid: 5eefda45-b987-4d3e-b58e-33bb435c024b
Last edited:
Does compressing them possibly help? Otherwise you could upload them somewhere and send me the download link.
Can you please also indicate which traffic inside the capture is traffic that you would not expect to see on the VM?
https://fex.net/uk/s/aerlnyp
I don't need traffic from .59.108 to .59.99 address
PVE version:
pve-manager/8.3.0/c1689ccb1065a83b (running kernel: 6.8.12-4-pve)
I don't need traffic from .59.108 to .59.99 address
PVE version:
pve-manager/8.3.0/c1689ccb1065a83b (running kernel: 6.8.12-4-pve)
Last edited:
.59.108 - VM in the same vlan 4048 but on the same node in a cluster.
In attached file
Code:
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet manual
mtu 9000
auto eno2
iface eno2 inet manual
mtu 9000
auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer3+4
mtu 9000
bond-lacp-rate fast
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000
auto vmbr0.4040
iface vmbr0.4040 inet static
address 10.50.253.2/24
gateway 10.50.253.254
mtu 9000
#MGMT
auto vmbr0.4043
iface vmbr0.4043 inet static
address 10.50.252.2/24
mtu 9000
#CLUSTER
auto vmbr0.4045
iface vmbr0.4045 inet static
address 10.50.251.2/24
mtu 9000
#CEPHCLUSTER
auto vmbr0.4053
iface vmbr0.4053 inet static
address 10.50.250.2/24
mtu 9000
#CEPHPUBLIC
Code:
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 20000
link/ether 38:ea:a7:8e:9b:e4 brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65535
bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 1 vlan_protocol 802.1Q bridge_id 8000.38:ea:a7:8e:9b:e4 designated_root 8000.38:ea:a7:8e:9b:e4 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 1.84 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 no_linklocal_learn 0 mcast_vlan_snooping 0 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536In attached file
I was out of office for a few days and then a bit busy - sorry for the late response.
I looked at the logs and it seems like the traffic that you are talking about has MAC addresses that do not start with 'BC:24:11' (which would be the prefix for Proxmox VMs). The MAC address for the .99 IP is not in the forwarding database, and because of that the packet gets flooded to all ports. Is anything possibly setting the MAC inside the VM manually, leading to this behavior? Is your VM possibly some kind of virtual switch / router? What are the MAC addresses of the bond interface as well as its members?
Thanks for the reply. Yes, the virtual machine that receives the direct traffic is a Mikrotik CHR virtual router. It has one external interface that is not part of the virtual machine bridge.
is one of the MAC addresses of the interfaces inside the VM starting with 82:e4:f3 ? It seems as though the VMs communicating with the Mikrotik are learning this MAC address, and the bridge doesn't have an entry for that MAC address - so it floods all ports with the unicast traffic for that VM.
7, and the mac address of the bridge ends with: 41:08 But ether1 is not part of the bridge - it is a WAN interface.How does the setup look like exactly of the MikroTik Router? In any case, I think the issue has to do with traffic from / to there landing on the bridge and the bridge doesn't know what to do with other than flooding, since it doesn't know the MAC addresses. It might be that the VMs learn the wrong MAC address via ARP, but there's not any relevant ARP request/reply in the tcpdump that I could go on.
You'd have to investigate where the learned MAC address is originally coming from, since it is not equivalent to the MAC address that the VM interface has.
How are the VMs configured? Is the router supposed to route the traffic between those VMs or should it just be switched (by the bridge)? I'd assume the latter since it's the same subnet.
You'd have to investigate where the learned MAC address is originally coming from, since it is not equivalent to the MAC address that the VM interface has.
How are the VMs configured? Is the router supposed to route the traffic between those VMs or should it just be switched (by the bridge)? I'd assume the latter since it's the same subnet.
Last edited:
There are 3 interfaces in Mikrotik, 2 of them have MAC addresses. The external interface created by Proxmox, the MAC addresses of which match both inside the virtual machine and in the virtual machine configuration in PVE. The second interface is an internal bridge in the virtual machine, which includes the interface without MAC is EOIP - memer of internal bridge in VM. Router is used for NAT for remote network. All work fine, but counters of traffic on eth1 - is terrible.
But you said .59.108 and .59.99 are the IPs of the network interfaces of VMs on the same bridge, is that correct? And both of the interfaces are on the same node? Because the MAC-Addresses in the respective UDP packets from .59.108 to .59.99 are as follows:
Code:
Ethernet II, Src: fa:5b:52:62:e5:05 (fa:5b:52:62:e5:05), Dst: 82:e4:f3:93:39:1b (82:e4:f3:93:39:1b)82:e4:f3:93:39:1b has no entry in the fdb of the bridge, so it will get flooded. Proxmox VMs have BC:24:11 by default - unless you override it.I think this is because the bridge doesn't know the destination MAC address in the packets, because it differs from the MAC address from the tap interface, so the packet gets flooded to all ports on the bridge - correct me if I am misunderstanding your setup. You wrote nodes - are those nodes or VMs? Just to be sure.