Guest VM can "see" outgoing traffic of other hosts

Panagiotis Botos · May 1, 2020

Hello,

I have a strange issue with my proxmox implementation.
Proxmox version is 4.4-24 .

The problem is that when I run a tcpdump in one of my Guest VM's I can see OUTGOING traffic that has nothing to do with this VM's IP ( so source and destination is another VM ). This happens on all nodes ( currently 7 ) and the outgoing traffic is only visible where it should not be - per node. So if lets say I have a node1 and a VM inside this node, this VM can "see" all outgoing traffic sourced by this node VM's.

I paste the network configuration :

Code:

auto bond0
iface bond0 inet manual
#    slaves eth0 eth1 eth2 eth3
    slaves eth8 eth9
    bond_miimon 100
    bond_mode 802.3ad
    bond_downdelay 2000
    bond_updelay 4000

auto bond1
iface bond1 inet manual
    slaves eth4 eth6
    bond_miimon 100
    bond_mode 802.3ad
    bond_downdelay 2000
    bond_updelay 4000

auto bond2
iface bond2 inet manual
    slaves eth5 eth7
    bond_miimon 100
    bond_mode 802.3ad
    bond_downdelay 2000
    bond_updelay 4000

auto vmbr0
iface vmbr0 inet static
    address X.X.X.X
    netmask X.X.X.X
    gateway X.X.X.X
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet static
    address  10.0.0.1
    netmask  255.255.255.0
    bridge_ports bond1
    bridge_stp off
    bridge_fd 0

auto vmbr2
iface vmbr2 inet static
    address  192.168.0.1
    netmask  255.255.255.0
    bridge_ports bond2
    bridge_stp off
    bridge_fd 0

vmbr0 is the public bridge
vmbr1 is the Proxmox
vmbr2 is Ceph

I have already exhausted all my troubleshooting skills regarding this issue. So if anyone can help me figure this out I would be really glad.

The switches ( stacked ) connected to these nodes are Dell Switches, and the configuration is quite simple ( with vlt ).

Example of Port-channel for Node1

Code:

interface Port-channel 1
 no ip address
 portmode hybrid
 switchport
 ip access-group multicast out
 vlt-peer-lag port-channel 1
 no shutdown

Again I will appreciate an assistance .

Best Regards,
Panagiotis Botos

Panagiotis Botos · May 4, 2020

Anyone that know's or can provide any insight on this issue ? Still haven't figured out why this is happening.

Best Regards,
Panagiotis Botos

LnxBil · May 8, 2020

Panagiotis Botos said:
Anyone that know's or can provide any insight on this issue ? Still haven't figured out why this is happening.

You're running a very old version of PVE, please upgrade to the newest and see if the problem persists. I'm unable to see other traffic in my cluster (besides the normal stuff like broad- and multicast, ARP, DHCP etc.), so it may have something to do with your old version.

Bruno Garcia · May 9, 2020

i think theres something to do with your bonds. Try to link your bridge directly at physical interface and run a tcpdump again.

Panagiotis Botos · May 13, 2020

Bruno Garcia said:
i think theres something to do with your bonds. Try to link your bridge directly at physical interface and run a tcpdump again.

Thank you for the suggestion. I will check the bonds and see if there are any inconsistencies.

Panagiotis Botos · May 13, 2020

@Bruno Garcia After checking the bonds, they are fine. No inconsistencies found, so the problem does not appear to be on the bonds.
This problem is messing with my nerves...

Panagiotis Botos · Jun 10, 2020

Anyone else has any idea on what could be going on so I can proceed with more troubleshooting?
I have currently used all my knowledge and didn't find anything.

DerDanilo · Jun 13, 2020

@pve Team. Can you reproduce this behaviour?

LnxBil · Jun 18, 2020

Why are we still trying to solve a problem that is in an ancient version of PVE? I already wrote that in never versions of PVE, I could not replicate the problem, so please upgrade to a new version and try again.

Panagiotis Botos · Jul 3, 2020

@LnxBil The issue here is that the cluster is in production state. Until we know for sure that this is an issue of the old PVE version, we can't possibly enforce a major downtime on our clients. That said, we will proceed with the update as soon as possible, as there are many things to consider ( ceph update etc ).

As said above also, we've tried to replicate the problem in our test cluster using the same outdated version and we could not.
This issue is bugging me for quite a while now, so I would like assistance on resolving it as soon as possible before even considering an update and the issues that might bring to the table.

LnxBil · Jul 5, 2020

Panagiotis Botos said:
As said above also, we've tried to replicate the problem in our test cluster using the same outdated version and we could not.
This issue is bugging me for quite a while now, so I would like assistance on resolving it as soon as possible before even considering an update and the issues that might bring to the table.

I understand, but If you can't replicate the problem and I also tried and couldn't how should we approach this?

From the top of my head, compare the working and non-working node settings of
- package versions of the nodes (should be identical, if not upgrade everything to the lastest possible version and try again)
- promiscuous mode on all nodes
- bridge configuration (brctl)
- firewall settings

manwe · Sep 6, 2021

Same issue here. Cluster with bonding and I can see traffic of another vm if I run tcpdump.
I'm using pve cluster v6.4
A vmbr0 on top of bond0 with public IPs for the vms.

auto bond0
iface bond0 inet manual
slaves eno3 eno4
bond-mode balance-rr
bond-miimon 100
bond_downdelay 200
bond_updelay 200

auto vmbr0
iface vmbr0 inet static
address 77.77.77.77/26
gateway 77.77.77.90
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge_maxage 0
bridge_ageing 0
bridge_maxwait 0

vms are in the same network 77.77.77.77/26

spirit · Sep 6, 2021

this is because of

Code:

bridge_maxage 0
bridge_ageing 0

with this, it's flooding always to all ports, like a hub.

itNGO · Jan 27, 2024

Sorry to bring this up again, but I belive this issue still exists even in PVE 8.1.4

We have a 3 Node CEPH-Cluster where the VMBR is set to BOND which is running in Active/Backup. When we have outgoing traffic anywhere in the Network which is in same Subnet as some VM-Firewalls on the PVE-Cluster, all VMs the that traffic on the Interface.

Where to start?

spirit · Jan 28, 2024

itNGO said:
Sorry to bring this up again, but I belive this issue still exists even in PVE 8.1.4

We have a 3 Node CEPH-Cluster where the VMBR is set to BOND which is running in Active/Backup. When we have outgoing traffic anywhere in the Network which is in same Subnet as some VM-Firewalls on the PVE-Cluster, all VMs the that traffic on the Interface.

Where to start?

can you share your /etc/network/interfaces.

They are really no bug in proxmox about this, until you have miss configured your vmbr (bridge_ageing 0, where vmbr will broadcast traffic to all ports) or have a network loop in your network.

itNGO · Jan 29, 2024

spirit said:
can you share your /etc/network/interfaces.

They are really no bug in proxmox about this, until you have miss configured your vmbr (bridge_ageing 0, where vmbr will broadcast traffic to all ports) or have a network loop in your network.

Hi,
sure....

Code:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp195s0f1
iface enp195s0f1 inet static
        address 10.255.183.11/24
#Corosync Cluster Traffic

iface enx4a3ab3036346 inet manual

auto enp195s0f0
iface enp195s0f0 inet manual

auto enp67s0f0
iface enp67s0f0 inet manual
        post-up/sbin/ip link set enp67s0f0 txqueuelen 10000
#10Gbe

auto enp1s0f0np0
iface enp1s0f0np0 inet static
        address 10.255.182.11/24
        mtu 9000
        post-up/sbin/ip link set enp1s0f0np0 txqueuelen 10000
        up ip route add 10.255.182.12/32 dev enp1s0f0np0
        down ip route del 10.255.182.12/32

auto enp1s0f1np1
iface enp1s0f1np1 inet static
        address 10.255.182.11/24
        mtu 9000
        post-up/sbin/ip link set enp1s0f1np1 txqueuelen 10000
        up ip route add 10.255.182.13/32 dev enp1s0f1np1
        down ip route del 10.255.182.13/32
#CEPH2

auto enp67s0f1
iface enp67s0f1 inet manual
        post-up/sbin/ip link set enp67s0f1 txqueuelen 10000
#10Gbe

iface enx8a6dafd2c190 inet manual

iface enx3a6a5e15de89 inet manual

iface enx062c055c2a8c inet manual

iface enxe26795262c70 inet manual

iface enxba8adde23f02 inet manual

iface enx36a74f9dbfde inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp67s0f0 enp67s0f1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp67s0f0

auto vmbr0
iface vmbr0 inet static
        address 10.255.181.11/24
        gateway 10.255.181.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

spirit · Jan 29, 2024

config looks fine.

do you use multiple physical switch ?
if yes, as you use active-backup bond, are your sure that all servers primary nics are on the same switch ?

itNGO · Jan 29, 2024

spirit said:
config looks fine.

do you use multiple physical switch ?
if yes, as you use active-backup bond, are your sure that all servers primary nics are on the same switch ?

Maybe thats the point. We will check this first...
Thx....

itNGO · Feb 7, 2024

itNGO said:
Maybe thats the point. We will check this first...
Thx....

At least it was a not configured interface in a firewall.... so virtual link was up, but no IP-Config. This lead to massive broadcast.... disabled interface and problem is gone....

Guest VM can "see" outgoing traffic of other hosts

Member

Member

Distinguished Member

Member

Member

Member

Member

Renowned Member

Distinguished Member

Member

Distinguished Member

New Member

Distinguished Member

Well-Known Member

Distinguished Member

Well-Known Member

Distinguished Member

Well-Known Member

Well-Known Member