Guest VM can "see" outgoing traffic of other hosts

P

Panagiotis Botos

Member
Proxmox Subscriber
Feb 9, 2017
11
2
23
Hello,

I have a strange issue with my proxmox implementation.
Proxmox version is 4.4-24 .

The problem is that when I run a tcpdump in one of my Guest VM's I can see OUTGOING traffic that has nothing to do with this VM's IP ( so source and destination is another VM ). This happens on all nodes ( currently 7 ) and the outgoing traffic is only visible where it should not be - per node. So if lets say I have a node1 and a VM inside this node, this VM can "see" all outgoing traffic sourced by this node VM's.

I paste the network configuration :


Code: 
auto bond0
iface bond0 inet manual
#    slaves eth0 eth1 eth2 eth3
    slaves eth8 eth9
    bond_miimon 100
    bond_mode 802.3ad
    bond_downdelay 2000
    bond_updelay 4000

auto bond1
iface bond1 inet manual
    slaves eth4 eth6
    bond_miimon 100
    bond_mode 802.3ad
    bond_downdelay 2000
    bond_updelay 4000

auto bond2
iface bond2 inet manual
    slaves eth5 eth7
    bond_miimon 100
    bond_mode 802.3ad
    bond_downdelay 2000
    bond_updelay 4000

auto vmbr0
iface vmbr0 inet static
    address X.X.X.X
    netmask X.X.X.X
    gateway X.X.X.X
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet static
    address  10.0.0.1
    netmask  255.255.255.0
    bridge_ports bond1
    bridge_stp off
    bridge_fd 0

auto vmbr2
iface vmbr2 inet static
    address  192.168.0.1
    netmask  255.255.255.0
    bridge_ports bond2
    bridge_stp off
    bridge_fd 0

vmbr0 is the public bridge
vmbr1 is the Proxmox
vmbr2 is Ceph

I have already exhausted all my troubleshooting skills regarding this issue. So if anyone can help me figure this out I would be really glad.

The switches ( stacked ) connected to these nodes are Dell Switches, and the configuration is quite simple ( with vlt ).

Example of Port-channel for Node1

Code: 
interface Port-channel 1
 no ip address
 portmode hybrid
 switchport
 ip access-group multicast out
 vlt-peer-lag port-channel 1
 no shutdown

Again I will appreciate an assistance .

Best Regards,
Panagiotis Botos
 
  • Like
Reactions: DerDanilo
Panagiotis Botos said:
Anyone that know's or can provide any insight on this issue ? Still haven't figured out why this is happening.
Click to expand...

You're running a very old version of PVE, please upgrade to the newest and see if the problem persists. I'm unable to see other traffic in my cluster (besides the normal stuff like broad- and multicast, ARP, DHCP etc.), so it may have something to do with your old version.
 
i think theres something to do with your bonds. Try to link your bridge directly at physical interface and run a tcpdump again.
 
Why are we still trying to solve a problem that is in an ancient version of PVE? I already wrote that in never versions of PVE, I could not replicate the problem, so please upgrade to a new version and try again.
 
@LnxBil The issue here is that the cluster is in production state. Until we know for sure that this is an issue of the old PVE version, we can't possibly enforce a major downtime on our clients. That said, we will proceed with the update as soon as possible, as there are many things to consider ( ceph update etc ).

As said above also, we've tried to replicate the problem in our test cluster using the same outdated version and we could not.
This issue is bugging me for quite a while now, so I would like assistance on resolving it as soon as possible before even considering an update and the issues that might bring to the table.
 
Panagiotis Botos said:
As said above also, we've tried to replicate the problem in our test cluster using the same outdated version and we could not.
This issue is bugging me for quite a while now, so I would like assistance on resolving it as soon as possible before even considering an update and the issues that might bring to the table.
Click to expand...

I understand, but If you can't replicate the problem and I also tried and couldn't how should we approach this?

From the top of my head, compare the working and non-working node settings of
- package versions of the nodes (should be identical, if not upgrade everything to the lastest possible version and try again)
- promiscuous mode on all nodes
- bridge configuration (brctl)
- firewall settings
 
Same issue here. Cluster with bonding and I can see traffic of another vm if I run tcpdump.
I'm using pve cluster v6.4
A vmbr0 on top of bond0 with public IPs for the vms.

auto bond0
iface bond0 inet manual
slaves eno3 eno4
bond-mode balance-rr
bond-miimon 100
bond_downdelay 200
bond_updelay 200


auto vmbr0
iface vmbr0 inet static
address 77.77.77.77/26
gateway 77.77.77.90
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge_maxage 0
bridge_ageing 0
bridge_maxwait 0

vms are in the same network 77.77.77.77/26
 
Last edited:
Sorry to bring this up again, but I belive this issue still exists even in PVE 8.1.4

We have a 3 Node CEPH-Cluster where the VMBR is set to BOND which is running in Active/Backup. When we have outgoing traffic anywhere in the Network which is in same Subnet as some VM-Firewalls on the PVE-Cluster, all VMs the that traffic on the Interface.

Where to start?
 
itNGO said:
Sorry to bring this up again, but I belive this issue still exists even in PVE 8.1.4

We have a 3 Node CEPH-Cluster where the VMBR is set to BOND which is running in Active/Backup. When we have outgoing traffic anywhere in the Network which is in same Subnet as some VM-Firewalls on the PVE-Cluster, all VMs the that traffic on the Interface.

Where to start?
Click to expand...
can you share your /etc/network/interfaces.

They are really no bug in proxmox about this, until you have miss configured your vmbr (bridge_ageing 0, where vmbr will broadcast traffic to all ports) or have a network loop in your network.
 
spirit said:
can you share your /etc/network/interfaces.

They are really no bug in proxmox about this, until you have miss configured your vmbr (bridge_ageing 0, where vmbr will broadcast traffic to all ports) or have a network loop in your network.
Click to expand...
Hi,
sure....

Code: 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto enp195s0f1
iface enp195s0f1 inet static
        address 10.255.183.11/24
#Corosync Cluster Traffic

iface enx4a3ab3036346 inet manual

auto enp195s0f0
iface enp195s0f0 inet manual

auto enp67s0f0
iface enp67s0f0 inet manual
        post-up/sbin/ip link set enp67s0f0 txqueuelen 10000
#10Gbe

auto enp1s0f0np0
iface enp1s0f0np0 inet static
        address 10.255.182.11/24
        mtu 9000
        post-up/sbin/ip link set enp1s0f0np0 txqueuelen 10000
        up ip route add 10.255.182.12/32 dev enp1s0f0np0
        down ip route del 10.255.182.12/32

auto enp1s0f1np1
iface enp1s0f1np1 inet static
        address 10.255.182.11/24
        mtu 9000
        post-up/sbin/ip link set enp1s0f1np1 txqueuelen 10000
        up ip route add 10.255.182.13/32 dev enp1s0f1np1
        down ip route del 10.255.182.13/32
#CEPH2

auto enp67s0f1
iface enp67s0f1 inet manual
        post-up/sbin/ip link set enp67s0f1 txqueuelen 10000
#10Gbe

iface enx8a6dafd2c190 inet manual

iface enx3a6a5e15de89 inet manual

iface enx062c055c2a8c inet manual

iface enxe26795262c70 inet manual

iface enxba8adde23f02 inet manual

iface enx36a74f9dbfde inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp67s0f0 enp67s0f1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp67s0f0

auto vmbr0
iface vmbr0 inet static
        address 10.255.181.11/24
        gateway 10.255.181.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
 
spirit said:
config looks fine.

do you use multiple physical switch ?
if yes, as you use active-backup bond, are your sure that all servers primary nics are on the same switch ?
Click to expand...
Maybe thats the point. We will check this first...
Thx....
 
itNGO said:
Maybe thats the point. We will check this first...
Thx....
Click to expand...
At least it was a not configured interface in a firewall.... so virtual link was up, but no IP-Config. This lead to massive broadcast.... disabled interface and problem is gone....
 
  • Like
Reactions: spirit
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom