VLANs and Broadcast traffic

mattym

Renowned Member
Mar 8, 2011
30
0
71
Hi, Hopefully someone can help me with this :) I have 2 proxmox servers one running version 2.1 and the other running 1.9. I have one interface on the server which connects to a Cisco 3560G switch which is trunked and then vlans attached to different KVM servers. All seems to work ok but after a period of time traffic entering the vlan directed to the KVM host it seems to get broadcast to all servers in the same vlan. If I reboot the KVM host all is ok again for a period of time then for no reason that I can see the traffic starts to broadcast around the VLAN again as it cant see the KVM host directly. Ive seen this happening on both my Proxmox servers which are attached to different Cisco switches on physically separate networks. Does anyone have any ideas because im getting lots of broadcast traffic for KVM's just snowballing and trying to kill off my switches :( Thanks in advance, Matt
 
hi,

they are Dell Poweredges R710 and the other a R510. Broadcom NICs.

thanks

matt
 
I rebooted a KVM host which I thought was causing issues, it was a Debian host and the CPU was up at 100% within the Debian host. The migration/0 process was sucking all the CPU. I connected a wireshark server into the VLAN and couldn't see any broadcast traffic before I rebooted it like I could on another of our proxmox servers vlan. Once the Debian VPS rebooted the traffic dropped off on my graphs and the CPU on the switch has returned back to a much lower level. Strange, have no idea what it was but will monitor to see if it happens again. I will most likely reboot the whole proxmox physical server tonight as it hasn't been rebooted since a new switch was connected.
 
I'm not sure if it's applicable, but a very common issue in a switched environment that will lead to broadcasting unicast traffic is arp table exhaustion. We use this sometimes to snoop on more secure networks where MITM attacks are tougher. Basically if the ARP Tables fill up some (including Cisco) switches will begin to broadcast all traffic.

Again, not sure if it's something to think about in this situation or not. Just a thought.
 
thanks for the replies :)

Ok Im not sure why this is happening on my 2.1 but -

UDP traffic is being broadcast around the whole vlan which is set to receive UDP (Netflows) to one KVM host.
I reboot the host running CentOS 6.3, broadcast traffic stops and UDP is being directed directly to the correct KVM host.
After a period of time 3 to 4 hours approx the broadcast traffic starts again. I checked the Cisco switch and noticed the MAC address is no longer in its table. Shortly after about 20 to 30 minutes the MAC address is back again.

I would expect the MAC address to stay in the table for as long as its receiving traffic but maybe because its primarily UDP it just drops the MAC from the table on the switch? Only thing I can think of is just to enter a static entry for this host but not ideal. :/

thanks

matt
 
I see, so it's specifically netflow traffic that's being broadcast.

Can you post your netflow config?
 
Why not separate this specific broadcast traffic to a dedicated vlan and then have hosts for which this traffic is usefull subscribe to this vlan?
This will prevent you from flooding the rest of the network.
 
it is sort of on a dedicated VLAN, the point being though it shouldnt work ok for a few hours then the MAC address vanish then come back again then it just continues to throw the traffic around the VLAN until I reboot the KVM host. :(

netflow config is from a router outside the vlan not attached to the Cisco switch in question -

ip flow-cache timeout inactive 10
ip flow-cache timeout active 5
ip flow-export source Loopback0
ip flow-export version 9
ip flow-export destination x.x.x.x 9996

thanks

matt
 
Have you done a tcpdump to verify what IP the traffic is headed toward? For example is it broadcasting to 255.255.255.255 or is it broadcasting to the subnet's broadcast address, or is it actually targeted at the x.x.x.x IP in your config?

I would assume the last, but it's worth verifying. What I'm confused about is what else is in this VLAN to even see this broadcast traffic (as in, why does it even matter if the traffic is being broadcast)? Did you have netflow working fine before on a dedicated host and are attempting to move it to a kvm, or is this a new implementation entirely?
 
its the latter for sure, ive had wireshark in the vlan and can see the UDP traffic being thrown around. Ive just added a static mac entry on the switch and its happy now but dont see why I should need to do that. Other devices are in the VLAN like the management of the PM box itself, DRAC's etc. Its a newish implementation, its been doing it for a while and ive just ignored it because it was like 300kbps of traffic being chucked to each port/device in the VLAN. Coupled with the issue I had with another switch and PM 1.9 (sorted for now) I thought id bring it up as im a bit stumped. :/
 
Only thing I can possibly think of is the VM may not be generating any outbound traffic for awhile, and the mac address is somehow falling out of the table (even though you'd think arp would stuff it right back in as soon as it falls out). I'm curious to know if the static arp entry fixes it.
 
yeah thats what I was thinking, I think UDP is stateless too. ARP is more layer 3, if i look at the arp table the only entries in there are for the switch itself and the default gateway which is an ASA5510.
 
Yeah, you want to look at mac-address-table on the switches...the arp is just for the switches layer 3 interface. But yeah, there's no response coming from your netflow monitor back to the router, so that's one way traffic. I'm thinking the issue is, the router still knows the MAC of the server, so it doesn't broadcast an arp request, and thus nothing 'tickles' the mac-address-table to re-insert it.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!