Proxmox Multicast ip doing multicast storm

[Chris Rivera]

The mutlicast ip that is found when using pvecm status is hitting all nodes on cluster with over 60Mb

if i run iftop on all nodes i see large amounts of communication traffic to this multicast ip... this is causing the cluster to not stay online and act up.... any ideas how or why this is happening? cacti also sees this in the graphs...

its so bad that some nodes are not able to to accept ssh connections since the port is overutilized.

 

[spirit]

Hi, how many nodes do you use ? Do you use bonding ?

On a 8 nodes cluster, I don't see more than 1mbit/s of corosync multicast traffic.

 

[Chris Rivera]

15 nodes and no bonding in use.

We are using openvz and have noticed some clients ability to utilize full bandwidth usage to send DDoS attacks so we keep each host limited to a 1gbps port.

multicast bandwidth is normally 1-2Mb.... sometimes increases to 6-10 megs steady.... but yesterday and a previous time it has become a storm and eats up all available bandwidth making nodes inaccessible. When this happens cman is not able to be started but i figured out a way around that already. I created a cron script to remove the dlm module (when cman is crashed it will remove without warning) then stop, cman & pve-cluster... then restart pvedaemon, pve-statd, then start pve-cluster (to receive the latest config file), then start cman.... this seems to be the fix for our weekly (out of sync) occurrence.

To solve this problem with proxmox multicast storm you basically need to take all nodes offline for a few 10-30 seconds then reconnect everything and it goes back to normal. Any ideas?

 

[dietmar]

To solve this problem with proxmox multicast storm you basically need to take all nodes offline for a few 10-30 seconds then reconnect everything and it goes back to normal. Any ideas?

Sorry, I never saw that. Are there some unusual thing going on before that happens?