High traffic between Cluster nodes

[idijoost]

Hi Everyone,

I noticed that starting from today there is a huge network spike on my proxmox host. This spike is is only on 'netin'. As can be seen in the screenshot.
I quickly noticed that the other node (in the cluster) is have a very similar graph. So the screenshot is from one node but on the other node it is the same also 'netin'. My VM/CT's are not using that much traffic. Maybe 100-200k. This spikes sometimes to 3M!! and continues to do so for 15-30 minutes. Because this happens on both hosts and my VM/CT's are not using so much, I believe this is a cluster thing and would like to know why this happen and if this is expected.

I have no HA or Replication enabled.

Thanks in advance

 

[dakralex]

It's hard to tell what could cause the high network traffic without any logging information. What is currently running on your system (e.g. Ceph)? Does the output of journalctl --since '2024-08-29 11:25' --until '2024-08-29 12:25' indicate anything related to the high inward network traffic?

You can also always do a short tcpdump, use any tools to check which program currently uses up a lot of network traffic (e.g. nethogs, iftop), or list the currently opened internet connections with lsof -i.

 

[idijoost]

Hello,

Thanks for your response, I am currently not near the systems. I didn't saw anything related in it, but maybe I missed it. I will post the logging when I am near the systems. I don't run ceph or HA or anything. It happens a few days, but didn't see it happen recently.