Network stops working (was OSDs on one node keeps crashing)

[Alwin]

It was a very simple test, pinging another host every 20 seconds and checking when the ping failed (1 packets transmitted, 0 received, 100% packet loss, time 0ms).

There is nothing going through the interface at the time of the packet loss? Can you check with tcpdump?

 

[oz1cw7yymn]

There is nothing going through the interface at the time of the packet loss? Can you check with tcpdump?

Hi, took some time. Reinstalled the mystery host with a clean proxmox 6.0 (theory was that the other hosts didn't show this due to having been upgraded).

I ping another host once every 20 seconds and record that in a file with the date

while true;do date>>ping.txt; ping -c 1 ip|grep received>>ping.txt;sleep 20;done

On the same mystery host I run tcpdump

tcpdump -C 25M -e -xx -vvv -tttt -w tcpdump-detailed.log

On the other host I also run tcpdump

tcpdump host mystery-ip -C 25M -e -i interface -xx -n -tttt -vvv | tee -a tcpdump2.log

I monitor the ping.txt for failed pings, and they turn up like clockwork at 00:55-00:56. Matching those pings in the tcpdump, I see the ICMP requests going out and replies coming in, except when the ping failed. No reply shows up on the interface. Both the request and reply show up in the tcpdump of the other host -> a reply is sent but not received.

However, the tcpdump on the mystery host is full of normal ARP, STP, DHCP solicits, multicast and broadcast traffic. Nothing looks strange.