Node random reboot at around specific time

[esi_y]

Can you run it with -0. I just want to know there's not more than one machine that thinks it has that IP address on the segment.

(I forgot to mention there's two arpings, one is in arping package the other in iputils-arping, which I meant. They have different switches. )

 

[scania471]

(I forgot to mention there's two arpings, one is in arping package the other in iputils-arping, which I meant. They have different switches. )

Oh that is a different story
I installed iputils-arping now, arping -b 10.11.12.163 works perfectly, under 1ms. I guess you meant the -0 is for the other package as the iputils-arping one does not have this switch, right?

 

[esi_y]

Oh that is a different story
I installed iputils-arping now, arping -b 10.11.12.163 works perfectly, under 1ms. I guess you meant the -0 is for the other package as the iputils-arping one does not have this switch, right?

I kind of realised it mid-way and thought you would give me a pass on it if I just adapt my answer as if nothing happened. But once you went checking the man page, I better tell you. So ... no duplicate MACs there?

I still feel weird about that machine's behavious before 2:12. Could you perhaps keep a continuous ping in 1 sec intervals running from a screen/tmux somewhere to see when exactly it's unreachable by the network when these reboots occur?

 

[esi_y]

I still feel weird about that machine's behavious before 2:12. Could you perhaps keep a continuous ping in 1 sec intervals running from a screen/tmux somewhere to see when exactly it's unreachable by the network when these reboots occur?

I mean anything works, even just something like ping -D 10.11.12.163 > pinging.log & and then tail -f pinging.log to check ... on some machine that will not be impacted, ideally not a VM that is being HA migrated.

 

[scania471]

no duplicate MACs there?

Correct, only 1 responses and that is the correct one.

Could you perhaps keep a continuous ping in 1 sec intervals running from a screen/tmux somewhere to see when exactly it's unreachable by the network when these reboots occur?

Yup, will put VMs back on it and start a ping the node. Will report back with the results when it happens again!
Thank you again for your help!

 

[scania471]

I mean anything works, even just something like ping -D 10.11.12.163 > pinging.log & and then tail -f pinging.log to check ... on some machine that will not be impacted, ideally not a VM that is being HA migrated.

Hi again!

So finally it rebooted today. I attached the logs for it. In Uptime kuma, it shows a 1m downtime from 2024-03-09 02:15:20 to 2024-03-09 02:16:24 (it checks every 60s, I now set it to check every 30s), every VM got migrated.

Weirdly enough, I found a reboot on 27th Feb in the system logs but I did not get any notification about that, the VMs did not get migrated, Uptime kuma doesn't show any downtime for that timeframe. I also attached the logs for that timeframe.

Side note: since then, I also had a hard drive failure in a different node (anton1) and since then, I get a lot less replication timeouts, like a LOT less, so that's good.

+ I updated and rebooted the VM that was pinging the node and I totally forgot about it, so I don't have pinging information