Node reboots after other Node loses network

[greenapple]

Hello all,

we are facing a strange issue. We have a two node Cluster plus QDevice on the latest PVE version with HA enabled on the VMs. We are currently in the process of testing the cluster. In order to test the fencing and HA recovery, we pulled the network from one of the two nodes (Node 1). Everything went to plan, the remaining node and the QDevice still had quorum according to the logs, it was only after the offline node got fenced when the second node randomly rebooted as well. I'm a bit puzzled because the Logs don't indicate any reason at all for the reboot.

We are running all the traffic on a single bond, we know this isn't best practice but since the Servers are blade systems we'd rather have the redundancy of the bond. The latency when bulk migrating all VMs from one node to the other doesn't surpass 0,9ms, so I can't really imagine the cluster losing Quorum due to latency. The problem is reproducible. The logs are attached. Does anyone have an idea? Thanks in advance!

 

[greenapple]

Hello,

I would like to follow up on this. We have redone the test with debug logging activated for Corosync and while tracking the latency. I'm personally about as lost as before, but latency seems to be ruled out as an issue. I have attached the logs. I'd be thankful for any ideas! Thanks in advance.

Latency
root@Node 2:~# ping QDevice
PING QDevice (QDevice) 56(84) bytes of data.
64 bytes from QDevice: icmp_seq=1 ttl=64 time=0.343 ms
64 bytes from QDevice: icmp_seq=2 ttl=64 time=0.390 ms
64 bytes from QDevice: icmp_seq=3 ttl=64 time=0.138 ms
64 bytes from QDevice: icmp_seq=4 ttl=64 time=0.370 ms
64 bytes from QDevice: icmp_seq=5 ttl=64 time=0.381 ms
64 bytes from QDevice: icmp_seq=6 ttl=64 time=0.387 ms
64 bytes from QDevice: icmp_seq=7 ttl=64 time=0.371 ms
64 bytes from QDevice: icmp_seq=8 ttl=64 time=0.367 ms
64 bytes from QDevice: icmp_seq=9 ttl=64 time=0.354 ms
64 bytes from QDevice: icmp_seq=10 ttl=64 time=0.223 ms
64 bytes from QDevice: icmp_seq=11 ttl=64 time=0.417 ms
64 bytes from QDevice: icmp_seq=12 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=13 ttl=64 time=0.249 ms
64 bytes from QDevice: icmp_seq=14 ttl=64 time=0.368 ms
64 bytes from QDevice: icmp_seq=15 ttl=64 time=0.426 ms
64 bytes from QDevice: icmp_seq=16 ttl=64 time=0.397 ms
64 bytes from QDevice: icmp_seq=17 ttl=64 time=0.360 ms
64 bytes from QDevice: icmp_seq=18 ttl=64 time=0.369 ms
64 bytes from QDevice: icmp_seq=19 ttl=64 time=0.262 ms
64 bytes from QDevice: icmp_seq=20 ttl=64 time=0.403 ms
64 bytes from QDevice: icmp_seq=21 ttl=64 time=0.412 ms
64 bytes from QDevice: icmp_seq=22 ttl=64 time=0.366 ms
64 bytes from QDevice: icmp_seq=23 ttl=64 time=0.271 ms
64 bytes from QDevice: icmp_seq=24 ttl=64 time=0.365 ms
64 bytes from QDevice: icmp_seq=25 ttl=64 time=0.368 ms
64 bytes from QDevice: icmp_seq=26 ttl=64 time=0.366 ms
64 bytes from QDevice: icmp_seq=27 ttl=64 time=0.388 ms
64 bytes from QDevice: icmp_seq=28 ttl=64 time=0.356 ms
64 bytes from QDevice: icmp_seq=29 ttl=64 time=0.407 ms
64 bytes from QDevice: icmp_seq=30 ttl=64 time=0.360 ms
64 bytes from QDevice: icmp_seq=31 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=32 ttl=64 time=0.365 ms
64 bytes from QDevice: icmp_seq=33 ttl=64 time=0.387 ms
64 bytes from QDevice: icmp_seq=34 ttl=64 time=0.403 ms
64 bytes from QDevice: icmp_seq=35 ttl=64 time=0.396 ms
64 bytes from QDevice: icmp_seq=36 ttl=64 time=0.398 ms
64 bytes from QDevice: icmp_seq=37 ttl=64 time=0.366 ms
64 bytes from QDevice: icmp_seq=38 ttl=64 time=0.405 ms
64 bytes from QDevice: icmp_seq=39 ttl=64 time=0.361 ms
64 bytes from QDevice: icmp_seq=40 ttl=64 time=0.391 ms
64 bytes from QDevice: icmp_seq=41 ttl=64 time=0.391 ms
64 bytes from QDevice: icmp_seq=42 ttl=64 time=0.153 ms
64 bytes from QDevice: icmp_seq=43 ttl=64 time=0.373 ms
64 bytes from QDevice: icmp_seq=44 ttl=64 time=0.279 ms
64 bytes from QDevice: icmp_seq=45 ttl=64 time=0.290 ms
64 bytes from QDevice: icmp_seq=46 ttl=64 time=0.402 ms
64 bytes from QDevice: icmp_seq=47 ttl=64 time=0.393 ms
64 bytes from QDevice: icmp_seq=48 ttl=64 time=0.273 ms
64 bytes from QDevice: icmp_seq=49 ttl=64 time=0.400 ms
64 bytes from QDevice: icmp_seq=50 ttl=64 time=0.362 ms
64 bytes from QDevice: icmp_seq=51 ttl=64 time=0.384 ms
64 bytes from QDevice: icmp_seq=52 ttl=64 time=0.358 ms
64 bytes from QDevice: icmp_seq=53 ttl=64 time=0.375 ms
64 bytes from QDevice: icmp_seq=54 ttl=64 time=0.345 ms
64 bytes from QDevice: icmp_seq=55 ttl=64 time=0.388 ms
64 bytes from QDevice: icmp_seq=56 ttl=64 time=0.392 ms
64 bytes from QDevice: icmp_seq=57 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=58 ttl=64 time=0.398 ms
64 bytes from QDevice: icmp_seq=59 ttl=64 time=0.384 ms
64 bytes from QDevice: icmp_seq=60 ttl=64 time=0.403 ms
64 bytes from QDevice: icmp_seq=61 ttl=64 time=0.386 ms
64 bytes from QDevice: icmp_seq=62 ttl=64 time=0.388 ms
64 bytes from QDevice: icmp_seq=63 ttl=64 time=0.395 ms
64 bytes from QDevice: icmp_seq=64 ttl=64 time=0.393 ms
64 bytes from QDevice: icmp_seq=65 ttl=64 time=0.336 ms
64 bytes from QDevice: icmp_seq=66 ttl=64 time=0.369 ms
64 bytes from QDevice: icmp_seq=67 ttl=64 time=0.390 ms
64 bytes from QDevice: icmp_seq=68 ttl=64 time=0.389 ms
64 bytes from QDevice: icmp_seq=69 ttl=64 time=0.372 ms
64 bytes from QDevice: icmp_seq=70 ttl=64 time=0.366 ms
64 bytes from QDevice: icmp_seq=71 ttl=64 time=0.352 ms
64 bytes from QDevice: icmp_seq=72 ttl=64 time=0.392 ms
64 bytes from QDevice: icmp_seq=73 ttl=64 time=0.345 ms
64 bytes from QDevice: icmp_seq=74 ttl=64 time=0.383 ms
64 bytes from QDevice: icmp_seq=75 ttl=64 time=0.256 ms
64 bytes from QDevice: icmp_seq=76 ttl=64 time=0.369 ms
64 bytes from QDevice: icmp_seq=77 ttl=64 time=0.245 ms
64 bytes from QDevice: icmp_seq=78 ttl=64 time=0.126 ms
64 bytes from QDevice: icmp_seq=79 ttl=64 time=0.160 ms
64 bytes from QDevice: icmp_seq=80 ttl=64 time=0.414 ms
64 bytes from QDevice: icmp_seq=81 ttl=64 time=0.401 ms
64 bytes from QDevice: icmp_seq=82 ttl=64 time=0.399 ms
64 bytes from QDevice: icmp_seq=83 ttl=64 time=0.133 ms
64 bytes from QDevice: icmp_seq=84 ttl=64 time=0.405 ms
64 bytes from QDevice: icmp_seq=85 ttl=64 time=0.340 ms
64 bytes from QDevice: icmp_seq=86 ttl=64 time=0.332 ms
64 bytes from QDevice: icmp_seq=87 ttl=64 time=0.120 ms
64 bytes from QDevice: icmp_seq=88 ttl=64 time=0.378 ms
64 bytes from QDevice: icmp_seq=89 ttl=64 time=0.113 ms
64 bytes from QDevice: icmp_seq=90 ttl=64 time=0.370 ms
64 bytes from QDevice: icmp_seq=91 ttl=64 time=0.376 ms
64 bytes from QDevice: icmp_seq=92 ttl=64 time=0.400 ms
64 bytes from QDevice: icmp_seq=93 ttl=64 time=0.417 ms
64 bytes from QDevice: icmp_seq=94 ttl=64 time=0.207 ms
64 bytes from QDevice: icmp_seq=95 ttl=64 time=0.357 ms
64 bytes from QDevice: icmp_seq=96 ttl=64 time=0.330 ms
64 bytes from QDevice: icmp_seq=97 ttl=64 time=0.327 ms
64 bytes from QDevice: icmp_seq=98 ttl=64 time=0.394 ms
64 bytes from QDevice: icmp_seq=99 ttl=64 time=0.366 ms
64 bytes from QDevice: icmp_seq=100 ttl=64 time=0.377 ms
64 bytes from QDevice: icmp_seq=101 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=102 ttl=64 time=0.327 ms
64 bytes from QDevice: icmp_seq=103 ttl=64 time=0.395 ms
64 bytes from QDevice: icmp_seq=104 ttl=64 time=0.435 ms
64 bytes from QDevice: icmp_seq=105 ttl=64 time=0.431 ms
64 bytes from QDevice: icmp_seq=106 ttl=64 time=0.367 ms
64 bytes from QDevice: icmp_seq=107 ttl=64 time=0.393 ms
64 bytes from QDevice: icmp_seq=108 ttl=64 time=0.378 ms
64 bytes from QDevice: icmp_seq=109 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=110 ttl=64 time=0.226 ms
64 bytes from QDevice: icmp_seq=111 ttl=64 time=0.149 ms
64 bytes from QDevice: icmp_seq=112 ttl=64 time=0.380 ms
64 bytes from QDevice: icmp_seq=113 ttl=64 time=0.368 ms
64 bytes from QDevice: icmp_seq=114 ttl=64 time=0.362 ms
64 bytes from QDevice: icmp_seq=115 ttl=64 time=0.352 ms
64 bytes from QDevice: icmp_seq=116 ttl=64 time=0.209 ms
64 bytes from QDevice: icmp_seq=117 ttl=64 time=0.399 ms
64 bytes from QDevice: icmp_seq=118 ttl=64 time=0.255 ms
64 bytes from QDevice: icmp_seq=119 ttl=64 time=0.394 ms
64 bytes from QDevice: icmp_seq=120 ttl=64 time=0.396 ms
64 bytes from QDevice: icmp_seq=121 ttl=64 time=0.387 ms
64 bytes from QDevice: icmp_seq=122 ttl=64 time=0.391 ms
64 bytes from QDevice: icmp_seq=123 ttl=64 time=0.351 ms
64 bytes from QDevice: icmp_seq=124 ttl=64 time=0.386 ms
64 bytes from QDevice: icmp_seq=125 ttl=64 time=0.332 ms
64 bytes from QDevice: icmp_seq=126 ttl=64 time=0.346 ms
64 bytes from QDevice: icmp_seq=127 ttl=64 time=0.417 ms
64 bytes from QDevice: icmp_seq=128 ttl=64 time=0.384 ms
64 bytes from QDevice: icmp_seq=129 ttl=64 time=0.333 ms
64 bytes from QDevice: icmp_seq=130 ttl=64 time=0.378 ms
64 bytes from QDevice: icmp_seq=131 ttl=64 time=0.336 ms
64 bytes from QDevice: icmp_seq=132 ttl=64 time=0.351 ms
64 bytes from QDevice: icmp_seq=133 ttl=64 time=0.390 ms
64 bytes from QDevice: icmp_seq=134 ttl=64 time=0.378 ms
64 bytes from QDevice: icmp_seq=135 ttl=64 time=0.332 ms
64 bytes from QDevice: icmp_seq=136 ttl=64 time=0.250 ms
64 bytes from QDevice: icmp_seq=137 ttl=64 time=0.371 ms
64 bytes from QDevice: icmp_seq=138 ttl=64 time=0.395 ms
64 bytes from QDevice: icmp_seq=139 ttl=64 time=0.402 ms
64 bytes from QDevice: icmp_seq=140 ttl=64 time=0.347 ms
64 bytes from QDevice: icmp_seq=141 ttl=64 time=0.342 ms
64 bytes from QDevice: icmp_seq=142 ttl=64 time=0.391 ms
64 bytes from QDevice: icmp_seq=143 ttl=64 time=0.375 ms
64 bytes from QDevice: icmp_seq=144 ttl=64 time=0.404 ms
64 bytes from QDevice: icmp_seq=145 ttl=64 time=0.384 ms
64 bytes from QDevice: icmp_seq=146 ttl=64 time=0.378 ms
64 bytes from QDevice: icmp_seq=147 ttl=64 time=0.395 ms
64 bytes from QDevice: icmp_seq=148 ttl=64 time=0.366 ms
64 bytes from QDevice: icmp_seq=149 ttl=64 time=0.369 ms
64 bytes from QDevice: icmp_seq=150 ttl=64 time=0.375 ms
64 bytes from QDevice: icmp_seq=151 ttl=64 time=0.364 ms
64 bytes from QDevice: icmp_seq=152 ttl=64 time=0.370 ms
64 bytes from QDevice: icmp_seq=153 ttl=64 time=0.394 ms
64 bytes from QDevice: icmp_seq=154 ttl=64 time=0.348 ms
64 bytes from QDevice: icmp_seq=155 ttl=64 time=0.364 ms
64 bytes from QDevice: icmp_seq=156 ttl=64 time=0.376 ms
64 bytes from QDevice: icmp_seq=157 ttl=64 time=0.383 ms
64 bytes from QDevice: icmp_seq=158 ttl=64 time=0.355 ms
64 bytes from QDevice: icmp_seq=159 ttl=64 time=0.387 ms
64 bytes from QDevice: icmp_seq=160 ttl=64 time=0.412 ms
64 bytes from QDevice: icmp_seq=161 ttl=64 time=0.381 ms
64 bytes from QDevice: icmp_seq=162 ttl=64 time=0.354 ms
64 bytes from QDevice: icmp_seq=163 ttl=64 time=0.349 ms
64 bytes from QDevice: icmp_seq=164 ttl=64 time=0.381 ms
64 bytes from QDevice: icmp_seq=165 ttl=64 time=0.352 ms
64 bytes from QDevice: icmp_seq=166 ttl=64 time=0.404 ms
64 bytes from QDevice: icmp_seq=167 ttl=64 time=0.323 ms
64 bytes from QDevice: icmp_seq=168 ttl=64 time=0.349 ms
64 bytes from QDevice: icmp_seq=169 ttl=64 time=0.349 ms
64 bytes from QDevice: icmp_seq=170 ttl=64 time=0.357 ms
64 bytes from QDevice: icmp_seq=171 ttl=64 time=0.353 ms
64 bytes from QDevice: icmp_seq=172 ttl=64 time=0.415 ms
64 bytes from QDevice: icmp_seq=173 ttl=64 time=0.353 ms
64 bytes from QDevice: icmp_seq=174 ttl=64 time=0.400 ms
64 bytes from QDevice: icmp_seq=175 ttl=64 time=0.367 ms
64 bytes from QDevice: icmp_seq=176 ttl=64 time=0.442 ms
64 bytes from QDevice: icmp_seq=177 ttl=64 time=0.364 ms
64 bytes from QDevice: icmp_seq=178 ttl=64 time=0.319 ms
64 bytes from QDevice: icmp_seq=179 ttl=64 time=0.373 ms
64 bytes from QDevice: icmp_seq=180 ttl=64 time=0.373 ms
64 bytes from QDevice: icmp_seq=181 ttl=64 time=0.379 ms
64 bytes from QDevice: icmp_seq=182 ttl=64 time=0.353 ms
64 bytes from QDevice: icmp_seq=183 ttl=64 time=0.374 ms
64 bytes from QDevice: icmp_seq=184 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=185 ttl=64 time=0.373 ms
64 bytes from QDevice: icmp_seq=186 ttl=64 time=0.413 ms
64 bytes from QDevice: icmp_seq=187 ttl=64 time=0.362 ms
64 bytes from QDevice: icmp_seq=188 ttl=64 time=0.410 ms
64 bytes from QDevice: icmp_seq=189 ttl=64 time=0.384 ms
64 bytes from QDevice: icmp_seq=190 ttl=64 time=0.253 ms
64 bytes from QDevice: icmp_seq=191 ttl=64 time=0.348 ms
64 bytes from QDevice: icmp_seq=192 ttl=64 time=0.290 ms
64 bytes from QDevice: icmp_seq=193 ttl=64 time=0.372 ms
64 bytes from QDevice: icmp_seq=194 ttl=64 time=0.369 ms
64 bytes from QDevice: icmp_seq=195 ttl=64 time=0.276 ms
64 bytes from QDevice: icmp_seq=196 ttl=64 time=0.393 ms
64 bytes from QDevice: icmp_seq=197 ttl=64 time=0.395 ms
64 bytes from QDevice: icmp_seq=198 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=199 ttl=64 time=0.361 ms
64 bytes from QDevice: icmp_seq=200 ttl=64 time=0.390 ms
64 bytes from QDevice: icmp_seq=201 ttl=64 time=0.349 ms
64 bytes from QDevice: icmp_seq=202 ttl=64 time=0.373 ms
64 bytes from QDevice: icmp_seq=203 ttl=64 time=0.349 ms
64 bytes from QDevice: icmp_seq=204 ttl=64 time=0.361 ms
64 bytes from QDevice: icmp_seq=205 ttl=64 time=0.414 ms
64 bytes from QDevice: icmp_seq=206 ttl=64 time=0.372 ms
64 bytes from QDevice: icmp_seq=207 ttl=64 time=0.356 ms
64 bytes from QDevice: icmp_seq=208 ttl=64 time=0.380 ms
64 bytes from QDevice: icmp_seq=209 ttl=64 time=0.357 ms
64 bytes from QDevice: icmp_seq=210 ttl=64 time=0.383 ms
64 bytes from QDevice: icmp_seq=211 ttl=64 time=0.238 ms
64 bytes from QDevice: icmp_seq=212 ttl=64 time=0.120 ms
64 bytes from QDevice: icmp_seq=213 ttl=64 time=0.379 ms
64 bytes from QDevice: icmp_seq=214 ttl=64 time=0.400 ms
64 bytes from QDevice: icmp_seq=215 ttl=64 time=0.404 ms
64 bytes from QDevice: icmp_seq=216 ttl=64 time=0.371 ms
64 bytes from QDevice: icmp_seq=217 ttl=64 time=0.373 ms
64 bytes from QDevice: icmp_seq=218 ttl=64 time=0.405 ms
64 bytes from QDevice: icmp_seq=219 ttl=64 time=0.384 ms
64 bytes from QDevice: icmp_seq=220 ttl=64 time=0.378 ms
64 bytes from QDevice: icmp_seq=221 ttl=64 time=0.341 ms
64 bytes from QDevice: icmp_seq=222 ttl=64 time=0.364 ms
64 bytes from QDevice: icmp_seq=223 ttl=64 time=0.376 ms
64 bytes from QDevice: icmp_seq=224 ttl=64 time=0.400 ms
64 bytes from QDevice: icmp_seq=225 ttl=64 time=0.382 ms
64 bytes from QDevice: icmp_seq=226 ttl=64 time=0.372 ms
64 bytes from QDevice: icmp_seq=227 ttl=64 time=0.365 ms
--reboot--

 

[justinclift]

Purely as a test, you could try telling the watchdog software (it does the rebooting) to not actually reboot the server.

If the server then stops rebooting 1 minute after the other node fails, you'll then at least know what's doing the reboot bit.

You can do that by creating a file in the /etc/modprobe.d directory with the contents of "options softdog soft_noboot=1" (then rebooting, ironically).

I do it by running this from a shell / command line on the server:

echo "options softdog soft_noboot=1" > /etc/modprobe.d/disable-softdog-reboot.conf

 

[justinclift]

Oh, another thing to check is the Proxmox cluster status from the cluster management software's perspective.

Would you be ok to run this command and paste the entire output here?

pvecm status

 

[greenapple]

Oh, another thing to check is the Proxmox cluster status from the cluster management software's perspective.

Would you be ok to run this command and paste the entire output here?

pvecm status

Sure thing!

root@NODE 2:~# pvecm status
Cluster information
-------------------
Name:             Cluster
Config Version:   5
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu Jun  6 09:51:08 2024
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1.15a
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW IP_NODE_1
0x00000002          1    A,V,NMW IP_NODE_2 (local)
0x00000000          1            Qdevice

Purely as a test, you could try telling the watchdog software (it does the rebooting) to not actually reboot the server.

If the server then stops rebooting 1 minute after the other node fails, you'll then at least know what's doing the reboot bit.

You can do that by creating a file in the /etc/modprobe.d directory with the contents of "options softdog soft_noboot=1" (then rebooting, ironically).

I do it by running this from a shell / command line on the server:

echo "options softdog soft_noboot=1" > /etc/modprobe.d/disable-softdog-reboot.conf

Thanks for the suggestion, I will be trying that!

 

[justinclift]

Membership information
----------------------
Nodeid       Votes   Qdevice   Name
0x00000001       1   A,V,NMW   IP_NODE_1
0x00000002       1   A,V,NMW   IP_NODE_2 (local)
0x00000000       1             Qdevice

Hmmm, nothing looks weird or wrong there.

After you've disabled the watchdog rebooting capability, it'd be interesting to see the cluster status info when one of the nodes is down.

Actually, if you could grab the status info again (separately) from each node when "the other is down", that might help point to what the problem is.

 

[greenapple]

So, after more intensive testing, we figured out that the reboots were not caused by proxmox, but by another piece of software. I guess if something looks right, it might as well be right and you have to look elsewhere.

Thanks a lot for your help! It definitely pushed me in the right direction.

Cheers!

 

[justinclift]

Cool, glad you got it figured out.