Pve node stops working when I join a node to its cluster.

[FaisalALi92]

Hello Everyone,

So I have two PVE nodes

I edited the hosts file on both machines so they look something like this:

IP1 stage-ve-01
IP2 stage-ve-testing

then I went to PVE1 and created my cluster then copied its information, after that I went to PVE2 to join it to the cluster.

After Joining everything seems ok but I lost access to PVE1. on The Web UI it appears to be online with the second node having a ? symbol.



On the second node both appear to be online but PVE1 is not working only the second one seems to be functioning.



I tried to log out then log back in but now it seems am locked out from both servers but I still have ssh access to both servers



Any help on how to troubleshoot this will be appreciated.

 

[Moayad]

Hi,

What says the journalctl -f command?
can you access to both nodes using SSH?

 

[FaisalALi92]

On node 1

root@stage-ve-01:~# journalctl -f
-- Journal begins at Sun 2022-07-17 11:48:43 +03. --
Jul 21 17:20:52 stage-ve-01 pvestatd[1579]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Jul 21 17:20:52 stage-ve-01 pvestatd[1579]: status update time (6.039 seconds)
Jul 21 17:21:02 stage-ve-01 pvestatd[1579]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Jul 21 17:21:03 stage-ve-01 pvestatd[1579]: status update time (6.042 seconds)
Jul 21 17:21:13 stage-ve-01 pvestatd[1579]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Jul 21 17:21:13 stage-ve-01 pvestatd[1579]: status update time (6.040 seconds)
Jul 21 17:21:22 stage-ve-01 pvestatd[1579]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Jul 21 17:21:22 stage-ve-01 pvestatd[1579]: status update time (6.046 seconds)
Jul 21 17:21:32 stage-ve-01 pvestatd[1579]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Jul 21 17:21:32 stage-ve-01 pvestatd[1579]: status update time (6.044 seconds)
Jul 21 17:21:42 stage-ve-01 pvestatd[1579]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 31 retries
Jul 21 17:21:42 stage-ve-01 pvestatd[1579]: status update time (6.041 seconds)

 

[FaisalALi92]

on node 2
root@stage-ve-testing:~# journalctl -f
-- Journal begins at Tue 2022-07-19 12:39:45 +03. --
Jul 21 17:22:19 stage-ve-testing corosync[983532]: [TOTEM ] Retransmit List: 12 19 1b 1d 1f 21 25 d3 d9 dc df e2 e5 10c 146 16c 16f
Jul 21 17:22:20 stage-ve-testing corosync[983532]: [TOTEM ] Retransmit List: 12 19 1b 1d 1f 21 25 d3 d9 dc df e2 e5 10c 146 16c 16f
Jul 21 17:22:20 stage-ve-testing corosync[983532]: [TOTEM ] Retransmit List: 12 19 1b 1d 1f 21 25 d3 d9 dc df e2 e5 10c 146 16c 16f
Jul 21 17:22:20 stage-ve-testing corosync[983532]: [TOTEM ] Retransmit List: 12 19 1b 1d 1f 21 25 d3 d9 dc df e2 e5 10c 146 16c 16f
Jul 21 17:22:21 stage-ve-testing corosync[983532]: [TOTEM ] Retransmit List: 12 19 1b 1d 1f 21 25 d3 d9 dc df e2 e5 10c 146 16c 16f
Jul 21 17:22:22 stage-ve-testing corosync[983532]: [TOTEM ] Retransmit List: 12 19 1b 1d 1f 21 25 d3 d9 dc df e2 e5 10c 146 16c 16f
Jul 21 17:22:22 stage-ve-testing corosync[983532]: [TOTEM ] Retransmit List: 12 19 1b 1d 1f 21 25 d3 d9 dc df e2 e5 10c 146 16c 16f

 

[FaisalALi92]

Hi,

What says the journalctl -f command?
can you access to both nodes using SSH?

Yes I have ssh connection. and I shared the output above